Ask HN: Is anyone working on a Reddit archive?

by nidnoggon 6/30/2023, 6:59:02 PM with 4 comments
  • by uniqueuidon 6/30/2023, 7:03:03 PM

    Yes.

    There is the pushift dataset covering posts and comments through 2022 [1].

    And the ArchiveTeam has begun crawling reddit as well some time ago [2]

    [1] https://old.reddit.com/r/pushshift/comments/10bwxke/updated_...

    [2] https://news.ycombinator.com/item?id=36254172

  • by cookiengineeron 6/30/2023, 8:15:42 PM

    I was focussing mostly on cyber security related subreddits because the vulnerability and exploit discussions were of great value to me.

    I built a little scraper in golang that stores the JSON data (instead of the HTML which the archive warrior stores) to save hdd storage. [1]

    The problem with reddit's API is that it only shows 1000 entries over 10 pages in every api. Meaning hot/top/new, and search results are limited. If you have more links related to the keyword, you won't discover more.

    So you need a very specific keyword list to be able to discover more posts, and search each subreddit for each entry in the keyword list.

    [1] https://github.com/cookiengineer/reddit-archivar

  • by minimaxiron 6/30/2023, 7:02:29 PM

    Pushshift was the Reddit archive but apparently recent agreements with Reddit may have changed that.

    Anyone else creating a Reddit archive will likely get a C&D.

  • by simonblackon 6/30/2023, 8:20:54 PM

    It's next on my list after I finish the MySpace archive.

    Seriously, why would anybody do this? Reddit has such a high noise-to-signal ratio that it would be a waste of resources. There may be value in keeping an archive of some individual subreddits, but not the main bulk of Reddit itself.