OpenAI scraping Reddit through redlib instances

  • The author used to run a public Arch mirror under mirror.ext4.xyz, so it's not exactly an unknown domain.

    Combined with the fact that a lot of their self-hosted stuff, including the Reddit front-ends, are in the Certificate Transparency logs [1], it's not hugely surprising that web crawlers would run into them.

    [1]: https://crt.sh/?q=ext4.xyz

  • that's not scraping, that's web search.

    their scrapers wouldn't identify themselves