Show HN: Webcrepe – SQL for the Internet

  • I think there is a trend towards using SQL to do the T part of ELT. For example, see the rise in popularity of dbt. Analysts are often limited by the data that's in the warehouse already. Instead of asking a data engineer or developer to do something so that more data gets pushed to the warehouse, I wanted to be able to pull it in myself.

    Because of that, I'm starting an open source project, WebCrepe, to empower analysts to pull data directly into their databases using SQL. The idea is that we pair a database extension with a web app to enable searching the internet and pulling in structured data. It's really early right now. I have a docker-compose file you can use to spin up a postgres database and the backend. I still need to write some better documentation on how to write queries but it's basically using the advanced google search language.

    I'm interested in analytics folks that have use cases I can build out and engineers interested in working on it. If there is any interest then I'll write up better docs and build more functionality.