This is really cool considering how expensive DataDog can get. I'm the author of LogLayer (https://loglayer.dev), which is a structured logger for TypeScript that allows you to use multiple loggers together. I've written transports that allows shipping to other loggers like pino and cloud providers such as DataDog.
I spent some time writing an integration for HyperDX after seeing this post and hope you can help me roll it out! Would love to add a new "integrations" section to my page that links to the docs on how to use HyperDX with LogLayer.
I liked Otel for traces and maybe logging -- but I think the Otel metrics is over-engineered.
Does ClickStack have a way to ingest statsd data, preferably with Datadog extensions (which adds tagging)?
Does ClickStack offer correlations across traces, logging, and metrics via unified service tagging? Does the UI offer the ability to link to related traces, logging, and metrics?
Why does the Elixir sdk use the hyperdx library instead of the otel library?
Are Notebooks in the roadmap?
Can I say it's similar to Signoz, in that both are ClickHouse-powered and available as both open-source and hosted versions? How are you guys different compared to Signoz?
(The UI looks similar too, although I guess a lot of observability tools seem to adopt that kind of UI).
How are you different than Signoz, another YC company that also does Observability using clickhouse ?
I'm looking for a new logging solution to replace Kibana. I have very good experience with ClickHouse, and HyperDX looks like a decent UI for it.
I'm primarily interested in logs, though, and the existing log shipping pipeline is around Vector on Kubernetes. Admittedly Vector has an OTel sink in beta, but I'm curious if that's the best/fastest way to ship logs, especially given that the original data comes out of apps as plain JSON rather than OTel.
The current system is processing several TB/day and needs fairly serious throughput to keep up.
I’m not sure what this is intended to do, but when I created an account, I saw in the left sidebar a widget saying “Was this search result helpful?” with thumbs up and thumbs down buttons. I hadn’t searched for anything. I pressed the “Hide” button instead, and the widget changed to an “Any feedback?” button. I thought I would tell you about this weird bug, so I clicked the feedback button. The widget changed back into the “Was this search result helpful?” widget.
I found the UX very difficult to read. The monospace font, the unusually small text, the bold white and bright green text on a dark background… I found it a little more readable by changing the font to system-ui, but not by much. Please consider a more traditional style instead of leaning into the 80s terminal gimmick. This factor alone makes me want to not use it. It needs to be easy to read, not a pain to read.
I am absolutely amazed at the amount of garbage being "logged", enough that it is not just a huge business, but also one of the primary task for some devops guys. It's like a goal in itself, you have a look at the output and it is absolutely scary, HUGE messages being "logged" for purpose unknown.
I've seen single traces over 100KB of absolute pure randomness encoded as base64... Because! Oh and also, we have to pay for the service, so it looks important.
Sure they tell you it is super helpful for debugging issues, but in a VERY large proportion of cases, it is 1) WAY too much, and 2) never used anyway. And most of the time what's interesting is the last 10 minutes of the debug version, you don't need a "service" for that.
/me gets down his horse :-)
Datadog is expensive this is true. But I have never felt it be slow. Speed is not its killer feature. It’s everything you can do with it once you have logs and or metrics flowing into it.
The dashboards and their creation are intuitive. Creating alerts and things from airflow logs is easy using their DSL. Connecting and sending notifications to things like slack just works tm.
So this is how we justify the datadog costs because of all the engineering time (engineers are still expensive, ai hasn’t replaced us yet) it saves and how quickly we can move from raw logs and metrics to useful insights.
This is really interesting.
Is Clickhouse the only stateful part of this stack? Would love to see compatbility with Rotel[0], a Rust implementation of the OTEL collector, so that this becomes usable for serverless runtime environments.
One key thing Datadog has is their own proprietary alternative to the OTEL collector that is much more performant.
Very cool, reminds me of SigNoz.
How would I self host this in k8s? Would I deploy a ClickHouse cluster using the Altinity operator and then connect it using the HyperDX local mode or what is the recommended approach to self-host ClickStack?
We run a full Grafana stack (Loki, tempo, Prometheus, alloy agent, Grafana) and back out with self hosted S3 (we are all onprem physical hardware).
While I do like the stack we have, it is a lot of components to run and configure. Don’t think we have ever had any issues once it was up and running.
Does anyone have any thoughts about how this compares? We don’t have a huge amount of days, 1 month of metrics is about 200GB and logs isn’t a whole lot more, less than a TB I think for 2 weeks.
I remember back in the day Mike was building Huggingface before Huggingface was a thing. He was ahead of his time. It's a pity model depot is no longer around.
There’s so many of these log aggregators I’ve completely lost track. I used Datadog extensively and found it overpriced and a very confusing UI.
Comparison to the other player in this space, Signoz? Also uses clickhouse as backend
Really interesting, Unfortunately, it looks like HyperDX depends on Mongo? I wonder if there are any open source document stores (possibly a mongo compatible one)( that could work with it?
Why did they decide to build a custom front end vs leverage Grafana that’s already as the front layer?
Do i need to sign-in when using the docker container?
It would have even much better if the link was pointing to https://github.com/hyperdxio/hyperdx the actual source code.
Because right now without the message on HN here, I wouldn't know what "open source observability stack" meant when the webpage does not explain what HyperDX is, nor does it provide a link to it or its code. I was expecting the whole thing "Open Source Datadog" to be ClickStack Repo inside Clickhouse Github. Which is not found anywhere.
But other than that congrats!. I have long wondered why no one has built anything on top of Clickhouse for Datadog / New Relic competition.
The Clickhouse DB opened up the ocean of open source "Scalable" Web Analytics that wont previously available or possible. I am hoping we see this change again to observability platform as well.
[flagged]
I like and use HyperDX in production and like it a lot. So kudos to the team for building and merging with Clickhouse. I found a lot of monetary value switching over to HyperDX considering it's significantly more cost efficient for our needs.
Should we be starting to prepare for the original HyperDX product to be deprecated and potentially move over to ClickStack?