As someone who has some experience in observability at scale, the issue with SigNoz, Prom, etc is that they can only operate on the data that is exposed by the underlying infrastructure where the IaaS has all the information to provide a better experience. Hence CloudWatch.
That said, if you own your infrastructure, I’d build out a signoz cluster in a heartbeat. Otel is awesome but once you set down a path for your org, it’s going to be extremely painful to switch. Choose otel if you’re a hybrid cloud or you have on premises stuff. If you’re on AWS, CloudWatch is a better option simply because they have the data. Dead simple tracing.
I spent some time working on this. First I tried to make a GitHub action that was triggered on completion of your other actions and passed along the context of the triggering action in the environment, then used the GitHub api to call out extra details of the steps and tasks etc, and the logs and make that all into a process trace and send it via an otel connection to like jaeger or grafana, to get flamchart views of performance of steps. I thought maybe it would be better to do this directly from the runner hosts by watching log files, but the api has more detailed information.
I have thought about that before, but I was blocked by the really poor file support for OTel. I couldn't find an easy way to dump a file from the collector running in my CI job and load it on my laptop for analysis, which is the way I would like to go.
Maybe this has changed?
noob question, i'm currently adding telemetry to my backend.
I was at first implementing otel throughout my api, but ran into some minor headaches and a lot of boilerplate. I shopped a bit around and saw that Sentry has a lot of nice integrations everywhere, and seems to have all the same features (metrics, traces, error reporting). I'm considering just using Sentry for both backend and frontend and other pieces as well.
Curious if anyone has thoughts on this. Assuming Sentry can fulfill our requirements, the only thing taht really concerns me is vendor-lockin. But I'm wondering other people's thoughts
How does SigNoz compare to the other "all-in-one" OTel platforms? What part of the open-core bit is behind a paywall?
That's a genius idea. So obvious in retrospect.
That's a poor title: the article is not about CI/CD, it is particularly about GitHub CI/CD and thus is useless for the most CI/CD cases.
Has anyone seen OTel being used well for long-running batch/async processes? Wonder how the suggestions stack up to monolith builds for Apps that take about an hour.