core.async was actually the "killer app" that sold Clojure to me. I've always liked Go's CSP-style concurrency, but I wanted a proper functional language to go with it. It turns out that Channels/Queues are just an insanely good abstraction for getting threads to talk to each other. The closest general-purpose library I've found that helps bring that to the rest of the world has been ZeroMQ, which is great but not as easy and nice to use as core.async.
That said, I really don't think RabbitMQ is so bad. The default docker configuration for it is fine for most cases you're likely to come across for all but the biggest applications, and I do think that having the ability to have jobs restart/requeued when a worker crashes out of the box is worth the tradeoffs for slightly increased complexity. I usually just use a single docker compose and glue everything together in one big ol' YAML.
Still, there's obvious value in avoiding dependencies, so it of course depends on the size and scope of your project. If you think you're going to end up distributing this over 100 nodes, something like RabbitMQ or ActiveMQ might be worth playing with, but if you're just doing relatively small (at you appear to be in this project), it's probably the correct choice to mimic whatever behavior you need with core.async.
Yes! core.async is a great tool and I've used it to solve a number of problems effectively. I'm very happy with what I get: reliable, predictable systems.
A real "whoa" moment comes when you realize you can also use core.async in ClojureScript :-)
Thank you for this post Janet.
I think my technical perspectives have moved in a similar direction: keep things extremely simple with minimum moving parts.
I maintained (automated upgrades) a RabbitMQ cluster and while it is powerful software it is operationally expensive. For a side project you probably just batch process in a cron.
If I were to take the approach in this blog post I would want everyone on the team to be extremely familiar with the model of task running: stuck jobs, timeouts, duplicate jobs, client disconnects and retries, stuck "poison" jobs seem like issues you might face.
I appreciate Janet’s aversion to adding RabbitMQ to the system. I had to use RabbitMQ with Common Lisp a year ago and in was a pain.
It looks like Clojure has better RabbitMQ client options than Common Lisp, but still, very cool to build something on core.async and keep things cleaner and simpler.
I appreciate this emphasis on simplicity and I recommend it to others. I've seen trends in the other direction which I think are dangerous. I notice that as some companies adopted a microservices approach there was a tendency to allow in more technologies than necessary.
"Premature polyglot programming" is a disease that afflicts certain startups. While any sufficiently big company will be polyglot, a small startup needs to stay focused on a limited tech stack, for as long as possible. After all, each new technology requires someone on staff who has the skill for that technology, and when your startup is small, your talent pool will also be small, and so it becomes common that you only have 1 person on the team who knows how to run some particular technology (Kafka, RabbitMQ, Redshift, DynamoDB, MongoDB, Tornado, etc).
So you end up with a lot of single point of failures, where the "single point of failure" is the single engineer who knows how to run the technology that you've made critical to the company. Be wary. Avoid this if possible.
I notice, especially, as the tech industry developed better tools for managing complex devops situations (Docker, Kubernetes, Terraform) there was a tendency from some engineers to think "Nowadays it is easy to run 10 different technologies, therefore we should run 10 different technologies." Be wary of this.
Janet Carr's emphasis on simplicity is something we should all imitate.