Ask HN: How can I learn to build distributed systems?

  • I have a different solution. I was lucky early in my career where I got to work on a few projects that were distributed not for scale but for super high reliability. I studied what I was given but studying was only so helpful.

    To that point, I recently had someone ask me how I learned some of the things I did on distributed. I am not disagreeing with other commenters about reading up and studying but I think that it isn't as good as experimenting with things yourself.

    Pick something like "leader election" and implement a naive/simple version of it so you have a better understanding. Take that and run it in AWS on 3 (or more) micro instances and play with failure scenarios. Alternatively you can do the same with local computers on your network, or raspberry pi's or any other SBC. For less than the cost of a text book you'll have real world experience implementing and experimenting.

    From there, try other scenarios and keep building and testing. Doing this is far more valuable then just reading about it. Then if you do hit a job where you need those skills it isn't just academic to you, you'll have played with how it works and seen how to debug issues etc.

    Speaking of debugging issues, one of the hardest things in distributed systems is just that, debugging. So build a basic (again maybe naive version) distributed logging/tracing solution so you can see the issues first hand. You'll then understand (at least somewhat better) the way to evaluate the common solutions/ideas that are on market or that people push.

    You can learn a lot even by just setting up something like zookeeper or etcd and learn how they work, failure modes and how they can help or hurt you.

    Just some ideas, but the basic concept is to actually implement things. Even just run your tests in containers (not as good IMO but still better than just reading about it) if you can't afford to spend a little money in AWS or on SBCs etc.

    *edit fixed a word mistake

  • Kleppman's Designing Data-Intensive Applications book should be able to hook you up with the basics.

  • Look into an open source project which is similar to your work, and workout a similar benchmark

  • Sure you can. Look into 'load testing'.