HN was down

  • Founder and CEO of M5 Hosting here. We did have a network outage today that affected Hacker News. As with any outage, we will do an RCA and we will learn and improve as a result.

    I'm a big fan of HN and YC in general, we host of other YC alum, and I have taken a few things through YC Startup School. During this incident, I spoke to YC personally when they called this morning.

  • HN is one of the few sites I always keep zoomed-in (around 200%), which led to me finding an interesting bug in Chrome while HN was down: Chrome's internal "This site can't be reached" page uses the zoom level of the site you would be visiting (if it were up), rather than Chrome's default zoom.

    Screenshot: https://i.imgur.com/VwFtgQh.png

  • Hacker News is hosted at M5 and they are having a network outage:

    http://status.m5hosting.com/pages/incident/5407b8e2b00244251...

    edit: Unrelated to the Azure outage.

  • I think I have tried to visit HN for more than 10 times in last 2 hours and failed. This made me realize how much I'm addicted to HN

  • Had me quite confused because I'm also having home internet issues. I was trying to get my laptop to switch to my mobile hotspot and HN is one of the sites I used to test connectivity because a) it's almost always available and b) loads very quick.

    A bit of a mindfuck trying to assess my actual internet connectivity via a site that was also down : )_

  • The title should be updated to "Productivity was up".

  • my fingers automatically just start typing in "news.y" when I'm idle, I definitely didn't know what to do when greeted with a 404!

    Is there any way to put the HN homepage on an edge cache so at least the homepage shows up? Or am I admitting that I'm addicted to checking HN too many times a day?

  • I never expect HN to be down... I asked my wife - hey is the internet down? She said - no, it's working for me. I clicked on another site and my mouth dropped.

  • It says a lot that @hnstatus has not tweeted since 2018.

  • @dang

    Would love an updated post on what the current hardware / software stack that’s running HN.

    It’s been years since I’ve seen a post/comment on this topic.

    Are you still running FreeBSD, on a few high frequency cores (iirc)?

  • Azure AAD also had an outage at this time - perhaps linked in some domino effect, or perhaps a coincidence?

    https://status.azure.com/en-us/status

  • Seeing HN unresolved was a bit weird as it is the best performing site I ever visit on my low bandwidth phone so several times I thought the problem was on my end. But in the end it helped me realize how frequently I dial into HN while it was down. I have a bit of a problem and I think I need to turn on that no procrastination flag on.

  • I've been on this site 12+ years, and I don't ever remember it being down. I assumed we were under nuclear attack.

  • HN is my homepage and when it wouldn't load, I told my coworkers the internet was down and took a 3 hour lunch.

  • > Back now. Got to write some code for a change... - 5:39

    https://twitter.com/HNStatus/status/1371576822748487683

  • It's seems an odd coincidence of this and the Azure AD outage -- I was trying to get to HN to see what people were saying about it!

  • Something that I learned from this is that HN has a status Twitter. Rarely used though, which is a testament to the team.

    https://twitter.com/HNStatus/status/1371525940656803848?s=20

  • I thought pg posted "Memphis".

  • I didn't notice unfortunately due to the Azure outage blowing everything up :(

  • So strange that this coincided with Azure authentication eating it.

  • The one time I'm actually reading HN for actually relevant information for actual work, it's down for half a day. Made for a great excuse to take a nap.

  • I have a massive learning project[1], and I think 2/3 of my "to get through as soon as sensible" content is news.ycombinator links.

    Needless to say, this site is my own personal StackOverflow, and I think there's something about ingratitude bouncing around in my mind somewhere.

    [1]https://github.com/PhilosAccounting/ts-learning

  • Is HN fully back? Looks like this was a little less than 3 hours total, is that right?

  • Funny, my first thought was "oh no they've blacklisted VPNs", can't remember when HN was ever down!

  • It did not seem to affect the Firebase feed.

  • Curious, what is the uptime of HN - is there some data about that?

    My guess is around 99.9% ... but maybe that's too optimistic?

  • Who needs so many 9's when there's actually interesting content we keep coming back for?

  • We also had issues with our YC application earlier today, was that related to this issue?

  • It’s OK. The skies haven’t fallen. (And I was able to accomplish something today.)

  • There was a noticeable increase in productivity during the last hour or so.

  • It's 3pm.... do you know where your servers are?

  • And now it looks like there is an outage at reddit

  • Is this related to the big Microsoft outage?

  • If only they used Kubernetes! /s

  • Since I couldn't get to HN, I wrote up how to make the site resilient to outages: https://gist.github.com/peterwwillis/ce2bfaba7fc72e4af44c281...

    tl;dr 1 server x 2 providers, different regions, replicate content

  • Looks like a startup opportunity

  • Good to see posting purely for karma isn't just a Reddit thing.

  • This site is so reliable, I thought my I.P. had gotten banned.