Heroku's Managed DB's have been down for 2+ hours

by kentfon 9/27/2021, 5:15:00 AM with 9 comments
  • by pgn674on 9/27/2021, 5:42:23 AM

    It mentions an issue at an upstream service provider. Is it AWS and their Degraded EBS Volume Performance in Northern Virginia? https://status.aws.amazon.com/

  • by throwdecroon 9/27/2021, 6:20:11 AM

    Is there an "uncanny reliability" range where increasing reliability on the part of a service provider makes things worse, by being so close to 100% reliable that any failure is a shock?

    Maybe it's better to go with cheaper services that fail more often, thus keeping customers in good practice for how to deal with it.

  • by forgingaheadon 9/27/2021, 6:18:28 AM

    Heroku has been strangely unreliable the past few weeks. Even their ticket response team has been slow, with their support engineers often talking past the issue to just send a scripted reply.

    We have the majority of our client apps hosted with them, but most don't require 24/7 availability. This is still concerning though, and we do have one high-availability app hosted on them now that we're trying to plan contingencies for.

    Open to any suggestions for alternatives! Ideally I'd keep things on Heroku, but it would be nice to have failsafes that could be activated relatively quickly in the event of similar issues.

  • by Daniel_skon 9/27/2021, 5:45:20 AM

    Signal is down too due to an outage of a service provider (I assume AWS).

  • by supermatton 9/27/2021, 6:40:14 AM

    Whatever happened to 5-nines uptime? It seems no cloud service provider these days is able to offer what was considered an industry standard.

    AWS even have documents telling people how to achieve exactly this! https://docs.aws.amazon.com/wellarchitected/latest/reliabili...

    Why don't "premium" service providers like heroku, etc, do this?

  • by kentfon 9/27/2021, 5:16:07 AM

    Luckily we had followers in different zones by chance. Still scary though. What are the best solutions for replicating Heroku in different clouds?

  • by thomaslordon 9/27/2021, 2:11:09 PM

    Interestingly, I have an app using Heroku Postgres that seems to have had zero issues during this outage. I can see data that was stored during this period of time and Rollbar doesn't show any DB connection errors.

  • by pvsukale3on 9/27/2021, 10:00:14 AM

    I have been trying to deploy fix to a bug we deployed yesterday. I think they have stopped deploys as well. As the deploys are being rejected without any explanation.

  • by TedShilleron 9/27/2021, 6:25:00 AM

    Heroku may have been down for 2+ hours, but MongoDB has been unreliable for 10+ years.