I usually send #hugops on notable outages, but Sportsnet+ essentially repeated the *exact same* outage of game 3, at exactly the same inning, on the first #WorldSeries run in 32 years. They had all they needed to make things right for game 7 and still didn't. What a shame.

A great post-mortem read from Matrix.org

https://matrix.org/blog/2025/10/post-mortem/

@jimsalter @joeress here's one for your 2.5 admins. #25admins #zfs #postgresql

https://matrix.org/blog/2025/10/post-mortem/ great write-up and #HugOps to those involved over that very stressful period.

- it's great to see they had a comprehensive backup/recovery strategy in place
- well done, really well done. Multiple fallback layers. Sad that they had to use them, but hey that's why we do this.
- Kudos to @beasts hosting for moral & technical support, once again I keep hearing good things about them

A story in SQL backup/recovery from matrix.org with three key lessons:

- always do critical recovery work with 2+ people checking and reviewing together (they did this), and rotate regions because sleep is critical
- never actually delete stuff during a crisis. Ever (narrator: they learned this the hard way)
- ZFS would have made this recovery significantly easier, in so many ways

It would have been almost trivial to recover from their failed storage with ZFS, and perhaps avoid either the failover, or the remote restore.

Scheduled ZFS snapshots would have meant a rollback instead of a recover in at least 2 of their high-risk moments.

It would have also meant higher storage costs because snapshots are almost but not quite free.

ZFS snapshots can be sent/received from an alternate system, over LAN at very high rates, much much faster than a remote S3-based streaming restore.

#hugOps story time! Quote this and tell me the biggest incident you ever saw in production. It’s inevitable, it’s gonna happen, and learning from incidents is way better than shitting on people trying to fix them.

I’ll start :)

https://hazelweakly.me/blog/mother-of-all-outages/

Did AWS join the government shut down?

Hey Amazon friends at US EAST 1. You had a shitty few hours. But you got stuff fixed. Thank you. #hugops

I am seeing *so many* bad takes about the AWS outage, so many.
Everybody is smug until they get fucked with their pants on.
Even if you've done everything to run your own stuff and host it, it is very difficult to avoid services that will be impacted and there's *no way* the majority of businesses are running all that stuff themselves.
Comms tools, status pages, payment systems, monitoring systems, build tool, deployment tools, planning tools, the list goes on.
If you're telling me you're running absolutely everything yourself, well done, I have no idea how, and if you're better at all of that than all the SAAS providers, I struggle to believe you've time left in the day to run your actual business.
100% you should be doing your due diligence to make sure you're resilient where it matters, but being caught up in something like this is almost inevitable if you're a non trivial online company.
Stop throwing rocks, start sending #hugOps

#HugOps is now trending across Mastodon

#HugOps to those dealing with today's outage and those impacted by it.

https://health.aws.amazon.com/health/status has a fascinating stream of updates if you are into that kind of thing. #hugops for everyone involved, it looks like an uphill battle. #aws

much as I love AWS (and I do), having a third of the Internet on a single provider is probably not ideal from a resiliency perspective.

#hugops to everybody on service teams today (and all the enterprise TAMs who are getting paged by customers constantly)

been there

#HugOps to all the IT and business people whose start of week has been borked by the AWS outage, while they try to restart everything from the ground up.

Massive #HugOps to AWS and their ecosystem. Outages are stressful, let alone international ones, so let's be extra kind to each other today

Is #hugops still a thing? #hugops to all those dealing with the AWS outage today, many of whom probably pleaded with their bosses for time to make critical services multi-cloud or at least multi-region.

Also #HugOps to all affected by the AWS outage

Fucking us-east-1 strikes again!

I see everyone is getting their unscheduled chaos monkey tests. #hugops

I see a lot of people here being smug about AWS having a major outage.

What happened to #hugops?

I would kind of want to send #hugops, but then again it's AWS, and my empathy is not really that high

Unser aktueller Test zeigt: us-east-1 ist für den Betrieb des #FFRN nicht nötig.

Es könnte nur sein das ihr aktuell gewisse Einschränkungen bei den Webseiten habt die ihr ggf. aufrufen wollt.