Wiki/Operation/Incident_Reports.md

1.1 KiB

These are cybersecurity and availability incidents that the AniNIX has had to remedy due to some failure in our detection and prevention systems within the last two years.

Note: We explicitly exclude routine incidents, such as IP's banned for SSH brute-force, files quarantined after virus scanning, and other routine housekeeping. We are also not including maintenance outages or short-term (<8 hours) ISP events.

2024MAY21 Major Local Power/Internet Outage

Timeline

  • 2024MAY21 20:29:47 -- Initial outage notification by FreshPing
  • 2024MAY21 21:10:00 -- Outage notification by CloudNS
  • 2024MAY22 06:55:00 -- Outage notification by Alliant Energy
  • 2024MAY22 06:57:00 -- Outage notification by Spectrum ISP
  • 2024MAY22 20:15:00 -- Power restoration notification from Alliant.
  • 2024MAY22 22:11:00 -- Services restored.

RCA

Major storm took out power and network across the region. Response teams from power & ISP were overwhelmed providing the response.

Improvements

  • Further business continuity design
  • Generator installation at MSN0