Network Outage Today

[Bob Proulx]

This morning there was a long network outage which separated the VMs
from their root file system holding the OS files on a Ceph partition.
This started at 8:18am US/Eastern time and seemed to mostly conclude
about 9:38am US/Eastern time.

That's a long time for a system to be without a root file system.  The
system times out trying to load code and data.  Some daemons die.
Others get killed.  The Savannah git server was in a broken state.
Thanks to Amin Bandali for handling the issue this morning with the
git server vcs2 and rebooting it.

This last happened 41 days ago.  The recovery including rebooting all
of the systems so that they all get reset to a known working state.

I am walking through all of the VMs I have access to, peeking at their
state, and rebooting all of them to make sure they are reset to a good
known working state.

I suggest that all VM owners reboot their VMs in order to ensure that
they have no problems.  I'll be rebooting the debbugs systems when I
get down to them on my list.

I note that right now 5pm US/Eastern time that Mailman on lists is now
sending out its daily housekeeping messages that normally are sent out
at 5am so things there are acting 12 hours delayed.  Seems like a wide
spread issue.

Just trying to communicate the news of the day.

Bob