Monday, 5 May 2008

Networking conspiration

Last thursday and friday (1st and 2nd May) we had a scheduled downtime for the yearly electrical maintenance of the building. Actually, this was the "easter intervention" that was moved in the last minute due to the unscheduled "lights off" that we had on the 13th March. Anyway, following what seems to be now a tradition, this time we had also a quite serious unscheduled problem just before our so-nicely-scheduled downtime. This time it was the network who caught us by surprise. On monday the 28th April, our Regional NREN had a scheduled intervention to deploy a new router to separate the switching and routing functionalities. We had been notified about this intervention. They told us we could see 5-10 minutes outages in a window of 4 hours.
In the end, the reality was that the intervention completely cut our network connectivity at 23:30 on the 28-Apr. The next morning, at 6:00, we recovered part of the service (the link to the OPN), but the non-OPN connectivity was not recovered until 17:30 on the 29-Apr.
When one thinks on the network one tends to assume "it's always there". On that N-day we decided to challenge this popular belief, so we did not have just one network incident but two. Just four hours after we recovered the OPN link, somwhere near Lyon a bulldozer destroyed part of the optical fiber that links us to CERN. This kept our OPN link completely down from 10:00 a.m. 29-Apr until 01:00 30-Apr.

Not bad as an aperitive, hours before a two-day scheduled intervention...

1 comment:

Xavier Espinal said...

I don't want to publicize this... but I can't stand to paste the following link to a web page alerting about possible dangers on earth produced by the LHC:

Needless to say that the information given there cannot be treated seriously, I found it hilarious.

Hope that bulldozer's driver was not one of this alarmists...