Yes, The ‘Net Is Hosed Today
Actually it started yesterday with a firmware update to Juniper routers which wasn’t ready for beta testing, much less deployment, and it had to be reversed. Many larger systems, like Time-Warner cable, saw it as as 30 second interruption in everything.
The wonderful part of deploying these changes via the ‘Net is that there is no single point of failure, so changes ripple through the system. Today, those who didn’t notice anything yesterday are experiencing the joy of the outage and then the reversal.
Just like kidney stones, this too will pass.
5 comments
One thing that constantly annoys me about today’s Internet is that the backbone providers are so tight-lipped about conditions on their network, leading those of us who are trying to diagnose WTF is going on to have to rely on sending random traffic into a black hole and seeing what we get back. I remember back in the day when UUNET had a status page you could go to that would tell you exactly what the status was of their backbone. Worldcom axed that, of course, knowing just how badly Worldcom was treating the backbone would have been a competitive disadvantage, and everybody else followed their lead. SIIIIIiiiiiigh!
Oh yeah, so it’s a BGP issue? Color me unsurprised. BGP provably cannot work. The fact that it does is just one of those arguments for the existence of a deity :).
– Badtux the Snarky Penguin
Yeah, I know what you mean. My host has an off-site status report, twitter feed, Facebook page, etc. that are used to report any problems with their system, while my ISP [The Phone Company] requires me to call them and wait on hold to be told that they have screwed up another update. It apparently doesn’t occur to them to send out an e–mail to subscribers to warn about these ‘previews of coming attractions’ so you won’t be running around with a data on CD that has to be transmitted to a client two time zones away within hours. If you knew they were going to be playing with the network, you wouldn’t schedule things like that.
Eric at Wampum has all of the details of the mess, but they lack of testing on a firmware update is just amazing to me, but I go back to a time when new machines were subjected to 48 hours of diagnostics before they were sold and the report was included with the machine.
I’d be satisfied with a “coming attractions” web page. But the only people who get access to such things are their direct subscribers, who for backbone providers, uhm, doesn’t include me (doh!).
In other news, Compost *finally* has a network status page for your locality *if* you log in to your account on their web site. Which can be difficult while the network is down :). And people wonder why I continue paying $30/month extra to AT&T for 3G tethering? 😕
I was really annoyed when The Phone Company dropped Earthlink and used their own servers. Earthlink had local dial-up service when the system went down, and The Phone Company doesn’t. Go figure. They require you to keep a landline phone, but they don’t provide dial-up.
I’m on the verge of getting a ‘smarter’ phone, that will enable me to connect when the DSL dies. I’m uncomfortable without backup.