T-Mobile & Microsoft – What Could Go Wrong?
Well, according to an AP article on MSNBC, your Sidekick “super-cell” may have amnesia because of a Microsoft server failure.
All of the information you input into your Sidekick is stored at the Microsoft server farm. There is no way of making a local copy, and it isn’t stored on internal memory. Apparently the Microsoft server farm doesn’t understand fault tolerance, RAID, or any of the other techniques that are “industry standards”. I get the feeling from the article that they backup servers occasionally, when someone thinks of it, if there is nothing else going on, rather than on a standard schedule.
No big deal, it’s only about a million people who have lost their address books.
Update: I should add that T-Mobile is going to give their customers a $20 $100 credit for the inconvenience.
19 comments
I’ve been following this, and it’s pathetic. The core problem is the same one Sprint has with their Nextel acquisition, or the reason I left a prior employer. Microsoft came in and acquired Danger because they wanted to build an iPhone competitor based around their Zune franchise. So everybody who was competent ended up leaving because a) the new management didn’t acknowledge or care about their achievements, b) the brown-nosers used this opportunity to shove the real innovators out of the design loop, and c) they didn’t want to deal with programming under Windows CE (abbreviated WINCE, because that’s what you do if you have to program under it). So like with Nextel after the Sprint acquisition, all that was left was deadweight and a minimal skeleton staff to keep the servers up and going. Apparently at some point in time the big Oracle database that held all the data quit making backups because some backup program’s plugin got out-of-date compared to the database version, but nobody noticed because it was a skeleton staff of utter cretins and nobody ever bothered checking the logs to see if anything unusual was happening (or not happening).
So they apparently managed to crash the Oracle database server during a software upgrade and wipe out its data and went to retrieve the backups and found that, uhm. The backups aren’t there!
Note that T-Mobile is a victim here too. They had a contract with Danger/Microsoft, and Microsoft violated this contract every way possible. Microsoft failed to maintain the server farm the way the contract called for, failed to work on the next version of the Sidekick the way the contract called for, and has caused serious and irreperable harm to T-Mobile’s business. All 1,000,000 Sidekick owners will probably have to be released from contract, and most of them will probably be going to AT&T.
Anybody who relies on Microsoft for *any* critical infrastructure is playing with fire. They simply don’t know, don’t care, and are incompetent from top to bottom. This little incident with T-Mobile is probably going to cost them close to a billion dollars by the time the courts finish counting up the damages and the funny part is that the incompetents who did this are probably going to get promoted. Monkey Boy Balmer needs to be shown the door and adults brought in from outside to get things under control, because ever since Windows 2000 they’ve been going downhill, with each software release being buggier, more bloated, and harder to use than the last. I won’t say that Microsoft ever produced great software…but for a time in the period 1995-2000 they at least produced good-enough software, software good enough to be better than most of their competitors. I don’t know whether it was Bill Gates retiring that unhinged them, or the server competition from Linux, the antitrust lawsuits, or what, but something’s happened to Microsoft since then and it’s not for the best.
They forgot they were a technology company and became just another corporation. The same think that happened to Apple when they hired Scully of Pepsi to run things. He didn’t understand the product, he understood marketing of products. This is the great fallacy of the current corporate system, that CEOs don’t have to understand what companies sell, to manage the company.
It is hard to understand how you could even consider updating Oracle without backing up everything first. Oracle upgrades are notorious for eating everything. I have a cousin who works only with Oracle. Most of her time is spent dealing with these sorts of problems as a military contractor. The company she works for has some local installations, and she shows up ready to spend time on the beach, knowing that no one thought to backup before her arrival. She wouldn’t even consider starting without a complete, verified backup, She has had to back out of too many upgrades to simply do it on a live system.
It’s hard to believe that they didn’t have a spare test bed system to try this on before doing it on the main system.
A billion dollars would have bought a hell of a lot of hardware, and paid for a lot of people for a long time. They all do this crap.
I think it was just after Y2K that they went heavily to temps and stopped the team system that had been their core development effort. It seemed like a lot of veteran MS people starting circulating resumes about then, It may just have been the old hands cashing out to participate in the tech bubble, or people deciding that they were tired of the hours and work load, but there seemed to be a lot of them out and about.
Oracle — no bloody wonder. Works great, when it works, but I have my cousin on speed dial when I have to deal with it, because she lives and breathes the program.
This is a lesson that everyone should learn. Never ever trust your data to an outside entity and especially if that entity is a large corporation. I have never trusted off site storage(to save me the trouble of backing up my own data) and never will.
As to Oracle, as Bryan says, you would have to be an absolute idiot to try any maintenance or even any database tuning without a fully verified backup. Not just Oracle but MSSQL and all the rest are just as weird.
From personal experience I can tell you that the tech companies are pinching pennies everywhere they can and one of their first moves is to get rid of the experienced and expensive employees. The ones that have learned the hard way why you don’t trust anything to work properly and that the only way to survice is to CYA every way you can. If that means backups of backups and spending an hour or two a day reading server logs then so be it.
.-= last blog ..Chicken Three Ways =-.
This is one of the reasons I don’t consider leaving my current hosting service. They have survived everything, including having the electrical mains shut off without warning, and I have never lost anything, or had to use my local backups. Despite their record, I still make local backups, and always include a backup immediately before I upgrade WordPress.
You can’t have too many backups.
I was not aware that a large number of the deaf population use T-Mobile Sidekicks.
.-= last blog ..Whole Foods is finally open in Santa Barbara =-.
It makes sense, Jill, because it does have a reasonable keyboard for its size, and it much easier to use than most of the others. There is normally a special fund that will kick in extra money to a telco for accommodating the hearing impaired.
I took notes for a girl at college, and she had the cheapest service I had ever heard of for her terminal. We communicated either via an ASCII link by modem, or the Relay system supported by the state. The Relay system was cheaper, because it was a long distant call for the direct connection, but there were no long distant charges uses the in-state relay system.
This was during the dark days of dial-up service, and a lack of local ISPs.
Microsoft has recovered the Sidekick data… days later, with little explanation and very little compensation to users. Apparently, it was never really gone… yeah, right. Why do I still not feel compelled to rush out, buy that “smart” phone and subscribe to that service?
This attitude is not new to M$. They’ve shamelessly exhibited it for years. It’s one reason I don’t have working WiFi on my laptop: Win XP SP3 has bugs specific to the combination of wireless chip and router I happen to have; no one on the forums seems to know a workaround,and M$ of course doesn’t give a good damn, because that’s all old technology now, and doesn’t affect their sales.
My experience with Linux was not a happy one, admittedly mostly due to my own ignorance. But I’m reconsidering my decision to stick with Windows, just because M$ has such a bad attitude.
They rebuilt the data base, which, if it is properly designed….. [sorry, but for some reason typing “properly designed” in reference to anything dealing with M$ sets off uncontrollable fits of laughter] will recover the data. Let me just say that I wouldn’t be surprised to find out that some people now have a new “friends and family” plan, while others have become Raskolnikov, a desperate outsider cut off from the world.
I am planning a new post on this as a juxtaposition to another tech post on data storage.
In addition to “too big to fail”, there is the grand American tradition of The Phone Company – “too big to care”. Customer satisfaction is irrelevant to monopolies – if you want it, you have to buy it from them.
With every passing month Linux gets easier to use, and the word on Windows 7 is that you have to wipe your disk to install it, and re-install everything again. Apparently you can’t just upgrade. The need for a new box is the only way that will happen.
The database was apparently stored on NetApp file servers, so if the snapshotting was set up correctly they should have been able to get an (inconsistent) version of the database back as of a few minutes or at most an hour before the destruction. My guess is that they couldn’t figure out how to get Oracle to re-import that data into the upgraded database (because it wasn’t a “clean” copy of the data thus Oracle would have snorted “not on your life, monkey boy!”), they lacked the internal competency to get Oracle to do the right thing, fuddled around for days, and *finally* pulled in someone like your cousin to get the data back into Oracle via duct tape and bungee cords and the occasional tire iron.
Linux has become ludicrously easy to install upon supported hardware. Usability… meh. The biggest problem is two different rival desktop GUI’s introducing gratuitous incompatibilities. There’s no reason for two GUI’s anymore (the original split was because of licensing reasons, not because of any core technological reasons), other than inertia and NIH (Not Invented Here). Anywho, this situation has basically kneecapped multimedia on Linux… there isn’t even a single way to play sound on Linux, you could have ALSA, ESD, ARTS, or OSS, all of which fight over the sound hardware and stomp on each other such that half the time my Skype conversations get played into /dev/null! I finally gave up on Linux on the desktop, too much of a mess to be worth it. One heckuva server though!
Regarding Windows 7, I’ll upgrade to it when Windows XP is EOL’ed and not one minute before. I simply have no need or desire to throw gratuitous incompatibilities into my computing infrastructure, especially since I don’t develop Windows software and thus have no need to test whether my software works under Windows 7.
.-= last blog ..A sad comment on Republicans =-.
Thinking back to the NT servers I’ve dealt with, it could take them this long to reboot.
It was a network, the data had to be there if you knew where to look.
I saw it reported [CNet, maybe] that they had outside consultants from Hitachi doing the upgrade. I can believe they used an outside firm, but why didn’t the consultants check before they touched the server?
The delays may have been a conference with all of the various attorneys for MS, Hitachi, and T-Mobile deciding who was going to pay for this mess.
Yes, Oracle has a thing about importing that is a true PITA, and you have to babysit the process or it blows up. If you catch it, you can halt, fix, and re-start from that point, otherwise it’s back to the beginning. Not quite as bad as the scan before installing Novell in the old days, but close.
OK, now I get it. I don’t use the multimedia features, so I would never encounter those problems. Hell, 99% of what I do is still text, with a few jpegs or gifs. If I was doing sound or video, then I would probably go with Mac. If you stay with text and pictures, Linux does everything without hassle, and has for years.
What all this really shows, is a distinct lack of serious planning and management. They obviously had no disaster recovery plan that had actually been tested, reviewed, and approved, with all stakeholders on-board and with each knowing exactlty what their roles and responsibilities were in case of disaster. And this is such a BASIC first-choice for a potential disaster that ANY competent Risk Manager SHOULD have had a working plan for! People would be amazed at just how common this scenario is, especially with Oracle! (I much prefer Sybase! Much easier to work with, and much less fragile!) Oracle as a company are more arrogant than M$ & HP combined (speaking from personal experience when I worked at HP).
This should never have happened… and I don’t mean that the servers failed, that happens often. It’s what happened after the failure that shows a complete contempt and lack of care for the customers.
As for linux… M$ once tried to migrate Hotmail after they acquired it to M$ server based systems. It was a dismal failure and Hotmail continued to run on FreeBSD (though I don’t know if they have since moved it, there was a rumor a couple years ago about RedHat). M$ secretly use Unix/Linux and other Open Source technologies, but woe-betide ANY employee (even a CIO) that mentions it!
Has Microsoft Fired Its CIO Because Hotmail Secretly Runs on Free Software?
M$ is the perfect example of an American company. Totally hypocritical to the core. 🙂
To be Continued… (stoopid spam system!) 😛
…continued 😛
I use Linux all the time, as a dev environment, for which it’s perfectly suited, especially if developing for target linux server environment’s, as I do. There are now several linux variants aimed specifically at multimedia, such as StartCom MultiMedia Edition, Ubuntu Studio, 64 Studio and others such as Dreamlinux Multimedia GL, Linux Mint — I use this as well as CentOS (which is an Enterprise Server and I wouldn’t even attempt to do any multimedia on it!) 😆
And…. To Be Continued, again… *sigh*
COntinued, part duex…
And if you want to build a multimedia appliance, or entertainment center, there is MythTV (http://www.mythtv.org/) and others. Though… all of these, as does any Windoze system, still has some problems depending on what you try to make them do. 😉
I am now running Win7 Ultra x64 (with Virtual PC and XP SP3 Mode) build 7600.16393 RTM on my main design system (for about 2 weeks), and so far, it’s a hell of a lot better than Vista ever was! It is a resource pig though (of course!) 😉 I have a dual-core 3.2GHz CPU with 8GB RAM (and you need at least 4GB to run the VPC/XP Mode). I still hate windoze but! I only use it because I sadly have no real choice… but that is changing! 😀
OK! Hopefully… THE END! Hmmph! 😛
Maybe it’s your Asian IP that is setting it off, because I’m logging a lot of Asian spam in the last few days. Actually the spam counter is spinning and I’m seeing the Asian flags on Feedjit. Most of them are coming from Digg and such, and I don’t even participate in their program. The references seem to be for various products that I don’t want or need.
If they had something to eliminate cat fits or fire ants, I might care.
At this point I assumed someone would bundle a graphics friendly system for Linux, but I just don’t have the particular need, because I can’t convince clients to break away from MS.
Hell, MS doesn’t even include a decent backup system with their OS, so you know they don’t put any importance on it. They make it such a total PITA to restore from a hardware failure, that even if you do make backups, it’s takes entirely too long to rebuild the system you had.
Being better than Vista is kind of like being better than Bush or Howard, not much of a challenge.
Errrm… I’m not using that Chinese proxy… The s/w don’t (yet) work on Win7! 😉 Sadly! 😆 Was good while it lasted. 😉
Vista was good for one thing (two actually, possibly 3) 😉 It was so bad that Dell forced M$ to extend the life of XP, and Dell began shipping Linux as an alternative desktop OS. The third is that it made many ordinary people annoyed at how bad Vista was! Even some of the sheeple woke up! 😆
I am thankful that some of the dev tools I use are finally being ported to Linux! If the trend continues, it won’t be long before I can dump windoze entirely!! YES!! Then I will joyfully do the *happy dance*! 😀
I found a very good vector drawing program (open source, for a bonus) for linux that means I can finally do away with Ill.ustra.tor and the exorbitant fees the other crooks at A.d.o.b.e (no free plugs from me for them!) charge! it’s called Inkscape! Looks good so far. 🙂 I still use Canvas (that runs on Mac/Win), but it’s not the same since AC.D S.ystems took over Deneba! 🙁 *sigh* Still… it could have been worse!! The crooks at a.d.o.b.e might have gotten it!!
inkscape, eh? you must be psychic, kryten. i was thinking just the other day, as i was idly doodling in gimp and paint, that i missed corel draw from back in the day.
.-= last blog ..Once more unto the breach, single payer advocates, once more =-.
LOL Good timing I guess h! 😀 Here’s a link if you need it (assuming Bryans blog system, that really hates me for some reason… must be owned by M$!) 😛
Inkskape – Main Site
Inkscape Tutorials blog
Inkscape Wiki
If you (or anyone) are looking for other good GPL (or other free licensed) software, just ask. 🙂
Enjoy! 😀
oops! I forgot… Here’s a really good 3rd part manual for Inkskape in English, french, SPanish & Italian. Take your pick! 😀 (Sorry Bryan… No German or Russian! Though I am sure they are online somewhere.) 😉 😆
Inkscape: Guide to a Vector Drawing Program
Righto… errrm… back to work then. 😉
Well, it is possible that the people who wrote the filter don’t know that Australia is a continent, as well as an island and a nation. As I told a friend who is a Kiwi, she shouldn’t expect Americans to know about New Zealand – they don’t know about the old one. [Actually a lot of people who do know about New Zealand, don’t know about the original.]
I still limp along with Paint Shop Pro 4. Corel Draw was just too full featured for my abilities. I have gimp, but don’t have the time to read enough about it to migrate. I’ll keep Inkscape in mind, because if I accidentally become wealthy and buy a new box, it will be for Linux. I’ll leave this one on XP, just like I have left its predecessor on Win95.