Sigh…
For some reason I missed the connection in Ellroon’s Friday Blog Sprinkles. At the end there is a story that seems to trace back to Brian Krebs of the Security Fix blog at the Washington Post site about a crack of the server at the Virginia Department of Health Professionals. A data base of drug prescriptions was stolen, and the original and several back-ups were apparently deleted. The cracker [the computer type, not the cowboy type] is demanding $10 million for the return of the data base.
It finally clicked that at the end of April Virginia’s Secretary of Technology, Aneesh Chopra, was selected as the first Federal Chief Technology Officer [CTO], despite having only public health degrees.
It gets worse. The Washington Examiner reports:
The Virginia agency recently attacked by a hacker has yet to receive a computer security upgrade ordered five years ago, a spokeswoman for Gov. Tim Kaine told The Examiner.
…
On Friday, Kaine spokeswoman Lynda Tran said the Department of Health Professionals’ small size meant the agency was among the last on the list to receive the security upgrades Gov. Mark Warner ordered five years ago. The upgrades were scheduled to be finished by the end of 2009, and state workers are on pace to get it done, Tran said. The 2004 order created the Virginia Information Technologies Agency, which is now coming under fire from lawmakers for the hacker’s Web site breach.
I’m not familiar with the Washington Examiner, but this is just a straight report of the words of a government spokesperson, which was good enough for the Washington Post for eight years.
Being geeks, people had to check to see what kind of server software was cracked … OK, what they really did was verify that it was Microsoft’s IIS running on the system.
Five years without a security upgrade? FIVE FREAKING YEARS!?!?!?
Sorry, but that isn’t quite up to industry standards for home offices, and the guy in charge is now going to lead the effort to convert the nation to electronic medical records.
14 comments
Yep. I said Chopra was an IT imbecile. This proves it. Curiously, I’m not even minutely moved to say *unbelievable* as it’s entirely believable. I am certain there have been many more successful attacks on Gov systems than are being reported. Wouldn’t surprise me if departments didn’t even know they’d been breached.
I thought Obama was smarter than Bushmoron and supposedly had a clue?
They should hire a bunch of high school students to manage their security. It’d be a vast improvement.
That all depends upon the meaning of the word “security upgrades”. My impression was that the upgrades in question were infrastructure upgrades, the usual package for modern businesses nowdays — firewalls between the various components of the system with firewall rules to allow only approved transactions through, intrusion detection systems, yada yada yada. That web server would have been toast a long time ago if it hadn’t gotten the regular Mickysoft security patches on at least a quarterly basis — you know and I know that Mickysoft has way too many exploits to have remained unhacked for that long if there hadn’t been any of the security patches installed.
Badtux´s last blog post..Grumpy about the newspaper industry
Oh yeah, regarding Chopra, he’s one of them Indian fellers, and everbody knows them thare Indians are, like, just technical geeky wizards and stuff who know all about them thare computer thangies, y’know? I mean, they’re just born like that, knowin’ all about them thare computer thingies. Y’all quit hatin’ on Chopra!
– Badtux the Snarky Penguin
(Channeling Bubba the Suthern Penguin)
Badtux´s last blog post..Grumpy about the newspaper industry
IT courses, classes and certificates aren’t a guarantee of competence, but they do show a willingness to get involved with the subject, and it is damn difficult to oversee technology if you don’t have a basic grounding in the terminology. There are too many good people in the field, to go out of the field for a choice like this.
There isn’t any clarification of what the mean in any of the articles I could find, but it has to be more than the patches, unless there is no one available to install the patches over their network. Microsoft certainly notified them of the patches in the monthly broadcast, but there’s no way of telling if they have an on-site sys admin.
The prioritizing of the upgrades is skewed. This server was actively used by people from around the state, so, while the number of people in the department may have been small, the users included every pharmacy in the state. With that much outside access over the ‘Net, it was an obvious target.
I would also look at the program they were using, and whether whoever created it used good security practices. Good application security practices can stop a lot of cracks, even if the underlying server software isn’t that great.
It is always possible that this was an inside job by a disgruntled current or former employee, or contractor.
Yes, Badtux, I have to wonder if “diversity” wasn’t part of this decision, but I know people from a lot of different cultures who have the “chops” for a major technology job. There were certainly a lot of different groups represented in Silicon Valley, who have education and experience in the field. Hell, they could have picked a penguin, that would have been real diversity.
I have no words.
I guess I was making fun of the tendency to stereotype all Indians as “technology gurus”. Of course they aren’t, any more than all Russians are drunk or all Germans are anal.
My reason for guessing that this was an infrastructure issue have to do with the sheer scale of the data loss. Web servers getting compromised is an everyday occurance, but it is rare that there’s this scale of data loss on a properly designed network. The firewall between the web server and the database server keeps data from leaving the company (the database server only being allowed to talk to a small set of internal addresses and only on specific ports with specific protocols), the web proxy between the web server and the Internet only allows web requests in and out, the IDS detects any breaches of the proxy and web server long before they can somehow figure a way to worm into the database server, and in general a properly designed infrastructure just isn’t going to be utterly and catastrophically breached like this, regardless of whether IIS patches have been applied or not, and regardless of whether the application was breached or the OS was breached.
But of course government IT is done by the lowest bidder, and I know from first-hand experience how that works — underpaid peons (because lowest bidder can’t afford to pay market rates) who have little opportunity to advance their skills (because lowest bidder can’t afford to buy the latest stuff for them to play with and overworks them to keep from having to hire more people meaning they have no time to just advance their skills), and just generally second-rate work. Add in insane agencies that don’t know what they want and keep changing their minds (most budget slips come not because of the contractor, but because the contracting agency decides “oh yeah, we need this one more feature” — hundreds and hundreds of times), and the wonder is that any government IT project actually works.
Add in the fact that Virginia was one of the first states on the Internet, and like most early government attachments to the Internet assumed it was connecting to a secure government network (only government agencies and defense contractors allowed to connect to the Internet back then, remember?) and not to a hostile network succeptible to Russo-Chinese hackers and such, and thus originally every single computer on Virginia’s network *had a public IP address*, and you start to see the magnitude of the infrastructure issues Virginia faced upon the dawn of the modern Internet era…
Badtux´s last blog post..Grumpy about the newspaper industry
You are “the snarky Penguin” and everyone who has read you for any length of time would know what your intention was, as I did. I was addressing a separate issue with the administration. Diversity is good, but there are a lot of good people who have diverse backgrounds and need a job, so why select a political appointee? Don’t ask a governor, ask someone in the business for recommendations, ask a geek.
Even a software firewall would have stopped or set off all kinds of alerts at this wholesale damage. This job required administrative privileges, and even without the hardware, you can throw up a “hedge of thorns” until the permanent fortifications get built. There are well documented ways of locking down IIS, but they seem to have been waiting for someone else to do something.
This is the problem with low bid – the same groups keep winning the bids and getting contracts even though they continue to fail. The contractor names occasionally change, but the people who profit remain the same.
It would have been cheaper and faster to build a state server farm that could be secured, and have the agencies use it just like most of the world uses hosting companies.
I assumed that Chopra was political payoff to somebody, put into the CTO position because it fits the stereotype of Indians as technology geeks and thus wouldn’t get an eyeblink from Senators who know no better when it came comfirmation time. But I’m just cynical that way.
Regarding the state server farm: But Bryan, whose budget would have financed that state server farm? Will the Federal Medicaid administrative overhead grant cover hosting costs for a shared server? I mean, c’mon. You’re talking sense, but we’re talking *government* here. When I was doing school automation, the attendance system wouldn’t talk to the free lunch system because they were on two entirely different funding mechanisms and purchased by two entirely different departments from two entirely different venders via two entirely different bidding processes. If the IT department had a server to share between the school lunch and attendance systems under the Federal rules they would have had to bid it out as if they were a private vender, which would work only if both the attendance and lunch systems were bid out at the same time because otherwise you have IT carrying that cost and where’s the funding stream for that? Take all the bureaucratic nonsense you have to go through in any large business, and scale it up a thousand times, and you might *start* to appreciate the difficulties here of what you proposed…
Goverment work is just… different. I’m sure glad I’m out of that business now!
Badtux´s last blog post..Grumpy about the newspaper industry
Government entities can submit bids just like everyone else, and if someone can underbid and provide the same level of service, go for it.
Yeah, they have the same stupid segregation of funding in Florida, which makes government services more expensive. Coupled with the rules against government agencies competing for contracts, is guaranteed to increase costs.
In hurricane country I would co-locate the server farm with the emergency management center so you only have to build a single hardened structure with back-up power and satellite communications capabilities.
I keep thinking like a systems analyst when there really isn’t a system to analyze.
Well, there *is* a system to analyze, but it’s one whose sole intent is to make it as easy to track taxpayer money as possible to make it easier to detect anybody not politically connected who is siphoning taxpayer money into their own pockets — not one whose intent is to make the most efficient use of taxpayer money. It’s all about “accountability”. It’s a wonder that government systems work as well as they do, given the hurdles set up to make sure that not one sent intended to go for, say, school lunches, gets spent on infrastructure that could also benefit, say, special education. Nevermind that both are Federal money streams, god forbid that one dime of special ed money gets spent on something that might accidentally benefit the school lunch program or vice-versa.
Now, granted, you have these kinds of issues in private enterprise too. But you don’t have the politicians and outside special interest groups involved in that case, just the normal internal politics. In private enterprise, you could share a student database server between school lunch and special education without a problem, each department would simply get invoiced by the IT department for the IT services involved, shuffle the money in their budget, and so it goes. With government you’d end up having to do a RFC, RFB, bid, bid response, blah blah blah *plus* you’re guaranteeing an audit by the Feds that’ll chew up even more time and money to guarantee that not one dime of special ed money is benefiting free lunch and vice-versa, to the point where it’s cheaper just to buy a second server for special education rather than to share one with school lunch (especially considering the manpower shortage that most government IT bodies have, where there’s not enough to people to handle all normal responsibilities, much less any added ones like putting together bids). And all this despite the fact that the money for both comes from the exact same damned place…
Like I said, I am *so* glad to no longer have any involvement in government IT.
Badtux´s last blog post..Dr. Doom: We’ve probably averted a total collapse
We have brand new computers sitting a store room covered in plastic because there was money to buy the computers, but no money to fix the roof or to pay for the electricity required to run the computers. The money would have to come from three separate accounts, and only the equipment account has any money in it.
Those are the state requirements, not even considering any Federal requirements.
One year budgets, locked accounts, and no local flexibility makes for a totally screwed-up non-system. Charter schools have none of these restrictions, and people wonder why they look more efficient on paper.
Epitomy of insanity: I worked at a computer repair shop around 1991 or so. A modem came in — a 2400 baud modem, over five years old, from the university. “Fix it,” they said. “It’d be cheaper to buy a new modem,” the boss man said. “We can’t do that, we have money in the repair budget, but not in the capital budget.” So the boss man thought about it a few minutes, and invoiced them for a new modem circuit board, new modem case, and new modem power supply, and handed’em a brand new USR 14,400 baud modem for $95.
Sad to say, that’s still how government IT works 🙁 .
Badtux´s last blog post..Hmm, he don’t like Dick Cheney
Oh yeah. My company back then used to sell truck loads of *spare parts* to Gov agencies. Computer and other office equipment in the 80’s and 90’s was notoriously unreliable! Always needed new parts. 😉 Curiously, they always got enough parts to build a complete system. For software, we’d just pad the price of the parts to cover the price and just give it to them. Ahhh… There are a lot of strange stories I know from that time. The World has always been totally crazy. 🙂
And the need and use of these work-arounds makes it impossible to create a logical budget. Because of the replacement schedule for vehicles, we used to buy a half a car a year, i.c. at the end of a year we bought the front-half and the first day of the next year we bought the back-half. Money couldn’t be carried over for large purchases; leasing wasn’t possible; and they wouldn’t approve a jump for a car every other year. 🙄