They Can’t “Account” For The Money
Dan Dupont at reports on Pentagon Financial Systems: Still a Mess
Given that some of the computer systems in question “run on old-style I.B.M. mainframes and rely on Cobol, the ancient Sumerian of computer languages,” none of this should be much of a surprise.
“This was a bunch of systems patched together,” says Greg Bitz, a former director of the center. “I never went home at night without worrying about one of them crashing.” Bitz predicts a crisis as older programmers retire. “Try to find somebody today who knows Cobol,” he says.
Portfolio’s story is accompanied by a time line of efforts to reform the system since 1968, though it misses a key moment from 1997, when John Hamre, then the Pentagon comptroller, was being grilled by a lawmaker during a hearing on the woeful state of Pentagon financial computer systems. His answer: “The good news, sir, is they’re all going to collapse in the year 2000.”
So the problem is COBOL. Wikipedia notes:
COBOL is one of the oldest programming languages still in active use. Its name is an acronym for COmmon Business-Oriented Language, defining its primary domain in business, finance, and administrative systems for companies and governments.
The COBOL 2002 standard includes support for object-oriented programming and other modern language features.
As for where to find COBOL programmers, you could just grab some intelligent accounting types and send them to The COBOL Center, where they can locate the instructional materials to learn how to program, or you could go to an IT job site and start interviewing the thousands of programmers who learned how to program in COBOL in school. It isn’t a difficult language and it is expressly suited for accounting in the military, as it was created with major military support.
It is a hell of a lot easier to create and maintain an accounting system in COBOL than C, which is why it is still in use in many corporations. The military once had the programmers to take care of these systems before it was decided to “privatize” all of the non-core functions.
17 comments
I’m reminded of a joke told around the Y2K transition. A contractor, having done modifications to a couple hundred COBOL-based systems for different companies to accommodate Y2K, is so stressed out immediately before the change-over that he hires a company to put him in some sort of cryonic chamber that will put him in suspension until the year 2000 is past. In he goes, and into hibernation he goes.
When he awakens, it is to a scene where nothing is familiar. The technology looks very advanced. People’s clothing is utterly strange. People speak English, but so much of it is new to him that he has difficulty conversing with the team that opens his chamber. Still, he manages to understand this:
“The clock on your chamber was not Y2K compliant, so you were not awakened in 2000 as scheduled. By the calendar you knew, the year is now 9,998 A.D. We awakened you because we are facing the Y10K transition, and it says in your biography that you know COBOL…”
There are probably a billion or more line of COBOL code still running. It doesn’t do anything fancy, but it does do double-entry accounting like a charm. You can get a COBOL compiler that runs on almost anything still able to boot up. The last COBOL I personally tweaked was on a PDP 11 running DEC Unix.
I think I’ve got a version that ran on MS-DOS on a diskette somewhere.
The problem is that nothing has ever been created that does what it does, as well as it does it. You cut a deal with a Community College to train a certain number of students to write COBOL with an offer of a job if they complete the specified courses. Apprenticeships, that the quick and easy way. Two semesters of COBOL, a CICS course, and a year of accounting, in addition to the regular required courses.
*SIGH* I get sick of (usually ignorant) people blaming COBOL for everything. I learned to program in COBOL (and later Pascal) in the 70’s & 80’s. The main problem with software written in COBOL is that the software is written to do things COBOL wasn’t designed to do. I know Adm. Hopper (who I was lucky enough to meet at a conference in the mid 80’s) was often exasperated by some of the things COBOL was being used for.
Steve: LOL I think I heard a variant on that around the time of the Y2k terrors! 😀 I actually had to remember my COBOL to work on an accounting system here to make it Y2k compliant. It was a nightmare! Took 3 of us about 8 Months to find all the bad code and fix it. Still, I was paid well. 😀
Cheers!
Oh, yes, there was a lot of that going on, trying to make it dance and sing, when it was design to keep books. I’ve written accounting software in other languages, and the garbage you have to go through to keep the accounting honest, like storing the money values as integers, rather than floating point and then playing with divide by 100 for display, is a major PITA. COBOL on the 360/370 systems just did it. No drama, no graphics, no errors, just keeping books like Bob Cratchit did at Marley & Scrooge -a humble but honest computer language.
Yup! 😀
I have fond memories of my time at DEC. Started working on PDP11/03’s (we nicknamed it ‘The Angel’, though I forget why now) then up to the bigger PDP-11/70’s. I have been trying to remember the OS we used… (RSX-11M I think… yeah) and when we got our first disk drive (which looked kinda like a washing machine, but had removable disk packs with three platters, if my memory serves) we also used DOS-11 well before the lying & thieving dweeb Gates stole DOS from a school buddy (Does that make gates the first ever PC Pirate? Or just a cheap con artist?) 😉 LOL.
Then I got to play with VMS on VAX-11/750 & 780’s. 🙂 The first computer I ever worked on was a Sperry Univac 1108 (I think it was the 08) for an Airline reservation system we had here. I believe it was the only one of it’s kind here at the time. I do remember we got into trouble because we programmed it to play tic-tac-toe (naughts & crosses)! LOL Fun times! 😀
At last! I was beginnning to suspect a conspiracy, as ever time I’ve tried to visit lately your site was unavailable. And I apparently didn’t bookmark your ‘annex’.
You should talk to my brother-in-law, database honcho at the Secret Service. He gets called in to work regularly at 3 a.m. to mend their crappy, pieced-together system that has only gotten worse under Homeland Security.
He’s definitely planning on early retirement.
I had a relative who worked for DEC [and other minicomputer corporations], so I had a range of machines available, after working on a probable IBM mainframe [based on the programming languages used, as everything input on punched paper tape and shipped to the computer site] in the military and then college afterward. Then I got involved with minis [Data General Nova and DEC] before dropping down to the micro world and CP/M.
There were too many people looking over your shoulder on the mainframes, the minis were a lot more fun, especially when you were the system administrator.
Now I trudge through the most restrictive environment since IBM OS/VS. I’m beginning a migration to Linux, as soon as my client sign-on. It would have already happened if Gates had pushed his new licensing agreement that no business could afford to agree to, but Vista has certainly strengthened my hand.
Sorry about that, Andante, but something changed several weeks ago that makes my address hard to reach some days. I still am not sure exactly what changed, but I have to hit retry multiple times to get here.
The government definitely needs to update their systems, there is no doubt about that. Most of the current infrastructure is old and inefficient. Updating would save time and electricity, while being more reliable. The hardware has plummeted in price, and you don’t have to start from scratch. They just don’t have the senior managers who understand IT, and fewer and fewer full-time people available who do.
This attitude of ignoring scheduled maintenance and upgrades to save money, is costing us now that the systems are failing. They won’t do anything until there is a catastrophic failure.
Yanno what scares me? I’m a barely conscious, inexperienced programmer, and I know what all those old DEC computers are. I’ve even worked on a couple of them…
Scary.
Indeed, nothing wrong with Cobol for keeping books. Ah yes, “pictures”. Being able to store data as decimal numbers with two digits to the right of the decimal place is what you want for dollars and cents, and the IBM mainframes have the hardware to handle it efficiently, unlike personal computers. Only other language I’d want to do this with would be PL/1, mostly because I prefer the more Algol-like syntax, but must admit that the recent COBOL standards have taken away a lot of the “ick” from the days when we thought 3270 terminals were high tech…
The DEC PDP-11 series was an extremely popular machine, Аня, so it isn’t surprising that anyone who has programmed in an medium-sized company or college, has worked on them.
Now, if you had time on a PDP-10 mainframe, that would be a surprise. I bought time on one for a couple of things, but it was done with a terminal connected via a modem.
Badtux, if you had been using a 029 keypunch for a while, the 3270 was high tech. I only had exposure to PL/1 at college, and the required programs weren’t very exciting, but the possibilities and logic were definitely more interesting than COBOL.
Face it, all you do with COBOL is re-write the same program in a slightly different fashion. I have a suspicion that some people didn’t like it because it produced good standard accounting results, and they wanted something a little more “adventurous” and “interesting”.
You are right about the maintenance issue Bryan, and outsourcing has been a disaster! In fact, there was a recent report here that Aus companies are beginning to bring back their outsourced service & support locally because the grand experiment in India & elsewhere is a failure. Cutting corners has *NEVER* solved any problems. It’s always a short-term, narrow minded person’s solution. By the time the shortcut bites the company or organization in the ass, the irresponsible moron has left and pocketed a huge bonus on the way.
Anya: Not scary at all! LOL 😉 Ya know what’s so great about those old systems? Predictability and reliability. 🙂
Badtux: The thing I always saw about COBOL was a curious dichotomy. It’s strength is also it’s weakness. 🙂 It was quite an *inflexible* language. Very simple, and very structured. But it was very predictable. A line of code did one simple thing (usually). That made complex systems quite large and certainly an equivalent ‘C’ program would be much more compact, but C goes the other way in being too flexible and generally less predictable. PL/1 scared me! It had a huge syntax! LOL But I also had to learn Algol (and Fortran) then PL/1 (eventually the simpler PL/M). COBOL Has evolved, and as you say, has less “ick”! 😀
When I began working in Industrial Design, I had to learn BCPL initially (which most people never heard off!) Then forth and Prolog, and a strange language called Occam which was used to program the amazing Inmos Transputers! *sigh* I loved those things! The Transputer and Occam were a dream come true for any engineer trying to develop complex automated systems. A UNIX company called Apollo had a Transputer Development System (TDS) for it. Apollo were a competitor to Sun Microsystems, and was eventually taken over by HP sadly. Apollo were amazing at the time as the OS it ran, called Aegis, was fully interoperable and compatible with Sys V, BSD & POSIX. IT had the fastest networking capabilities built in as standard (12Mb/s Token Ring, IBM’s at that time was 4Mb/s) and 10Mb/s Ethernet and fully supported diskless workstations (as drives were quite expensive back then, this was a big selling point).
Sadly, Inmos was taken over by Thompson and they buried commercial development of the Transputer in favor of Military development.
The thing that annoys me the most today (and for the past couple decades) is that companies that produce an excellent product get swallowed up by a big shark and the product either becomes mediocre or eventually is simply killed off in favor of the big shark’s mediocre products. Many companies, especially M$, HP, Symantec and others, do this often. It is difficult to produce excellence when the tools one is forced to use are mediocre.
Cheers.
The best product that I ever architected and was principle designer on was originally targetted for death by a company that planned to bury it for fear of competition. The product was saved only because of some real skulduggery that I won’t go into but it’s still sold today by a different company, a fact I’m quite proud of.
This brings up a real problem with capitalism — the guys doing the innovation are not money guys and really don’t like running a business, so anybody offers them enough zeros on the check, they grab it and run and go do something else. Problem is, it’s *hard* to innovate, and they rarely end up doing anything else that’ll give them the same rush they got the first time.
Bryan — time on PDP-10 timesharing computers was quite common and available during the mid to late 1970’s and many high schools purchased time on Tymshare or its equivalent prior to the Apple II making it feasible for high schools to own their own computers. So if someone was born in the early 1960’s it wouldn’t surprise me if they had exposure to a PDP-10 during their high school years, presuming they were in a school district capable of handling the rather extravagant costs of having all those phone lines for 300 baud modems to talk to Tymnet and the hefty per-byte charges and per-CPU-minute charges. If I’m looking at timelines right, you were flying around the sky looking at the Ruskies at the time, so you probably missed out on that brief but important era in computing history when folks had access to mainframes via modem… the PC killed that concept deader than dead because why rent part of a mainframe when you can own a whole PC for less? So it goes…
One might almost think all on this thread are a bunch of old guys ‘n’ gals reliving their glory days. Almost. 😈
FWIW, I cut my teeth on the original Rice University Computer, designed and built there in the late 1950s both as an experiment in architecture and as a practical computing facility for the theoretical physicists etc. who kept exceeding the available horsepower on the IBM 7094. It was still in use in the late 1960s when I was an undergraduate.
The R-1 (yes, there was eventually an R-2, sort of) was an impressive piece of hardware for its time. I am glad I arrived after the CRT main memory had been replaced with core!
And yes, occasionally, standing at the main console that looked as if it could run the Starship Enterprise (the latter conceived of many years later), debugging some odd program or another, I reached out and turned the crank. Yes, that computer had a crank. The crank was connected to a cam that drove a microswitch, about 50 instructions per turn. While one cranked, one watched the registers in lights through the first Plexiglas case, which stood just beyond the console. The lights one watched were attached to the actual register circuits.
Those were the days! (I do not miss those days.)
cobol schmobol. fortran rulez!
IBM was rather infamous for buying up patents to bury them if it felt they threatened core products in any way. When I was in San Diego I did a lot of consulting work for the biggest computer store there and got to play with some wonderful, innovative products that died as soon as the little companies that made them were bought out. The big guys didn’t want the competition, so they bought the companies and patents. That doesn’t even address the “clone wars”, when people finally figured out it didn’t have to say IBM to be a PC.
Even in the mini world you saw the innovation die when “professional” management took over successful tech companies from the engineers who founded them. The “professionals” weren’t interested in new products, they wanted to suck all of the profit possible from the old products. This concept is behind the decline of the manufacturing sector in the US, as plants and processes weren’t modernized, the work was shifted to new factories in other places, first in the South, and then overseas.
Badtux, I used the DEC-10 in Rochester, NY in the late 1970s. It was a local machine and system. Time sharing was common in Rochester, with local companies buying time on mainframes at the local universities, and some corporations selling excess capacity via modem terminals. It was a lot cheaper than any of the national systems, but there were limits on the windows of use.
That crank sounds like a better system than the toggle switches on most minis, Steve.
As an aside, Bryan…
Legend has it that when your uncle (my father) was going to MIT in the early ’50’s his office was located next door to the room in which the Model Railroad Club was developing the first electronic computer. He’d warm his office in the wintertime with the heat from the vacuum tubes by opening the vents between the two rooms.
Ya don’t suppose that’s how I came by my natural geekiness, via radiation?
Your uncle was messing about with missile guidance systems, and nuclear warheads, so radiation is always a possibility, but genetics is an easier explanation, without the radiation.
The Rochester Institute of Technology heats one of its buildings during the winter with waste heat from the computers.