Code Red Battling Google, Microsoft Changes How It Builds Software

Code Red
Battling Google, Microsoft
Changes How It Builds Software

Delay in New Windows Version
Drove Giant to Develop
Simpler, Flexible Product

Engineers Get Trip to 'Bug Jail'

By ROBERT A. GUTH
Staff Reporter of THE WALL STREET JOURNAL
September 23, 2005; Page A1

REDMOND, Wash. -- Jim Allchin, a senior Microsoft Corp. executive, walked into Bill Gates's office here one day in July last year to deliver a bombshell about the next generation of Microsoft Windows.

"It's not going to work," Mr. Allchin says he told the Microsoft chairman. The new version, code-named Longhorn, was so complex its writers would never be able to make it run properly.

The news got even worse: Longhorn was irredeemable because Microsoft engineers were building it just as they had always built software. Throughout its history, Microsoft had let thousands of programmers each produce their own piece of computer code, then stitched it together into one sprawling program. Now, Mr. Allchin argued, the jig was up. Microsoft needed to start over.

Mr. Gates resisted at first, pushing for Mr. Allchin's group to take more time until everything worked. Over the next few months, Mr. Allchin and his deputies would also face protests from programmers who complained he was trying to impose bureaucracy and rob Microsoft of its creativity.

"There was some angst by everybody," says Mr. Gates of the period. "It's obviously my role to ask people, 'Hey, let's not throw things out we shouldn't throw out. Let's keep things in that we can keep in.' "

Ultimately, Mr. Allchin's warning proved cathartic and led to what he and others call a transformation in Microsoft's most important product. A key reason: the growing threat from rivals such as Google Inc., Apple Computer Inc. and makers of the free Linux operating system. In recent years these companies have been dashing out some software innovations faster than Microsoft. Google has grown particularly effective at introducing new programs such as email and instant messaging over the Internet, watching how they perform and regularly replacing them with improved versions.

Microsoft's Windows can't entirely replicate that approach, since the software is by its nature a massive program overseeing all of a computer's functions. But Microsoft is now racing to move in that direction: developing a solid core for Windows onto which new features can be added one by one over time.

As always, Microsoft's great fear is that it will lose its near-monopoly on computer operating systems and basic office software. In the short term, there is little danger of that. But the more Google and other software makers encroach on Microsoft's turf, the greater the chance that someday computer users will wake up and find Microsoft Windows superfluous.

"What happened when the American car companies failed to update their manufacturing lines? There was a more efficient way to bring cars to market for a lower price and they lost their market," says Microsoft Vice President Chris Jones. "We're in a little bit of a different industry but it's the same thing."

Microsoft's holy grail is a system that cranks out a new, generally bug-free version of basic Windows every few years, with frequent updates in between to add enhancements or match a competitor's offering.

The Longhorn crisis helps explain the sweeping restructuring that Microsoft Chief Executive Steve Ballmer announced this week to organize the company into three major business units. A key goal is to force Microsoft to be more nimble in producing and delivering software.

Mr. Allchin's reforms address a problem dating to Microsoft's beginnings. Old-school computer science called for methodical coding practices to ensure that the large computers used by banks, governments and scientists wouldn't break. But as personal computers took off in the 1980s, companies like Microsoft didn't have time for that. PC users wanted cool and useful features quickly. They tolerated -- or didn't notice -- the bugs riddling the software. Problems could always be patched over. With each patch and enhancement, it became harder to strap new features onto the software since new code could affect everything else in unpredictable ways.

The 53-year-old Mr. Allchin, who joined Microsoft in 1990 and is now co-head of the Platform Products and Services Division, says he always disdained the fast-and-loose culture of PC software. The holder of a doctorate in computer science, Mr. Allchin craved discipline in code writing. But in the booming 1990s, when it seemed Microsoft could do no wrong, there was little Mr. Allchin could do. As soon as Microsoft was done with one version it pushed on to the next. Mr. Allchin was haunted by what he calls his "little demons."

In 2001 Microsoft made a documentary film celebrating the creation of Windows XP, which remains the latest full update of Windows. When Mr. Allchin previewed the film, it confirmed some of his misgivings about the Windows culture. He saw the eleventh-hour heroics needed to finish the product and get it to customers. Mr. Allchin ordered the film to be burned.

When the Longhorn project to build an XP successor got started, teams of engineers set off to develop it as they always had. Mr. Gates was especially eager for them to add a fundamental change to Windows called WinFS that would let PC users search and organize information better. One goal was to let users scour their entire computer for work they had done on a subject without needing to go through every individual program or document.

Mr. Allchin says he soon saw his fears realized. In making large software programs engineers regularly bring together all the new unfinished features into a single "build," a sort of prototype used to test how the features work together. Ideally, engineers make a fresh build every night, fix any bugs and go back to refining their features the next day. But with 4,000 engineers writing code each day, testing the build became a Sisyphean task. When a bug popped up, trouble-shooters would often have to manually search through thousands of lines of code to find the problem.

Mr. Gates's WinFS project was so troublesome that engineers began talking about whether they could make the "pig fly." Images of pigs with wings started appearing in presentations and offices.

And Microsoft's culture was facing a new threat. The mass of patches and agglomerations that made up Windows turned it into an easy target for viruses and other Web-based attacks. Mr. Allchin had to divert top engineers into the effort to fix security problems in existing versions of Windows. "The ship was just crashing to the ground," Mr. Allchin says.

In late 2003, Mr. Allchin called on the help of two men. The first was one of Microsoft's best-known "shippers," people known for their ability to turn around troubled software projects. Windows veteran Brian Valentine had a reputation for booming motivational speeches, beer bashes and stunts like showing up to work functions as Elvis, the Easter Bunny or even once a hula girl with a coconut bra.

The second man Mr. Allchin tapped was Amitabh Srivastava, now 49, a fellow purist among computer scientists. A newcomer to the Windows group, Mr. Srivastava had his team draw up a map of how Windows' pieces fit together. It was 8 feet tall and 11 feet wide and looked like a haphazard train map with hundreds of tracks crisscrossing each other.

That was just the opposite of how Microsoft's new rivals worked. Google and others developed test versions of software and shipped them over the Internet. The best of the programs from rivals were like Lego blocks -- they had a single function and were designed to be connected onto a larger whole. Google and even Microsoft's own MSN online unit could quickly respond to changes in the way people used their PCs and the Web by adding incremental improvements.

In April 2004, Google, seemingly out of nowhere, introduced its Gmail service, competing with Microsoft's Hotmail program. Tiny Internet browser maker Mozilla Foundation beat Microsoft to market with browser features planned for Longhorn.

Most alarming: By July 2004, it became clear that Google was working on a "desktop search" tool for finding information on a PC -- offering some of the features that Mr. Gates's WinFS program was supposed to bring to Longhorn. Google, previously focused exclusively on the Internet, was now stepping onto Microsoft's turf as the creator of software inside the PC.

While Windows itself couldn't be a single module -- it had too many functions for that -- it could be designed so that Microsoft could easily plug in or pull out new features without disrupting the whole system. That was a cornerstone of a plan Messrs. Srivastava and Valentine proposed to their boss, Mr. Allchin. Microsoft would have to throw out years of computer code in Longhorn and start out with a fresh base. It would set up computers to automatically reject bug-laden code. The new Longhorn would have to be simple. It would leave bells and whistles for later -- including Mr. Gates's WinFS, Messrs. Srivastava and Allchin say.

Mr. Allchin signed on to the plan and broke the news to Messrs. Gates and Ballmer. Mr. Allchin remembers that Mr. Gates pushed him to keep going with the original version of Longhorn, saying if the software writers needed more time Microsoft could ship a scaled-down version in the interim. The executives agreed to reserve a final decision until Mr. Ballmer returned from a business trip, according to Mr. Allchin and Mr. Valentine, who was also present.

Over the next few weeks, Mr. Gates expressed frustration. At one meeting on Aug. 17, he berated Longhorn engineers for the mess, say people familiar with the meeting. (Mr. Gates says he doesn't remember it.) Afterward, Mr. Srivastava says he called his team together, acknowledging that he had underestimated the scope of the challenge they faced in fixing Longhorn, though he was heartened by the group's apparent willingness to change.

As Microsoft's chief software architect, Mr. Gates says that his role is "almost paradoxical" because he has to push for innovation while being the "ultimate realist" when problems arise on that quest. In this case, he says he and Mr. Ballmer needed to make sure that the recommendations from Mr. Allchin's group were sound.

On Aug. 27, 2004, Microsoft said it would ship Longhorn in the second half of 2006 -- at least a year late -- and that Mr. Gates's WinFS advance wouldn't be part of the system. The day before in Microsoft's auditorium, Mr. Allchin had announced to hundreds of Windows engineers that they would "reset" Longhorn using a clean base of code that had been developed for a version of Windows on corporate server computers.

As he started to learn more about Mr. Srivastava's broader plan, Mr. Gates was concerned that the unproven tools for keeping the Windows core clean would levy a "tax" on engineers -- in other words, that they would spend so much time trying to meet Mr. Srivastava's standards that they wouldn't be able to devise innovations for Windows users. At a meeting on Sept. 8, Mr. Srivastava's team was walking Mr. Gates through the plan when he challenged them. Why, he wondered, weren't the reformers asking the mass of Windows engineers for their view of the changes?

"It was all just, 'Hey, bless this process,' which I was unwilling to do," Mr. Gates says. "They're just talking about process and I'm frustrated we're not talking about how the teams are responding to it."

By late October, Mr. Srivastava's team was beginning to automate the testing that had historically been done by hand. If a feature had too many bugs, software "gates" rejected it from being used in Longhorn. If engineers had too many outstanding bugs they were tossed in "bug jail" and banned from writing new code. The goal, he says, was to get engineers to "do it right the first time."

Recognizing Mr. Gates's concerns over the impact on programmers, Mr. Srivastava hit on a plan to win their hearts and minds. On Nov. 5, he visited the computer-filled office of Dave Cutler, a revered elder statesman among Windows engineers and a stickler for good code writing. Would he publicly throw his weight behind the new approach? Mr. Srivastava asked.

On Dec. 1, Mr. Srivastava escorted Mr. Cutler to Microsoft's auditorium where the software guru told 1,000 engineers that he had used the tools to build Windows code that was nearly bug-free. That Mr. Cutler -- famous for never attending meetings -- would emerge to back Mr. Allchin's revolution helped persuade some engineers to drop their objections.

Others weren't so easily convinced. Responding to an attendee questioning the merits of the new regime, Mr. Valentine, the enforcer, shot back, "Is your code perfect? Are you perfect? If not, you should shut up and support this effort," according to one of his team members, G.S. Rana. (Mr. Valentine says he doesn't remember the remarks but doesn't dispute Mr. Rana's recollection.)

As engineers began cooperating and Mr. Srivastava's team worked overtime to refine the tools, the quality of the code flowing into Longhorn began to improve. The time to create a new "build" fell to just a few days, allowing a faster cycle of writing and testing new code. After the Windows group was able to install a workable version of the system on their PCs four days before Christmas, Mr. Srivastava says the group celebrated by not working over the holidays.

Not everything went so quickly, as engineers grappled with the challenge of making Longhorn more like Lego blocks. Microsoft missed its June deadline for the first "beta" or test version of Longhorn. On the Fourth of July Mr. Srivastava monitored the progress on his wireless laptop, set up next to his grill as he cooked veggie burgers and teriyaki chicken for family guests. Mr. Srivastava was so preoccupied with Longhorn that he inadvertently agreed to his wife's plan to remodel their bedroom. He recalls that when he protested, she joked, "You got the Windows job. I get this. It's a small price to pay."

On July 27, Microsoft shipped the beta of Longhorn -- now named Windows Vista -- to 500,000 customers for testing. Experience had told the Windows team to expect tens of thousands of reported problems from customers. Instead, there were a couple thousand problem reports, says Mr. Rana, the team member.

And last month, Microsoft delivered a test version of Mr. Gates's WinFS idea -- not as a part of Longhorn but as a planned add-on feature. Microsoft this month said it would issue monthly test versions of Windows Vista, a first for the company and a sign of the group's improved agility.

It could take years before Windows can be as flexible as Microsoft needs it to be to pump out new features quickly. But the cultural shift is in swing. Hours after showing off Windows Vista to software makers this month, Mr. Gates in an interview noted how Microsoft's Office group is now using some of Mr. Srivastava's tools to improve its code. "It's amazing the invention those guys have brought forward," he said. "I wish we'd done it earlier."

This week Mr. Allchin announced that as part of the restructuring he will retire next year after Windows Vista is in customers' hands. In a recent interview he said his demons aren't fully exorcised. "There're weaknesses in everything we're doing today," Mr. Allchin says. "But it's such a huge step up from where we were."

Write to Robert A. Guth at rob.guth@wsj.com