Windows
A Software Engineering Odyssey
Mark Lucovsky
Distinguished Engineer
Microsoft Corporation
<br /> <p><font color="#660066" size="7" face="Tahoma">Agenda</font> <br /> </p> <p><font color="#40458C" size="7" face="Tahoma">History of NT</font></p> <p><font color="#40458C" size="7" face="Tahoma">Design Goals/Culture</font></p> <p><font color="#40458C" size="7" face="Tahoma">NT 3.1 Development vs. Windows 2000 Development</font></p> <p><font color="#40458C" size="7" face="Tahoma">Development for the next 10 years</font></p> <br /> <p><font color="#660066" size="7" face="Tahoma">NT Timeline first 10 years</font> <br /></p> <p><font color="#40458C" size="7" face="Tahoma">2/89 Coding Begins</font></p> <p><font color="#40458C" size="7" face="Tahoma">7/93 NT 3.1 Ships</font></p> <p><font color="#40458C" size="7" face="Tahoma">9/94 NT 3.5 Ships</font></p> <p><font color="#40458C" size="7" face="Tahoma">5/95 NT 3.51 Ships</font></p> <p><font color="#40458C" size="7" face="Tahoma">7/96 NT 4.0 Ships</font></p> <p><font color="#40458C" size="7" face="Tahoma">12/99 NT 5.0 a.k.a. Windows 2000 ships</font></p> <br /> <p><font color="#660066" size="7" face="Tahoma">Unix Timeline first 20 years</font> <br /></p> <p><font color="#40458C" size="6" face="Tahoma">’69 Coding Begins</font></p> <p><font color="#40458C" size="6" face="Tahoma">’71 First Edition – PDP 11/20</font></p> <p><font color="#40458C" size="6" face="Tahoma">’73 Fourth Edition – Rewritten in C</font></p> <p><font color="#40458C" size="6" face="Tahoma">’75 Fifth Edition – Leaves Bell Labs, basis for BSD 1.x</font></p> <p><font color="#40458C" size="6" face="Tahoma">’79 Seventh Edition – One of the best</font></p> <p><font color="#40458C" size="6" face="Tahoma">’82 System III</font></p> <p><font color="#40458C" size="6" face="Tahoma">’84 4.2 BSD </font></p> <p><font color="#40458C" size="6" face="Tahoma">’89 SVR4 Unification of Xenix, BSD, System V</font></p> <ul><p><font color="#40458C" size="6" face="Tahoma">NT development begins</font></p></ul> <br /> <p><font color="#660066" size="7" face="Tahoma">History of NT</font> <br /> </p> <p><font color="#40458C" size="6" face="Tahoma">Team forms November 1988</font></p> <p><font color="#40458C" size="6" face="Tahoma">Six guys from DEC</font></p> <p><font color="#40458C" size="6" face="Tahoma">One guy from Microsoft</font></p> <p><font color="#40458C" size="6" face="Tahoma">Build from the ground up</font></p> <ul><p><font color="#40458C" size="6" face="Tahoma">Advanced PC Operating System</font></p></ul> <ul><p><font color="#40458C" size="6" face="Tahoma">Designed for for desktops and servers</font></p></ul> <ul><p><font color="#40458C" size="6" face="Tahoma">Secure, scalable SMP design</font></p></ul> <ul><p><font color="#40458C" size="6" face="Tahoma">All new code</font></p></ul> <p><font color="#40458C" size="6" face="Tahoma">Schedule: 18months (only missed our date by 3 years)</font></p> <br /> <p><font color="#660066" size="7" face="Tahoma">History of NT (cont.)</font> <br /> </p> <p><font color="#40458C" size="6" face="Tahoma">Initial effort targeted at Intel i860 code-named N10, hence the name NT which doubled as N-Ten and New Technology</font></p> <p><font color="#40458C" size="6" face="Tahoma">Most development done on i860 simulator running on OS/2 1.2 (took about 30 minutes) </font></p> <p><font color="#40458C" size="6" face="Tahoma">Microsoft built a single board i860 computer code named Dazzle including the supporting chipset and actually ran a full kernel, memory management, etc on the machine.</font></p> <p><font color="#40458C" size="6" face="Tahoma">Compiler came from Metaware with weekly UUCP updates sent to my Sun-4/200.</font></p> <p><font color="#40458C" size="6" face="Tahoma">Microsoft wrote a PE/Coff linker as well as a graphical cross debugger </font></p> <br /> <p><font color="#660066" size="7" face="Tahoma">Design Longevity</font> <br /> </p> <p><font color="#40458C" size="6" face="Tahoma">OS Code has a long lifetime</font></p> <p><font color="#40458C" size="6" face="Tahoma">You have to base your OS on solid design principles</font></p> <p><font color="#40458C" size="6" face="Tahoma">You have to set goals, and not everything can be at the top of the list</font></p> <p><font color="#40458C" size="6" face="Tahoma">You have to design for evolution in hardware, usage patterns, etc.,</font></p> <p><font color="#40458C" size="6" face="Tahoma">Only way to succeed is base your design on a solid architectural foundation</font></p> <p><font color="#40458C" size="6" face="Tahoma">Development environments never get enough attention…</font></p> <br /> <p><font color="#660066" size="7" face="Tahoma">Goal Setting</font> <br /> </p> <p><font color="#40458C" size="6" face="Tahoma">First job was to establish high level goals.</font></p> <ul><p><font color="#40458C" size="5" face="Tahoma">Portability – Ability to target more than one processor, avoid assembler, abstract away machine dependencies. We purposely started the i386 port very late in order to avoid falling into a typical, Microsoft, x86 centric design.</font></p></ul> <ul><p><font color="#40458C" size="5" face="Tahoma">Reliability – Nothing should be able to crash the OS. Anything that crashes the OS is a bug. Very radical thinking inside of Microsoft considering Win16 was cooperative multi-tasking in a single address space, and OS/2 had many similar attributes with respect to memory isolation </font></p></ul> <ul><p><font color="#40458C" size="5" face="Tahoma">Extensibility – Ability to extend the OS over time</font></p></ul> <ul><p><font color="#40458C" size="5" face="Tahoma">Compatibility – With DOS, OS/2, POSIX, or other popular runtimes. This is the foundation work that allowed us to invent windows two years into NT OS/2 development.</font></p></ul> <ul><p><font color="#40458C" size="5" face="Tahoma">Performance – All of the above are more important than raw speed!</font></p></ul> <br /> <p><font color="#660066" size="7" face="Tahoma">NT OS/2 Design Workbook</font> <br /> </p> <p><font color="#40458C" size="6" face="Tahoma">Design of executive captured in functional specs</font></p> <p><font color="#40458C" size="6" face="Tahoma">Written by engineers, for engineers</font></p> <p><font color="#40458C" size="6" face="Tahoma">Every functional interface was defined and reviewed</font></p> <p><font color="#40458C" size="6" face="Tahoma">Small teams can do this efficiently, </font></p> <ul><p><font color="#40458C" size="6" face="Tahoma">making this process scale is an almost impossible challenge</font></p></ul> <ul><p><font color="#40458C" size="6" face="Tahoma">Senior developers are inundated with spec reviews and the value of their feedback becomes meaningless</font></p></ul> <ul><p><font color="#40458C" size="6" face="Tahoma">You have to spread review duties broadly, and everyone must share the “culture”</font></p></ul> <br /> <p><font color="#660066" size="7" face="Tahoma">Developing a Culture</font> <br /> </p> <p><font color="#40458C" size="6" face="Tahoma">To scale a development team, you need to establish a culture</font></p> <ul><p><font color="#40458C" size="6" face="Tahoma">Common way of evaluating designs, making tradeoffs, etc.</font></p></ul> <ul><p><font color="#40458C" size="6" face="Tahoma">Common way of developing code and reacting to problems (build breaks, critical bugs, etc.)</font></p></ul> <ul><p><font color="#40458C" size="6" face="Tahoma">Common way of establishing ownership of problems</font></p></ul> <p><font color="#40458C" size="6" face="Tahoma">Goal setting can be the foundation for the culture</font></p> <p><font color="#40458C" size="6" face="Tahoma">Keeping a culture alive as a team grows is a huge challenge</font></p> <br /> <p><font color="#660066" size="7" face="Tahoma">The NT Culture</font> <br /> </p> <p><font color="#40458C" size="6" face="Tahoma">Portability, Reliability, Security, and Extensibility ingrained as the teams top priority</font></p> <ul><p><font color="#40458C" size="6" face="Tahoma">Every decision was made in the context of these design goals</font></p></ul> <p><font color="#40458C" size="6" face="Tahoma">Everyone owns all the code, so whenever something is busted anyone has a right and a duty to fix it</font></p> <ul><p><font color="#40458C" size="6" face="Tahoma">Works in small groups (< 150 people) where people cover for each other</font></p></ul> <ul><p><font color="#40458C" size="6" face="Tahoma">Fails miserably in large groups</font></p></ul> <p><font color="#40458C" size="6" face="Tahoma">Sloppiness is not tolerated</font></p> <ul><p><font color="#40458C" size="6" face="Tahoma">Great idea, but very difficult to nurture as group grows</font></p></ul> <ul><p><font color="#40458C" size="6" face="Tahoma">Abuse and intimidation gets way out of control, can’t keep calling people stupid and expect them to listen</font></p></ul> <p><font color="#40458C" size="6" face="Tahoma">A successful culture has to accept that mistakes will happen</font></p> <br /> <p><font color="#660066" size="7" face="Tahoma">Development Environment</font> <br /> </p> <p><font color="#40458C" size="7" face="Tahoma">NT 3.1 vs. Windows 2000</font></p> <ul><p><font color="#40458C" size="6" face="Tahoma">Development Teams</font></p></ul> <ul><p><font color="#40458C" size="6" face="Tahoma">Source Code Control System</font></p></ul> <ul><p><font color="#40458C" size="6" face="Tahoma">Process Management</font></p></ul> <ul><p><font color="#40458C" size="6" face="Tahoma">Serialized Development</font></p></ul> <ul><p><font color="#40458C" size="6" face="Tahoma">Defects</font></p></ul> <br /> <p><font color="#660066" size="7" face="Tahoma">Development Team</font> <br /> </p> <p><font color="#40458C" size="6" face="Tahoma">NT 3.1</font></p> <ul><p><font color="#40458C" size="6" face="Tahoma">Starts very small (6), grows very slowly to 200 people</font></p></ul> <ul><p><font color="#40458C" size="6" face="Tahoma">NT Culture was commonly understood by all</font></p></ul> <p><font color="#40458C" size="6" face="Tahoma">Windows 2000</font></p> <ul><p><font color="#40458C" size="6" face="Tahoma">Mass assimilation of other teams into the NT team</font></p></ul> <ul><p><font color="#40458C" size="6" face="Tahoma">NT 4.0 had 800 developers, Windows 2000 had 1400</font></p></ul> <ul><p><font color="#40458C" size="6" face="Tahoma">Original NT culture practiced by the old timers in the group, but keeping the culture alive was very difficult due to growth, physical separation, etc.</font></p></ul> <ul><p><font color="#40458C" size="5" face="Tahoma">Diluted culture leads to much conflict</font></p></ul> <ul><p><font color="#40458C" size="5" face="Tahoma">Accountability: I don’t “own” the code that is busted, see Markl</font></p></ul> <ul><p><font color="#40458C" size="5" face="Tahoma">reliability vs. new features</font></p></ul> <ul><p><font color="#40458C" size="5" face="Tahoma">64-bit portability vs. new features</font></p></ul> <br /> <p><font color="#660066" size="7" face="Tahoma">Source Code Control System (NT 3.1)</font> <br /></p> <p><font color="#40458C" size="6" face="Tahoma">Internally developed, maintained by a non-NT tools team</font></p> <ul><p><font color="#40458C" size="6" face="Tahoma">No branch capability, but with small team, it was not needed</font></p></ul> <p><font color="#40458C" size="6" face="Tahoma">10-12 well isolated source “projects”, 6M LOC</font></p> <p><font color="#40458C" size="6" face="Tahoma">Informal project separation worked well</font></p> <ul><p><font color="#40458C" size="6" face="Tahoma">minimal obscure source level dependencies</font></p></ul> <p><font color="#40458C" size="6" face="Tahoma">Small hard drive could easily hold entire source tree</font></p> <p><font color="#40458C" size="6" face="Tahoma">Developer could easily stay in synch with changes made to the system</font></p> <br /> <p><font color="#660066" size="7" face="Tahoma">Source Code Control System (Windows 2000)</font> <br /></p> <p><font color="#40458C" size="6" face="Tahoma">Windows team takes ownership of source code control system which at this point is on life support</font></p> <p><font color="#40458C" size="6" face="Tahoma">Branch capability sorely needed, tree copies used as substitute, so merging is a nightmare</font></p> <p><font color="#40458C" size="6" face="Tahoma">180 source “projects” 29M LOC</font></p> <p><font color="#40458C" size="6" face="Tahoma">No project separation, reaching “up and over” was very common as developers tried to minimize what they had to carry on their machines to get their jobs done</font></p> <p><font color="#40458C" size="6" face="Tahoma">Full source base required about 50Gb of disk space</font></p> <p><font color="#40458C" size="6" face="Tahoma">To keep a machine in synch was a huge chore (1 week to setup, 2 hours per-day to synchronize)</font> <br /> </p> <br /> <p><font color="#660066" size="7" face="Tahoma">Process Management (NT 3.1)</font> <br /></p> <p><font color="#40458C" size="6" face="Tahoma">Safe synch period in effect for ~4 hours each day, all other times the rule is check-in when ready</font></p> <p><font color="#40458C" size="6" face="Tahoma">Build lab synchs during morning safe synch period, and starts a complete build. </font></p> <ul><p><font color="#40458C" size="6" face="Tahoma">Build breaks are corrected manually during the build process (1-2 breaks was normal)</font></p></ul> <p><font color="#40458C" size="6" face="Tahoma">Complete build time is 5 hours on ~486/50</font></p> <p><font color="#40458C" size="6" face="Tahoma">Build is boot tested with some very minimal testing before release to stress testing</font></p> <ul><p><font color="#40458C" size="6" face="Tahoma">Defects corrected with incremental build fixes</font></p></ul> <p><font color="#40458C" size="6" face="Tahoma">4pm, stress testing on ~100 machines begins</font></p> <br /> <p><font color="#660066" size="7" face="Tahoma">Process Management (Windows 2000)</font> <br /></p> <p><font color="#40458C" size="6" face="Tahoma">Developers are not allowed to change the source tree without explicit, email/written permission</font></p> <ul><p><font color="#40458C" size="5" face="Tahoma">Build lab manually approves each check-in using a combination of email, web, and bug tracking database</font></p></ul> <p><font color="#40458C" size="6" face="Tahoma">Build lab approves about 100 changes each day and manually issues the appropriate synch and build commands</font></p> <ul><p><font color="#40458C" size="5" face="Tahoma">Build breaks are corrected manually, and when they occur, all further build processing is halted</font></p></ul> <ul><p><font color="#40458C" size="5" face="Tahoma">A developer that mistypes a build instruction can stop the build lab, which in turn stops over 5,000 people</font></p></ul> <p><font color="#40458C" size="6" face="Tahoma">Complete build time is 8 hours on 4 way PIII Xeon 550 with 50Gb disk and 512k RAM</font></p> <p><font color="#40458C" size="6" face="Tahoma">Build is boot tested and assuming we get a boot, extensive baseline testing begins</font></p> <ul><p><font color="#40458C" size="5" face="Tahoma">Testing is a mostly manual, semi-automated process</font></p></ul> <ul><p><font color="#40458C" size="5" face="Tahoma">Defects occurring in the boot or test phase must be corrected before build is “released” for stress testing</font></p></ul> <p><font color="#40458C" size="6" face="Tahoma">4pm, stress testing on ~1000 machines begins</font></p> <br /> <p><font color="#660066" size="7" face="Tahoma">Team Size</font> <br /> </p> <p><font color="#40458C" size="6" face="Tahoma">1700</font> <br /> </p> <p><font color="#40458C" size="6" face="Tahoma">700</font> <br /> </p> <p><font color="#40458C" size="6" face="Tahoma">325</font> <br /> </p> <p><font color="#40458C" size="6" face="Tahoma">230</font> <br /> </p> <p><font color="#40458C" size="6" face="Tahoma">140</font> <br /> </p> <p><font color="#40458C" size="6" face="Tahoma">Test Team Size</font> <br /> </p> <p><font color="#40458C" size="6" face="Tahoma">Dev Team Size</font> <br /> </p> <p><font color="#40458C" size="6" face="Tahoma">Product</font> <br /> </p> <p><font color="#40458C" size="6" face="Tahoma">1400</font> <br /> </p> <p><font color="#40458C" size="6" face="Tahoma">Win2k</font> <br /> </p> <p><font color="#40458C" size="6" face="Tahoma">800</font> <br /> </p> <p><font color="#40458C" size="6" face="Tahoma">NT 4.0</font> <br /> </p> <p><font color="#40458C" size="6" face="Tahoma">450</font> <br /> </p> <p><font color="#40458C" size="6" face="Tahoma">NT 3.51</font> <br /> </p> <p><font color="#40458C" size="6" face="Tahoma">300</font> <br /> </p> <p><font color="#40458C" size="6" face="Tahoma">NT 3.5</font> <br /> </p> <p><font color="#40458C" size="6" face="Tahoma">200</font> <br /> </p> <p><font color="#40458C" size="6" face="Tahoma">NT 3.1</font></p> <br /> <p><font color="#660066" size="7" face="Tahoma">Serialized Development</font> <br /> </p> <p><font color="#40458C" size="6" face="Tahoma">The model from NT 3.1 -> Windows 2000</font></p> <p><font color="#40458C" size="6" face="Tahoma">All developers on team check-in to a single main line branch</font></p> <p><font color="#40458C" size="6" face="Tahoma">Master build lab synchs to main branch and builds and releases from that branch</font></p> <p><font color="#40458C" size="6" face="Tahoma">Checked in defect affects everyone waiting for results</font></p> <br /> <p><font color="#660066" size="7" face="Tahoma">Defect Rates and Serialization</font> <br /> </p> <p><font color="#40458C" size="6" face="Tahoma">Compile time or run time bugs that occur in a developers office only affect that developer</font></p> <p><font color="#40458C" size="6" face="Tahoma">Once a defect is checked-in, the number of people affected by the defect increases</font></p> <p><font color="#40458C" size="6" face="Tahoma">Best developers are going to check-in a runtime or compile time mistake at least twice each year</font></p> <p><font color="#40458C" size="6" face="Tahoma">Best developers will be able to cope with a checked-in compile time or run time break very quickly (about 20 minutes end-to-end) </font></p> <p><font color="#40458C" size="6" face="Tahoma">As the code base gets larger, and as the team gets larger, these numbers typically double</font></p> <br /> <p><font color="#660066" size="7" face="Tahoma">Defect Rates Data</font> <br /> </p> <p><font color="#40458C" size="5" face="Tahoma">10.2 hours</font> <br /> </p> <p><font color="#40458C" size="5" face="Tahoma">15.3</font> <br /> </p> <p><font color="#40458C" size="5" face="Tahoma">40 minutes</font> <br /> </p> <p><font color="#40458C" size="5" face="Tahoma">4</font> <br /> </p> <p><font color="#40458C" size="5" face="Tahoma">Win2k, 1400</font> <br /> </p> <p><font color="#40458C" size="5" face="Tahoma">3.8 hours</font> <br /> </p> <p><font color="#40458C" size="5" face="Tahoma">6.6</font> <br /> </p> <p><font color="#40458C" size="5" face="Tahoma">35 minutes</font> <br /> </p> <p><font color="#40458C" size="5" face="Tahoma">3</font> <br /> </p> <p><font color="#40458C" size="5" face="Tahoma">NT 4.0, 800</font> <br /> </p> <p><font color="#40458C" size="5" face="Tahoma">1.2 hours</font> <br /> </p> <p><font color="#40458C" size="5" face="Tahoma">2.5</font> <br /> </p> <p><font color="#40458C" size="5" face="Tahoma">30 minutes</font> <br /> </p> <p><font color="#40458C" size="5" face="Tahoma">2</font> <br /> </p> <p><font color="#40458C" size="5" face="Tahoma">NT 3.51, 450</font> <br /> </p> <p><font color="#40458C" size="5" face="Tahoma">41 minutes</font> <br /> </p> <p><font color="#40458C" size="5" face="Tahoma">1.6</font> <br /> </p> <p><font color="#40458C" size="5" face="Tahoma">25 minutes</font> <br /> </p> <p><font color="#40458C" size="5" face="Tahoma">2</font> <br /> </p> <p><font color="#40458C" size="5" face="Tahoma">NT 3.5, 300</font> <br /> </p> <p><font color="#40458C" size="5" face="Tahoma">20 minutes</font> <br /> </p> <p><font color="#40458C" size="5" face="Tahoma">1</font> <br /> </p> <p><font color="#40458C" size="5" face="Tahoma">20 minutes</font> <br /> </p> <p><font color="#40458C" size="5" face="Tahoma">2</font> <br /> </p> <p><font color="#40458C" size="5" face="Tahoma">NT 3.1, 200</font> <br /> </p> <p><font color="#40458C" size="5" face="Tahoma">Total</font></p> <p><font color="#40458C" size="5" face="Tahoma">Defect Fix Time</font> <br /> </p> <p><font color="#40458C" size="5" face="Tahoma">Defects:</font></p> <p><font color="#40458C" size="5" face="Tahoma">Per Day</font> <br /> <br /></p> <p><font color="#40458C" size="5" face="Tahoma">Time to Fix:</font></p> <p><font color="#40458C" size="5" face="Tahoma">Per Defect</font> <br /> </p> <p><font color="#40458C" size="5" face="Tahoma">Defects:</font></p> <p><font color="#40458C" size="5" face="Tahoma">Per year</font></p> <p><font color="#40458C" size="5" face="Tahoma">Per Dev</font> <br /> </p> <p><font color="#40458C" size="5" face="Tahoma">Product</font></p> <p><font color="#40458C" size="5" face="Tahoma">and</font></p> <p><font color="#40458C" size="5" face="Tahoma">Team Size</font> <br /> </p> <p><font color="#40458C" size="6" face="Tahoma">With serialized development:</font></p> <ul><p><font color="#40458C" size="6" face="Tahoma">Good, small teams operate efficiently</font></p></ul> <ul><p><font color="#40458C" size="6" face="Tahoma">Even the absolute best large teams are always broken, and always serialized</font></p></ul> <br /> <p><font color="#660066" size="7" face="Tahoma">Development Environment Summary</font> <br /></p> <p><font color="#40458C" size="6" face="Tahoma">NT 3.1</font></p> <ul><p><font color="#40458C" size="6" face="Tahoma">Fast and loose development, lots of fun, lots of energy</font></p></ul> <ul><p><font color="#40458C" size="6" face="Tahoma">Few barriers to getting your work done</font></p></ul> <ul><p><font color="#40458C" size="6" face="Tahoma">Defects serialized parts of the process, but didn’t stop the whole machine, minimal down time</font></p></ul> <p><font color="#40458C" size="6" face="Tahoma">Windows 2000</font></p> <ul><p><font color="#40458C" size="6" face="Tahoma">Source code control system bursting at the seams</font></p></ul> <ul><p><font color="#40458C" size="6" face="Tahoma">Excessive process management serialized the entire development process, 1 defect stops 1400 devs, 5000 team members!</font></p></ul> <ul><p><font color="#40458C" size="6" face="Tahoma">Resources required to build a complete instance of NT were excessive giving few developers a way to be successful</font> <br /></p></ul> <br /> <p><font color="#660066" size="7" face="Tahoma">Focused Fixes</font> <br /> </p> <p><font color="#40458C" size="6" face="Tahoma">Source Code Control System</font></p> <p><font color="#40458C" size="6" face="Tahoma">Source Code Restructuring</font></p> <p><font color="#40458C" size="6" face="Tahoma">Make the large team work like a set of small teams</font></p> <ul><p><font color="#40458C" size="6" face="Tahoma">Windows is already organized into reasonable size development teams</font></p></ul> <ul><p><font color="#40458C" size="6" face="Tahoma">Goal is to allow these teams to work as a team when contributing source code changes rather than as a group of individuals that happen to work for the same VP</font></p></ul> <ul><p><font color="#40458C" size="6" face="Tahoma">Parallel Development, Team Level Independence</font></p></ul> <p><font color="#40458C" size="6" face="Tahoma">Automated Builds</font></p> <br /> <p><font color="#660066" size="7" face="Tahoma">Source Code Control System</font> <br /></p> <p><font color="#40458C" size="7" face="Tahoma">New source code control system identified 3/99 (SourceDepot)</font></p> <p><font color="#40458C" size="7" face="Tahoma">Native branch support</font></p> <p><font color="#40458C" size="7" face="Tahoma">Scalable, high speed, client server architecture</font></p> <p><font color="#40458C" size="7" face="Tahoma">New machine setup 3 hours vs. 1 week</font></p> <p><font color="#40458C" size="7" face="Tahoma">Normal synch 5 minutes vs. 2 hours</font></p> <br /> <p><font color="#660066" size="7" face="Tahoma">Source Code Control System (cont.)</font> <br /></p> <p><font color="#40458C" size="7" face="Tahoma">Transition to SourceDepot done on LIVE Win2k code base</font></p> <p><font color="#40458C" size="7" face="Tahoma">Hand built SLM -> SourceDepot migration system allowed us to keep in synch with the old system while transitioning to SourceDepot and changing the source code layout</font></p> <br /> <p><font color="#660066" size="7" face="Tahoma">Source Code Restructuring</font> <br /> </p> <p><font color="#40458C" size="7" face="Tahoma">16 Depots for covering each major area of source code</font></p> <p><font color="#40458C" size="7" face="Tahoma">Organization is focused on:</font></p> <ul><p><font color="#40458C" size="6" face="Tahoma">minimizing cross project dependencies to reduce defect rate</font></p></ul> <ul><p><font color="#40458C" size="6" face="Tahoma">Sizing projects to compile in a reasonable amount of time</font></p></ul> <ul><p><font color="#40458C" size="6" face="Tahoma">To build a project, all you need is the code for that project, AND the public/root project</font></p></ul> <ul><p><font color="#40458C" size="6" face="Tahoma">Cross project sharing is explicit</font></p></ul> <br /> <p><font color="#660066" size="7" face="Tahoma">New Tree Layout</font> <br /> </p> <p><font color="#40458C" size="6" face="Tahoma">The new tree layout features</font></p> <ul><p><font color="#40458C" size="6" face="Tahoma">Root project houses public</font></p></ul> <ul><p><font color="#40458C" size="6" face="Tahoma">15 Additional projects hang off of the Root</font></p></ul> <ul><p><font color="#40458C" size="6" face="Tahoma">No nested projects</font></p></ul> <ul><p><font color="#40458C" size="6" face="Tahoma">All projects build independently</font></p></ul> <ul><p><font color="#40458C" size="6" face="Tahoma">Cross project dependencies resolved via Public, Publicinternal using checked in interfaces </font></p></ul> <br /> <p><font color="#660066" size="7" face="Tahoma">Explicit Internal Interface Sharing</font> <br /></p> <p align="center"><font color="#40458C" size="5" face="Times New Roman">The Base Project internal interfaces exposed here</font> <br /></p> <p align="center"><font color="#40458C" size="5" face="Times New Roman">The Admin Project internal interfaces exposed here</font></p> <br /> <p><font color="#660066" size="7" face="Tahoma">Team Level Independence</font> <br /> </p> <p><font color="#40458C" size="6" face="Tahoma">Each team determines its own check-in policy, enable rapid, frequent check ins</font></p> <p><font color="#40458C" size="6" face="Tahoma">Teams are isolated from mistakes made by other teams</font></p> <ul><p><font color="#40458C" size="6" face="Tahoma">When errors occur, only the team causing the error is affected</font></p></ul> <ul><p><font color="#40458C" size="6" face="Tahoma">A build, boot, or test break only affects a small subset of the product group</font></p></ul> <p><font color="#40458C" size="6" face="Tahoma">Each team has their own view of the source tree, their own mini build lab, and builds an entire installable build</font></p> <p><font color="#40458C" size="6" face="Tahoma">Any developer with adequate resources can easily duplicate a mini build lab</font></p> <ul><p><font color="#40458C" size="6" face="Tahoma">build and release a completely installable Windows System</font></p></ul> <br /> <p><font color="#660066" size="7" face="Tahoma">Parallel Development (cont.)</font> <br /></p> <p><font color="#40458C" size="5" face="Times New Roman">Main Build Lab Branch</font> <br /></p> <p><font color="#40458C" size="5" face="Times New Roman">Debbl’s Windows Branch</font> <br /></p> <p><font color="#40458C" size="5" face="Times New Roman"> Main branch is built by the master build lab.</font></p> <p><font color="#40458C" size="5" face="Times New Roman"> Quality is always high because only well tested complete group check-ins are done here</font> <br /></p> <p><font color="#40458C" size="5" face="Times New Roman">Robs Base Branch</font> <br /> </p> <p><font color="#40458C" size="5" face="Times New Roman"> Team branches have their own check-in policy.</font></p> <p><font color="#40458C" size="5" face="Times New Roman"> Team members check-in to their branch, not the main branch</font></p> <p><font color="#40458C" size="5" face="Times New Roman"> Each team branch has their own mini-build lab that produces full builds for that team</font> <br /></p> <p><font color="#40458C" size="5" face="Times New Roman">Robs team checks into his own branch without affecting Main, or Debbl</font> <br /> </p> <p><font color="#40458C" size="5" face="Times New Roman">Debbl’s team makes similar check-ins without impacting Rob or Main</font> <br /> </p> <p><font color="#40458C" size="5" face="Times New Roman">Rob integrates build lab changes to his tree</font> <br /></p> <p><font color="#40458C" size="5" face="Times New Roman">Rob builds resulting tree, tests tree and then integrates to the main branch</font> <br /> </p> <p><font color="#40458C" size="5" face="Times New Roman">Debbl makes a few changes then integrates from the main build lab branch</font> <br /> </p> <p><font color="#40458C" size="5" face="Times New Roman">During test, more changes are needed to make system work well. When done, integration into main build lab branch occurs. During this period, robs team continues to check-in to their branch</font> <br /></p> <p><font color="#40458C" size="5" face="Times New Roman">N+1 Dev Branch</font> <br /> </p> <p><font color="#40458C" size="5" face="Times New Roman">N+1 Team checks into their branch and periodically integrates from main</font> <br /> </p> <p><font color="#40458C" size="5" face="Times New Roman">QFE/SP Branch</font> <br /> </p> <p><font color="#40458C" size="5" face="Times New Roman">At RTM, a QFE/SP branch is created. N+1 Branch integrates from main, then integrates their branch into main</font></p> <br /> <p><font color="#660066" size="7" face="Tahoma">Parallel Development</font></p> <br /> <p><font color="#660066" size="7" face="Tahoma">Team Level Independence (cont.)</font> <br /></p> <p><font color="#40458C" size="6" face="Tahoma">Teams integrate their changes into the “main” trunk one at a time, so there is a high degree of accountability when something goes wrong in “main”</font></p> <p><font color="#40458C" size="6" face="Tahoma">Build breaks will happen, but they are easily localized to the branch level, not the main product codeline</font></p> <p><font color="#40458C" size="6" face="Tahoma">Teams are isolated from mistakes made by other teams</font></p> <ul><p><font color="#40458C" size="5" face="Tahoma">When errors occur, they affect smaller teams</font></p></ul> <ul><p><font color="#40458C" size="5" face="Tahoma">A build, boot, or test break only affects a small subset of the windows development team</font></p></ul> <p><font color="#40458C" size="6" face="Tahoma">Each team has their own view of the source tree and their own mini build lab</font></p> <ul><p><font color="#40458C" size="5" face="Tahoma">I.e. Each team’s lab is enlisted in ALL projects and builds ALL projects</font></p></ul> <ul><p><font color="#40458C" size="5" face="Tahoma">Each team needs resources able to build an NT system</font></p></ul> <p><font color="#40458C" size="6" face="Tahoma">Each team’s build lab builds, tests, and mini-bvt’s a complete standalone system</font></p> <br /> <p><font color="#660066" size="7" face="Tahoma">Automated Builds</font> <br /> </p> <p><font color="#40458C" size="6" face="Tahoma">Build lab runs 100% hands off</font></p> <p><font color="#40458C" size="6" face="Tahoma">10am and 10pm full synch and full build</font></p> <ul><p><font color="#40458C" size="6" face="Tahoma">Build failures are auto detected and mailed to the team</font></p></ul> <ul><p><font color="#40458C" size="6" face="Tahoma">Successful builds are automatically released with automatic notification to the team</font></p></ul> <ul><p><font color="#40458C" size="6" face="Tahoma">Each VBL can build:</font></p></ul> <ul><p><font color="#40458C" size="6" face="Tahoma">4 platforms (x86 fre/chk, ia64 fre/chk) = 8 builds each day, 56 each week</font></p></ul> <ul><p><font color="#40458C" size="6" face="Tahoma">No manual steps at all. 100% Hands off automatic</font></p></ul> <ul><p><font color="#40458C" size="6" face="Tahoma">7 VBLs in Win2k Group</font></p></ul> <ul><p><font color="#40458C" size="6" face="Tahoma">Majority of builds work, but failures when they occur are isolated to a single team</font></p></ul> <br /> <p><font color="#660066" size="7" face="Tahoma">Productivity Gains</font> <br /> </p> <p><font color="#40458C" size="6" face="Tahoma">Developers can easily switch from working on release N to release N+1</font></p> <p><font color="#40458C" size="6" face="Tahoma">Developers in one team will not be impacted by mistakes/changes made by other teams</font></p> <p><font color="#40458C" size="6" face="Tahoma">Developers have long, frequent checkin windows (Base team has a 24x7 open checkin window, vs. 2-3 hour per day checkin window with manual approval used during W2K</font></p> <p><font color="#40458C" size="6" face="Tahoma">Source code control system is fast and reliable</font></p> <p><font color="#40458C" size="6" face="Tahoma">Testing is done on complete builds instead of assorted collections of private binaries</font></p> <ul><p><font color="#40458C" size="6" face="Tahoma">What is in the source code control system is what is tested</font></p></ul> <br /> <p><font color="#660066" size="7" face="Tahoma">How is it working?</font> <br /> </p> <p><font color="#40458C" size="6" face="Tahoma">Source code control system is working very well</font></p> <ul><p><font color="#40458C" size="6" face="Tahoma">No scaling problems, easily handling 5100 total user enlistments and 411,000 files</font></p></ul> <p><font color="#40458C" size="6" face="Tahoma">Source code restructuring is working well</font></p> <ul><p><font color="#40458C" size="6" face="Tahoma">No new depots added, explicit sharing between projects still the rule</font></p></ul> <p><font color="#40458C" size="6" face="Tahoma">Parallel Development is working very well</font></p> <ul><p><font color="#40458C" size="6" face="Tahoma">Teams feel independent and able to control their own destiny</font></p></ul> <ul><p><font color="#40458C" size="6" face="Tahoma">Per-team serialization only occurs when a team “reverse integrates” their changes into the main branch</font></p></ul> <br /> <p><font color="#660066" size="7" face="Tahoma">Summary</font> <br /> </p> <p><font color="#40458C" size="6" face="Tahoma">The initial NT development environment and culture worked well for the first few years</font></p> <p><font color="#40458C" size="6" face="Tahoma">Ten years of team and code growth forced a major re-design of the development environment and culture</font></p> <p><font color="#40458C" size="6" face="Tahoma">With the new environment in place, the team is working a lot like they did in the NT 3.1 days with a small, fast moving, development team</font></p> <br /> <p><font color="#660066" size="7" face="Tahoma">Questions</font>
Jake Savin 