On the importance of other people’s hard work

NOTE: As I return from a long hiatus from blogging, please excuse the overly agressive tone of this post. I don't want to disparage any of the teams I worked with at Microsoft, just point out some opportunities we had for getting more done by doing less.

At the start of the Visual Studio 2008 product cycle I was working on the Indigo Tools team, tasked with adding WCF tooling to Visual Studio. On what would later be known as the “Add Service Reference” dialog, or this guy:

The hardest decision we had to make during the product cycle was the very first: how will we implement our feature? We had to choose between two very different approaches.

The first was to build a wrapper on top of the existing WCF tools, specifically svcutil.exe. Svcutil.exe did everything users would want from the Add Service Reference dialog and more. So all we would conceivably need to do is add some hooks to the tool and put a UI on top of it.

The second option was to start from scratch and duplicate the functionality of svcutil.exe. This would mean rewriting the guts of the tool, but make it much easier to integrate into Visual Studio.

After much debate we went with option B: coding our own thing from scratch. Ultimately we erred on the side of making it easier to integrate into Visual Studio.

As you can probably tell from the title of this post, in hindsight that was a mistake. Visual Studio doesn’t support nearly as many WCF service types as we would have liked, it generates poor code, and has a poor user experience to boot. (Click “Show All Files” in Solution Explorer and just see how much crap is added when adding a service reference.)

The problem was that we did exactly what Joel Spolsky claims is “the single worst strategic mistake that any software company can make”, and that is to decide to rewrite code from scratch. Instead of building off of an existing code base and refactoring it to suit our new needs, we instead chose to rewrite the stack from scratch.

Let me expound on the perceived pros and cons of using existing code (EC) or rewriting from scratch and using new code (NC).

The aversion to reusing hypothetical existing code or EC is that it never does quite does exactly what you want it to do. (Because, obviously, if it did you wouldn’t have a problem using it.) EC either doesn’t support the features you need, doesn’t implement them in the way you want, or perhaps is just formatted in a way you don’t like. Also, it would be naive to expect the documentation for EC to be complete, well written, and thorough.

In reality however, EC is actually perfectly fine. (Think about it -- someone used it for something, so it can’t be fundamentally flawed.) The real problem with EC has is that you didn’t write it, and therefore it seems too complicated to work with. But, more on EC later...

Suppose that you bought into the assumption that reusing / refactoring EC was a bad idea. So you instead introduce new code or NC. Hypothetical NC is a well-researched solution, engineered to solve your problem elegantly and efficiently.

Most likely you will start writing NC with lots of unit tests, but even so, eventually bugs will creep in. (Most likely from features NC must support which you didn’t envision.) You will fix these bugs, but leave TODO comments for the larger ones. And of course, since you are busy coding you don’t have any time for real documentation. So you scale back your vision for NC and the second it half-works you forget about NC and move onto the next problem. (Thereby abandoning NC and perpetuating the EC vs. NC struggle.)

NC always seems like a great idea at first, but it runs into the same problems as EC: writing clear code is hard. Documenting code is hard. Finishing code is hard.

My thesis is that reimplementing anything is a waste of time. That’s right, I used the totally bold and flame-bait word "anything".

If you work to improve existing code, then you and the code’s existing developers/users benefit. You fix some bugs, contribute some documentation, and leave the world better than you found it. The word here is synergy -- everybody wins.

But in the case that you try to go about things on your own, nobody wins. Even if NC has more bells and whistles than EC, the world is worse off. Why?

Because you’ve made an even bigger mess for the next guy. Previously the choice was just between reusing EC and writing NC. Now, they have to chose between EC, NC, and newer code (NC++?).

And, even if this EC, NC, and NC++ situation wasn’t bad enough think of the critical fixes that now need to be made to all the duplicate code in the world. Security updates, breaking library changes, compiler updates, and so on. By introducing new projects you increase the overall cost of housekeeping.

Let me break away from the hypothetical and talk about real examples. Look through these lists and think to yourself: does every one of these contribute something novel to the universe? Or would mankind be better off if the effort spent in creating that new thing had been spent improving the old one? http://en.wikipedia.org/wiki/Category:Open_source_content_management_systems

http://en.wikipedia.org/wiki/List_of_build_automation_software
http://en.wikipedia.org/wiki/List_of_text_editors
http://en.wikipedia.org/wiki/List_of_compiler-compilers



Especially for the text editors list. Is there anything you can’t do in Emacs? What else could you possibly want?

Despite what Richard Stallman and the hippies at the FSF tell you, I think the real benefit of FOSS comes form people not having to worry about solved problems.

Certainly licenses, politics, and egos may make it difficult to follow another developer’s lead and work on an existing project. But I urge you to think twice before writing any new code. Ask yourself, has this function already been written in the history of mankind? And if so, how can you take advantage of that and not write that function. Just go search for it!

Let me provide a specific example of the benefit of not rewriting code with another Microsoft war story. Fast forward three years from my WCF experience and I’m working on the F# team trying to ship Visual Studio 2010. We were a small and agile team with an immense amount of work to do. But due to Don Syme’s amazing leadership we were able to produce a great v1.0 product on time and with a few more features than we expected.

That is not a typeo. We actually able to ship more language features than we thought we would.

One such feature came in during a weekend Don Syme was in town. The team decided to spend a Saturday afternoon hacking away on the codebase, but rather than just fixing bugs we wanted to add something bigger and meatier. Of all the options we considered, we settled on a new language feature of dynamic support.

One of the big features the C# and VB.NET teams were adding to their compilers for Visual Studio 2010 was support for dynamic types.

It took a team of people -- dev and test -- working on adding this feature to the C# and VB compilers months, and yet we were able to add it to F# in an afternoon*.

How? By simply doing the easy part and reusing all of their hard work.

Implementing these dynamic types in F# required two pieces. First, syntax support for differentiating dynamic calls from regular ones. We did this in F# by adding the (?) and (?<-) operators. Second, you need a runtime binder to determine how dynamic calls get dispatched. (I.e. how to deal with method overloading, name resolution, and so on.)

Adding syntax support for two operators is quite easy, the bulk of the work for adding dynamic support would have been in implementing the F# runtime binder. However, the key observation is that we didn’t actually need to write a new runtime binder. There already was an excellent dynamic method binder for C# which worked just fine.

All users need to do is copy and paste this code (by Matt Podwysocki) and you are set. While the corner cases for name/method overloading for F# don’t exactly match up with C#, for 99% of cases you won’t notice.

That, in a nutshell is the importance of leveraging other people’s hard work.

It is easy to get distracted with ‘how’ to implement something, and forget it is ‘what’ you want to enable. It is important to leverage any and all pre-existing work. Not only do you have an opportunity to save time, but you also get to build up on the insights, innovations, and efforts of those before you. This sets you up for long term success, both in terms of maintenance.