OpenDS: Observations from an open-source project maintainer/screwup

[Other posts on this topic: first previous ]

L’Affairre OpenDS continues to make the rounds of the blogosphere. My modest efforts to date include the following, in the order published:

• OpenDS: On Being Bitten By The Hand That Once Fed You
• OpenDS Exegesis Ecclesiastes 5:13
• On OpenDS: Time for Sun to make a call

Recent contributions to this contretemps include the following:

• Mike Dolan’s Comparing “open source” projects? Start by asking why does the project exist
• Redmonk’s Michael Coté’s link to Mike’s post, links for 2007-12-08, with Michael’s comment, “Control vs. no control when a company does open source.”
• Redmonk’s Stephen O’Grady’s link to Mikes post links for 2007-12-09, with Stephen’s comment, “very interesting piece by mdolan; a bit cynical towards Sun, as usual, and i do think comparisons are possible between projects managed under different governance models, but there’s a lot in here that i agree with. be interesting if he went introspective.”

Additional observations can be found via the comments to Mike’s blog post:

• Ian Skerrett, The Importance of Open Source Project Governance
• Sean Dague’s link to Mike’s post, links for 2007-12-09, with the comment, “This post brings up a lot of good points on what “open” means.”
• Robert Galoppini’s link to his earlier post, Capability Coordination in Modular Organization: Voluntary FS/OSS Production and the Case of Debian GNU/Linux
• Christian Geisart’s comment, “It’s about time to establish the term “Open Development””

Both Mike Dolan and Sean Dague are IBM colleagues, though of course their comments, as are mine, are being posted as personal views, and are not intended in any way, shape or form to reflect the opinions or strategy of our employer.

As best as I can tell, only Sean and I have had actual experience starting and running an open-source project. Sean only offers a brief comment that Mike’s comment makes some good points. I agree with Sean, and hope he takes the time to give some of his own thoughts in a future blog post.

Here are some of my own further observations, based on my experience starting and running an open-source project, Jikes.

What is an open project?

There are two kinds of “open” projects. The first is the publication of software in source code form under a license approved by the Open Source Initiative (OSI).

The second is the publication of content, usually not computer software, under either an OSI-approved license or a Creative Commons (CC) license. An example of an open project that does not publish software can be found at Groklaw.net. If you look at the bottom of the page you will find that the content is licensed under the CC Attribution-Noncommercial-No Derivative Works 3.0 Unported License.

Some open-source licenses explicitly allow for the inclusion of content that is not software code; for example, the Eclipse Public License v1.0, which says in part:

a) in the case of the initial Contributor, the initial code and documentation distributed under this Agreement,

Since our main purpose here is to study how projects are governed, not the form of the content they distribute, I take “open-source” and “open-content” projects to mean the same thing.

Why does an an individual or organization start an open project?

The Jikes project came into being for many reasons:

Jikes was one of the first programs that IBM made available in binary form for Linux, in the middle of July, 1998. The publication of this binary version was followed by exponential growth. Before the announce we had been averaging about 20 downloads/day. We got several thousand within the next few days.

Soon thereafter we got requests, which we had expected, that IBM publish the code in source form. I told our users that while I didn’t think IBM would approve the release of the source code, I would make a good-faith effort to present the case to IBM management.

The first reason was that users asked for it.

The second reason was that I saw this as a challenge, and I wanted to see if I could convince IBM to release the source code.

The third reason was that I wanted to buy some more time for the project, as I knew that if Jikes were to be released in open-source form, then we would probably be given more time to work on it.

The fourth reason was that I knew that Jikes was not going to be part of any IBM product, so that unless I got the code out in open-source form, then it would wind up on some hard disk in IBM Research in Hawthorne, and no one would ever know what a great job my colleage, Philippe Charles, had done in designing and implementing it. (I wrote part of the code; he wrote most of it.)

Another reason was I wanted to get Philippe some of the recognition that was his due. He had then worked on Jikes non-stop for over two-years, in the most sustained, high-quality programming effort I had ever personally witnessed (I still feel that way).

The proposal I wrote offered a business reason for IBM to release the code, and I will discuss this below.

Finally, I don’t really know the reason Jikes went out in open-source form. I do know it was approved by management. Whether it was because they accepted my business case, or didn’t pay proper attention, and just did it on a hunch, remains an open question.

This is a key point: it doesn’t really matter why an open-source project gets started, just that it does.

Initial publication of open-source code?

Once you, as an individual or on behalf of your employeer, decided to publish code in open-source form, you have several options.

As I have stated earlier, the best strategy, at least in my view, is to give the code to someone else, and let them take over its governance. For example, no matter how good you think you — or your company — is, the folks at Apache Software Foundation will do a much better job than you can ever hope to do, so the best option is to give it to them, if they will accept it, and then work under their guidance and auspices.

Some of the lessons learned running an open-source project for company

Ken Coar, both an IBMer and a member of the Board of Directors of the Apache Software Foundation, refers in his blog to his employer as “BigCo.” In the rest of this post I’ll relate some of my experiences and lessons learned running the Jikes project, though I will refer to BigCo, and not just IBM, as I think the lessons learned apply to other corporations, including Sun for example, that initiate an open-source project.

The business case for BigCo to release the source of Jikes was as follows, and was written at a time when BigCo had not yet started an open-source project, back in the days (1998 in my case) when open-source was just showing up on the corporate radar. I argued that if BigCo were to ever engage seriously in open-source activity it had to do the following:

1. Publish the source code of software recognized as valuable. (It would not suffice to throw just anything over the firewall; it had to be code that would be of interest. In this case, the users had already asked for the code.)
2. Make the code available under an open-source license, writing a new one if necessary;
3. Publish the code at a URL ending in “Bigco.com,” to show BigCo was not afraid of publishing open-source code;
4. Run the project at that URL as an open-source project, to show that BigCo both understood the open-source rules and was willing to play by them.

The last point is crucial, so let me repeat it:

It was not enough to release Jikes. We had to build a project around it, to show we understood the rules and would live by them

This of course required that someone be found to run the project. Since BigCo had never done this before, that meant BigCo would have to place the responsibility for running this project into the hands of a complete novice, fully understanding that this novice would probably screw things up, not only once, but probably many times.

The question then was who would be the open-source project-manager screwer-upper?

That would be me, Dave Shields.

Moreover, I spent a year at this task, and in the process thus became BigCo’s recognized expert in screwing up the management of open-source projects.

This is the main reason I’m writing this post. Though you can learn something from someone who has enjoyed success in a new venture, the best lessons are to be found in close observation of the failures and screw-ups.

Here then a brief history of the Jikes project, emphasizing what went wrong more than what went right, from the pen of someone who was not only SU the “super user,” but also “SU” the “Screwer Upper.”

What is the goal for the project

The project began knowing it would have a limited duration, at least from the point of BigCo. My guess was about a year, and that turned out to be right.

While it would be acceptable to just run the project for a year and see what happened, as that could have shown the BigCo had gained first-hand experience in accepting the responsbility and risk of running an open-source project, my own goal from the start was to have the project continue after BigCo stopped participating in it, so that I not only had to build a community but wanted to build one that was self-sustaining.

This is generally the case. A BigCo starts a project not to provide a lifetime job for some of its developers, but in the hopes the initial seed code will be adopted, expanded, and then taken over by a community not controlled by BigCo.

This is but another way of stating what everyone who has ever engaged in starting, running, or leading an open-source project will tell you.

A successful open-source project requires three things:

• Good code. Open-source developers have a keen sense of smell. They can smell a rat seconds after trying the code. If the code isn’t good you are doomed.
• Good leadership. Even if the code is good, you need to show you can manage it well. Ineffective or bad management usually spells doom; the best you can hope for otherwise is that someone with more skills will fork the project.
• Vibrant, self-sustaining community. Community is all, for users will not put code in serious applications unless they are confident that it is either perfect (which is rarely the case), or has a community behind it with a lifetime measured in years, if not decades. To use open-source code is to place the fate of your code in the hands of the open-source project. Only idiots would place their fates in the hands of idiots. (That’s the theory at least, though most elections prove this is not always the case.)

You must recruit open-source developers

You need to recruit open-source developers to take over the project, unless you are willing to spend the rest of your life working on the project; otherwise the project will die when you and the other BigCo employees leave it.

Code rusts, and sometimes poops, too.

Software is not a stable form of writing. [1] The surrounding envrironment is constantly changing, so to think you can perfect the code and then leave it is as chimera. Moreover, most code has bugs, and you need someone to take care of the poop. Unless you like poop — and I don’t — then not only do you need to attract developers who will do the neat extensions and polishing, you also need to find devlopers to tend to the poop.

Most open-source folks do not trust big companies

Soon after Jikes first went out I started to see comments and such in Slashdot suggesting it was just some sort of conspiracy on the part of my employer. After spending a day or two trying to sort this out by claiming BigCo really had the community’s best interests at heart, I gave up further boasting of BigCo’s sterling reputation, but I did take away one important lesson:

If you are running a project of behalf of a company, accept that most people do not trust companies, and spare yourself the grief of trying to change their minds. Also accept that all they will know of the company going forward is how you act. As is often the case, words mean nothing, character and behavior is all.

As a corollary of the above, it doesn’t even matter what your company thinks the project is about. All that matters is how you participate and run the project.

Most open-source folks don’t care about many of the things that big companies do

I have often dealt with press releases, position papers and such, some of which were prepared on tight deadlines. I try to help as best as I can, but I do keep in mind that most of this kind of work is of absolutely no interest to open-source developers, and it is those developers you need to attract to make a project sustainable. For example, open-source developers do not even know what the Gartner quadrant is, nor do they care, nor do they read InformationWorld and most of the other trade rags. They have better things to do, reading the project mail lists being the main one.

There are open-source “rules”

There are well-defined rules and practices for running an open-source project project, what is called open-source “governance.” If you believe, as I do, that open-source programming is just the application of the scientific method to programming, and thus programming at the highest level, then it follows that the best way to govern a project is to also use the scientific method: peer review; giving credit where credit is due, with extra credit to the first to do something; promotion based on merit and not the arbitrary exercise of power; lack of dogma; willingness to try new ideas; and so forth.

These rules are quite simple. Indeed, they are so simple that the open-source community takes it for granted that you know them. You will get help if you ask for it, but most folks will not volunteer it. After all, why try to help someone who is so dumb they don’t even know they need help?

If you do ask for help you will get it. Moreover, if you make clear that you want to play by the rules, then a lot will be forgiven. It’s ok to screw up, as long as you demonstrate you are willing to fix things. On the other hand, protestations that something is not a screw-up, but a requirement of corporate process, as was recently the case in some obscure change to OpenDS project rules that probably only a handful of Sun employees had even read, will for the most part be met by the response, “The screwup doth protest too much.”

Email is hopeless for resolving disputes

Open-source developers are extremely skilled in email. They can deal with many threads, often at a deep technical level, and are usually much more skilled at this than corporate developers.

Corporate folks, especially those new to the open-source world, tend to bring their culture with them. Attempts to use that culture’s process to run an open-source project will only cause friction, and in some cases needless disputes.

Once a dispute goes beyond the irksome email or two into a full-scale “firestorm,” then wise hands will immediately cease and desist.

Sometimes you have to give a message in person

I once worked as part of a team that was asked develop a presentation that was deemed important. When we were done we sent it off as an attachment to the responsible exec, saying it was ready for distribution via email to the execs for whom it was intended.

We got a response saying, “Some messages are too important for email. If we just send this out it will be ignored. Find out the key folks, the key sites, schedule some meetings, then get on a plane, and deliver the message in person.”

This explains why I suggested in the post, “Time for Sun to make a call,” that only a public statement by a senior Sun executive, or a direct face-to-face meeting, could rescue OpenDS from its current sad state.

You don’t need am ombudsman, you need a sheriff

I have mentioned in the past that it is useful in large projects to have a “sheriff,” an employee who is also a recognized expert in open-source. The sheriff can serve as a bridge between the open-source community and the corporate developers, most of whom will be dealing in an open-source project for the first time. The sheriff can buy time while the firestorms are put out, the feathers unruffled, and tempers cooled down, so that everyone can be back to work.

This role can also be thought of as similar to that of an ombudsman, but the ombudsman needs to be someone who is a recognized expert in running a project. Morever, the notion of “ombudsman” itself carries it a connotation of authority, something that itself is anathema to well-run projects; see for example, Yogi Yarns — On forking and plenipotentiaries

[I’m going to draw this note to close here, though I could go on. I started a post, “history of an open-source screw up,” several days ago, but as it kept growing and growing I realized it might best be published in serial form, and so offer this as a first chapter in that larger work.]

Notes:

1. Don Knuth’s Tex is a notable exception, as is Knuth himself.