Open innovation in software means open source

July 2, 2010

Here’s a “reprint” of an article that I wrote recently for the H, to introduce my talk at TransferSummit last week.

According to Henry Chesbrough[1], Open Innovation consists of using external ideas as well as internal ideas, and internal and external paths to market, to advance a company’s technology.

Software architects and developers are usually not short of ideas, but which of those ideas are the really good ones? How do you select the winning options and avoid wasting energy and money on the useless ones?

Feedback is the key to separating the wheat from the chaff. Fast and good quality feedback is required to steer any fast vehicle or sports device, and it works the same in software: without a performant feedback loop, you’re bound to fall on your face – or at least to be slower than your competitors on the road to success.

Innovation is not invention – it’s about value

In a recent blog post on the subject, Christian Verstraete, CTO at HP, rightly notes that innovation is not invention. Whereas the value of a new invention might be unknown, the goal of innovation is to produce value, often from existing ideas.

The output of our feedback loop must then be a measurement of value – and what better value for a software product than happy stakeholders? Other developers adopting your ideas, field testers happy with performance, experts suggesting internal changes which will make them feel good about your software’s structure. That kind of feedback is invaluable in steering your innovative software product in the right direction, quickly.

How fast is your feedback loop?

If you have to wait months to get that high-quality feedback, as you might in a corporate setting, your pace of innovation will be accordingly slow.

In the old world of committees, meetings and reports, things move at the speed of overstuffed schedules and overdue reports – slowly. In the new world of agile open source projects, fast and asynchronous Internet-based communication channels are your friends, helping people work at their own pace and on their own schedule, while collectively creating value quickly.

Open source organizations like the Apache Software Foundation provide standardised tools and best practices to foster efficient communications amongst project members. Shared source code repositories generate events to which project members can subscribe, to be informed immediately of any changes in modules that they’re interested in. Web-based issue trackers also use events and subscriptions to make it easy to collaborate efficiently on specific tasks, without requiring the simultaneous online presence of collaborators. Mailing lists also allow asynchronous discussions and decisions, while making all the resulting information available in self-service to new project members.

It is these shared, event-based and asynchronous communications channels that build the quick feedback loop that is key to software innovation. It is not uncommon for a software developer to receive feedback on a piece of code that they wrote, from the other end of the world, just a few minutes after committing that code to the project’s public code repository. Compared to a written problem report coming “from above” a few weeks later, when the developer has moved on to a different module, the value of that fast feedback is very high. It can feel a bit like a bunch of field experts looking over your shoulder while you’re working – scary but extremely efficient.

How good are your feedback “sensors”?

Fast feedback won’t help if it’s of low quality, and fortunately open source projects can also help a lot here. Successful projects can help bring together the best minds in the industry, to collectively solve a problem that benefits all of them. The Apache HTTP server project is one of the best examples, with many CTO-level contributors including a few that were involved in defining the protocols and the shape of today’s Web. If software developers (God forbid) were sold between companies the way soccer players are transferred between teams, we’d see millions of dollars flowing around.

Open source projects are very probably the best way to efficiently bring software experts together today. Industry associations and interest groups might fulfill that role in other industries, but developers like to express themselves in code, and open source projects are where that happens today.

You could of course hire experts to give feedback on your software inside your company, but it’s only a handful of companies who have enough money to bring in the level and number of experts that we are talking about – and that might well turn out to be much slower than the open source way of working.

What’s the right type of project?

Creating or joining an open source project that helps your business and attracts a community of experts is not that easy: the open source project space is somewhat crowded today, and those experts are busy people.

Judging from the Apache Software Foundation’s achievements in the last ten years, infrastructure projects have by far the highest success rate. If you can reduce (part of) your problem to a generalised software infrastructure that appeals to a wide range of software developers, those experts will see value in joining the project. Apache Hadoop is another very successful example of software architects and developers from different companies joining forces to solve a hard problem (large scale distributed computing) in a way that can benefit a whole industry. On a smaller scale, Apache Jackrabbit , one of the projects in which my employer is very active, brings together many experts from the content management world, to solve the problem of storing, searching and retrieving multimedia content efficiently. Those types of software modules are used as central infrastructure components in systems that share a similar architecture, while offering very different services to their end users.

Projects closer to the user interface level are often harder to manage in an open group, partly because they are often more specific to the exact problem that they solve, and also because it is often hard for people coming from different companies and cultural backgrounds to agree on the colour of the proverbial bike shed. An infrastructure software project can be well defined by an industry specification (such as JCR in Jackrabbit’s case), and/or by automated test suites. These are usually much easier to agree on than user interface mock-ups.

Where next?

I hope to have convinced you that open source projects provide the best feedback loop for innovative software. As a next step, I would recommend getting involved in open source projects that matter to you. There are many ways to contribute, from reporting bugs in a useful way, to writing tutorials, contributing new modules or extensions, or simply reporting on your use of the software in various environments.

Contributing, in any small or big way, to a successful open source project is the best way to see this high-quality feedback loop in action. You might also try to use the open source ways of working inside your company, to create or improve your own high-quality “innovation feedback loop”.

I cannot pretend to have the definitive answer to the “how do you select and execute the right ideas to innovate?” question. When it comes to software, however, the fast and high-quality feedback loop that open source projects provide is, in my opinion, the best selection tool.

[1] Chesbrough, H.W. (2003). Open Innovation: The new imperative for creating and profiting from technology. Boston: Harvard Business School Press


The single mailing list dream

June 13, 2009

The ASF uses a (way too) large number of mailing lists for all its internal and project communications.

Having crosscutting discussions is quite hard – for example, many projects use OSGi these days, and the only way for them to share their OSGi experience would be to create yet another list, or to subscribe to all of each other’s lists, which means a lot more traffic to manage.

One of my current technical dreams is to have a single list for all of the ASF, using tags to define the audience and visibility of messages – a la Twitter hashtags.

A message about the maven-scr-plugin on the Sling list, for example, would be tagged

#sling #osgi #maven-scr-plugin #scr #public

so that people subscribing to the #osgi and #scr tags, for example, would see it.

Another obvious use case is to easily ignore all discussions about a given topic (like #budget maybe? ;-), in a reliable way and without losing other communications within the same group.

I’m not sure how to implement this today (particularly the access control part for things like the #asf-private tag), but that would in my opinion be a huge improvement on what we have now.

Update: it’s now 2017 and I heard from a colleague that his company the OpenStack dev list is using this model for group communications. That’s using mailing lists, but the model would apply to any shared channel that supports threaded discussions with searchable thread titles. I’m so happy to hear that this actually works! And too bad I haven’t been able to use it myself so far.


Why I hate press releases and polls

November 26, 2008

It’s about the filters.

When I read a blog post like ApacheCon 2008: inspiration, I can feel the author’s way of thinking, even though I don’t know him. Obviously liked the event and people, and maybe too enthusiastic to mention the inevitable negative aspects of their overall experience. An unfiltered report allows me to see through the plain words.

Most people don’t use mental filters when writing on their own blog, or if they do it is usually obvious. Ideas flow freely from their mind to mine, with only my reading filter in between.

Press releases are a different story: the original author of the idea often has to filter it down so that the PR people understand it (not that they’re dumb, but they’re often not subject matter experts either), and said PR people too often apply some form of “make sure we look good” or even “smoke and mirrors are fine” filters, common stuff in this type of writing.

That’s two or three filters more in the flow of ideas. Way too much.

Polls are even worse: if you were to ask me what word processor I use, with the choice between MagicTextWizard, OpenTextGenius or Other, I would reply Other, of course.

But am I really using the Other word processor? Not at all, my real answer is that I hate all word processors equally, and will use durable formats for any important writing: plain text, HTML, TeX if I have to. Editing with vi if needed, or anything that provides a full-screen view with no noisy menus.

Filters again – the poll’s limited set of answers is forcing my answer into a dumbed-down checkbox.

I hope this explains better why a press release about a dumbed-down poll will not make me too enthusiastic…grassroots PR is the best PR in my opinon, so press releases which sound like that will make me much happier.


Good technology makes things easier than you think

October 3, 2008

Working on macosx, I often find that things are easier than I thought – just drop something where it makes sense, or do what you would naturally do, and things work as expected. Often, at least.

I had the same experience today playing with OSGi services – one of my services needs to be informed of all configuration changes. OSGi provides a ConfigurationListener interface for that, but how are those registered with the framework?

That’s actually much easier than I thought at first – here’s the complete source code:

import org.osgi.service.cm.ConfigurationEvent;
import org.osgi.service.cm.ConfigurationListener;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

/**
 *  @scr.component 
 *      immediate="true"
 *      metatype="no"
 *  @scr.service
 */
public class DebuggingConfigurationListener implements ConfigurationListener {

	private final Logger log = LoggerFactory.getLogger(getClass());	
	
	public void configurationEvent(ConfigurationEvent e) {
		log.info("Got ConfigurationEvent, pid={}, type={}", e.getPid(), e.getType());
	}

}

Not bad hey? The scr.service annotation causes the class to be registered as a ConfigurationListener service, and the framework uses a whiteboard pattern to send configuration events to all such services.

Note to self: make things easier than what people expect.


QOTD: on compiling open source projects

September 26, 2008

Rod Johnson, Spring founder, in the TSS thread about the new SpringSource maintenance policy:

Anyone who refuses to compile an open source project under any circumstances doesn’t really believe in open source: they believe in other people working for them for free.

Well said.


QOTD – about dwarfs

July 31, 2008

Being considered the best speaker in a computer science department is like being known as the tallest of the Seven Dwarfs.

This is from Randy Pausch The Last Lecture book, I got it yesterday and find it really hard to put it down.



Why I love unixish configs

June 30, 2008

I wrote this today while discussing configuration mechanisms with my colleagues.

I love the unixish way of configurating things because the configs are:

Discoverable
With find, xargs and grep I can (very often) find out where something
is configured in my unixish system, with minimal initial knowledge.
Commentable
It is easy to add comments to configuration items in a text-based
config, and comments are not mangled when that config is later
modified from a GUI, if that’s available.

Cloneable
Copying the right config files from one system to another allows
configs to be cloned.

Traceable
By putting my configs under subversion control, I know what happened to them.
Documentable
Processing my configs with simple tools allows me to create reports or
dashboards easily.

The opposite of this is the M$ hell of opaque configurations managed by (sometimes even more opaque) GUIs, and unfortunately Sling leans more towards that opposite at the moment. We’ll have to fix this.


Give us today our daily social network invitation

May 22, 2008

I’ve been getting at least one, if not several invitations for new social networks every day lately.

So, please don’t feel offended if I ignore them, it is sometimes hard to select between timewasters and…more timewasters ;-)

(seriously: some of this stuff is very useful, but as with any new tool it looks like we’re in the explosion of new offerings phase right now).


QOTD: Albert Einstein on the teaching power of examples

May 19, 2008

Example isn’t another way to teach, it is the only way to teach (Albert Einstein).

Sounds so obvious…you don’t teach babies to speak by explaining how to move all the organs involved in speech: you start by talking to them, and once they get the basics you explain a bit and move to the next stage of examples.


Automated tests == reference documentation

March 13, 2008

Someone was asking about ujax:redirect in Sling. We have no docs on that yet, but how about some (readable) automated test code?

Using automated tests as reference documentation means that the docs will always be in sync with the code – assuming you’re using continuous integration to run those tests often.

People might object to having to read source code, so maybe creating a filter that presents the code in a nicer way would help?

Literate programming for test cases…the idea sounds worth pursuing.