Here's a "reprint" of an article that I wrote recently for the H, to introduce my talk at TransferSummit last week.

According to Henry Chesbrough[1], Open Innovation consists of using external ideas as well as internal ideas, and internal and external paths to market, to advance a company's technology.

Software architects and developers are usually not short of ideas, but which of those ideas are the really good ones? How do you select the winning options and avoid wasting energy and money on the useless ones?

Feedback is the key to separating the wheat from the chaff. Fast and good quality feedback is required to steer any fast vehicle or sports device, and it works the same in software: without a performant feedback loop, you're bound to fall on your face - or at least to be slower than your competitors on the road to success.

Innovation is not invention - it's about value

In a recent blog post on the subject, Christian Verstraete, CTO at HP, rightly notes that innovation is not invention. Whereas the value of a new invention might be unknown, the goal of innovation is to produce value, often from existing ideas.

The output of our feedback loop must then be a measurement of value - and what better value for a software product than happy stakeholders? Other developers adopting your ideas, field testers happy with performance, experts suggesting internal changes which will make them feel good about your software's structure. That kind of feedback is invaluable in steering your innovative software product in the right direction, quickly.

How fast is your feedback loop?

If you have to wait months to get that high-quality feedback, as you might in a corporate setting, your pace of innovation will be accordingly slow.

In the old world of committees, meetings and reports, things move at the speed of overstuffed schedules and overdue reports - slowly. In the new world of agile open source projects, fast and asynchronous Internet-based communication channels are your friends, helping people work at their own pace and on their own schedule, while collectively creating value quickly.

Open source organizations like the Apache Software Foundation provide standardised tools and best practices to foster efficient communications amongst project members. Shared source code repositories generate events to which project members can subscribe, to be informed immediately of any changes in modules that they're interested in. Web-based issue trackers also use events and subscriptions to make it easy to collaborate efficiently on specific tasks, without requiring the simultaneous online presence of collaborators. Mailing lists also allow asynchronous discussions and decisions, while making all the resulting information available in self-service to new project members.

It is these shared, event-based and asynchronous communications channels that build the quick feedback loop that is key to software innovation. It is not uncommon for a software developer to receive feedback on a piece of code that they wrote, from the other end of the world, just a few minutes after committing that code to the project's public code repository. Compared to a written problem report coming "from above" a few weeks later, when the developer has moved on to a different module, the value of that fast feedback is very high. It can feel a bit like a bunch of field experts looking over your shoulder while you're working – scary but extremely efficient.

How good are your feedback "sensors"?

Fast feedback won't help if it's of low quality, and fortunately open source projects can also help a lot here. Successful projects can help bring together the best minds in the industry, to collectively solve a problem that benefits all of them. The Apache HTTP server project is one of the best examples, with many CTO-level contributors including a few that were involved in defining the protocols and the shape of today's Web. If software developers (God forbid) were sold between companies the way soccer players are transferred between teams, we'd see millions of dollars flowing around.

Open source projects are very probably the best way to efficiently bring software experts together today. Industry associations and interest groups might fulfill that role in other industries, but developers like to express themselves in code, and open source projects are where that happens today.

You could of course hire experts to give feedback on your software inside your company, but it's only a handful of companies who have enough money to bring in the level and number of experts that we are talking about – and that might well turn out to be much slower than the open source way of working.

What's the right type of project?

Creating or joining an open source project that helps your business and attracts a community of experts is not that easy: the open source project space is somewhat crowded today, and those experts are busy people.

Judging from the Apache Software Foundation's achievements in the last ten years, infrastructure projects have by far the highest success rate. If you can reduce (part of) your problem to a generalised software infrastructure that appeals to a wide range of software developers, those experts will see value in joining the project. Apache Hadoop is another very successful example of software architects and developers from different companies joining forces to solve a hard problem (large scale distributed computing) in a way that can benefit a whole industry. On a smaller scale, Apache Jackrabbit , one of the projects in which my employer is very active, brings together many experts from the content management world, to solve the problem of storing, searching and retrieving multimedia content efficiently. Those types of software modules are used as central infrastructure components in systems that share a similar architecture, while offering very different services to their end users.

Projects closer to the user interface level are often harder to manage in an open group, partly because they are often more specific to the exact problem that they solve, and also because it is often hard for people coming from different companies and cultural backgrounds to agree on the colour of the proverbial bike shed. An infrastructure software project can be well defined by an industry specification (such as JCR in Jackrabbit's case), and/or by automated test suites. These are usually much easier to agree on than user interface mock-ups.

Where next?

I hope to have convinced you that open source projects provide the best feedback loop for innovative software. As a next step, I would recommend getting involved in open source projects that matter to you. There are many ways to contribute, from reporting bugs in a useful way, to writing tutorials, contributing new modules or extensions, or simply reporting on your use of the software in various environments.

Contributing, in any small or big way, to a successful open source project is the best way to see this high-quality feedback loop in action. You might also try to use the open source ways of working inside your company, to create or improve your own high-quality "innovation feedback loop".

I cannot pretend to have the definitive answer to the "how do you select and execute the right ideas to innovate?" question. When it comes to software, however, the fast and high-quality feedback loop that open source projects provide is, in my opinion, the best selection tool.

[1] Chesbrough, H.W. (2003). Open Innovation: The new imperative for creating and profiting from technology. Boston: Harvard Business School Press