List all your Maven dependencies

July 8, 2010

Here’s a one-liner (well, two) that neatly lists all the Maven dependencies from your project. Useful to check their licenses, for example.

# first grab all dependencies
mvn dependency:resolve

# then list them with -o to keep noise low,
# remove extra information and duplicates
mvn -o dependency:list \
| grep ":.*:.*:.*" \
| cut -d] -f2- \
| sed 's/:[a-z]*$//g' \
| sort -u 

The output looks like this:

asm:asm:jar:1.5.3
asm:asm:jar:3.1
biz.aQute:bnd:jar:0.0.169
cglib:cglib:jar:2.1_3
classworlds:classworlds:jar:1.1
classworlds:classworlds:jar:1.1-alpha-2
...

so it’s also useful to detect multiple versions of the same dependency in a multi-module project.


Open innovation in software means open source

July 2, 2010

Here’s a “reprint” of an article that I wrote recently for the H, to introduce my talk at TransferSummit last week.

According to Henry Chesbrough[1], Open Innovation consists of using external ideas as well as internal ideas, and internal and external paths to market, to advance a company’s technology.

Software architects and developers are usually not short of ideas, but which of those ideas are the really good ones? How do you select the winning options and avoid wasting energy and money on the useless ones?

Feedback is the key to separating the wheat from the chaff. Fast and good quality feedback is required to steer any fast vehicle or sports device, and it works the same in software: without a performant feedback loop, you’re bound to fall on your face – or at least to be slower than your competitors on the road to success.

Innovation is not invention – it’s about value

In a recent blog post on the subject, Christian Verstraete, CTO at HP, rightly notes that innovation is not invention. Whereas the value of a new invention might be unknown, the goal of innovation is to produce value, often from existing ideas.

The output of our feedback loop must then be a measurement of value – and what better value for a software product than happy stakeholders? Other developers adopting your ideas, field testers happy with performance, experts suggesting internal changes which will make them feel good about your software’s structure. That kind of feedback is invaluable in steering your innovative software product in the right direction, quickly.

How fast is your feedback loop?

If you have to wait months to get that high-quality feedback, as you might in a corporate setting, your pace of innovation will be accordingly slow.

In the old world of committees, meetings and reports, things move at the speed of overstuffed schedules and overdue reports – slowly. In the new world of agile open source projects, fast and asynchronous Internet-based communication channels are your friends, helping people work at their own pace and on their own schedule, while collectively creating value quickly.

Open source organizations like the Apache Software Foundation provide standardised tools and best practices to foster efficient communications amongst project members. Shared source code repositories generate events to which project members can subscribe, to be informed immediately of any changes in modules that they’re interested in. Web-based issue trackers also use events and subscriptions to make it easy to collaborate efficiently on specific tasks, without requiring the simultaneous online presence of collaborators. Mailing lists also allow asynchronous discussions and decisions, while making all the resulting information available in self-service to new project members.

It is these shared, event-based and asynchronous communications channels that build the quick feedback loop that is key to software innovation. It is not uncommon for a software developer to receive feedback on a piece of code that they wrote, from the other end of the world, just a few minutes after committing that code to the project’s public code repository. Compared to a written problem report coming “from above” a few weeks later, when the developer has moved on to a different module, the value of that fast feedback is very high. It can feel a bit like a bunch of field experts looking over your shoulder while you’re working – scary but extremely efficient.

How good are your feedback “sensors”?

Fast feedback won’t help if it’s of low quality, and fortunately open source projects can also help a lot here. Successful projects can help bring together the best minds in the industry, to collectively solve a problem that benefits all of them. The Apache HTTP server project is one of the best examples, with many CTO-level contributors including a few that were involved in defining the protocols and the shape of today’s Web. If software developers (God forbid) were sold between companies the way soccer players are transferred between teams, we’d see millions of dollars flowing around.

Open source projects are very probably the best way to efficiently bring software experts together today. Industry associations and interest groups might fulfill that role in other industries, but developers like to express themselves in code, and open source projects are where that happens today.

You could of course hire experts to give feedback on your software inside your company, but it’s only a handful of companies who have enough money to bring in the level and number of experts that we are talking about – and that might well turn out to be much slower than the open source way of working.

What’s the right type of project?

Creating or joining an open source project that helps your business and attracts a community of experts is not that easy: the open source project space is somewhat crowded today, and those experts are busy people.

Judging from the Apache Software Foundation’s achievements in the last ten years, infrastructure projects have by far the highest success rate. If you can reduce (part of) your problem to a generalised software infrastructure that appeals to a wide range of software developers, those experts will see value in joining the project. Apache Hadoop is another very successful example of software architects and developers from different companies joining forces to solve a hard problem (large scale distributed computing) in a way that can benefit a whole industry. On a smaller scale, Apache Jackrabbit , one of the projects in which my employer is very active, brings together many experts from the content management world, to solve the problem of storing, searching and retrieving multimedia content efficiently. Those types of software modules are used as central infrastructure components in systems that share a similar architecture, while offering very different services to their end users.

Projects closer to the user interface level are often harder to manage in an open group, partly because they are often more specific to the exact problem that they solve, and also because it is often hard for people coming from different companies and cultural backgrounds to agree on the colour of the proverbial bike shed. An infrastructure software project can be well defined by an industry specification (such as JCR in Jackrabbit’s case), and/or by automated test suites. These are usually much easier to agree on than user interface mock-ups.

Where next?

I hope to have convinced you that open source projects provide the best feedback loop for innovative software. As a next step, I would recommend getting involved in open source projects that matter to you. There are many ways to contribute, from reporting bugs in a useful way, to writing tutorials, contributing new modules or extensions, or simply reporting on your use of the software in various environments.

Contributing, in any small or big way, to a successful open source project is the best way to see this high-quality feedback loop in action. You might also try to use the open source ways of working inside your company, to create or improve your own high-quality “innovation feedback loop”.

I cannot pretend to have the definitive answer to the “how do you select and execute the right ideas to innovate?” question. When it comes to software, however, the fast and high-quality feedback loop that open source projects provide is, in my opinion, the best selection tool.

[1] Chesbrough, H.W. (2003). Open Innovation: The new imperative for creating and profiting from technology. Boston: Harvard Business School Press


My new Flyer ebike: fast and fun!

July 1, 2010

flyer-t8hs.jpgI recently bought a new Flyer electric bike, got a faster T8 HS ex-demo for a good price.

I sold the previous C8+ to my nephew, happy that it’s staying in the family! In about 15’000km all year in any weather (including snow and accompanying salt on the road) over 4 1/2 years, I have had exactly zero problems with the C8, which says a lot about the build quality and maturity of those bikes. It needed just the usual bike maintenance, and an expected change of battery after about 600 charging cycles, but zero maintenance related to the electronics or motor. Not to mention only two punctures in 15’000km, thanks to the Schwalbe Marathon puncture-proof tires. Over time, those tires get full of small superficial holes which are mostly punctures that didn’t happen, very cool!

The new bike is an HS model, as in high speed: contrary to the old one which would only assist me until 25kmh (so you’d be faster on flat with a good bike), this one happily helps up to 45kmh or more, and it’s also a better bike to start with: 28″ wheels, more rigid frame and thinner higher-pressure tires. As with all so-called pedelec bikes, the Flyers don’t go anywhere if you don’t pedal, the assistance only kicks in (very naturally) when you ride normally, and gives you more power.

And this thing is fast: I just beat my record on the commute back from the office, 350m elevation over 12km, getting home in 29:30 which means 24kmh average speed on Lausanne’s steep hills. Not bad – you do have to pedal hard to reach such speeds uphill, but it’s a lot of fun and I get home almost as fast as any other transportation, considering the traffic density – and I don’t need to spend time at the gym after that, so I’m probably saving time all in all! The morning downhill ride takes about 20 minutes, unbeatable at 8AM unless you ride a helicopter.

The equipment is very good: Magura hydraulic rim brakes (almost as good as discs, I guess newer models have them), LED lights, lockable front fork and the SRAM dual drive which combines a 3-gear hub with an 8-gear rear derailleur to get 24 usable combinations (no “forbidden” ones like dual derailleurs). You cannot have a front derailleur on the Flyer due to the motor wheel which drives the chain, and the dual drive is really the best of both worlds in the city: gear hub for quick downchanging when stopping or surprised, and derailleur for fine tuning.

All in all, an excellent commuter’s bike if your ride is steep, or just for the fun of riding faster. The big plus with the ebike is that you can use less of your own energy if you’re tired or if conditions are bad, while still getting to your destination in a reasonable amount of time.

You do have to ride very carefully as sleepy car drivers and pedestrians often don’t realize how fast you ride on that thing, nor that you’re actually faster than cars in many tight or bumpy places. After years of motorcycling and cycling I’m used to being very clear about my intentions on the road, using obvious positioning in lanes, and that helps a lot! The city of Lausanne is also doing an excellent job in helping cyclists find safe space to ride, and most of my commute is on very low-traffic roads as a result.

Do I sound enthusiastic? That’s because I am – electric bikes are by far the best way of commuting in a steep city like Lausanne. They are somewhat expensive to buy, but maintenance costs almost nothing, and you save a lot on gym costs (and doctor’s fees I guess – cycling is good for your health). And if you drive a car or motorbike to work, you should really calculate how much that costs and draw the right conclusions!


Can I haz web?

June 15, 2010

How many people today still think information is only valid or “serious” when represented on an A4 piece of paper?

Way too many, if you ask me.

I’m always disappointed when people push out important content as PDF documents (or much worse…I won’t even name that format) attached to web pages or email messages, instead of just including the content in those web pages or messages, as a first-class citizen.

For some reason, people seem to think that information presented in A4 format has more value than the same information presented as a simple and clean web page. It is quite the opposite actually: web pages can be linked to, easily indexed, reformatted for efficient reading (thanks readability), etc.

Ted Nelson, the inventor of hypertext, wrote in 1999 already [1]:

We must overthrow the paper model, with its four prison walls and peephole one-way links

And also, in the same paper:

WYSIWYG generally means “What You See Is What You Get” — meaning what you get when you print it out. In other words, paper is the flat heart of most of today’s software concepts.

Granted, we haven’t fully solved the two-way links problem yet, but I hope you get the idea. Who needs paper or A4 pages? This is 2010, and this is the Web.

Please think about it next time you publish an important piece of information. Does it really need to live in the prison walls of a “document”? In what ways is that more valid than a web page or plain text email message?

Most of the time, almost always, the answer is: it’s not more valid, it’s just less usable.

Can I haz web? kthxbye.

[1] http://people.artcenter.edu/~vanallen/web_techniques/tednelson_liberate.htm


Open Innovation in Software means Open Source

December 7, 2009

I’m giving a talk today at the Open Source, Open Development, Open Innovation workshop in Oxford:

Open source software is more than just a licence, it is also a software development methodology that allows companies to share resources and collaborate on non-core parts of their software/service offering. When managed well, open development enables a reduction in cost, and an increase in innovation as a result of the convergence of the best minds in the problem space. In this presentation Bertrand Delacretaz will describe how Day Software has embraced open development by positioning itself as the leaders in both open standards and open source software. We will examine how Day’s active engagement with 25 open source projects and numerous standards groups has enabled the company to become a world leader in their market and in the open source projects they participate in.

The funny thing is that the above abstract was written by Ross Gardler while waiting for my own version of it – and it says exactly what I was trying to say, only better ;-)

The event is covered by a live blog, and you can ask questions there.

To put it simply, my conclusion is that quick feedback from users and customers is key to open innovation – and open source. if done right, provides lots of feedback, fast.


What Makes Apache Tick?

November 17, 2009

apachecon_us_09_twleung.jpgLooking at the diversity of Apache Software Foundation communities, one can see a recipe for failure: people from different cultural backgrounds, different mother tongues, different employers, different timezones…all working together to create some of the best software on the planet? You must be kidding.

How can this very loose collage of disparate people pump out dozens of high-quality releases every year, often working better than more structured corporate teams? This “mystery” has been on my mind for a while, and I have identified four drivers that influence the way we use collaboration tools that play a major part in our success.

The first driver is a common vision amongst project members. The Biblical saying, “Without a vision, my people perish” is quite valid for our projects. Both using a central development mailing list for each project, and spending time to collectively define our project’s charter, helps us foster this common vision amongst project members. Every member should have the same answer to the “what are our goals?” question, so it’s important to get them to talk in a central place, where they all get the same information, as opposed to undocumented, one-to-one discussions.

Secondly, providing real-time status updates to project members is key in helping them stay on track. At Apache, this is implemented by the many events generated by our collaboration tools: commit events to indicate code changes, issue tracker events to provide updates about the status of bugs and new features, success/failure events from continuous build systems, and standardized ways of announcing releases so that other projects are informed. Project members subscribe freely to as few or as many event channels that they want to, so as to stay on top of things in near real-time, and without having to actively ask others about what happened. Status meetings? No need for those, as the information is flowing all the time.

The third success driver lies in enabling real-time help requests. In an immediate crisis of the “we need to deliver this by tomorrow” type, especially when working with a big team, you need to be able to ask for help without necessarily 1) knowing who specifically will help, and 2) bothering others with direct person-to-person requests, especially if they work in a different timezone. The key here is using issue trackers, where one web page stores key data and parts of the dialog that leads to resolving an issue. Posting an issue on the tracker, with sufficiently detailed instructions about how to reproduce the problem, along with attributes such as severity level, affected modules, etc., is the best way to expose a problem to the group quickly and with precision. Using an issue tracker also allows you to quickly and efficiently change priorities as well as re-assign issues and tasks – key elements that make all the difference in a crisis.

Finally, having searchable archives of this information allows new project members, or those returning after a period of absence, to learn what transpired and why things have been done in a certain way. Without self-service archives, new participants would have to talk to everybody to find out about the project’s history, past decisions, conventions, etc., which is neither efficient nor scalable. Most of our archives are automatically built as project activities progress: mailing lists are archived, source code control history is kept forever, and issue trackers write the full history of the project’s micro-decisions.

Combined with Apache’s principles of meritocracy and consensus-based decision making, these four collaboration drivers allow our project teams to work very efficiently, and, in many cases, even more so than structured teams that do not establish those central hubs of information exchange.

Does your project team foster a common vision and provide tools for real-time status updates, real-time help requests and self-service archives to its members? If yes, congratulations: you’re on a good track to becoming as effective as an Apache project!

Many thanks to Sally Khudairi for reviewing and copy editing.

ApacheCon US 2009 picture by
Ted Leung / Creative Commons License (CC BY-NC-SA 2.0).

See also my Open Source Tools are Good For You presentation, which discusses the tools that Apache projects use to implement this.


What does Apache provide that other code repositories don’t?

November 16, 2009

People thinking about creating an open source project might rightly consider hosting on one of the various hosting services available: Google Code, SourceForge, kenai, bitbucket and github come to mind. Quick and easy, create a repository or request some resources and you’re in business.

Incubating a project at the Apache Software Foundation (ASF) takes a lot more effort than just requesting a hosting space on one of those services, so why would you do that? One can perfectly host code on one of those services with an Apache License, so what’s the difference?

I think the big difference lies in the governance model, and in fact calling the ASF just a code repository is very wrong. Let’s discuss some key elements of that.

The Apache voting process has been tried and tested since 1999, or even earlier. This is one of the things that projects coming through the incubator have to learn, led by their mentors. Learning is usually very easy as people quickly see the benefits of those simple no-nonsense rules.

The ASF also provides a well-defined structure for managing projects, and the foundation as a whole, in a fair and consensus-driven way. One could argue that structure gets in the way, and sometimes it does, but when things go wrong having a well-defined way of getting back on track helps tremendously. And this structure leaves a lot of freedom to the project’s management committee (PMC), there’s a lot of room for adapting a project’s way of working to its community and goals.

Creating an Apache project is certainly not required for all open source projects (and the foundation couldn’t scale to thousands of projects right now anyway), but for the critical infrastructure parts of one’s business (what’s sometimes called “open core”), having an established governance model makes all the difference.

The governance model is just one of the benefits that Apache projects get – there’s also the visibility, brand recognition, nice build services, and other tools, and, last but not least, the many friends that you make along the way! As everybody now knows, there are no jerks at Apache!


JCR in 15 minutes

November 3, 2009

We had a great NoSQL meeting yesterday evening colocated with ApacheCon. Thanks Jukka for organizing!

I was in track B for the second part, and found it very interesting to compare three different approaches to non-relational content storage: MarkLogic server, JCR and Pier Fumagalli’s Lucene+DAV technique.

I also quite liked Steve Yen’s “horseless carriage” way of looking at NoSQL. Defining things by what they are, as opposed to what they are not, sounds like a good idea.

I gave a short talk about JCR, find the slides below. Of course, as usual, they’re not as good as when I’m here to talk about them ;-)


Life in Open Source Communities, live at ApacheCon!

October 30, 2009

notmuch.jpgI have just finished my slides for next week at ApacheCon. Though the topic of how to “survive” in our open source communities has been on my mind for a while, this is a totally new presentation, which is both great (in the blank slate sense) and a lot of work.

cocoon-step.jpgHaving recently read Presentation Zen (very recommended if you do presentations and/or like beautiful books), I started adding full-screen pictures to the first few slides, and couldn’t stop! The presentation will then consist of me ad-libbing (or more precisely trying to tell stories) on a series of nice pictures grabbed from morguefile.com (don’t worry about that name).

allabout.jpgI’ll post the slides here later, for now they are super secret, so you’ll just get the teasers…images courtesy of morguefile.com (update: slides added now).

Hope to see you next week! In any case I have collected a number of useful links in my delicious bookmarks, I’ll point people to them in the presentation.


How well does the french-speaking world know the Apache Software Foundation?

October 27, 2009

emmanuel-tivipro.jpg

I was at OpenWorldForum in Paris a few weeks ago, together with fellow Apache members Sander Striker and Emmanuel Lécharny.

My first impressions (apart from the fact that Paris is always nice – I knew that already) were that the French tend to wear suits and say “vous” (polite form of “you”) instead of “tu” (the familiar form) which I would tend to use in geeky circles. Cultural differences…

But more seriously, how well does the french-speaking world know the Apache Software Foundation? Not well, it seems to me. In most of our discussions people could associate the ASF with the Apache HTTP Server project, but not much more. 2’000 committers? 300 members? Really?

bertrand-tivipro.jpgTo help improve this, I hope that the ASF can take a more active role in the conference next year, I’ll bring this up next week at ApacheCon with our conference people.

In the meantime french-speaking folks are welcome to learn a bit more about it thanks to TiViPRO’s interviews of Emmanuel and myself, shot during the conference.