bertrand’s brain grep

Adobe, Day and Open Source: a dream and a nightmare

July 30, 2010

What does the acquisition of Day by Adobe mean for Day’s open source activities? Some people are disappointed by the lack of comments about this in the official announcements to date.

Thankfully, Erik Larson, senior director of product management and strategy at Adobe, commented on Glyn Moody’s blog post quite early in the frenzy of tweets and blog posts that followed yesterday’s announcement.

Quoting him:

…we are very excited for Day’s considerable “open source savvy” to expand Adobe’s already significant open source efforts and expertise. That is a strategic benefit of the combination of the two companies. I have personally learned a lot from David Nuscheler and his team in the past few months as we put the deal together.

Not bad for a start, but we’re engineers right? Used to consider the worst case, to make sure we’re prepared for it.

Me, I’m an engineer but also an optimistic, and I’m used to start with the ideal, happy case when analyzing situations. It helps focus my efforts on a worthy goal.

So let’s do this and dream about the best and worst cases. This is absolutely 100% totally my own dreams, I’m not speaking for anyone here, not wearing any hat. Just dreamin’, y’know?

The Dream

This is late 2011.

The last few months have more than confirmed that Day’s acquisition by Adobe, one year ago, happened for strategic reasons: a big part of the deal was filling up gaps in Adobe’s enterprise offering, but Day’s open source know-how and network have brought a lot of value as well.

Day folks have played an important role in expanding the open development culture inside Adobe; Photoshop will probably never be fully open source, but moving more key components of the Adobe technology stack to open source, and most importantly open development, has paid off nicely. In terms of reaching out to developers and customers, in getting much better feedback at all levels, and in terms of software quality of course. It’s those eyeballs.

The Apache Software Foundation’s Incubator has been quite busy in the last few months. The new platinum sponsor enjoys a fruitful relationship with the foundation.

With JCR moving to their core, Adobe’s enterprise applications are starting to reach a new level of flexibility. Customers are enthusiastic about being able to access their data via simple and standards-based interfaces. Enterprise-level mashups, anyone?

JCR is not just that minor content repository API pushed by that small swiss software vendor anymore: being adopted by a major player has made a huge difference in terms of market recognition (I’m sure my friends at Hippo, Jahia and Sakai, among others, will love that one). The added resources have also helped improve the implementations, and people love the book!

With this, Apache Jackrabbit and Apache Sling have reached new levels of community participation and quality. Although quite a few new committers are from Adobe, a number of other companies have also pushed their developers to participate more, due to the increased market visibility of JCR.

Adobe’s additional resources, used wisely to take advantage of the Day team’s strengths, have enabled them to fully realize the CQ5 vision. Everything is content, really.

As in all fairy tales, the former Day team and Adobe live happily ever after. (Editor’s note: this is not Disney, can we strike that one please?)

The Nightmare

This is late 2011, and I can hear the programmers complaining in their bland cubicles.

Aaarrggghhhhh.

The few Day folks who still work at Adobe did try to convince their management to continue on the open source and open development track. No luck – you can’t argue with an US company making 4 billion a year, can you?

CQ5 customers are too busy converting their websites to native PDF (this is about documents, right?) to realize what’s going on. The most desperate just switched to DrooplaPress, the newest kid on the LISP-based CMSes block. That won’t help business much but at least it’s fun to work with. If you love parentheses, that is.

Adobe’s competitors who really jumped on the open source and open development train are gone for good, it is too late to catch up. You should have sold you shares a year ago.

Luckily, Apache Jackrabbit and Apache Sling are still alive, and increased involvement of the “Benelux Gang” (ex-Day folks spread over a few Benelux content management companies) in those projects means there’s still hope.

You wake up wondering why you didn’t accept that job at the local fast food. Computers are so boring.

Coda

I know life is more complicated than dreams sometimes, but I like dreams much better than nightmares, and I’m a chronic optimistic. So you can easily guess which scenario I’m going to work towards!

I’ll keep you posted about what really happens next. Once I wake up, that is.

Just dreamin’, y’know?

Dear Oracle, can we have our nice javadoc URLs back?

July 21, 2010

If you support this request, please vote for it in the comments below and/or on twitter using the #E17476 hashtag!

Update (2010/07/24): it looks like the old java.sun.com URLs are back, thanks Oracle and especially @mreinhold!

Update (2010/07/27): see also Good Feedback and Happy Endings – The Ugly URLs.

Dear Oracle,

A while ago you bought Sun, and IIRC promised to do good things for Java. Or at least indicated you would. Or something like that.

Now, a bad thing happened a few days ago. Not a bad bad bad thing, just a tiny annoying change in the cool URLs that Sun used to publish the JDK’s javadocs. Not annoying annoying annoying but not nice.

Even Google remembers: today if I search for IndexOutOfBoundsException on Google it returns the following URL:

http://java.sun.com/j2se/1.5.0/docs/api/java/lang/IndexOutOfBoundsException.html

Which is a cool URL that shouldn’t change.

Now, requesting this URL today causes a redirect to:

http://download.oracle.com/docs/cd/E17476_01/javase/1.5.0/docs/api/java/lang/IndexOutOfBoundsException.html

Which is also somewhat cool, but not as much. Factor 10 down in coolness. It makes me assume that you’re serving javadocs from a CD, and that CD’s identifier is E17476_01. That’s useful info if you’re the filesystem driver who’s reading the CD, but I doubt filesystem drivers are searching for javadocs on Google. Also, I’m not looking at downloading anything. Just browsing, okay?

Cool URLs shouldn’t change.

Can we have the old one back? Ok, maybe with java.oracle.com instead of java.sun.com – you bought them anyway. But please please please, let the poor CD filesystem driver alone!

Thanks.

P.S. we’re having a little vote on Twitter about this, check it out at http://search.twitter.com/search?q=%23E17476

23 Comments | Java, The Web | Permalink
Posted by bdelacretaz

This is how we work at Apache

July 16, 2010

I just had to (re-)explain how the Apache way of working makes a difference by enabling a continuous flow of information between developers.

No more begging for reports, no more boring meetings where you only exchange information: who could say no to that?

Here it is for your enjoyment. This is the same thing that I’ve been saying in my recent talks on this topic, reduced to the bare minimum.

All technical discussions and decisions on public mailing lists.
Speak in URLs: if you reference something (discussion, vote,
code…anything) include its URL, which must be permanent.
Shared code repository, commit early, commit often (as in: daily at least, from day one)
Commit events sent to mailing lists and/or RSS feeds to which people
can subscribe.
Shared issue tracker and “if you’re working on something it must be
an issue in the tracker” so that progress reports are automatic. Also generates mail/RSS events.
Commits are linked to tracker issue IDs – by speaking in URLs in your commit messages, mostly.
Automatic archiving of all this information, for self-service access.

All this is public and centrally accessible of course, so everybody
gets the same information.

The main reluctance that I see when trying to convince people to work
in this way is the fear of exposing your mistakes and initial bad
designs in public. My answer is to just get over it: you’d find tons
of such blunders if you were to analyze my work at Apache in the last
ten years, yet I’m reasonably alive and kicking.

1 Comment | Open Source, asf, collaboration | Tagged: opensource, collaboration | Permalink
Posted by bdelacretaz

List all your Maven dependencies

July 8, 2010

Here’s a one-liner (well, two) that neatly lists all the Maven dependencies from your project. Useful to check their licenses, for example.

# first grab all dependencies
mvn dependency:resolve

# then list them with -o to keep noise low,
# remove extra information and duplicates
mvn -o dependency:list \
| grep ":.*:.*:.*" \
| cut -d] -f2- \
| sed 's/:[a-z]*$//g' \
| sort -u

The output looks like this:

asm:asm:jar:1.5.3
asm:asm:jar:3.1
biz.aQute:bnd:jar:0.0.169
cglib:cglib:jar:2.1_3
classworlds:classworlds:jar:1.1
classworlds:classworlds:jar:1.1-alpha-2
...

so it’s also useful to detect multiple versions of the same dependency in a multi-module project.

1 Comment | tools | Permalink
Posted by bdelacretaz

Open innovation in software means open source

July 2, 2010

Here’s a “reprint” of an article that I wrote recently for the H, to introduce my talk at TransferSummit last week.

According to Henry Chesbrough[1], Open Innovation consists of using external ideas as well as internal ideas, and internal and external paths to market, to advance a company’s technology.

Software architects and developers are usually not short of ideas, but which of those ideas are the really good ones? How do you select the winning options and avoid wasting energy and money on the useless ones?

Feedback is the key to separating the wheat from the chaff. Fast and good quality feedback is required to steer any fast vehicle or sports device, and it works the same in software: without a performant feedback loop, you’re bound to fall on your face – or at least to be slower than your competitors on the road to success.

Innovation is not invention – it’s about value

In a recent blog post on the subject, Christian Verstraete, CTO at HP, rightly notes that innovation is not invention. Whereas the value of a new invention might be unknown, the goal of innovation is to produce value, often from existing ideas.

The output of our feedback loop must then be a measurement of value – and what better value for a software product than happy stakeholders? Other developers adopting your ideas, field testers happy with performance, experts suggesting internal changes which will make them feel good about your software’s structure. That kind of feedback is invaluable in steering your innovative software product in the right direction, quickly.

How fast is your feedback loop?

If you have to wait months to get that high-quality feedback, as you might in a corporate setting, your pace of innovation will be accordingly slow.

In the old world of committees, meetings and reports, things move at the speed of overstuffed schedules and overdue reports – slowly. In the new world of agile open source projects, fast and asynchronous Internet-based communication channels are your friends, helping people work at their own pace and on their own schedule, while collectively creating value quickly.

Open source organizations like the Apache Software Foundation provide standardised tools and best practices to foster efficient communications amongst project members. Shared source code repositories generate events to which project members can subscribe, to be informed immediately of any changes in modules that they’re interested in. Web-based issue trackers also use events and subscriptions to make it easy to collaborate efficiently on specific tasks, without requiring the simultaneous online presence of collaborators. Mailing lists also allow asynchronous discussions and decisions, while making all the resulting information available in self-service to new project members.

It is these shared, event-based and asynchronous communications channels that build the quick feedback loop that is key to software innovation. It is not uncommon for a software developer to receive feedback on a piece of code that they wrote, from the other end of the world, just a few minutes after committing that code to the project’s public code repository. Compared to a written problem report coming “from above” a few weeks later, when the developer has moved on to a different module, the value of that fast feedback is very high. It can feel a bit like a bunch of field experts looking over your shoulder while you’re working – scary but extremely efficient.

How good are your feedback “sensors”?

Fast feedback won’t help if it’s of low quality, and fortunately open source projects can also help a lot here. Successful projects can help bring together the best minds in the industry, to collectively solve a problem that benefits all of them. The Apache HTTP server project is one of the best examples, with many CTO-level contributors including a few that were involved in defining the protocols and the shape of today’s Web. If software developers (God forbid) were sold between companies the way soccer players are transferred between teams, we’d see millions of dollars flowing around.

Open source projects are very probably the best way to efficiently bring software experts together today. Industry associations and interest groups might fulfill that role in other industries, but developers like to express themselves in code, and open source projects are where that happens today.

You could of course hire experts to give feedback on your software inside your company, but it’s only a handful of companies who have enough money to bring in the level and number of experts that we are talking about – and that might well turn out to be much slower than the open source way of working.

What’s the right type of project?

Creating or joining an open source project that helps your business and attracts a community of experts is not that easy: the open source project space is somewhat crowded today, and those experts are busy people.

Judging from the Apache Software Foundation’s achievements in the last ten years, infrastructure projects have by far the highest success rate. If you can reduce (part of) your problem to a generalised software infrastructure that appeals to a wide range of software developers, those experts will see value in joining the project. Apache Hadoop is another very successful example of software architects and developers from different companies joining forces to solve a hard problem (large scale distributed computing) in a way that can benefit a whole industry. On a smaller scale, Apache Jackrabbit , one of the projects in which my employer is very active, brings together many experts from the content management world, to solve the problem of storing, searching and retrieving multimedia content efficiently. Those types of software modules are used as central infrastructure components in systems that share a similar architecture, while offering very different services to their end users.

Projects closer to the user interface level are often harder to manage in an open group, partly because they are often more specific to the exact problem that they solve, and also because it is often hard for people coming from different companies and cultural backgrounds to agree on the colour of the proverbial bike shed. An infrastructure software project can be well defined by an industry specification (such as JCR in Jackrabbit’s case), and/or by automated test suites. These are usually much easier to agree on than user interface mock-ups.

Where next?

I hope to have convinced you that open source projects provide the best feedback loop for innovative software. As a next step, I would recommend getting involved in open source projects that matter to you. There are many ways to contribute, from reporting bugs in a useful way, to writing tutorials, contributing new modules or extensions, or simply reporting on your use of the software in various environments.

Contributing, in any small or big way, to a successful open source project is the best way to see this high-quality feedback loop in action. You might also try to use the open source ways of working inside your company, to create or improve your own high-quality “innovation feedback loop”.

I cannot pretend to have the definitive answer to the “how do you select and execute the right ideas to innovate?” question. When it comes to software, however, the fast and high-quality feedback loop that open source projects provide is, in my opinion, the best selection tool.

[1] Chesbrough, H.W. (2003). Open Innovation: The new imperative for creating and profiting from technology. Boston: Harvard Business School Press

Leave a Comment » | Open Source, Thoughts | Tagged: innovation, opensource | Permalink
Posted by bdelacretaz

My new Flyer ebike: fast and fun!

July 1, 2010

I recently bought a new Flyer electric bike, got a faster T8 HS ex-demo for a good price.

I sold the previous C8+ to my nephew, happy that it’s staying in the family! In about 15’000km all year in any weather (including snow and accompanying salt on the road) over 4 1/2 years, I have had exactly zero problems with the C8, which says a lot about the build quality and maturity of those bikes. It needed just the usual bike maintenance, and an expected change of battery after about 600 charging cycles, but zero maintenance related to the electronics or motor. Not to mention only two punctures in 15’000km, thanks to the Schwalbe Marathon puncture-proof tires. Over time, those tires get full of small superficial holes which are mostly punctures that didn’t happen, very cool!

The new bike is an HS model, as in high speed: contrary to the old one which would only assist me until 25kmh (so you’d be faster on flat with a good bike), this one happily helps up to 45kmh or more, and it’s also a better bike to start with: 28″ wheels, more rigid frame and thinner higher-pressure tires. As with all so-called pedelec bikes, the Flyers don’t go anywhere if you don’t pedal, the assistance only kicks in (very naturally) when you ride normally, and gives you more power.

And this thing is fast: I just beat my record on the commute back from the office, 350m elevation over 12km, getting home in 29:30 which means 24kmh average speed on Lausanne’s steep hills. Not bad – you do have to pedal hard to reach such speeds uphill, but it’s a lot of fun and I get home almost as fast as any other transportation, considering the traffic density – and I don’t need to spend time at the gym after that, so I’m probably saving time all in all! The morning downhill ride takes about 20 minutes, unbeatable at 8AM unless you ride a helicopter.

The equipment is very good: Magura hydraulic rim brakes (almost as good as discs, I guess newer models have them), LED lights, lockable front fork and the SRAM dual drive which combines a 3-gear hub with an 8-gear rear derailleur to get 24 usable combinations (no “forbidden” ones like dual derailleurs). You cannot have a front derailleur on the Flyer due to the motor wheel which drives the chain, and the dual drive is really the best of both worlds in the city: gear hub for quick downchanging when stopping or surprised, and derailleur for fine tuning.

All in all, an excellent commuter’s bike if your ride is steep, or just for the fun of riding faster. The big plus with the ebike is that you can use less of your own energy if you’re tired or if conditions are bad, while still getting to your destination in a reasonable amount of time.

You do have to ride very carefully as sleepy car drivers and pedestrians often don’t realize how fast you ride on that thing, nor that you’re actually faster than cars in many tight or bumpy places. After years of motorcycling and cycling I’m used to being very clear about my intentions on the road, using obvious positioning in lanes, and that helps a lot! The city of Lausanne is also doing an excellent job in helping cyclists find safe space to ride, and most of my commute is on very low-traffic roads as a result.

Do I sound enthusiastic? That’s because I am – electric bikes are by far the best way of commuting in a steep city like Lausanne. They are somewhat expensive to buy, but maintenance costs almost nothing, and you save a lot on gym costs (and doctor’s fees I guess – cycling is good for your health). And if you drive a car or motorbike to work, you should really calculate how much that costs and draw the right conclusions!

3 Comments | Fun, cycling | Tagged: cycling, ebike, flyer | Permalink
Posted by bdelacretaz

Can I haz web?

June 15, 2010

How many people today still think information is only valid or “serious” when represented on an A4 piece of paper?

Way too many, if you ask me.

I’m always disappointed when people push out important content as PDF documents (or much worse…I won’t even name that format) attached to web pages or email messages, instead of just including the content in those web pages or messages, as a first-class citizen.

For some reason, people seem to think that information presented in A4 format has more value than the same information presented as a simple and clean web page. It is quite the opposite actually: web pages can be linked to, easily indexed, reformatted for efficient reading (thanks readability), etc.

Ted Nelson, the inventor of hypertext, wrote in 1999 already [1]:

We must overthrow the paper model, with its four prison walls and peephole one-way links

And also, in the same paper:

WYSIWYG generally means “What You See Is What You Get” — meaning what you get when you print it out. In other words, paper is the flat heart of most of today’s software concepts.

Granted, we haven’t fully solved the two-way links problem yet, but I hope you get the idea. Who needs paper or A4 pages? This is 2010, and this is the Web.

Please think about it next time you publish an important piece of information. Does it really need to live in the prison walls of a “document”? In what ways is that more valid than a web page or plain text email message?

Most of the time, almost always, the answer is: it’s not more valid, it’s just less usable.

Can I haz web? kthxbye.

[1] http://people.artcenter.edu/~vanallen/web_techniques/tednelson_liberate.htm

6 Comments | The Web, collaboration | Permalink
Posted by bdelacretaz

Open Innovation in Software means Open Source

December 7, 2009

I’m giving a talk today at the Open Source, Open Development, Open Innovation workshop in Oxford:

Open source software is more than just a licence, it is also a software development methodology that allows companies to share resources and collaborate on non-core parts of their software/service offering. When managed well, open development enables a reduction in cost, and an increase in innovation as a result of the convergence of the best minds in the problem space. In this presentation Bertrand Delacretaz will describe how Day Software has embraced open development by positioning itself as the leaders in both open standards and open source software. We will examine how Day’s active engagement with 25 open source projects and numerous standards groups has enabled the company to become a world leader in their market and in the open source projects they participate in.

The funny thing is that the above abstract was written by Ross Gardler while waiting for my own version of it – and it says exactly what I was trying to say, only better ;-)

The event is covered by a live blog, and you can ask questions there.

To put it simply, my conclusion is that quick feedback from users and customers is key to open innovation – and open source. if done right, provides lots of feedback, fast.

Leave a Comment » | Open Source, conferences | Permalink
Posted by bdelacretaz

What Makes Apache Tick?

November 17, 2009

Looking at the diversity of Apache Software Foundation communities, one can see a recipe for failure: people from different cultural backgrounds, different mother tongues, different employers, different timezones…all working together to create some of the best software on the planet? You must be kidding.

How can this very loose collage of disparate people pump out dozens of high-quality releases every year, often working better than more structured corporate teams? This “mystery” has been on my mind for a while, and I have identified four drivers that influence the way we use collaboration tools that play a major part in our success.

The first driver is a common vision amongst project members. The Biblical saying, “Without a vision, my people perish” is quite valid for our projects. Both using a central development mailing list for each project, and spending time to collectively define our project’s charter, helps us foster this common vision amongst project members. Every member should have the same answer to the “what are our goals?” question, so it’s important to get them to talk in a central place, where they all get the same information, as opposed to undocumented, one-to-one discussions.

Secondly, providing real-time status updates to project members is key in helping them stay on track. At Apache, this is implemented by the many events generated by our collaboration tools: commit events to indicate code changes, issue tracker events to provide updates about the status of bugs and new features, success/failure events from continuous build systems, and standardized ways of announcing releases so that other projects are informed. Project members subscribe freely to as few or as many event channels that they want to, so as to stay on top of things in near real-time, and without having to actively ask others about what happened. Status meetings? No need for those, as the information is flowing all the time.

The third success driver lies in enabling real-time help requests. In an immediate crisis of the “we need to deliver this by tomorrow” type, especially when working with a big team, you need to be able to ask for help without necessarily 1) knowing who specifically will help, and 2) bothering others with direct person-to-person requests, especially if they work in a different timezone. The key here is using issue trackers, where one web page stores key data and parts of the dialog that leads to resolving an issue. Posting an issue on the tracker, with sufficiently detailed instructions about how to reproduce the problem, along with attributes such as severity level, affected modules, etc., is the best way to expose a problem to the group quickly and with precision. Using an issue tracker also allows you to quickly and efficiently change priorities as well as re-assign issues and tasks – key elements that make all the difference in a crisis.

Finally, having searchable archives of this information allows new project members, or those returning after a period of absence, to learn what transpired and why things have been done in a certain way. Without self-service archives, new participants would have to talk to everybody to find out about the project’s history, past decisions, conventions, etc., which is neither efficient nor scalable. Most of our archives are automatically built as project activities progress: mailing lists are archived, source code control history is kept forever, and issue trackers write the full history of the project’s micro-decisions.

Combined with Apache’s principles of meritocracy and consensus-based decision making, these four collaboration drivers allow our project teams to work very efficiently, and, in many cases, even more so than structured teams that do not establish those central hubs of information exchange.

Does your project team foster a common vision and provide tools for real-time status updates, real-time help requests and self-service archives to its members? If yes, congratulations: you’re on a good track to becoming as effective as an Apache project!

Many thanks to Sally Khudairi for reviewing and copy editing.

ApacheCon US 2009 picture by
Ted Leung / Creative Commons License (CC BY-NC-SA 2.0).

See also my Open Source Tools are Good For You presentation, which discusses the tools that Apache projects use to implement this.

Leave a Comment » | Open Source, asf | Permalink
Posted by bdelacretaz

What does Apache provide that other code repositories don’t?

November 16, 2009

People thinking about creating an open source project might rightly consider hosting on one of the various hosting services available: Google Code, SourceForge, kenai, bitbucket and github come to mind. Quick and easy, create a repository or request some resources and you’re in business.

Incubating a project at the Apache Software Foundation (ASF) takes a lot more effort than just requesting a hosting space on one of those services, so why would you do that? One can perfectly host code on one of those services with an Apache License, so what’s the difference?

I think the big difference lies in the governance model, and in fact calling the ASF just a code repository is very wrong. Let’s discuss some key elements of that.

The Apache voting process has been tried and tested since 1999, or even earlier. This is one of the things that projects coming through the incubator have to learn, led by their mentors. Learning is usually very easy as people quickly see the benefits of those simple no-nonsense rules.

The ASF also provides a well-defined structure for managing projects, and the foundation as a whole, in a fair and consensus-driven way. One could argue that structure gets in the way, and sometimes it does, but when things go wrong having a well-defined way of getting back on track helps tremendously. And this structure leaves a lot of freedom to the project’s management committee (PMC), there’s a lot of room for adapting a project’s way of working to its community and goals.

Creating an Apache project is certainly not required for all open source projects (and the foundation couldn’t scale to thousands of projects right now anyway), but for the critical infrastructure parts of one’s business (what’s sometimes called “open core”), having an established governance model makes all the difference.

The governance model is just one of the benefits that Apache projects get – there’s also the visibility, brand recognition, nice build services, and other tools, and, last but not least, the many friends that you make along the way! As everybody now knows, there are no jerks at Apache!

3 Comments | Open Source, asf | Permalink
Posted by bdelacretaz