Open source is done. Welcome to Open Development!

December 14, 2017

I originally published this article on SD Times, republishing it to keep it around for posterity…

If you’re looking at embracing open source today, you might be a bit late to the game. Using open-source software is mainstream now, and being involved in open-source projects is nothing to write home about either. Everybody does it, we know how it works, its value is proven.

But what’s next? Sharing source code openly is a given in open-source projects, but in the end it’s only about sharing lines of text. The real long-term power of successful open-source projects lies in how their communities operate, and that’s where open development comes in.

Shared communications channels. Meritocracy. Commit early, commit often. Back your work by issues in a shared tracker. Archive all discussions, decisions and issues about your project, and make that searchable. All simple principles that, when combined, make a huge difference to the efficiency of our corporate projects.

But, of course, the chaotic meritocracies of open-source projects won’t work for corporate teams, right? Such teams require a chain of command with strictly defined roles. Corporate meritocracy? You must be kidding.

I’m not kidding, actually: Open development works very well in corporate settings, and from my experience in both very small and fairly large organizations, much better than old-fashioned top-to-bottom chains of command and information segregation principles. Empower your engineers, trust them, make everything transparent so that mistakes can be caught early, and make sure the project’s flow of information is continuous and archived. Big rewards are just around the corner—if you dive in, that is.

What’s open development?
Open development starts by using shared digital channels to communicate between project members, as opposed to one-to-one e-mails and meetings. If your team’s e-mail clients are their knowledge base, that will go away with them when they leave, and it’s impossible for new project members to acquire that knowledge easily.

A centralized channel, like a mailing list, allows team members to be aware of everything that’s going on. A busy mailing list requires discipline, but the rewards are huge in terms of spreading knowledge around, avoiding duplicate work and providing a way for newcomers to get a feel for the project by reviewing the discussion archives. At the Apache Software Foundation, we even declare that “If it didn’t happen on the dev list, it didn’t happen,” which is a way of saying that whatever is worth saying must be made available to all team members. No more discrepancies in what information team members get; it’s all in there.

The next step is sharing all your code openly, all the time, with all stakeholders. Not just in a static way, but as a continuous flow of commits that can tell you how fast your software is evolving and where it’s going, in real time.

Software developers will sometimes tell you that they cannot show their code because it’s not finished. But code is never finished, and it’s not always beautiful, so who cares? Sharing code early and continuously brings huge benefits in terms of peer reviews, learning from others, and creating a sense of belonging among team members. It’s not “my” code anymore, it’s “our” code. I’m happy when someone sees a way to improve it and just does it, sometimes without even asking for permission, because the fix is obvious. One less bug, quality goes up, and “shared neurons in the air” as we sometimes say: all big benefits to a team’s efficiency and cohesion.

Openly sharing the descriptions and resolutions of issues is equally important and helps optimize usage of a team’s skills, especially in a crisis. As in a well-managed open-source project, every code change is backed by an issue in the tracker, so you end up with one Web page per issue, which tells the full history of why the change was made, how, when, and by whom. Invaluable information when you need to revisit the issue later, maybe much later when whoever wrote that code is gone.

Corporate projects too often skip this step because their developers are co-located and can just ask their colleague next door directly. By doing that, they lose an easy opportunity to create a living knowledgebase of their projects, without much effort from the developers. It’s not much work to write a few lines of explanation in an issue tracker when an issue is resolved, and, with good integration, rich links will be created between the issue tracker and the corresponding source code, creating a web of valuable information.

The dreaded “When can we ship?” question is also much easier to answer based on a dynamic list of specific issues and corresponding metadata than by asking around the office, or worse, having boring status meetings.

The last critical tool in our open development setup is in self-service archives of all that information. Archived mailing lists, resolved issues that stay around in the tracker, source-code control history, and log messages, once made searchable, make project knowledge available in self-service to all team members. Here as well, forget about access control and leave everything open. You want your engineers to be curious when they need to, and to find at least basic information about everything that’s going on by themselves, without having to bother their colleagues with meetings or tons of questions. Given sufficient self-service information, adding more competent people to a project does increase productivity, as people can largely get up to speed on their own.

While all this openness may seem chaotic and noisy to the corporate managers of yesterday, that’s how open-source projects work. The simple fact that loose groups of software developers with no common boss consistently produce some of the best software around should open your eyes. This works.


Status meetings are a waste of time and money

November 23, 2017

Last Monday I presented on Asynchronous Decision Making at the (excellent) FOSS Backstage Micro Summit in Berlin and there were some questions about me saying that status meetings are a waste of time.

My opinion on status meetings hasn’t changed in a long time and I’m very happy to see Jason Fried loudly confirm it in his status meetings are the scourge post.

Quoting him:

How would you feel if you had to regularly expense $1200 so you could “tell a few teammates something”. Think that would go over well?

If your team shape allows you to run status meetings, you should first reflect on their actual cost. And if you still want to run them after that I suggest:

a) Requiring people to briefly report their status in writing before the meeting, asynchronously

b) Requiring people to read other people’s status before the meeting, asynchronously

c) Choosing a maximum of 3 items to discuss in your meeting,
based on those reports, and timebox those topics during the meeting

d) If you don’t get enough items to deserve interrupting your whole team right now: cancel that meeting! Or maybe limit it to managers to avoid interrupting the Makers.

That’s just the essentials, Jason Fried has more detailed suggestions in a similar spirit, make sure to read his post!

I like the 3P format for brief written status reports:

– Progress: what concrete, measurable progress has been made since last report

– Problems: what’s blocking you from progressing

– Perspectives: what are your plans for the next period

If this post (and Jason’s) help you save tons of money by eliminating useless meetings, feel free to make a donation to a good cause ;-)


Large Mailing Lists Survival Guide

November 10, 2017

I initially published this blog post on the Adobe Open Development blog at http://blogs.adobe.com/ but that’s been archived so I’m re-publishing it here for convenience. Thanks to Adobe for allowing me to spend a portion on my time reflecting on such topics!

Here’s a “survival guide” that we use at Adobe to help our colleagues make sense of our busy Open Development mailing lists.

To send or not to send

TS1. Before asking a question, search the list archives (and issue trackers, etc.) as the answer might already be in there.

TS2. Keep noise low – consider how many people will receive your message. If you’re just saying “thank you” to one or two persons, do it off-list.

TS3. Does your message really belong in an email? If it’s information that’s supposed to last and that you expect future coworkers to know, a wiki, website or blog post is a much better place. Just send the URL of that wiki/website page in email then. Writing more than 3-4 paragraphs is often a sign of something that does not belong in email. And email is where information goes to die!

TS4. Do not cross-post. If you really think other lists might be interested, let them know about your message by sending them its URL, but do not copy the message itself. Cross-posting tends to create split discussions that are impossible to follow.

TS5. Don’t be shy – if a mailing list exists it is meant to be used, so any on-topic question with a sensible subject line is welcome, assuming you do your homework as explained in this guide.

Starting new discussion threads

ST0. To start a new topic, do not reply to an existing message – use the “new message” function of your email client to create a new discussion thread.

ST1. When starting a new discussion thread, include at least one [TAG] in the subject line that points to the product, technology or topic that your message is about.

ST2. The goal of the subject line is to help people decide if they want or need to read your message. If that’s not sufficient, use your first phrase as a summary to clarify. On busy lists, spending time on crafting a good subject line avoids wasting other people’s time, and gives you much better chances to get an answer.

ST3. One question/topic per thread please, it’s much easier to follow, helps people notice your question and makes much more sense in the archives later.

Writing your message

WR1. The shorter your message or reply, the more likely people are to read it.

WR2. Speak in URLs – if something has an URL or unique identifier, point to it. Don’t say “the about page on the new website” or “the memory problem”, but rather http://www.example.com/about and SLING-9362 if the latter is a well-known shortcut to the URL of the actual issue in your tracker. This helps avoid misunderstandings, as well as creating valuable archives with a rich Web of links.

WR3. If you’re describing a bug, include all the necessary information to reproduce the problem, while being as concise as you can: what did you do, what did you expect, what did you get, what was your environment.

WR4. No large attachments or stack traces: open a bug with that info or put it somewhere where people can download it and include only the URL in your message.

WR5. Be direct, ask the root question instead of a derived one. If you need to buy food at 4AM, for example, and you think Burrito Unlimited might be open, asking where can I buy food now is more likely to provide a helpful response that asking where the nearest Burrito Unlimited is. Also known as an “XY problem” where you ask Y but what you really need is X.

Replying and quoting

RQ1. Quote sparingly and precisely. Opinions differ on this, but the lazy top-quoting that’s unfortunately the norm depending on which email client you use tends to make discussions shallow, as it’s impossible to know precisely what people are replying to. So many years later, Usenet quoting still rules on busy lists and/or for complex discussions.

RQ2. Give sufficient context for people to follow your thoughts. Closely related to precise quoting.

RQ3. Remember WR1 – the more concise message usually wins.

Filtering tips

Here are some tips for coping with many emails from multiple high-traffic lists. The goal is to give you the option to quickly skim over the lists traffic to find the few threads that are actually relevant to you.

F1. Have a folder for each list, and a filtering rule that moves the relevant messages there.

F2. To make sure you quickly see responses to threads you take part in, have a special rule for emails/threads that include you. This is a rule that you can have across all your emails: it could be a special “to me” folder, or a different inbox.

F3. To filter out the topics that interest you, set up a rule based on keywords in the subject. Same for people.
This works especially well if people consistently use the ST1 subject line tags mentioned above.

See Also


Apache: lean and mean, durable, fun!

May 19, 2017

Here’s another blog post of mine that was initially published by Computerworld UK.

My current Fiat Punto Sport is the second Diesel car that I own, and I love those engines. Very smooth yet quite powerful acceleration, good fuel savings, a discount on state taxes thanks to low pollution, and it’s very reliable and durable. And fun to drive. How often does Grandma go “wow” when you put the throttle down in your car? That happens here, and that Grandma is not usually a car freak.

Diesel engines used to be boring, but they have made incredible progress in the last few years – while staying true to their basic principles of simplicity, robustness and reliability.

The recent noise about the Apache Software Foundation (ASF) moving to Git, or not, made me think that the ASF might well be the (turbocharged, like my car) Diesel engine of open source. And that might be a good thing.

The ASF’s best practices are geared towards project sustainability, and building communities around our projects. That might not be as flashy as creating a cool new project in three days, but sometimes you need to build something durable, and you need to be able to provide your users with some reassurances that that will be the case – or that they can take over cleanly if not.

In a similar way to a high tech Diesel engine that’s built to last and operate smoothly, I think the ASF is well suited for projects that have a long term vision. We often encourage projects that want to join the ASF via its Incubator to first create a small community and release some initial code, at some other place, before joining the Foundation. That’s one way to help those projects prove that they are doing something viable, and it’s also clearly faster to get some people together and just commit some code to one of the many available code sharing services, than following the ASF’s rules for releases, voting etc.

A Japanese 4-cylinder 600cc gasoline-powered sports bike might be more exciting than my Punto on a closed track, but I don’t like driving those in day-to-day traffic or on long trips. Too brutal, requires way too much attention. There’s space for both that and my car’s high tech Diesel engine, and I like both styles actually, depending on the context.

Open Source communities are not one-size-fits-all: there’s space for different types of communities, and by exposing each community’s positive aspects, instead of trying to get them to fight each other, we might just grow the collective pie and live happily ever after (there’s a not-so-hidden message to sensationalistic bloggers in that last paragraph).

I’m very happy with the ASF being the turbocharged Diesel engine of Open Source – it does have to stay on its toes to make sure it doesn’t turn into a boring old-style Diesel, but there’s no need to rush evolution. There’s space for different styles.


Shared neurons and the Shadok’s First Law of Failure

January 30, 2017

This blog post of mine was originally published by Computerworld UK in 2010.

French-speaking fortysomethings might remember the Shadoks, a two-minute TV comics show that aired on ORTF when I was a kid.

Those silly (and funny) creatures need to get to Earth as their own planet, on the left of the sky, is falling into pieces. They spend their time pumping on bicycle-like contraptions to produce Cosmogol 999 fuel to power their rocket towards Earth, while respected Professor Shadoko keeps them motivated with his motto: “the more you fail, the more likely your are to eventually succeed”. So they keep failing, every single time, hoping that statistics will prove them right some day.

A researcher colleague recently asked me to demonstrate that the work done by Apache communities is more than the sum of the individual capabilities of its members. I think the answer is yes, and that how we cope with failure has a lot to do with it.

Demonstrating this with hard numbers would be difficult, but having been active in Apache projects for the last ten years I can think of a number of examples where what we sometimes call “shared neurons” make all the difference.

It usually starts with a random thought – an often half-baked idea, that might be totally unrealistic or impossible to implement, but is nevertheless exposed to the community as is, and often starts a fruitful discussion. People bounce off each other’s ideas, in a written brainstorm that’s slower than if you were talking face to face, but sometimes more efficient as people have more time to think about their contributions to the discussion.

Brainstorming in the same room is faster, but brainstorming with a worldwide group of specialists – that’s much more powerful than Cosmogol 999, even if it has to happen in writing. But sometimes that turns into a Shadok fuel-producing exercise that fails to produce useful results.

Some Apache (and other) projects use those random thoughts as a first-class design and architecture tool, marking such email discussions with [RT] in their subject lines. This serves as a warning that the discussion is not necessarily about something feasible, it’s more about putting people’s neurons together to shape and refine potentially useful ideas. And sometimes those ideas are just too silly or unrealistic to go anywhere – yet the corresponding discussions stay archived forever on our mailing lists.

Some people are afraid of exposing their unfinished ideas to their communities so early, and having them archived for posterity whatever the outcome. Your idea might turn out to be a stupid one once people nicely demonstrate why it won’t work, or someone might point you to that part of code that does exactly what you were dreaming about, and that you had forgotten about. Silly you.

In my opinion, being ready to accept such failures makes you a much more efficient contributor to open communities. Although you don’t need to fail as consistently as the Shadoks, being ready to fall on your face once in a while allows you to contribute your wildest ideas to the discussion. Combined with respect for other’s crazy ideas, and with the ability to explain nicely when people are wrong, we have a recipe for efficient sharing of neurons, where the sum is definitely greater than its parts.

I’m seeing this neuron sharing in action on our Apache mailing lists all the time, and I think accepting the risk of being wrong when exposing one’s ideas on our public lists makes a big difference. We’re not Shadoks after all – our failure stats are way too low.


Microservices? Nah…Federated Services!

March 17, 2016

As much as I like the microservices concept (*) the name does not really make sense to me.

Why would decoupled services necessarily need to be micro?

Focusing on a service’s size is the wrong crusade for me, I’m happy as long as a service does one thing and one thing well – but that thing might be too large for a micro service.

We had good discussions about this with my colleagues and on Twitter recently, and after some virtual brainstorming it’s my good Apache friend Santi Gala who came up with the Federated Services term (worth a drink-of-your-choice Santi of course, next time we meet!).

I think this describes what we are after much better. Rightsizing makes much more sense than necessarily making things very small.

So here we go, with a Twitter-compatible less-than-140-characters definition:

Federated Services are independently deployable, scalable and swarm-friendly software components with language agnostic interfaces.

That’s it. I don’t think we need more than that to define those services.

WDYT?

(*) I’m working on a blog post about some elements the history of service oriented systems with another good Apache Friend. Stay tuned!


Redeploy OSGi bundles automatically with fizzed-watcher-maven-plugin

September 17, 2015

Being able to redeploy your OSGi bundles automatically when you make changes to their source code is very useful when developing Apache Sling applications, for example.

Today I tried the fizzed-watcher-maven-plugin on a Sling sample bundle, and it seems to work quite well. I just had to add the following to my POM:


<plugin>
  <groupId>com.fizzed</groupId>
  <artifactId>fizzed-watcher-maven-plugin</artifactId>
  <version>1.0.6</version>
  <configuration>
    <watches>
      <watch>
        <directory>src/main</directory>
      </watch>
    </watches>
    <goals>
      <param>clean</param>
      <param>install</param>
    </goals>
    <profiles>
      <param>autoInstallBundle</param>
    </profiles>
  </configuration>
</plugin>

And changing any file under src/main causes the bundle to be rebuilt and (via Sling’s default autoInstallBundle profile) reinstalled in my test Sling instance.

To start the plugin with that setup I used


mvn fizzed-watcher:run -Dsling.url=http://localhost:8080/system/console

See https://github.com/fizzed/maven-plugins for more info.

Filed under: very useful.