Pragmatic validation metrics for third-party software components

Earlier this week at the IKS general assembly I was asked to present a set of industrial validation metrics for the open source software components that IKS is producing.

Being my pragmatic self, I decided to avoid any academic/abstract stuff and focus on concrete metrics that help us provide value-adding solutions to our customers in the long term.

Here’s the result, for a hypothetical FOO software component.

Metrics are numbered VMx to make it clear what we’ll be arguing about when it comes to evaluating IKS software.

VM1
Do I understand what FOO is?
VM2
Does FOO add value to my product?
VM3
Is that added value demonstrable/sellable to my customers?
VM4
Can I easily run FOO alongside with or inside my product?
VM5
Is the impact of FOO on runtime infrastructure requirements acceptable?
VM6
How good is the FOO API when it comes to integrating with my product?
VM7
Is FOO robust and functional enough to be used in production at the enterprise level?
VM8
Is the FOO test suite good enough as a functionality and non-regression “quality gate”?
VM9
Is the FOO licence (both copyright and patents) acceptable to me?
VM10
Can I participate in FOO’s development and influence it in a fair and balanced way?
VM11
Do I know who I should talk to for support and future development of FOO?
VM12
Am I confident that FOO still going to be available and maintained once the IKS funding period is over?

VM1 can be surprisingly hard to fulfill when working on researchy/experimental stuff ;-)

Suggestions for improvements are welcome in this post’s comments, as usual.

Thanks to Alex Conconi who contributed VM11.

2 Responses to Pragmatic validation metrics for third-party software components

  1. bdelacretaz says:

    See also http://jboye.com/blogpost/find-the-needle-in-the-haystack-that-is-semantic-web/ for a more end-user oriented view on this. I particularly like Brian’s “speak human” recommendation.

  2. I think it might be good to add some intermediate questions between VM1 and the questions actually requiring to have integrated (or at least tried to) FOO. VM9 might currently be the only question falling into this category.

    Dome ideas:
    – Is the scope of FOO (not the concrete implementation) has the potential to provide added value to my product?
    – Is the documenation on how to use FOO easy to understand?
    – Does the FOO-Website describe the product and its benefits in a way I understand and trust?

%d bloggers like this: