TAP vs XML?

Curtis Poe has written an article on ONLamp titled "TAP vs XML".

TAP (Test Anything Protocol) is a specification for the test report format as generated by Test::More and family in perl.

Very popular in the perl world, TAP is language agnostic and you can find test frameworks generating TAP in many languages.

There is supposedly one in python but I couldn't find when I started my python project, so I fell back to unittests, ex-PyUnit, port of the jUnit test framework (the perl member of the xUnit family being Test::Unit).

The other thing I like about TAP is its simple and predictable format that makes its easily machine readable. That triggered a plethora of perl modules that "do things" with a TAP report.

Thanks to a module like Test::TAP::HTMLMatrix, we have been able to generate report like that:

It is based on Test::TAP::Model which I've seen being used to store the test reports in a SQLite database to facilitate a fined grained control of the evolution of tests (for which revision of the code they are passing, which ones are failing, if there is a pattern when a group of tests starts all to fail, ...)

The ONLamp article presents TAP as opposed to XML.

TAP is good as long as you can build an ecosystem around it like we did but there's a whole world out there where XML is deeply anchored for messaging and configuration.

It just so happens that I've been observing this world for a while from my perl island. What have I seen? I've seen tools, great tools grown up to support modern software engineering practices.

my options were:

• moaning about these tools being unfriendly to perl

• try to recreate these tools for perl projects

• subvert these tools by using them with perl

I've tried the first one, it was for good for me but didn't make things moving.
The Second one has been tried too: too much code that needs maintain (and we are not paid for working on such things) and not good enough to be open sourced (because not being paid to work on such things, we rushed them).

We've started to look at the third option which seems to be the One.

In the case I'm interested in, it's to fully make use of Cruise Control to support continuous integration and automated testing for my perl project.

As I am in a position where I've got the basic functionalities working along with some nice perl-CC integration (our HTMLMatrix test reports are now generated and archived within CC), I want to further integrate the perl testing framework to Cruise Control reporting facilities.

And this is where the TAP vs XML ends for me.

I need my TAP based perl framework to talk nicely to the Cruise Control build report.
For example, I want him to stops telling me that there's no test run when I know that are, because CC doesn't understand the format test output.

I don't mind the unreadability of the xml, as XML is not for human to read.
It will be transformed by XSLT (after merging it with CC build xml log) in a human-readable form to be part of the CC build report.

I don't mind xml generation being prone to errors by human (remember xml is not for human to use) as xml syntax can normally be validated by machines using Schemas.

I found a possible helper module on CPAN in the shape of Test::TAP::XML that needs to be investigated.

So it's more like TAP+XML then?

For me, yes, but I just found someone who also tries to bury XML in the world of project automation :-)