Often when writing test automation, you run into bugs.  Arguably, the greatest usefulness of test automation is to catch regressions when new changes are made.  But finding bugs is part of it, too, of course.

In other schemes, I’d not been sure how to handle bugs that I know are there, have a bug written against them, and are likely to exist for a while longer.  Sure, it’d be great to have the bug fixed right away, but sometimes it won’t.  How do you mark it as a known failure?  Is it right to put a bug number in the source code to give a heads up where to look for more info on the failure?

My feeling has been that it IS ok to put bug numbers in the source code of your test automation.  But I had a friend who argued that this is wrong.  That source code should never contain bug numbers.  While I don’t agree, I admit, that the issue gave me pause.  It did seem kind of hacky.  But I wanted to have a clear reference to the bug for context.

Anyway, with lettuce the controversy is kind of removed.  Since you don’t define your tests in source code, you define them in .feature files.

However, having them mixed in with your passing tests can be misleading for non-test authors.  “I just want to know which ones are supposed to pass!” one might say.  So here is how I organized things with Lettuce:

In my features folder, I broke my .features files into 3 sections:

• Bugs – Here are the feature scenarios that are broken with known bugs.
• Passing – All the scenarios that are expected to pass.
• Staging – A working area for scenarios.  Ones I’m not done with yet.

This makes it easy for a CI tool like Jenkins, or the developer considering checking in a change, to just execute the passing tests like so:

lettuce features/passing/

You can also, of course, execute an individual feature by doing:

lettuce features/passing/
roles.feature

Break it down any way you like.  I’ve considered making a Smoke folder, too.  Since lettuce discovers all the files named .feature, it’s a snap.