estimated reading time: 10min.

Before dispensing any advice about design, infrastructure or any work methodology I thought it best to start with the foundations.
Testing is one of the core aspects of programming. I know it’s one of those tasks that no one wants to do and is often considered a nuisance by developers, but pound for pound it is by far the most rewarding.

However, in order for tests to be effective they must protect the developer from making mistakes he might be oblivious to. If you commit a change to the code that causes an error in someone else’s test you need to know about it as soon as possible, the longer you wait the harder it will be for you to find the bug later on or even to fix it.

So what you need is system that is aware of all tests available in your code, can execute all these tests whenever a change to the code is committed (or every few minutes) and in case of a failure in the tests notify all the developers that changed the code.

In other words what you need is continuous testing…

Continuous testing is probably one of the most useful things any application can use. It requires very little time to set up and has far reaching benefits in both development time and quality.

In an attempt to provide you with a real world example of how to apply continuous testing in your project I will take a different approach from my counterparts and address the most difficult aspect of using a continuous testing framework: convincing your coworkers it’s good for them.

For that reason I am not going to give you many code snippets or talk about available technologies, but rather on the human side of the equation starting with…

Your boss

As we all know there is always something really urgent that has to be done right now and can’t possibly be postpones by something as superfluous as a testing server, right?

Wrong! If time and effort are important then continuous testing is paramount. Try explaining the following points:

Efficiency

using continuous testing (CT) allows developers to find bugs as they are introduced to the system. Because of this if a developer accidently breaks some previously existing behavior, he is notified immediately (or fast enough), before many other changes had been done to the code and while the developer still remembers all the changes he made.
The bottom line is, less time is spent on fixing previously working code and your boss gets more bang for his buck.
If he is still not convinced try this…
Assuming you only save 5 min. a day by using CT (from my experience it s significantly more). Multiply that by the amount of developers working on the project and you will get the daily saving of a CT system.
For instance if you are 3 developers then is one month (22 work days) you will save
22*3*5/60 = 5.5 work hours, so even if you spend half a day setting up some basic form of CT your work will repay itself within a month, keep in mind that as your application and team grows in size and complexity so will the time earned per day.

Quality

By testing everything all the time and notifying people about the status of their tests you are preventing code degradation from occurring, there is a significantly smaller chance that your developers will break some backward compatibility or some existing feature if all such features are tested every time a change is introduced.
Furthermore, by continuously providing feedback to the developers about the status of their tests and by combining tool such as code coverage, method complexity and coding style as part of the report, developers are forced to think about testing whenever they write a new piece of code, making writing tests an integral part of the work process and not some deferred after though.

Time To Market (TTM)

The holy grail of management, in my previous work place reducing the TTM was a major thing, mainly because it consistent of a product developed by 4 teams and about 30 developers over the course of months.
When the product was ready for delivery, each team would merge all their changes to the code and then make sure all the tests were passing. Since some usually aren’t passing this requires a difficult operation by which the merge process requires the involvement of many developers and may take several days. Once the tests pass a code freeze is announced and functional tests ensue followed by a round of fixes and then functional tests again.
When this is over the product was given to integration and QA for testing, overall deploying new code would take about 1-2 weeks.
If we were to use CT that time would drop significantly since merging and unit testing the code is free (it is done on every change). Not only that but with some more effort functional tests can also be integrated into a CT environment making them less likely to fail as well, and if that is not enough then consider the fact the since the quality of code and efficiency of your developers had increased the process of fixing bugs has also decreased in size and complexity.
If you feel that that does not apply to you I can tell you that I currently work at a startup company developing a large scale internet application server and since employing CT the deployment process has changes dramatically.
Before CT deployment was a fearful process where we would all take a break from our jobs and publish the new code version to a single server. We would than monitor all the servers to see that nothing was crashing if it was we would revert the change and try to find the bug if not we would package the code and publish it to all the other servers.
We would then continue to monitor these servers and see that they do not crash once all the code has been published. All in all it was a lengthy nerve wrecking process that took hours of work for all people involved.
Now that we have switched to CT code deployment is essentially free, the code is already thoroughly tested and production ready at any stage in development. So pushing the changes to the servers is just a matter of uploading the new code, in fact after you use some good automated functional tests the need for active deployment can all but disappear and you can have your build server deploy the code to the servers on a daily basis (nightly-builds) and thus deploying a new version of your project just becomes the act of deciding which nightly build will be used as the “stable” version.

So now your boss is convinced, great! But still to come is your biggest challenge… your coworkers.

Developers

Getting your coworkers on board is harder than you might initially think. Instead of giving out main bullets like I did before I’ll try to address some the resistance points you might encounter.

Fix all my previous tests? I don’t think so

Some projects already have many tests integrated in the source code, however, since they are not maintained in an automatic incremental way ghost tests from previous builds can exist all over the code.
What happens when you have many old deprecated tests in your code which don’t pass? Once you start the automated tests all the old unmaintained tests will fail and no one wants to start spending all of his time getting some old deprecated tests to pass on top of all the stuff the developer has to do anyway.
Let’s dump the self righteous claim that you should not have failed tests in your project and deal with the actual problem.
You don’t need to get all tests to pass as a first stage of automated testing, instead only get the ones you are running anyway to pass and any new ones as well just comment out any old test you aren’t running on a regular basis.
Remember the goal is not to be as correct as possible but to get people used to the idea that tests are good. The old tests that aren’t passing will be fixed in time when they become important.

I don’t want some machine to notify everyone whenever I mess up

The big fear people have is usually that once some automated testing server is up every bad move you make is broadcasted to all other developers. First of all, it doesn’t have to be that way most automated testing servers allow notifications to be sent only to the people that committed any changes since the last check, and keep in mind that most times that will only be the one person that introduced the bug.
Secondly, once automated tests are up and running and a full test is conducted every few minutes failures will stop being some sort of blame notification like developers are used to and instead will become the development tool they are supposed to be.
A developer will commit his changes and then after a few minutes will get a notification if he introduced any errors so the developer no longer has a need to run a full system test every time he commits a change it will be run for him protecting him from costly mistakes later on.
And finally, since every commit will cause a full test to run you can expect failed builds (execution of tests) to be a thing of the norm and part of virtually any major code change. I can tell you that in my current project the build fails several times a day, it stops being an issue and no one gives it any other thought apart from the developers that introduced the changes.

I don’t want to spend the time to write tests to everything I do

You’re not really spending time think about it as an insurance policy. If someone changes the code in a way that causes your component not to work he will be notified in time. No more trying to find bugs in your code with the bosses breathing down your neck only to find it resulted from someone else’s change.
Anyway, not writing tests is just like sticking your head in the sand. Your code either works as it’s supposed to and in that case your tests will make sure it stays that way or it doesn’t in which case you’d better know about it sooner than later. The more time and code passes from the time of introducing a bug to the time the bug is recognized the harder it will be to find and fix said bug.

Conclusion

Continuous testing is a great tool for any developer in any project. And with so many easy out-of-the-box testing tools readily available it only takes a minimal amount of time and skill to bring up a continuous testing server. I tried to give you the “How-To” guide for people rather than software, the tools themselves aren’t really important but for the sake of being thorough I will add a small post about the testing architecture I use later on.

Please tell me what are some of the resistance points you’ve encountered and how you handled them…