Sites Inria

Version française

Project - Europe

Jean-Michel Prima - 1/12/2016

Bringing Automated Test Generation to DevOps

Benoit Baudry, head of DIVERSE team

A powerful software development methodology, DevOps has become a very popular tool in the US for it greatly helps web giants to deliver more frequent updates. True enough, Europe hasn't catch up yet. Coordinated by Inria, STAMP is a consortium whose aim is to provide a critical missing piece to this methodology. To wit: an automated test generation tool that would make DevOps less risky and incidentally more palatable to European companies. This projet has just been selected as part of the EU H2020 call for proposals ICT-10-2016 ‘Software Technologies’.

Release early, release often. Such is the mantra of Amazon, Twitter, Netflix and the like. These pionneers in the engineering of software applications that run in the cloud now routinely perform hundreds of code updates per day in what has become a thrust of continuous delivery around the clock. This stunning agility turns out to be a decisive competitive edge as it dramatically cuts time-to-market and hikes revenue. Behind the feat lies DevOps: a powerful development methodology that brings high degrees of automation at all steps of construction and deployment.

Whereas DevOps has gained huge traction in the US, the concern is raised that the more conservative European enterprises may be “missing the train ”. For instance, a survey by Rackspace LCC has found that only 40% of UK organizations have adopted this methodology, compared to 66% of their American counterparts. This disinclination is thought to reflect a different cultural attitude toward risk. Indeed, a hasty deployment may propagate a regression bug into production due to lack of sufficient testing. Having this risk very much in mind, developers often prefer adding new portions of code rather than venturing into the hazardous removal of old lines. A perfect recipe for accumulating obsolete chunks of code and thus increasing the so-called technical debt. Fear of breaking things is all the more justified as testing in DevOps mostly relies on manual effort for which, obviously, there is awfully little time given the short development cycles imposed by the continuous delivery.

Test Amplification

And that's where STAMP comes into play. Funded by the EU, this three-year project gathers three academic partners, as well as five software companies and an open source wiki platform. Leveraging advanced research in automatic test generation, “it aims at pushing automation in DevOps one step further through innovative methods of test amplification, says project coordinator and Inria scientist Benoît Baudry. It will reuse existing assets such as test cases, API descriptions and dependency models in order to generate more test cases and test configurations each time the application is updated.

STAMP will enhance automatic testing at three different stages in the DevOps process. First: unit testing. “Manually written test cases cover a very small portion of possible behaviors. STAMP will automatically analyze them in order to generate new ones and reduce the time necessary to detect regression bugs. ” Second: configuration testing. “STAMP will generate large quantities of configuration variants and resource conditions, and automatically deploy all of them to test the scalability of a system. ”  Third: runtime testing. “The feedback from operations to development provides rich data about the system’s behavior. But this is currently very costly to exploit. STAMP will automate the analysis of production logs in order to improve online testing and re-inject production-level test cases in the continuous testing process.

Purposely, these three test amplification techniques will be delivered as independent micro-services. “One will be able to assemble them in order to get continuous amplification, or use them separately and integrate them in different tool chains.

Keywords: STAMP Benoit Baudry INRIA Rennes - Bretagne Atlantique DIVERSE H2020 Europe