A recent post on LinkedIn had this image:
The problem is as old as development. The tester finds a bug but the developer can’t reproduce the bug so closes it as unreproducible.
… but how else can testing show development the bug? Should the developer come over to the testers desk and work on the testers systems while the tester waits?
The problem is that there is a difference in the developer’s environment and the tester’s environment. To solve this problem we have to track down why and then see if we can address it. The industry has been addressing many of these why’s. We are able to share exact machine copies with things like virtual machines where a whole machine can be exported as an OVA file (open virtualization archive). We made sharing applications and builds even faster and more efficient with things like Docker. We’ve made build configurations repeatable and automated with tools like Vagrant, Chef, Puppet, Ansible etc.
Data is the problem
What we haven’t done is make sure the data sets in the developer and QA environments are the same. Insuring the same data has historically been deemed too costly, too slow and in some cases simply not possible. It’s not possible, or so it seems, to be able to guarantee that data sets are the same especially when these data sets are multi terabytes in size especially if you are talking about DevOps and/or continuous delivery where we might want to refresh the testing data set multiple times a day after each destructive test and those tests might need to be run multiple times a day if doing continuous delivery.
Anti-Pattern
What companies have done instead of sharing full exact copies of production data in development and testing is to instead use subsets of data or small copies of data that are easier to provision and that are suppose to be representative of production data. The number of bugs found at each stage, development, testings, UAT and production varied. It generally looks like this (these numbers are taken from an actual customer – large financial institution base out of New York)
The problem is that each of these stages (dev, test, uat, prod) lasts a different about of time. Development and testing tend to last several months and the user acceptance testing on a full size copy of production is done near the end of the development cycle like:
Where UAT testing on a full copy of production usually happens near the end of the development cycle. Now if we look at the number of bugs found we see that most bugs are found in the UAT testing but often UAT testing is only scheduled for a couple of weeks before release but so many bugs are found that it requires either extending the release date or releasing code with bugs.
Another problem is that the longer it takes to address a bug the more expensive it is to fix the bugs. Not only is it more expensive but the expense goes up exponentially. This was pointed out in Barry Boehm in his book Software Engineering Economics back in 1981:
Here is a recent paper by Dynatrace showing the exponentially increasing cost of fixing bugs:
Chart by Dynatrace from their paper “Goodbye war room, hello DevOps 2.0“
It makes sense that the longer one waits to fix a bug the more expensive it is. I wonder if you’ve ever been in the position, or can imagine, when writing some code, if the code is run immediately and it returns an error, you know what we were writing, why you were writing it and what lines you had just added, so it’s relatively easy to identify, understand and fix the bug. Now on the other hand imagine if a tester reports the problem 2 months from now. Now you will have to remember why you were writing that code, what that code did, how that code worked and how it could have a bug. It takes longer. It’s harder. Now of course we all know how expensive that bug can be it hits production and not just the developer has to get involved but a whole teams of people have to get involved to track it down and worse yet if there is some business data corruption or loss.
Solution
The solution is easy. The solution is Data as a Service (DaaS) where data is treated like code and can be branched, bookmarked, shared, versioned etc. DaaS has a foundation on virtual data. Virtual data is data that shares duplicate blocks with other copies. The shared data is never modified. Modified data is written to a new location, avoiding over writing the existing data, and the new data is only visible to the virtual copy the change was made on. With DaaS a tester can find a bug and not only have the exact code version but also can bookmark the exact data state to reproduce the bug.
Applications can involve several layers such as the application stack, the database, files system configuration files etc. Complex applications can involve an application stack on top of more than one data source. Getting a copy of all of these at the same point in time and then sharing that data can be almost impossible unless you have DaaS. With DaaS, one can track multiple different layers and bookmark them all at the same point in time. For example a developer or tester can be given a “container” that contains the application stack, the data files and the database binary distribution. A tester could bookmark the container and pass that bookmark to a developer giving the developer not only the correct version of code but also the correct version of the database and data to reproduce the bug.
With DaaS we’ve seen bugs being discovered earlier by development and the number of late stage bugs is decreased.
Find bugs early not only keeps bugs out of production but also makes it cheaper to fix the bugs as the bugs are fixed earlier where it’s cheaper
By investing in DaaS we can shift left, reducing costs, shortening timelines, improving quality and pulling in more revenue earlier.
You may say “that’s great and fine for you but I don’t have DaaS technology at work and even if I did, it sounds complicated”
Well today is your day as Delphix supplies DaaS simply with point and click of a mouse or a CLI or restful web services and there is even a free version of Delphix called “Delphix Express” that comes with a full demo lab that you can download and play with on your laptop, Mac, Windows, Linux etc:
