When software is being updated it receives a new version number that describes the magnitude of the changes. Major changes may rev the product from version 3.x to version 4.0, while smaller changes might rev only from 3.3 to 3.4. The problem creeps in when dealing with rewrites.

When Awesome Inc. decide that their AwesomeWare 3.5 software needs to be rewritten it is likely that the company will see this rewrite as version 4.0 of the software. It is assumed that new features will be introduced in addition to the "engineering need" for a new technology stack. Software rewrites are often costly and new features are promised as a means to soften the blow of the expense of the project. In reality, however, this is not version 4.0 of the software. A software rewrite means delivering version 1.0 of the sequel to the original product. This is where the problem occurs and becomes a hard pill to swallow.

What starts as a difference in the naming/branding of the product will quickly flourish into lengthy discussions about software quality and feature parity. When software is being updated it is understood that the new version will add some new features, refine some existing features, and solve a few bugs. Existing features should not be removed unless specifically requested, and old bugs should certainly not be reintroduced. When software is being updated this is very achievable with an appropriate development process. However, when software is being rewritten these premises rarely hold true.

Software rewrites involve revisiting fundamental assumptions about the way the product works. Every department wants the opportunity to revisit the decisions they made when the first version of the product was being developed. Database architects want to fix the flaws in their schema that have crept in over the years. Product managers want to fix workflows that don’t quite work correctly, and developers will be rewriting almost every line of code from scratch. This creates a monster project that is larger in scope than the first 1.0 of the software but with much loftier expectations.

All of this leads to the development of a new product that is supposed to do everything the old product did but better and cooler and in newer technology. In reality this version 1.0 software will have all of the blemishes and warts of a first version release. Bugs that haven’t been seen since the "early years" will start to manifest themselves again. Features that were hiding in dark corners of the original product will be forgotten or just not work the same way. The product itself will change and taken on a whole new tempo. All of this comes with the benefits of having fixed fundamental architectural flaws in all layers of the product as well as having adopted a new technology stack. However, those benefits are of most visibility to technical departments and that value is hard to translate to other areas of the business.

From a branding perspective this should be AwesomeWare version 4.0 not AwesomeWare 2 version 1.0. As a result it often retains the version 4.0 branding and all of the assumptions that go along with it, for stakeholders and users alike. Somewhere in that transition it is lost that this is a version 1.0 rookie that has only been around for a few months, not a six year veteran that has been battle hardened over the years. Bug counts will spike, features from the version 3.x product will be lost, and fingers will start pointing in all directions looking for who to blame.

The reality is that performing a software rewrite the same undertaking as writing new software, no matter how much intellectual property and prior lessons learned are included. New software is new software and that means new bugs, new (and differently imagined) features, and a whole new journey towards refining what is produced into a solid product.

This isn’t to say that software rewrites aren’t sometimes the right option. Technical debt accumulates as a result of decisions made to meet business needs. Technology stacks age as better platforms and frameworks are developed. Rewriting large parts of the software is often the only means to address these issues. However, the assumption with most software from a marketing perspective is that it is a stable platform from which all new versions will be additive and progressively more stable.

It is important that staff in charge of technical communications make the reality of the software known to all levels of the business and to address the differences in understanding of what software versioning means. When a rewrite is undertaken it is important that all parties understand the technical reality of the situation and what can be expected as a result. Leaning on technical staff to "make it do everything the old product did but better" is not a realistic expectation and will not lead to a successful project. Instead, communicate the ramifications of decisions, and communicate them often. There will always be a difference in understanding of what a new software version means, but going into it with full communication can help lessen the impact of that difference earlier in the lifecycle and hopefully avoid some big surprises later on.

“All swans are white- until you reach Australia and discover the black swans paddling serenely. For science built on induction, the counterexample is always the ruffian waiting to mug the innocent hypothesis as they pass by, which is why the scientific method now deliberately seeks him out, sending assumptions into the zone of maximum danger. The best experiments deduce an effect from the hypothesis and then isolate it in the very context where it may be disproved. This falsifiability is what makes a hypothesis different from a belief- and science distinct from the other towers of opinion.”

from “Chances are…adventures in probability”, Michael Kaplan & Ellen Kaplan (ISBN-13: 978-0143038344)

This paragraph started me thinking about whether or not the quality assurance and software testing practices I have observed on several recent software projects are being quite as effective as they could be. Software quality, not unlike project documentation, is too often treated as an unnecessary evil within most software projects; a purely expense item whose perceived value and importance noticeably depreciate across the lifetime of the project. Amongst the first items to be cut from the list of software deliverables, some development teams have a hard time getting management to agree to have adequate quality assurance cycles within the project plan at all. Of course once the project has launched and is in production, quality could not be any more at the fore-front of everyone’s mind. This is because it is not until the first time critical data is lost or a production line is halted that the real value of testing and quality assurance is seen. Software is visible and prototypes have buttons that can be clicked and controls that can be manipulated. Testing leaves only a void on the timeline and a hole in the budget; and when was the last time you saw a QA person giving a corporate demo? The problem here is that the value of quality assurance is not tangibly communicated until it is too late for any corrective action to be taken.

I talked to a friend of mine who has been a software quality architect now for several years and despite the lack of value for his role continues to enjoy his work and perform his job with the utmost commitment to excellence. I asked him about the difference in the projects where QA has been valued from the beginning of the project. His answer didn’t form into useful information for me until the last couple of days. My friend felt that if the QA engineers had a clear understanding of the project and its direction from the beginning of every release cycle that they could better create and craft appropriate test plans and be seen as adding value right from the beginning of the project. With a clear understanding of what the software was actually trying to do, his team was able to provide useful feedback to the developers even within the first couple of release cycles. This information was able to help shape the software designs and decisions made by the developers and became itself a tangible deliverable within the project. Such visibility of the contribution they were making became an integral part of the release planning. Instead of simply “filing bugs” from day one (which is little more than an inconvenience to a developer who “knows it isn’t working yet”), they were able to leverage their knowledge of the project’s direction to get away from the typical bug-tracking path and focus more on actually testing.

To paraphrase the contribution outlined above, the QA team had ceased to be a bug-filing machine and had migrated into validating assumptions and testing concepts; providing a tangible output for their work that influenced the course of the project.

Exhaustive testing certainly has its place and I for one am a very large proponent of code-coverage, standards adherence, and testing to a high-level of detail. However, when was the last time that a bug filed in the first release of a mid- to large- size software project had any relevance to the final version of the software that was actually shipped. Certainly there are some simple errors that can be fixed by a developer within certain software sprints, but it has been my experience that subjecting incomplete alpha-builds of software to rigorous testing simply results in proving that indeed the software is not yet ready for release. Rather than taking the typical software reaction of performing QA at the end of the software cycle, I propose the integration of QA at different levels throughout the software development process.

Software testers do just that; they “test” things. However, the role of a software tester has become very divorced from that of a software engineer to the great detriment of many projects. Quality assurance engineers are often left in the dark about project requirements and are often brought in towards the end of a project and asked to “bang away on it” for a while, filing bug reports of their findings. Ultimately, several of the “high-priority” bugs are patched within the software to make it limp less noticeably and the product is pushed into production. This is neither satisfying for the bewildered quality assurance engineer performing the banging, nor for the developer who uses the appearance of such bugs to account for the “problems with late changes in requirements.” There are many tasks that a quality engineer may perform, especially early on in the project, that do not involve blindly banging away on something.

This returns me to the paragraph above and thinking about how classical and quantum science theories are tested. Certainly the minds that attempted to locate flaws or incorrect assumptions in some of Einstein’s theories of quantum mechanics (Einstein himself being one of them) had an intimate knowledge of the system they were trying to break. Further, they focused upon certain key assumptions that were underpinnings to the general overall theory and tried to find flaws there. It is much easier to verify and validate a small assumption than to assault an entire framework in one go. This probably leads the mind towards thoughts of unit testing and test-driven development as a whole, however if I might I’d like to divert for a moment and categorize those practices as another form of exhaustive testing; albeit one more commonly linked with software developers. Rather, consider that the importance was not the unit size of the piece of software being tested; rather the knowledge of potential flaws that led Einstein (or Podolsky/Rosen) to create a particular thought experiment that attacked a very specific part of the theory. Here, the importance falls upon the knowledge of which parts of the framework to test and in which specific ways. It is in that single aspect I feel most software quality efforts fall short. If I were to attempt to disprove the general theory of relativity, I would stab wildly in the dark testing the most basic assumptions that had more than likely already been tested by the original developer of the theory many times during its creation. Yet this is precisely what happens in many software quality assurance efforts. As a software developer, I have to test and debug my code many times during the natural creation cycle of that code. I go through countless cycles of testing it in many known ways to ensure that it functions in the way that I expect. Despite this, I know these same tests are going to be repeated many times over by an under-informed QA team during a time in the project where time is most limited. The problem doesn’t stop there however, as many developers could cite areas within the system that they feel are potential weak points. This information simply never makes it to the QA team and so they are left up to their own instincts to “find” the bugs within the software.

Consider a different approach, especially early on in the project life-cycle, that involves quality assurance engineers testing simple premises. It is common within software projects for a development team to settle upon one or more patterns that are appropriate for their solution. These aren’t formal software patterns such as the oft-abused factory, more the employment of a particular data-binding technique or a decision about the structure of a data access layer. As soon as the overall system architecture has been constructed there are weak points that begin to identify themselves, areas where the developers aren’t entirely happy with the solution at hand but can’t think of another way to solve the problem within the time and materials budgets that have been set. These areas are perfect for the initial attention of the quality engineers. Just as developers must ramp up on certain technologies for a project, so too must quality engineers. I can’t count the number of times a development team becomes short-tempered with a quality assurance team because they don’t understand why a Foo control behaves in a particular way, conveniently forgetting the struggles they had with the Foo control when first learning it. Had the testing team been present throughout the learning phases of the project, they too would have had the chance to learn about the intricacies of the Foo control, probably learning its quirks from another perspective and providing additional value to the team as a whole by sharing their experience from the end-consumer side of the software.

This brings me to my point (to cut a short story long as might be said) about focusing on falsifiability. The most valuable testing of any system seeks out the worst ruffians (bugs) that are waiting to mug (crash) that system, exposing them early enough in the development life-cycle to have an actual impact on the direction of the project. These most vile flaws are often the most insidious and will appear to deliberately hide from an unsuspecting team until the worst possible moment. Developers are not looking for these flaws. They are focused upon the solution to the problem not on whether their solution will fail. They have knowledge of the potential existence of such flaws but fully intend to code around the problem in a manner robust enough to hold off any coming assault. However, quality engineers are paid exclusively to seek such problems and alert people to the coming calamity while there is still time to avoid it. In order to do this they must know where to look and must focus upon the most uncomfortable parts of the solution, the parts where assumptions fail and things go wrong. It is in this that better communication and more shared knowledge is needed and it is in this that developers have a duty to include their quality personnel and expose to them their deepest fears. Due to the overly oppressive nature of most software deadlines and the expectation that software is a finitely-provable art, most developers would rather tear off a limb than tell the world about every part of their proposed solution that might fail. However, it is in doing just such a thing that a higher-quality product is built and a more robust solution is delivered. After the software launches successfully and proves itself aside from other failing products, developer and tester alike will be held aloft in the minds of product managers (and in some cases users) for having the diligence and skills to deliver. It is in focusing upon the areas of maximum danger, by actively striving to falsify every assumption within the solution that a truly successful software project is born.