On The Testing of Code

Birth of a Good Thing

Early in my career, I worked on small customer projects and consulting projects that did not have code testing mechanisms or processes in them. The development mode of work in those projects was simply delivery. Quality was something the Quality Assurance (QA) teams managed and Production Support dealt with. Developers coded and typically live tested projects on their journey to production. No additional testing processes were required.

Eventually, I transitioned to projects that had a greater number of factors to them than presentation or simple APIs. Dynamic views, flexible APIs, and integrated microservices added exponential layers of complexity to these projects. This meant that our interpretation of Quality Assurance (QA) had to grow according to the layers of dynamic elements we were working with.

The Testing Advantage

Code testing is the QA gap between developing code and releasing code. It has many forms ranging from targeted, sandboxed analysis to generic overview with many variations in between. Different languages and ecosystems support testing solutions differently, which also affects choices for each project. There is no singular solution, as optimal solutions for code testing will reflect the nature of the project. WYSIWYG style projects may only need generic overview whereas dynamic microservices benefit more from the introspection that targeted sandbox analysis provides.

Here are some of the common styles of code testing.

Live Testing

Definition

Live testing is the process of booting up a site or API live and using it directly.

Process

The project is fully connected to whatever dependencies or integrations it requires and running live. A live tester or testing harness operates the project from the designed controls. For a visual site, this would include URL navigation or UI controls. For an API, this would entail HTTP interactions on available routes.

Details

Live testing by a user is also a common part of the development process. We nickname that “happy path” testing sometimes as it engages the intended/happy processes, but doesn’t necessarily test unintended/unhappy processes.

Beyond

Beyond Live Testing, there are programmatic tools that will perform either browser or API interactions via code – comparing results to recorded expectations. These tools allow QA to analyze many pages or routes in a matter of seconds, rapidly speeding up this process.

Live Testing is the most basic and generic solution. Now, let’s discuss the opposite end of the spectrum.

Unit Testing

Definition

Unit testing is the process of isolating code to test it’s behavior in a targeted fashion. This is the most microscopic version of code testing as it includes:

1. Sandboxing: mocking (creating standardized fake version of) dependencies and integrations that the code has when it is run live
2. Isolation: targeting the smallest areas of code, typically a singular file, module, or even an individual function. The code is also isolated from other modules or functions we have written, meaning that code has to be included in the sandboxing process.

Process

The work of Sandboxing and Isolation for Unit Testing can be a daunting task, especially at first. The purpose of these elements is to allow isolated functionality to be tested as intentionally as one desires. This can be as simple as the “happy path” approach as we do with standard Live Testing or as intricate as performing multifaceted error testing and dynamic successes within the same suite.

The depth at which one Unit Tests code relates directly to the code’s complexity, flexibility, and importance on the project.

Details

The distinct advantage that Unit Testing has over other approaches is that Sandboxing and Isolation create consistently reproducible results. When future changes are implemented in the code, well written Unit Test suites will detect those changes based on written expectations.

Different language systems have various tooling for Unit Testing. The JavaScript and Python languages both have particularly strong communities that support Unit Testing and testing in general.

Beyond

As valuable as Unit Testing is, it only tests what we have isolated. Much of the behavior of our projects is dependent on the quality and stability of integrated code. That is something that Integration Tests address.

Integration Testing

Description

Integration Testing is the process of using a testing harness to run connected processes in a Sandboxed context.

Process

Similar tools that are used for Unit Testing can also be used for Integration Testing. External dependencies and integrations are Sandboxed, similar to Unit Testing, so that they neither effect, or are affected by the project’s code. However, Integration Tests widen the scope of the code being tested by no longer isolating project code, but rather running integrated sections of it. This can be scoped from something as targeted as two modules communicating with each other or as wide as an entire application process running end-to-end.

Details

The purpose of Integration Tests is to allow only the code we have developed in the project to be tested in its integrated state. This is why Integration Tests still utilize Sandboxing. We don’t want to add the complexity of external influences to alter the nature of how our own code relates within the project.

Beyond

The essential difference between widely scoped Integration Testing and using automated tooling for Live Testing is the Sandboxing element. Live Tests include the full integration of dependencies and integrations whereas Integration Tests are still Sandboxed.

One might think that’s all the options, but there still remains something that QA benefits from that helps analyze change versions. That is Regression Testing.

Regression Testing

Definition

Regression Testing is the process of using a test harness to run a suite of Live Tests over a site or application, testing them specifically against behavioral expectations for the purpose of establishing consistency.

Process

Regression Testing is un-Sandboxed Integration Testing, but with a specific purpose. Regression Testing suites are constructed with behavioral expectations for the purpose of establishing consistency.

While changing an application from one state to another, the development and testing team update expected changes in behavior in the Regression Testing suite. When ran, the Regression Tests are intended to identify other behaviors – particularly the unintended (unhappy) ones.

Details

Regression Testing is specifically intended for identifying these changes in state from iteration to iteration of a project. When constructed appropriately, they can affirm project behavioral state or alert the team of unintended changes. This is particularly valuable before releasing a collection of changes to production.

Beyond

Regression Testing is typically part of a development team’s pre-release process. This means that, if approved, the next step is the customer base in production. The only step beyond this is typically the observations and reports that come back to the team from production, something that Production Support typically handles.

Summary

Testing our code affirms its stability and validates its quality. The purpose of the testing processes described here is to reduce what Production Support has to handle by addressing developer errors, integration flukes, or unintended changes in project state as much as possible. They are not perfect. However, a development team that dedicates their efforts towards not only producing good code, but also using tools like these to affirm the integrity and stability of their code will reduce Production issues on their projects.

This benefits both business and customer concerns.