Automating large Regression Test Suites with Postman and Newman

Lately, our team has been streamlined the development of our larger systems by integrating our tests into an automated commit process. We have implemented a few test suites over the year, including unit tests, regression tests, and UI tests. Having tests are great, but as we use them during our development they slow us down getting new code committed to Git. Our Regression Test suite is created with Postman Collections and run after an app is built. Learn more about creating Postman Collections here. With our apps growing larger and larger, we’ve had to create larger and larger test Collections.

And with larger Collections comes great responsibility.

After working with large Collections and possibly more confusing environment variable files, I wanted something better to work with.

So I found a great NPM Module Newman-Run that will use a feed file to run any number of Collections paired with environments asynchronously. A feed is a JSON file with arrays that reference the collection and environment used for the Newman run. This feed file has made it so we can break out our huge Collections into several smaller, more manageable Collections. With this change to the collections, we can break them into smaller more succinct tests for sections of the app. The module uses a feed file for you to reference your Collection and environment variables files that you want run. A feed file looks like:

{
     "runs":[
         {
             "collection": "./reference/PostmanRegressionTesting/Initial-Scenarios.postman_collection.json",
             "environment": "./reference/PostmanRegressionTesting/Development.postman_environment.json"
         },
         {
             "collection": "./reference/PostmanRegressionTesting/Global-Tests.postman_collection.json",
             "environment": "./reference/PostmanRegressionTesting/Development.postman_environment.json"
         }
     ]
 }

When Newman-Run calls this file it makes ‘runs’ that are Collection and environment pairs that run your tests where you want them run. It uses Newman to run your tests, but it runs them asynchronously on your machine. Running these asynchronously means they will be run at the same time, completing them faster. We have seen them run faster by a factor of two on our machine, but the speed is based on how many you run at once in your feed file. Once you have your runs ready in the feed file, the Newman-Run command is very simple:

newman-run -f <FeedFile.json>

As they run you will get the output Newman normally creates. When a test fails it will show you, in length, what the problem was. So you can troubleshoot how to correct it.

This has been a great module for us to speed up our testing integrations and I hope it will help yours as well!