Regression Testing Best Practices (4293963)

What type of testing does KACE recommend?

KACE recommends regression testing with a slow rollout. In this context, this type of testing includes testing amongst development/test environments, small groups of real-world test devices (including hardware), and then rolling out in phases.  

When does KACE recommend this testing be completed?

This type of testing should be considered in situations like Operating System Upgrades, Windows Feature Upgrades, Patching, Scripting, or any other task that could potentially alter the state of production systems.

Does this type of testing prevent complications?

When integrating large software changes within an environment filled with other software and device drivers/firmware, it is nearly impossible to completely prevent all complications. That being said, when making changes that could affect production systems, thoroughly testing every possible combination known within your environment can greatly increase the chances of success and significantly reduce the severity of the issues seen at launch.

What do we do when we run into a complication even after having completed the testing?

Gather all of the exact details to recreate the situation you are experiencing, as well as any and all associated logs and search for similar reports. Check the vendor/publisher's site or release notes for the upgrade/update/patch/software. Ensure that the issue is already known. If not, please report it to the vendor/publisher.

If the issue is related to the KACE agent or one where KACE should be notified, please gather the exact details, logs associated, and KACE Agent logs to contact technical support.

Quick overview of how to complete regression testing:

1. Identify and research the task and all appropriate release notes
2. Create the task at hand and set it to deploy to a group of virtual test machines
   1. Consider the following:
      1. Operating system
      2. Installed software
      3. Custom Programs
   2. Test, review, and attempt to place these virtual machines in real-world situations
3. Identify a group of real-world devices that can be used for testing
   1. If this is not possible, consider identifying certain production machines that are less critical
   2. Be sure that test devices are using the same types of software, device drivers and firmware as production systems
      1. In order to gauge the impact on a specific environment, the test group should mimic production in every way possible
4.  Review all possible sources to assure that additional issues were not reported while you were testing

NOTE: Prior to rollout, KACE recommends preparing and verifying every possible backup and recovery methods for each device, virtual or physical.

Quick overview of a slow rollout:

1. After regression testing, backup/recovery preparation, and downtime approval, begin a slow rollout
2. Choose a small group of production devices in all departments to target for round one
   1. It is suggested that servers and/or critical production devices are evaluated separately and later in the rollout
3. Verify and review these devices for some time
   1. The amount of time will vary depending on how frequently these devices are used and how representative they are of the entire group
4. Choose a second group which contains about half of the normal production devices remaining and deploy to this group
5. Verify and review these devices for some time
   1. The amount of time will vary depending on how frequently these devices are used and how representative they are of the entire group
6. Deploy to the remaining production devices (non-server/critical)
7. Identify a small group of servers/critical production systems to start with
   1. Possibly consider one or two from each task group
8. Deploy to round one of the servers/critical production systems
9. Verify and review these systems for some time
   1. The amount of time will vary depending on how frequently these systems are used and how representative they are of the entire group
10. Choose a second group which contains about half of the normal servers/critical systems remaining and deploy to this group
11. Verify and review these systems for some time
    1. The amount of time will vary depending on how frequently these systems are used and how representative they are of the entire group
12. Deploy to the remaining servers/critical systems