I know that we all come from different walks and strategy when it comes to testing configuration changes prior to production. I wanted to open up what I've found that works well in my environment as well as hear out your success stories or worst nightmare scenarios.

My test environment has 2 tiers.

Tier 1 - 4 different virtual workstations running in VMware Workstation. I power these 4 on and run patches, software deployments, and scripts through them to ensure that nothing is majorly broken. These 4 machines span 3 Operating systems and run multiple different business applications. If something breaks here it's too easy to revert to the preinstall snapshot and try again.


Tier 2 - A test group of 10 physical production workstations that are at the same physical location as my department. Provided nothing broke in Tier 1 this is where I go next. I have 10 end users across multiple departments with different requirements. Once I've verified that they have received their change and they've gone through their checklist of items they do on a typical basis (prioritized by business need), our changes usually go to production level.


A success story using this method is that our Office 2010 deployment was totally debugged using this method. Across the 2 operating systems we deployed to everything worked as intended and remained silent.


A disaster was a software deployment that went production and triggered an automatic reboot that wasn't caught in the virtual environment. The push to the test group happened late one business day and rather than report the reboot the end users called it a day. The reboot also effected some production servers and critical systems that were down somewhere between 10-15 minutes. It is fortunate that when they came back online nothing else was wrong.