A few years ago, an undetermined application issue caused 9000 laptops to boot loop… Boot, 30secs, BoD, boot again, repeat…

The laptops were all over the country and about 1000+ of them were remote – meaning they were not near a company location.

I was in charge of the physical response while in parallel the engineers worked through the night to figure out how to fix this remotely, if at all possible.

While calling in all favors from many IT groups (ramping up temp admin rights for IT people that don’t know how to spell P-C so they could image systems, etc), the engineering team with the help of three vendors figured out a very creative way to fix the issue.

They created a top domain GPO that could stop a certain process from running, thus allowing the workstation to boot fully. The concern was will the workstation have enough time to ‘checkin’ prior to boot looping. At 6am, it worked and was deployed as fast as it could. We had techs swarming all the major office building floors (there were over 30 locations) to help people through.

As the morning progressed, people were able to receive the GPO and start work normally. The issue left was the 500+ fully remote people (some decided to drive over 2-3+ hours to get to a company location). The could not establish VPN with the laptop rebooting.

Yet another very creative solution. 1. A tech would call the end user and give them the root admin password for the workstation. 2. Have them boot in Safe Mode w/networking (Windows 7 still). 3. Have the end user establish a LogMeIn session with the tech. 4. The tech would run a script to do what the GPO did. 5. Tech would force reboot the workstation and the end user was good to go.

We had a few systems where the hard drive was on it’s last leg and with all the boot looping finally crashed. So, there was some minor clean up after the fact.

Amazing efforts by all involved.