On a call today, our new vendor asked that we verify every web site works before having them apply service packs. Our analyst said, “We can do that.” I pointed out the problem causing the present concern happened one in ten times on one site on one server of the instance. Therefore to catch it, they would need 10 views of the login page for 30 servers for each of 18 sites. That is 5,400 page views.
The conundrum came up because when the service pack was applied to test, some sites on one server failed this check. Over time they cleared and returned. We have monitoring in place to check a single site on each server works with a login and logout. This check is super-sensitive to changes. Originally this check was on a functional evaluation site, but it broke every other week because someone changed a color, icon, etc. That was with 7. With 111, we would go mad.
Clearly, I am going to have to develop automated testing to verify sites on each of their servers before and after server pack application. Too bad the vendor does not make sure everything works after they make changes to our systems.