Well, to be honest I much prefer - and trust - fully automated test pipelines to manuality. It affords more frequent and thorough verification of the final artifacts.
The problem comes when the automation is only partial because added at a later stage, and the existing software does not lend itself to testing.
The risk there is that some short-sighted line manager will go “YOLO, I have a deadline” and skip the tests rather than flag the problem and dedicate budget and effort to redesign the existing for testability.
That’s where guidance and oversight are key, and more often than not, sorely lacking.
The other problem with the type of "in the field" testing that Microsoft does is that it can only capture issues that it can physically report back to the head office.
I had two separate computers with very distinct hardware that both failed to upgrade to any of the semi-annual Windows releases. They would just get stuck in a reboot loop at the SSD storage driver loading section of the installer, which means that no errors could be written. Hence, no specific errors could ever be sent back to Microsoft other than "failed".
I eventually found a workaround myself, but the root cause was never fixed, and will never be fixed, because Microsoft is blind to it.
Similarly, anything that causes a BSOD gets reported back via telemetry, but hangs and freezes aren't. Random performance issues, glitches, etc... are in the same bucket of permanently invisible issues.
If you're wondering why Windows is crashing less but glitching more... there's your answer.
Gah, glitches. Just spent two weeks trying to figure out the sudden seizures a Win10 running on a spanking new T14s. After 15 years of Apple Mac it’s insane, how can people put up with this horror.
I"m a huge proponent of automate the crap out of your tests, but you also have to break out of the automating mindset to get at the better problems. People are not machines, and automated tests are machines verifying machines.
It's why I like having a few non-automators and program noodlers around.
Well, there’s something to humans - testers in particular, who seem to be wired like hackers rather that programmers - that is superior to any fuzzing or “QuickCheck” approach. It’s the intuition, the art of second-guessing the mental model of the programmer that authored the code under test. That’s why it’s such a shameful tragedy to waste human capacity on scripted tests, the human factor should be authoring the automation.
The problem comes when the automation is only partial because added at a later stage, and the existing software does not lend itself to testing.
The risk there is that some short-sighted line manager will go “YOLO, I have a deadline” and skip the tests rather than flag the problem and dedicate budget and effort to redesign the existing for testability.
That’s where guidance and oversight are key, and more often than not, sorely lacking.