Mm. On the Studio, my desktop switching is pretty quick - 17 frames from tapping the ctrl/arrow. I have an M4 Max which is actually 1 frame faster at 16 frames (though measurement-error probably accounts for that).
In either case that's less than 1/3 of a second. It's nowhere near what you're seeing. I'd be tempted to take the machine to an Apple Store and say "this is broken, please fix it".
What frame rate are you using on those monitors? I ask because this is faster at lower frame rates. This is surprising and I very much want what you have.
Are you using software like Rectangle, Better Touch Tool, etc?
In either case that's less than 1/3 of a second. It's nowhere near what you're seeing. I'd be tempted to take the machine to an Apple Store and say "this is broken, please fix it".