I think the reason is that people like to consider being able to move around environments that you don't have memorized without tripping on things. Do I have to step out of my virtual office to go get a beer from the fridge, or can I get the real-life obstacles that are in the way projected into my virtual office?

Imagine a virtual office with a thousand monitors that is your apartment that has been plopped down in the middle of a forest with the walls knocked out.