I run my development stack on a few VMs and honestly even with Vagrant or others solutions the overhead/resource usage is noticeable and it doesn't feel that nice. It's fine for dev stuff though.
I imagine you're suggesting something like X over SSH, right? Do you use something similar on a daily basis? I would probably commit sudoku if I had to use the entire VM (not headless).
seems like the amount of malware able to escape through an up-to-date firefox, running with ublock/umatrix, running in an up-to-date linux distribution, running inside an up-to-date virtualbox should be very, very small: in the end of course nothing is fully secure, but IMHO unless somebody has to fear state-level actors firefox + ublock/umatrix + virtualbox is more than good enough
Of course this does not mean doing all of the browsing within the same vm, multiple vms for different things (banking, development, internet radio with flash, ...) after all RAM at least on the desktop is still fairly cheaply available, and on laptops one can still run a few browsing vms in 16gb if the need arises.
I compartmentalize stuff in VirtualBox VMs. I never share clipboard, and only install guest additions when I want a large display or shared folder. There is substantial disk space and RAM overhead, but both are inexpensive. And CPU overhead is small.