What you are describing is what we did with a half remote team at least once a quarter. Fly everyone in, have collaborative meetings during the day, team bonding at night.
I do think one of the core issues with the current Covid situation is that companies which are now completely remote can't do the normal bonding things that remote companies used to do.
Given the ability to do so, I do wonder how much of the concerns around remote work could be resolved.
Yeah, though even simple efforts yield big dividends. We schedule about 20 minutes after our company all hands to 'zingle', which sometimes includes breakout rooms to just hangout with a dozen people in an unstructured way. We do weekly "pub trivia" and other social activities. I've gotten in the habit of booking social lunches with coworkers, where we just hop on a zoom and catch up over lunch.
I won't say it's a perfect substitute, but not doing those things would be a disaster.