Are you sure that they could?

Sure, you can decouple this to an extent. But, barring complete informational disconnection (read: airgap and no wireless communications) (and potentially not even then), this only reduces the attack surface, not removes it entirely.

What happens, for example, when your driver assist includes GPS data? Oh look... now you're downloading and decoding maps. Whoops! Attack vector.

What happens when your keyfob starts doing encrypted communication with the car (as other people in this thread are suggesting)? Whoops! Attack vector.

What happens when your entertainment console shows options to change the amount of time before the doors automatically lock? Whoops! Attack vector.

Cars are getting more complex - and it only takes one break in the defenses.

The infotainment system must be capable of taking address input, and that same computer would be internet connected for a variety of reasons. Taking over the steering as the above poster suggested would likely not be a desirable hack anyway. More likely someone would want to leave the low level control systems intact and just change the desired destination. A solid hack would actually wait until you get to your destination, and then once you send the vehicle to park itself, it instead just goes to the attacker's desired location. Might be a while before you even realized something was up!

It was noted that the Xbox One will accept voice commands from a video the console itself is playing. What happens when someone makes a hyper-targeted Pandora ad that uses your car's voice control function to enter a new destination address? If you are paying attention you will likely notice this, but many people have suggested that at some point you can sleep in your car and wake up at your destination, so even that isn't guaranteed.

No doubt direct control of steering and brakes will be highly locked down, but as you point out that in no way eliminates the possibility for mischief.