It still catches me off guard sometimes. Just a personal opinion, but it's a bit offputting because it's distracting to hear only one half of the conversation. My brain is dragged unwillingly into filling in the other half. Also we tend to talk kinda obnoxiously loud on the phone. But I guess that applies equally well to all public phone conversations.
To clarify, I'm able to ignore in person conversations. Phone conversations are more distracting especially when they're loud. I'm unwillingly drawn into conversations I'd rather ignore.