It's not you. Computer Modern is based on Monotype Modern 8A, one of the typefaces available for Monotype's ‘4-line system’ of typesetting mathematics. It's a Scotch Roman design, a 19th-century fashion that represents the nadir of typographic taste.
I'm not sure if I'd call it a low point unless you're simply describing your personal taste; I think there's far from uniform concensus that Scotch romans (and other "modern" faces, like the didones) were a mistake. I mean, Georgia and Miller (both by Matthew Carter, who also did Verdana and Tahoma) are both revivals in spirit, used for body copy, and they're less than three decades old.
But I'm really glad someone mentioned the Monotype 4-line system! Two papers come to mind that might be worth sharing, which I thought were really enjoyable and well-done:
