You are overestimating the jitter problem by a large amount. The first S/PDIF receiver IC I find via google (CS8416) claims to have an output jitter of typ. 200ps on the clock output, running from its internal PLL synced to an externally supplied S/PDIF input.