I remember in the 90s when Microsoft, Apple and RealNetworks pushed hard for their own codecs and tools. IMHO those where dark times for digital video. ffmpeg and Xiph.org played a large part in changing that.
... which is really disappointing when you learn that H.261/MPEG1, a broadcast-ready standard, was already frozen at that time. (Of course, even more worry-free formats are already here - it is just the time needed to replace existing equipment).
You can blame the lack of uptake of MPEG-2 in the personal computing space on the license terms set by the MPEG LA, which charged patent licensees an amount based on the number of encoders and/or decoders they wanted to ship [2], and because DCT codecs were too slow to decode on CPUs until the mid-1990s. There were competitive proprietary codecs that could be licensed more favorably and performed better. These got built into codec-agile platform media stacks like QuickTime, Video for Windows, and later, Real and Flash, and these stacks would gain support for newer codecs as the technology progressed.
It wasn't until 1999 with MPEG-4 that the licensing situation made MPEG codecs an option for desktops again, by which point other proprietary DCT codecs were widely deployed. Both Microsoft and Real dabbled with tech that came out MPEG-4, but it wasn't until that H.264 became decodable by mainstream computers that everyone finally settled down around an MPEG codec. Then, Google bought On2. (But that's a long story for a different thread [1].)
Here's some more detail for this thread:
H.261 pre-dates MPEG-1 by a bit. H.261 was compiled by the ITU-T chiefly to support video telephony, because the predecessor codec H.120 was conceptually neat, but practically garbage. H.261's design choices and JPEG's design choices factored heavily into MPEG-1 Video, and while the MPEG-1 standard also delivered a program stream and 3 techniques of audio compression, it was hardly broadcast-ready: it was missing a transport stream for unreliable media, and it was missing support for interlaced video. These deficiencies were rectified by MPEG-2 in 1996.
In the early 1990s, DCT-based video like MPEG-1 was too slow on contemporary mainstream desktop hardware, so people developed vector quantization codecs that would allow realtime playback: Cinepak, Indeo, MS Video 1, Smacker, and TrueMotion S. Some of these saw a lot of use in video games, and some were picked up by OS and platform-based stacks for multimedia, like QuickTime, Video for Windows, and then later, Real and Flash. Those platform stacks were designed with codec agility so that more advanced codecs could be switched in as developments came along. But correspondingly, good candidate codec needed to have a favorable IP situation for widespread deployment, so codecs were often homegrown from skimming contemporary sources (including standards), or licensed from smaller companies instead of the alternative: paying the MPEG LA for a license per decoder deployment, and then still having to procure decoder software.
Later, processors became more powerful and you could decode DCT in real time. In 1996, H.263 came out as an improvement over MPEG-2/H.262 at low bitrates. Other codecs followed suit and we ended up with RealVideo, Sorenson Spark in Flash Video, and VP3. The importance of OS-delivered or runtime-delivered media stacks rose; Real, QuickTime, Video for Windows, and Flash. These platforms would go on to dictate the containers, the video codecs, and audio codecs that were supported, so they had a dominating influence on multimedia on personal computers for the next decade.
MPEG-4 came out in 1999, which brought both MPEG-4 Part 2 (-> SP, ASP, DivX, Xvid) and H.264/AVC. The former was a bit of an improvement over H.263, while the latter had great potential but mainstream processors were too slow once again.
Microsoft cribbed MPEG-4 Part 2 to make a version of a Windows Media Video codec, RealVideo did the same, and DivX and later Xvid rose to prominence in the early 2000s. The latter two gained notoriety for being used for DVD rips and filesharing; one could convincingly compress a 480p main title from a DVD in MPEG-4 Part 2 to ~700-1000 MiB, 700 MiB would fit on a CD.
Right around this time, Microsoft was showing off their latest WMV to prove you can have HD movies on a DVD. But content owners wanted bigger disks with better DRM, in part to make filesharing more of a hassle, and were leaning towards H.264 as the codec, so Microsoft got their latest VMV standardized as VC-1, and based their marketing on easier decoding, interlacing support, and a favorable patent situation. They got their VC-1 written into the HD-DVD and Blu-ray standards alongside H.264 and the aging MPEG-2. But it turned out there are patents on VC-1 after all and a patent pool was set up by the MPEG LA just the same; and then hardware finally caught up to decode H.264, so everyone settled on an MPEG codec at last. Briefly.
Everything you said is right on but a small addition: RealVideo, and Sorensen Video 1/Spark were based on h.263 drafts. Sorensen Video 3 was based on the h.264 draft.
It's interesting because h.263 (coming out of the ITU-T) was developed for video conferencing like h.261. Real and Sorensen adapted it to streaming/disc delivery. The MPEG-4 spec was the first time the VCEG and MPEG got together and put the peanut butter and chocolate together from the start. It's maybe obvious in hindsight but I think it was the first time the needs of video conferencing, streaming, and disc/"digital" delivery really overlapped in terms of devices and environment.
VideoCDs (MPEG1 encoded) were around long before most in the US were even thinking about video on a computer. 352x240 CIF resolution in the early '90s. Much more popular in Asia. The US had to suffer with VHS until DVDs gained popularity in the early '00s.
I wouldn't say we "suffered with VHS" since VHS actually has higher resolution (it records the full 480/576 vertical lines of broadcast TV, with horizontal resolution similar to VCD), VHS tapes could hold 2-8 hours (vs 74-80 minutes for VCD, not even enough for a standard "feature length" film), and you could record your own VHS tapes (which was the original selling point of home VCRs, not pre-recorded tapes; by the time CD recorders became cheap DVD was already released).
"In modern-day digital terminology, NTSC VHS is roughly equivalent to 333×480 [...] while PAL VHS offers the equivalent of about 335×576 [...] S-VHS improved the horizontal luminance resolution to 400 lines"
https://en.wikipedia.org/wiki/VHS
I'm well aware of these "benefits" you speak of, but it was still a crap format for video. If you were satisfied with VHS, then I'm happy for you. However, VHS was the absolute worst video format. It was the penultimate example of how end users did not compare about quality. Early YouTube also demonstrated this. Laser Discs required flipping over. VCDs required a second disc. First gen DVDs were DVD-10s which required flipping over (until they eventually got DVD-9s working).
Even with VCD being half the resolution of the SD image, it was still a much better image than VHS could ever do.
VHS has a similar horizontal resolution to VCD (VHS has 240 lines per picture height, or about 320 lines across the whole width, while VCD has 352 pixels across the whole width) but has double the vertical resolution (and since it's an interlaced format, that also means it has double the temporal resolution). Take some interlaced 60Hz sports footage and convert it to VCD, and if you can tell me with a straight face that the VCD looks better, then I'll tell you that you're blind.
Well, analogue HDTV and digital music formats were already experimented in '70s and '80s by Japanese [MUSE, Hi-Vision, CDs] and European [HD-MAC, NICAM, also CDs] researchers, so it is no surprise that similar research into digital realm are being performed [H.26*, MPEG-1 Audio Layer I to III and AAC (MPEG-2 Audio), and DVB and DAB were spearheaded by EBU and ETSI so no surprise there].
How were those "dark times" for video? At least for Real and Apple they were trying to solve then-unsolved delivery problems. Apple's video stack also covered desktop video editing/production.
Video is hard. It's a lot of data that needs to be read, processed, and displayed under tight time constraints and needs to be synchronized with associated audio playback.
This is all made more challenging if you're trying to do it on storage, bandwidth, memory, and processing constrained consumer computing hardware. Compression somewhat solved the storage and bandwidth problems but not processing. Better fidelity codecs needed a lot more processing power to decode.
In the early 90s you had MPEG-1 at the high end of the quality scale but was so processor intensive you needed decoder ASICs in consumer hardware just to play it back. Then you had codecs like Cinepak that were far less processor intensive but middling quality. Then you had much lower fidelity like Microsoft's Video1, Apple Video, and even Smacker which had very low decoding requirements but didn't look great.
Network delivery of any of those in consumer hardware was a pipe dream when 14.4K modems were still rare and 9.6k were common. The h.261 codec, which MPEG-1 was based on, had a minimum bitrate of 64kbps which was out of reach of pretty much everyone.
Besides the hardware decoding requirements of MPEG-1 it was entirely unsuited for editing. Both QuickTime and Video for Windows were meant for editing on consumer desktop machines. The codecs they supported were meant for editing and then delivery (on CD-ROM typically).
In the mid to late 90s processing power had advanced such that MPEG-1 and h.261/3 could be decoded real-time in software. RealVideo and Sorensen Video 1 were both based on drafts of the h.263 spec which included video conferencing over POTS connections in its design criteria.
Again I'm not seeing dark times for digital video. There were lots of codecs because they had different uses, limitations, and strengths. The h.26x codecs were designed for video conferencing and it was Real and to a lesser extent Apple that realized they were also useful for streaming over the internet. Both MPEG-1/2 were unsuited for streaming as they didn't support variable frame rates and handled low dial-up bitrates poorly at best. It wasn't until the MPEG-4 überspec that internet streaming, video conferencing, and disc-based delivery settled under a single specification.
While ffmpeg is an amazing project and widely used, it didn't really do anything to settle the proliferation of codecs and containers. It really was MPEG-4 that allowed for that to happen to the extent it's happened.
> Both MPEG-1/2 were unsuited for streaming as they didn't support variable frame rates and handled low dial-up bitrates poorly at best
Nonsense. Variable frame rate is nothing but a very minor bandwidth saver. MPEG-1/2 have a tremendous amount in common with MPEG-4 ASP video and can perform almost as well in most scenarios you can draw up. MPEG-1/2 were mostly hobbled by the use of old, basic encoding hardware and the constrained parameters commonly used (GOP sizes of 12-15) to be compatible with old, basic decoding hardware. Even today you can use ffmpeg to encode MPEG-1, MPEG-2 and MPEG-4 (ASP) video from the same inputs using the same parameters and see very close to equivalent quality at a given bitrate. FFMpeg unfortunately changes it's defaults like resolution, GOP size, etc., to what it thinks you likely want, so you must watch and control for that.
First of all, MPEG-4 Part 2 was an evolution of the h.263 spec. While all the h.26x have commonalities but they don't all have a linear relationship. Their commonalities are in high level concepts, not necessarily implementations.
As I said and thought I made clear, MPEG-1/2 were unsuitable for streaming on consumer hardware of the era. The lowest practical bitrates for either was far higher than common dial-up rates. Neither spec supported Sub-QCIF frame sizes or frame rates below 15fps.
Similarly audio was a problem as the specs wanted to use MPEG audio which likewise did not have super low bitrate modes. A software decoder could handle such things but the content was outside of a spec so you'd need the encoder and decoder to work hand in hand to work reliably.
No one decided to develop h.263 and MPEG-4 just for funsies because MPEG-1/2 handled everything with aplomb. Low bitrate/high latency conditions (dial-up, cellular) were not well served by MPEG-1/2.
> thought I made clear, MPEG-1/2 were unsuitable for streaming on consumer hardware of the era. The lowest practical bitrates for either was far higher than common dial-up rates. Neither spec supported Sub-QCIF frame sizes or frame rates below 15fps.
You did make it clear, you're just 100% incorrect. You can encode MPEG-1 video with any resolution and any bitrate you choose, just like MPEG-4 ASP. It is entirely within spec.
> MPEG-4 Part 2 was an evolution of the h.263 spec
Not really. MPEG-4 ASP has much more in common with MPEG-2. MPEG-4 ASP for example omitted the in-loop deblocking in h.263, which reappeared in AVC/H.264.
> audio was a problem as the specs wanted to use MPEG audio which likewise did not have super low bitrate modes.
This is entirely wrong again. The MPEG-2 standard had defined audio modes down to 8kbits and 16000Hz.
> No one decided to develop h.263 and MPEG-4 just for funsies
h.263 was developed for very low latency encoding to support real-time video conferencing better than MPEG-1/2.
MPEG-4 SP was an attempt to develop a patent royalty-free codec, which MPEG-1/2 were not (they are, now). MPEG-4 part-2 video does offer some small improvements in quality over MPEG-1/2 (qpel, 4mv), but nothing dramatic. That was saved for AVC, which is of course a significant improvement over MPEG-1/2 at low bitrates.
