It depends. You aren't going to make a very fast modern codec encoder or decoder using Python. The hotspot ends up being the vast majority of the process. That management/glue layer becomes very thin, amounting only to feeding in the bitstream and reading back the raw video frames.