New WebM/H.264 Comparison

A new comparison of VP8 (the video codec in WebM), H.264 and XviD has been published by the Moscow State University's Graphics & Media Lab. The comparison is an addition to a comprehensive review of codecs. The team that performed the comparison makes it clear that it didn't do as extensive a test of VP8 as it did of the other codecs, since the study was apparently virtually completed by the time that Google announced that it was open-sourcing VP8 and including it in WebM. Nevertheless, the team came to a number of conclusions:

For movie encoding:

Comparing VP8 to XviD, VP8 is 5-25 times slower with 10-30% better quality (lower bitrate for the same quality). When comparing VP8 and x264 VP8 also shows 5-25 lower encoding speed with 20-30% lower quality at average. For example x264 High-Speed preset is faster and has higher quality than any of VP8 presets at average.

For HDTV:

Comparing VP8 to XviD, VP8 is 5-20 times slower with 10-20% better quality (lower bitrate for the same quality). When comapring VP8 and x264 VP8 shows 5-20 lower encoding speed with almost the same quality, excluding x624 High-Quality preset.

The source material that the team used for the Movie testing was clips from the films "Ice Age 3", "Raiders of the Lost Ark", "Enemy of the State" and "Up". For the HDTV test, the team used video shot in the Amazon taken from a Microsoft site, the trailer from "Iron Man 2", a close-up video of a calendar and a clip from the movie "Troy."

The WebM team responded to the Moscow State University results, admitting that a lot of work needs to be done to VP8 in order to improve its encoding speed. However, they contend that VP8 would have provided better-quality output had the source material not previously been encoded (only one sample, the calendar, was uncompressed.) Here are their comments:

We've been following the MSU tests since they began and respect the group's work. One issue we noticed in the test is that most input sequences were previously compressed using other codecs. These sequences have an inherent bias against VP8 in recompression tests. As pointed out by other developers, H.264 and MPEG-like encoders have slight advantages in reproducing some of their own typical artifacts, which helps their objective measurement numbers but not necessarily visual quality. This is reflected by relatively better results for VP8 on the only uncompressed input sequence, "mobile calendar.

Even with this limitation, VP8 delivered respectable results against other encoders, especially considering this is the first time VP8 has been included in the test and VP8 has not been specifically optimized for SSIM as some other codecs have.

To date, WebM developers have focused on the VP8 decoder performance and are only starting to optimize the encoder for speed. The WebM project has only been underway for three weeks, and we believe that our encoder speed will improve significantly in the near future.

The WebM team's comments about the source material have some merit, but in the real world, video that's previously been compressed is often included in projects. It may be difficult or impossible to get uncompressed source material--for example, AVCHD camcorders and some DSLRs output video using H.264. Therefore, the performance of VP8 on previously compressed video is important.

The Moscow State University comparison provides useful insight into the performance of VP8, and points out some of the areas that Google and its partners need to work on.