The Importance of Codecs & the MediaVerse way

by | Sep 16, 2021 | Content Distribution

Codecs are a key element when it comes to the creation and usage of media, one of the main challenges in MediaVerse.

It’s All About Codecs

We have all stumbled upon many different digital data extensions, such as .mp4, .mov, and many others. One might think: as long as I can play the video I want to watch, I don’t care about the codec that much. From a technical point of view each codec comes with different advantages and disadvantages. Users want to watch or listen to movies or songs as quickly as possible – but how? Let’s talk about compression.
We need to compress a huge quantity of data because we need lighter and faster content delivery. For instance, as a content creator you don’t want to wait hours to upload your audiovisual content. The figure impressively illustrates the power of compression: With the right codec, it is possible to deliver the content within much shorter time.

 

The Power of Compression: H.264 reduces download time by 99.6% compared with raw video

How does it work? We first need the compression to the right format. After compression we send it over the internet and once we have the file delivered we can start to decode and decompress on the device (with hardware and software locally). The encoding is useful to adapt the format of video or audio to different situations: For example, when we watch a movie on a mobile phone over the internet, there will be a trade-off  between quality and speed because we neither have a super strong internet connection nor a very strong CPU.

Glimpse of Codec Diversity

There is an enormous amount of different codecs for audiovisual media. The popular media platforms’ choices on which codecs they support have a huge impact on media providers and users. Let’s take a look at some reasoning by Netflix and YouTube.

Netflix

Netflix has switched streaming to encoding in H.264 High Profile and VP9 (profile 0), in response to increasing processor capabilities of decoding streams online. The fact that both compression formats have found broad decoder support from mobile hardware and Android OS respectively has motivated this change.

Netflix’s latest update prioritizes video quality for low bandwidth internet connections. Before the update, Netflix used AVCMain (H.264/AVC) profile for almost all videos and HEVC for videos in Ultra HD resolution. The two new compression formats that have been introduced are AVCHi – Mobile and VP9-Mobile.

YouTube

“YouTube encourages partners to upload videos that are as close to the original, high-quality source format as possible to increase the likelihood that your videos will play in higher quality (HQ)”, the video platform states on their website. YouTube re-encode all video to improve playback quality. Furthermore, it seems that YouTube adapts and converts all aspect ratios to 16:9 and 4:3. The format preferred is the original but users can also upload videos in MPEG2 or MPEG4. Indeed, many different codecs are supported by YouTube: MOV, MPEG4, MP4, AVI, WMV, MPEGPS, FLV, 3GPP, WebM, DNxHR, ProRes, CineForm, HEVC (h265). However, the best video format for YouTube, according to the platform, is the MP4 file type in combination with an H.264 codec, 15-20 Mbps, a standard aspect ratio of 16:9, and AAC-LC as audio codec.

Browser Compatibility

The MediaVerse framework will allow clients to upload, create, modify, and remove all kinds of media files. They will get access to the MediaVerse application framework through a browser, therefore we need to check all browser features like compatibility with audio or video codecs.

Video

Codec Year HDR Licence Browsers Pros and cons
AVC  (H.264) 2003 yes Commercial use requires a license    Works in
Chrome,
Edge,
Firefox,
Explorer,
Opera,
Safari
Pro
Very compatible Uncertain
Investigate license
HEVC  (H.265) 2013 yes Licensing requirements Works in
Edge,
Explorer,
Safari
Not in
Chrome,
Firefox,
Opera
Pro
Compress more than H.264
Contra
Not so compatible + license
Theora 2004 no Royalty-free, open standard Works in
Chrome,
Edge,
Firefox,
Opera
Not in
Explorer,
Safari
Pro
Comparable to H.264 but free
Contra
No Safari
VP8 2008 no Royalty-free, open spec Works in
Chrome,
Edge,
Firefox,
Explorer,
Opera
Not stable in
Safari
Pro
Very compatible Contra
Safari only uses vp8 over webRTC
VP9 2013 yes Royalty-free, open standard Works in
Chrome,
Edge,
Firefox,
Explorer,
Opera
Not in
Safari
Pro
Evolution of VP8,
Comparable to H.265 Contra
No Safari
AV1 2019 yes Royalty-free, open standard Works in
Chrome,
Edge,
Firefox,
Opera
Not in
Explorer
Not safe in
Safari?
Maybe in the future
Pro
Best compression
Contra
More CPU for encoding/decoding Uncertain
Safari
Comparison of available video codecs

Audio

Codec Licence Browsers Pros and cons
FLAC Fully open and free of any licensing requirements Works in
Chrome,
Edge,
Firefox,
Safari
Not in
Explorer,
Opera
Pro
Lossless
Contra
Big files
AAC For streaming or distributing AAC-encoded content: no license required; developers of codecs are required to obtain a patent license Works in
Chrome,
Edge,
Explorer,
Firefox,
Opera,
Safari
Pro
Very popular,
compress more than mp3
OPUS Fully open and free of any licensing requirements Works in
Chrome,
Edge,
Explorer,
Firefox,
Opera,
Safari
Pro
Low latency
Comparison of available audio codecs

The MediaVerse Approach

In the scope of MediaVerse the project partners are building a system that will allow us to use media files (images, video, audio, etc.) over the internet. We need to adapt not only software or architecture but data. Following requirements have to be met:

  • We must analyze which formats, type containers, and codecs fit better with everything. 
  • Working over the internet the best option needs to be flexible for playing media files. The device that plays media can be portable or not, for example, we need to adapt codecs when a phone is a device as opposed to if it is a laptop.
  • The hardware and software need to be compatible; we need to choose the correct codecs and try to include all software and hardware versions as much as possible.
  • Codecs with potential and future. We cannot choose deprecated or oldy codecs. We need to be sure that in the future the chosen codecs will still be valid.
  • We prioritize open-source and royalty-free codecs.

Based on the partners’ codec evaluation, a choice for video and audio codecs was made. For video content the technical MediaVerse partners chose VP9 as the default video codec because it is royalty-free, open-source, and is developed by Google. VP9 has good times for encodings (libvpx). For audio data Opus audio codec is selected because it has better performance over the internet (lower latency), is free, open, and fully compatible and addresses VR and 3D sounds. Additionally, Telegram and WhatsApp are using this audio codec for voice recordings functionality.

[/et_pb_text][/et_pb_column][/et_pb_row]