H.264#
H.264 also known as Advanced Video Coding (AVC) or AVC1 is a video compression standard for full HD Video and Audio.
The standard is defined in MPEG Part 10 (H.264) ISO/IEC 14496-10.
(H264 is patented but MPEG LA has allowed the free use of H.264 technologies for streaming Internet video that is free to end users.)
ITU H.264 is origin of H.264 and provides public access the AVC/H264 Codec https://www.itu.int/rec/T-REC-H.264-201304-S
https://en.wikipedia.org/wiki/Advanced_Video_Coding
https://mpeg.chiariglione.org/standards/mpeg-4/advanced-video-coding
https://doc-kurento.readthedocs.io/en/latest/knowledge/h264.html
https://developer.mozilla.org/en-US/docs/Web/Media/Formats/codecs_parameter#avc_profiles
Compression#
The Compression of H.264 uses an algorithm that uses previous or following frames to decode the stream. The System uses I-Frames/Key-Frames that don't need any previous Information to be decoded, to initialize a Stream.
- Intra Coded Picture (I-Frame)
- Predictive Coded Picture (P-Frame)
- Bidirectional Coded Picture (B-Frame)
An I-Frame gets send every x Frame to get an instantaneous decoding refresh (IDR). The Frame is needed to start the Stream and help to stabilize the video when packages get lost.
When the picture in a video footage change to much the frames can also be corrupted, this will cause the video to glitch and a I-Frame is needed to get back on track.
Many modern System have already a Hardware decoder for H264 integrated in the GPU and don't need to use the CPU, this improves the performance dramatically.
Profile and Level#
The profile and the added constraints indicate properties of the bitstream and video to allow a decoder and player to recognize the requirements for decoding and displaying it. The level indicates the bitrate and the required decoder performance for the profile.
The Codecs, Profile, Constraints and Level gets combined in on string following the Generic ISO BMFF syntax eg. "avc1.640028". Some Codes are much longer!
| codec | profile | constraints | level |
|---|---|---|---|
| avc1 | 0x64 | 0x00 | 0x28 |
| h264 | high | none | 4.0 |
| Profiles | code | Description |
|---|---|---|
| Base / Baseline | BP, 66, 0x42 | Primarily for low-cost applications that require additional data loss robustness, this profile is used in some videoconferencing and mobile applications. |
| Extended | XP, 88, 0x58 | Intended as the streaming video profile, this profile has relatively high compression capability and some extra tricks for robustness to data losses and server stream switching. |
| Main | MP, 77, 0x4D | This profile is used for standard-definition digital TV broadcasts that use the MPEG-4 format as defined in the DVB standard. |
| High | HiP, 100, 0x64 | The primary profile for broadcast and disc storage applications, particularly for high-definition television applications. This is the most commonly used profile. This Profile is also used in the default RPI Video Stream on /dev/video0 when H264 is selected as pixel format |
| More ... | eg. Hi444PP, 244, 0xF4 | Different Constraints set and also some special Profile properties. |
| Levels | code | Maximum video bit rate (kbits/s) |
|---|---|---|
| ... | ... | ... |
| 3 | 30, 0x1E | 10,000 |
| 3.1 | 31, 0x1F | 14,000 |
| 3.2 | 31, 0x20 | 20,000 |
| 4 | 40, 0x28 | 20,000 |
| 4.1 | 41, 0x29 | 50,000 |
| 4.2 | 42, 0x2A | 50,000 |
| ... | ... | ... |
https://doc-kurento.readthedocs.io/en/latest/knowledge/h264.html#profiles-and-levels
NAL Unit#
Network Abstraction Layer (NAL) is a Part of AVC video Coding standards for simple interfacing of the video stream with different network transport mechanisms. NAL Units are packets with data as bytes with a small header that defines the content.
NAL Units get differentiated in two types:
- Video Coding Layer (VCL) NAL Units contain data to a Video Picture
- Non-VCL NAL Units contain associated additional information
Each NAL Unit starts with a Access Unit Delimiter 0x00, 0x00, 0x00, 0x01. With this Delimiter a Buffer can be Split up in the single NAl Units.
The first Byte after the Delimiter holds the information, what type it is and what information it contains. By using an bitwise AND on the first Byte of the Nal, it can be easily detected if it is I-Frame, P-Frame, PPS or SPS.
The first 3 bit are not important to define what NALType the NAL Unit is and what it contains. In berryMSE we use nalType = nal_unit[0] & 0x1F to find out which information the NAL Unit Contains.
| nalType | Description |
|---|---|
| 1 | Non I Frame, P Slice, P Frame, NonIDR, VCL |
| 5 | I Frame, I Slice, Intra Frame, Key Frame, IDR, VCL |
| 7 | SPS, non-VCL |
| 8 | PPS, non-VCL |
https://doc-kurento.readthedocs.io/en/latest/knowledge/h264.html#nal-units-nalu
https://www.itu.int/rec/T-REC-H.264-201304-S Page 40/41/62/63 (59/60/84/85)
PPS and SPS#
The Sequence Parameter Set (SPS) and the Picture Parameter Set (PPS) contain information for the decoder. The SPS contains constraints for a series of Pictures and the PPS for one or more individual pictures of a sequence.
SPS contains in the first 3 Bytes (After NALType) the Codec info about Profile, Constraints and Level. This means it changes between different Cameras. It also contains Information about the macroblock Width and Height used for decoding. When changing the picture resolution this will change to fit.
PPS contains special codec infos for single Pictures in a Sequence. It most of the time doesn't change but will be sometimes be different, when changing the Bitrate.
E.g. Raspberry Pi 3B+ SPS/PPS for H.264 Main 4.0 1280x720
- Sequence Parameter Set
sps := []byte{
0x27, 0x64, 0x00, 0x28, 0xac, 0x2b, 0x40, 0x28,
0x02, 0xdd, 0x00, 0xf1, 0x22, 0x6a,
} - Picture Parameter Set
pps := []byte{
0x28, 0xee, 0x02, 0x5c, 0xb0, 0x00,
}
https://doc-kurento.readthedocs.io/en/latest/knowledge/h264.html#sps-pps
https://www.cardinalpeak.com/blog/the-h-264-sequence-parameter-set
https://www.itu.int/rec/T-REC-H.264-201304-S Page 40/41/45 (59/60/64)
H263 and H265#
H263 (MPEG4 Part 2) previous development and only focused on low bit rates for conferences with less motion in pictures. H264 reaches with its compression lower bitrate (almost 50%) and better video quality.
H265 (MPEG-H Part 2) latest development also known as High Efficiency Video Coding (HEVC). It got a much better compression, almost doubled.