Feature Video Coding With H.264/AVC - Purdue University

Slice 0Slice 1Slice 2Figure 5. Partitioning of an image into several slices.sentation into a format suitable for specific transport layers or storage media. For circuit-switched transport likeH.320, H.324M or MPEG-2, the NAL delivers the codedvideo as an ordered stream of bytes containing startcodes such that these transport layers and the decodercan robustly and simply identify the structure of the bitstream. For packet switched networks like RTP/IP orTCP/IP, the NAL delivers the coded video in packets without these start codes.This paper gives an overview of the working, performance and hardware requirements of H.264/AVC. In Section2, the concept of standardized video coding schemes isintroduced. In Section 3, we describe the major tools ofH.264/AVC that achieve this progress in video coding performance. Video coder optimization is not part of thestandard. However, the successful use of the encoderrequires knowledge on encoder control that is presentedin Section 4. H.264/AVC may be used for different applications with very different constraints like computationalresources, error resilience and video resolution. Section 5describes the profiles and levels of H.264/AVC that allowfor the adaptation of the decoder complexity to differentapplications. In Section 6, we give comparisons betweenH.264/AVC and previous video coding standards in termsof coding efficiency as well as hardware complexity.H.264/AVC uses many international patents, and Section 7paraphrases the current licensing model for the commercial use of H.264/AVC.2. Concept of Standardized Video Coding SchemesStandardized video coding techniques like H.263,H.264/AVC, MPEG-1, 2, 4 are based on hybrid video coding. Figure 3 shows the generalized block diagram of sucha hybrid video encoder.The input image is divided into macroblocks. Eachmacroblock consists of the three components Y, Cr andCb. Y is the luminance component which represents thebrightness information. Cr and Cb represent the colorinformation. Due to the fact that the human eye system isless sensitive to the chrominance than to the luminance10IEEE CIRCUITS AND SYSTEMS MAGAZINEthe chrominance signals are both subsampled by a factorof 2 in horizontal and vertical direction. Therefore, a macroblock consists of one block of 16 by 16 picture elementsfor the luminance component and of two blocks of 8 by 8picture elements for the color components.These macroblocks are coded in Intra or Inter mode.In Inter mode, a macroblock is predicted using motioncompensation. For motion compensated prediction a displacement vector is estimated and transmitted for eachblock (motion data) that refers to the correspondingposition of its image signal in an already transmitted reference image stored in memory. In Intra mode, formerstandards set the prediction signal to zero such that theimage can be coded without reference to previously sentinformation. This is important to provide for errorresilience and for entry points into the bit streamsenabling random access. The prediction error, which isthe difference between the original and the predictedblock, is transformed, quantized and entropy coded. Inorder to reconstruct the same image on the decoder side,the quantized coefficients are inverse transformed andadded to the prediction signal. The result is the reconstructed macroblock that is also available at the decoderside. This macroblock is stored in a memory. Macroblocks are typically stored in raster scan order.With respect to this simple block diagram (Figure 3),H.264/AVC introduces the following changes:1. In order to reduce the block-artifacts an adaptivedeblocking filter is used in the prediction loop. Thedeblocked macroblock is stored in the memory andcan be used to predict future macroblocks.2. Whereas the memory contains one video frame inprevious standards, H.264/AVC allows storing multiple video frames in the memory.3. In H.264/AVC a prediction scheme is used also in Intramode that uses the image signal of already transmitted macroblocks of the same image in order to predict the block to code.4. The Discrete Cosine Transform (DCT) used in formerstandards is replaced by an integer transform.Figure 4 shows the generalized block diagram of thecorresponding decoder. The entropy decoder decodesthe quantized coefficients and the motion data, which isused for the motion compensated prediction. As in theencoder, a prediction signal is obtained by intra-frame ormotion compensated prediction, which is added to theinverse transformed coefficients. After deblocking filtering, the macroblock is completely decoded and stored inthe memory for further predictions.In H.264/AVC, the macroblocks are processed in socalled slices whereas a slice is usually a group of macroblocks processed in raster scan order (see Figure 5). Inspecial cases, which will be discussed in Section 3.6, theFIRST QUARTER 2004

M AB C D E F G HM A B C D E F G HM A B C D E F G HIIIJJKKJ Mean (A, B,K C, D, I, J, K, L)LLLMode 0: VerticalAMMode 1: HorizontalMode 2: DC: Neighboring samples that are already reconstructed at the encoder and at the decoder side: Samples to be predictedFigure 6. Three out of nine possible intra prediction modes for the intra prediction type INTRA 4 4.processing can differ from the raster scan order. Five different slice-types are supported which are I-, P-, B-, SI,and SP-slices. In an I-slice, all macroblocks are encoded inIntra mode. In a P-slice, all macroblocks are predictedusing a motion compensated prediction with one reference frame and in a B-slice with two reference frames. SIand SP-slices are specific slices that are used for an efficient switching between two different bitstreams. Theyare both discussed in Section 3.6.For the coding of interlaced video, H.264/AVC supports two different coding modes. The first one is calledframe mode. In the frame mode, the two fields of oneframe are coded together as if they were one single progressive frame. The second mode is called field mode. Inthis mode, the two fields of a frame are encoded separately. These two different coding modes can be selectedfor each image or even for each macroblock. If they areselected for each image, the coding is referred to as picture adaptive field/frame coding (P-AFF). Whereas MPEG-2allows for selecting the frame/field coding on a macroblock level H.264 allow for selecting this mode on a vertical macroblock pair level. This coding is referred to asmacroblock-adaptive frame/field coding (MB-AFF). Thechoice of the frame mode is efficient for regions that arenot moving. In non-moving regions there are strong statistical dependencies between adjacent lines even thoughthese lines belong to different fields. These dependenciescan be exploited in the frame mode. In the case of movingregions the statistical dependencies between adjacentlines are much smaller. It is more efficient to apply thefield mode and code the two fields separately.3. The H.264/AVC Coding SchemeIn this Section, we describe the tools that make H.264such a successful video coding scheme. We discuss Intracoding, motion compensated prediction, transform coding, entropy coding, the adaptive deblocking filter as wellas error robustness and network friendliness.FIRST QUARTER 20043.1 Intra PredictionIntra prediction means that the samples of a macroblockare predicted by using only information of already transmitted macroblocks of the same image. In H.264/AVC,two different types of intra prediction are possible forthe prediction of the luminance component Y.The first type is called INTRA 4 4 and the second oneINTRA 16 16. Using the INTRA 4 4 type, the macroblock, which is of the size 16 by 16 picture elements(16 16), is divided into sixteen 4 4 subblocks and a prediction for each 4 4 subblock of the luminance signal isapplied individually. For the prediction purpose, nine different prediction modes are supported. One mode is DCprediction mode, whereas all samples of the current 4 4subblock are predicted by the mean of all samples neighboring to the left and to the top of the current block andwhich have been already reconstructed at the encoderand at the decoder side (see Figure 6, Mode 2). In additionto DC-prediction mode, eight prediction modes each for aspecific prediction direction are supported. All possibledirections are shown in Figure 7. Mode 0 (vertical prediction) and Mode 1 (horizontal prediction) are shownexplicitly in Figure 6. For example, if the vertical prediction mode is applied all samples below sample A (see Figure 6) are predicted by sample A, all samples belowsample B are predicted by sample B and so on.Using the type INTRA 16 16, only one predictionmode is applied for the whole macroblock. Four differentprediction modes are supported for the typeINTRA 16 16: Vertical prediction, horizontal prediction,DC-prediction and plane-prediction. Hereby plane-prediction uses a linear function between the neighboring samples to the left and to the top in order to predict thecurrent samples. This mode works very well in areas of agently changing luminance. The mode of operation ofthese modes is the same as the one of the 4 4 predictionmodes. The only difference is that they are applied for thewhole macroblock instead of for a 4 4 subblock. The effiIEEE CIRCUITS AND SYSTEMS MAGAZINE11

ciency of these modes is high if the signal is very smoothwithin the macroblock.The intra prediction for the chrominance signals Cband Cr of a macroblock is similar to the INTRA 16 16type for the luminance signal because the chrominancesignals are very smooth in most cases. It is performedalways on 8 8 blocks using vertical prediction, horizontal prediction, DC-prediction or plane-prediction. All intraprediction modes are explained in detail in [1].3.2 Motion Compensated PredictionIn case of motion compensated prediction macroblocksare predicted from the image signal of already transmitted reference images. For this purpose, each macroblockcan be divided into smaller partitions. Partitions withluminance block sizes of 16 16, 16 8, 8 16, and 8 8samples are supported. In case of an 8 8 sub-macroblockin a P-slice, one additional syntax element specifies if thecorresponding 8 8 sub-macroblock is further dividedinto partitions with block sizes of 8 4, 4 8 or 4 4 [8].The partitions of a macroblock and a sub-macroblock areshown in Figure 8.8811664337550Figure 7. Possible prediction directions for INTRA 4 4 mode.16 x 1616 x 88 x 16In former standards as MPEG-4 or H.263, only blocks ofthe size 16 16 and 8 8 are supported. A displacementvector is estimated and transmitted for each block, refersto the corresponding position of its image signal in analready transmitted reference image. In former MPEGstandards this reference image is the most recent preceding image. In H.264/AVC it is possible to refer to several preceding images. For this purpose, an additionalpicture reference parameter has to be transmitted together with the motion vector. This technique is denoted asmotion-compensated prediction with multiple referenceframes [9]. Figure 9 illustrates the concept that is alsoextended to B-slices.The accuracy of displacement vectors is a quarter of apicture element (quarter-pel or 1/4-pel). Such displacement vectors with fractional-pel resolution may refer topositions in the reference image, which are spatiallylocated between the sampled positions of its image signal. In order to estimate and compensate fractional-peldisplacements, the image signal of the reference imagehas to be generated on sub-pel positions by interpolation.In H.264/AVC the luminance signal at half-pel positions isgenerated by applying a one-dimensional 6-tap FIR filter,which was designed to reduce aliasing components thatdeteriorate the interpolation and the motion compensated prediction [8]. By averaging the luminance signal atinteger- and half-pel positions the image signal at quarterpel positions is generated. The chrominance signal at allfractional-pel positions is obtained by averaging.In comparison to prior video-coding standards, theclassical concept of B-pictures is extended to a generalized B-slice concept in H.264/AVC. In the classical concept,B-pictures are pictures that are encoded using both pastand future pictures as references. The prediction isobtained by a linear combination of forward and backward prediction signals. In former standards, this linearcombination is just an averaging of the two prediction signals whereas H.264/AVC allows arbitrary weights. In thisgeneralized concept, the linear combination of predictionsignals is also made regardless of the temporal x44x8Figure 8. Partitioning ofa macroblock and a submacroblock for motioncompensated prediction.4x4Sub-MacroblockPartitions12IEEE CIRCUITS AND SYSTEMS MAGAZINEFIRST QUARTER 2004

For example, a linear combination of two forward-prediction signals may be used (see Figure 9). Furthermore,using H.264/AVC it is possible to use images containing Bslices as reference images for further predictions whichwas not possible in any former standard. Details on thisgeneralized B-slice concept, which is also known as multihypothesis motion-compensated prediction can be foundin [10], [11], [12].of the luminance component. Finally, blocks “16” and “17”comprise the DC coefficients and blocks “18”–“25” the ACcoefficients of the chrominance components.Compared to a DCT, all applied integer transforms haveonly integer numbers ranging from 2 to 2 in the transform matrix (see Figure 10). This allows computing thetransform and the inverse transform in 16-bit arithmeticusing only low complex shift, add, and subtract operations. In the case of a Hadamard transform, only add andsubtract operations are necessary. Furthermore, due tothe exclusive use of integer operations mismatches of theinverse transform are completely avoided which was notthe case in former standards and caused problems.All coefficients are quantized by a scalar quantizer.The quantization step size is chosen by a so called quantization parameter QP which supports 52 different quantization parameters. The step size doubles with eachincrement of 6 of QP. An increment of QP by 1 results inan increase of the required data rate of approximately12.5%. The transform is explained in detail in [15].3.3 Transform CodingSimilar to former standards transform coding is appliedin order to code the prediction error signal. The task ofthe transform is to reduce the spatial redundancy of theprediction error signal. For the purpose of transformcoding, all former standards such as MPEG-1 and MPEG2 applied a two dimensional Discrete Cosine Transform(DCT) [13] of the size 8 8. Instead of the DCT, differentinteger transforms are applied in H.264/ AVC. The size ofthese transforms is mainly 4 4, in special cases 2 2.This smaller block size of 4 4 instead of 8 8 enables theencoder to better adapt the prediction error coding tothe boundaries of moving objects, to match thetransform block size with the smallest blockImages to CodeAlready Decoded Images as Referencesize of the motion compensation, and to generally better adapt the transform to the local prediction error signal.dt 2Three different types of transforms are used.dt 1The first type is applied to all samples of all predt 3diction error blocks of the luminance componentY and also for all blocks of both chrominancecomponents Cb and Cr regardless of whethermotion compensated prediction or intra predicFigure 9. Motion-compensated prediction with multiple referencetion was used. The size of this transform is 4 4.images. In addition to the motion vector, also an image referenceIts transform matrix H1 is shown in Figure 10.parameter dt is transmitted.If the macroblock is predicted using the typeINTRA 16 16, the second transform, aHadamard transform with matrix H2 (see Figure10), is applied in addition to the first one. It1 1111 111transforms all 16 DC coefficients of the already1 1H1 2 1 –1 –2H2 1 1 –1 –1H3 transformed blocks of the luminance signal. The1 –1 –111 –1 –111 –1size of this transform is also 4 4.1 –22 –11 –11 –1The third transform is also a HadamardFigure 10. Matrices H1, H2 and H3 of the three different transformstransform but of size 2 2. It is used for theappliedin H.264/AVC.transform of the 4 DC coefficients of eachchrominance component. Its matrix H3 is shownin Figure 10.3.4 Entropy Coding SchemesThe transmission order of all coefficients is shown in H.264/AVC specifies two alternative methods of entropyFigure 11. If the macroblock is predicted using the intra coding: a low-complexity technique based on the usage ofprediction type INTRA 16 16 the block with the label context-adaptively switched sets of variable length“ 1” is transmitted first. This block contains the DC coef- codes, so-called CAVLC, and the computationally moreficients of all blocks of the luminance component. After- demanding algorithm of context-based adaptive binarywards all blocks labeled “0”–“25” are transmitted whereas arithmetic coding (CABAC). Both methods representblocks “0”–“15” comprise all AC coefficients of the blocks major improvements in terms of coding efficiency comFIRST QUARTER 2004IEEE CIRCUITS AND SYSTEMS MAGAZINE13

CAVLC is the baseline entropycodingmethod of H.264/AVC. Its1617basic coding tool consists of a sin–1gle VLC of structured Exp-Golombcodes, which by means of individu18 1922 230154ally customized mappings is(Only for20 2124 252376applied to all syntax elements16 x 16 INTRAexcept those related to quantizedCbCrMode)8912 13transform coefficients. For the lat10 11 14 15ter, a more sophisticated codingChrominance Signalsscheme is applied. As shown in theLuminance Signal Yexample of Figure 12, a given blockof transform coefficients is firstFigure 11. Transmission order of all coefficients of a macroblock [14].mapped on a 1-D array accordingto a predefined scanning pattern.pared to the techniques of statistical coding traditionally Typically, after quantization a block contains only a fewused in prior video coding standards. In those earlier significant, i.e., nonzero coefficients, where, in addition, amethods, specifically tailored but fixed variable length predominant occurrence of coefficient levels with magnicodes (VLCs) were used for each syntax element or sets tude equal to 1, so-called trailing 1’s (T1), is observed atof syntax elements whose representative probability dis- the end of the scan. Therefore, as a preamble, first thetributions were assumed to be closely matching. In any number of nonzero coefficients and the number of T1scase, it was implicitly assumed that the underlying statis- are transmitted using a combined codeword, where onetics are stationary, which however in practice is seldom out of four VLC tables are used based on the number ofthe case. Especially residual data in a motion-compensat- significant levels of neighboring blocks. Then, in the seced predictive coder shows a highly non-stationary statis- ond step, sign and level value of significant coefficientstical behavior, depending on the video content, the are encoded by scanning the list of coefficients in reverscoding conditions and the accuracy of the prediction order. By doing so, the VLC for coding each individualmodel. By incorporating context modeling in their level value is adapted on the base of the previouslyentropy coding framework, both methods of H.264/AVC encoded level by choosing among six VLC tables. Finally,offer a high degree of adaptation to the underlying the zero quantized coefficients are signaled by transmitsource, even though at a different complexity-compres- ting the total number of zeros before the last nonzerosion trade-off.level for each block, and additionally, for each significantlevel the correspondingrun, i.e., the number of4 x 4 Block of QuantizedArray of Scanned Quantizedconsecutive precedingTransform CoefficientsTransform Coefficientszeros. By monitoring themaximum possible number of zeros at each coding stage, a suitable VLCis chosen for the codingof each run value. A totalnumber of 32 differentVLCs are used in CAVLCCABACCAVLCentropy coding mode,PreambleNumber signif. coeff: 5Coded Block Flag: 1where, however, theTrailing 1’s (T1): 3Signif. Coeff. Flag: 1,1,0,1,0,1,1structure of some ofLast Coeff. Flag: 0,0,0,0,1these VLCs enables simReverseSign T1: –1,1,1ple on-line calculation ofPrecodedLevels: 2,1Magnitude of Levels: 1,1,1,2,1any code word withoutSyntaxTotal Zeros: 2Level Signs: –1,1,1,1,1recourse to the storage ofElementsRun Before: 0, 1, 1, (0)code tables. For typicalcoding conditions andFigure 12. Precoding a block of quantized transform coefficients.test material, bit rate14IEEE CIRCUITS AND SYSTEMS MAGAZINEFIRST QUARTER 2004

reductions of 2–7% are obtained by CAVLC relative to aconventional run-length scheme based on a single ExpGolomb code.For significantly improved coding efficiency, CABAC asthe alternative entropy coding mode of H.264/AVC is themethod of choice (Figure 13). As shown in Figure 13, theCABAC design is based on the key elements: binarization,context modeling, and binary arithmetic coding. Binarization enables efficient binary arithmetic coding via aunique mapping of non-binary syntax elements to asequence of bits, a so-called bin string. Each element ofthis bin string can either be processed in the regular coding mode or the bypass mode. The latter is chosen forselected bins such as for the sign information or lowersignificant bins, in order to speedup the whole e

Moving Picture Experts Group and the ITU-T Video Coding Experts Group, is the latest standard for video coding. The goals of this standardization effort were enhanced compression effi-ciency, network friendly video representation for interactive (video telephony) and non-interactive applications (broadcast, streaming, storage, video on demand).