Swift: Adaptive Video Streaming With Layered Neural Codecs

SlowReaction2.5-foldBitrate Overhead75sec50sec(a) Most ABR algorithms (BOLA, Penseive) cannot upgrade thequality of a video segment once downloaded and are slow to reactto changing network conditions.75secFastReaction25secmate of the network capacity improves closer to the playbacktime, which is not possible in case of regular coding.Refilledhigh qualitysegmentsDiscardedlow quality segmentsBandwidthwaste(b) BOLA-FS does allow video quality to be upgraded by redownloading a higher quality segment. However, the previouslydownloaded segment is wasted.Figure 3: Limitations of today’s ABR algorithms becauseof regular coding: either slower reaction to network conditions or bandwidth wastage to achieve faster reaction time tohighest quality. The reaction latency includes time to noticethroughput increase as well as playing the buffered segments,and hence segment duration (5 sec here) plays a role. Penseiveaggressively controls video quality fluctuations to compensate for incorrect bandwidth prediction, and hence the suddenjump in quality compared to BOLA.coding. During the experiment, the throughput improves drastically at the 100 second mark. Two state-of-the-art streamingalgorithms, Pensieve [35] and BOLA [52], cannot upgradethe quality of a segment once the segment has been downloaded. This causes a slow reaction to adjust to the improvedthroughput. In Figure 3(b) however, BOLA-FS [51], a version of BOLA, does allow the higher quality segment to bere-downloaded when the network conditions improve. However, the previously downloaded lower quality segment isdiscarded, resulting in wasted bandwidth.2.2Layered CodingA more suitable coding method to address the above issues islayered coding, where a video segment is encoded into a baselayer (providing the lowest playback quality level) and multiple enhancement layers as shown in Figure 1. Clearly, layeredcoding gives much finer control on rate adaptation comparedto regular coding. For example, multiple enhancement layersfor the same segment can be fetched incrementally as the esti-USENIX AssociationFigure 4: Compression efficiency of traditional layered coding.We use H.265 [53] and its layered extension SHVC [12] toencode the videos (described in §6.1). The single layer bitratecurve is same for both, and the additional layers are for SHVC.As shown, SHVC requires 2.5 more bits for 4 layers ofSHVC compared to a single layer for the same quality.2.3Challenges of Adopting TraditionalLayered Coding in Video StreamingLayered coding has typically been developed and implemented as an extension to a regular coding technique. Published standards demonstrate this dependency: SHVC [12]has been developed as an extension of H.265 [53], similarly,older SVC [47] as an extension for H.264 [57]. Developinglayered coding as an extension on top of a regular codingintroduces multiple challenges in real-life deployments:1) Cross-layer compression overhead: The key to large compression benefits in current generation video coding standards(e.g., 2000 compression ratio for H.265 [53]) is interframe prediction – the consecutive frames are similar and soit is efficient to simply encode the difference between consecutive frames. However, using the inter-layer frame predictionacross enhancement layers of the current frame with respectto the previous frame makes video quality drift during decoding [29, 42, 61, 67]. To minimize or avoid the drift, mostof the layered coding methods do not use inter-frame prediction across layers and thus lose out on its compressionbenefits [11, 17, 31]. In effect, to achieve the same quality,layered coding (e.g., SHVC) requires significantly more bitscompared to its regular counterpart (e.g., H.265). In our study,we find that a 4-layer SHVC coding method needs 2.5 bitsper pixel compared to its regular coding counterpart, H.265(see Figure 4).2) High encoding and decoding latency: The computationalcomplexity of these algorithmic codecs mainly comes fromthe motion estimation process during inter-frame prediction [53, 57]. During the motion estimation, it is useful - foreach pixel - to encode its motion vector, i.e., where its relativelocation was in the previous frame. The motion vectors arecomputed for each frame by dividing the frame into thousands of blocks of pixels and searching a similar block in theprevious frames. In general, the codecs use a set of previous19th USENIX Symposium on Networked Systems Design and Implementation105

MS-SSIM 1.0Decoding Latency (ms)frames to search blocks in each frame making it computationally expensive. The process becomes even more complex incase of layered coding because each layer has to be decodedone after the other because of the dependency of a layer on theprevious one (to exploit the content redundancy) [11, 27, 30].This serial process of layered coding makes the latency tobe a function of number of layers, and therefore the latencyincreases progressively as we increase the number of layers.Figure 5 shows per-frame100SHVCdecoding latency of thex265state-of-the-art layered cod50ing (i.e., SHVC) of a 1-minvideo on a desktop with con0figuration described in §6.1.12345Number of LayersAs shown, it takes more than100ms to decode each frame Figure 5: Latency challengesfor 5 layers, an order of mag- of traditional layered coding.nitude increase in coding la- The decoder is run on a hightency compared to its reg- end Desktop (as described inular counterpart H.265 (an §5) using a single-threaded imx265 [7] implementation). plementation of SHVC [3].Despite several optimizations in the past, such range of latencies makes it infeasible to realize real-time decoding onheterogeneous platforms. Recent studies (e.g., Jigsaw [11])tackle this challenge by proposing a lightweight layered coding method (using GPU implementation), but the latency isstill a function of number of layers.Because of these challenges, traditional layered codingis not used in practice today. In this work, rather than approaching this problem with yet another extension, we seekto explore layered coding via a clean-slate, learning-basedapproach with a goal towards efficient layered compressionby embracing the opportunities of new hardware capabilities(e.g., GPUs and other data parallel accelerators).2.4Layered Coding using Neural CodecsVideo compression has recently experienced a paradigmshift in the computer vision community due to new advances in deep learning [32, 43, 60, 65]. The compression/decompression here is achieved using neural networksthat we refer to as neural video codecs.The basic idea is the use of an AutoEncoder (AE), a neuralnetwork architecture used to learn efficient encodings that haslong been used for dimentionality reduction purposes [20].The AE consists of an encoder and a decoder. The encoderconverts an input video to a code vector that has a lowerdimension than the input size, and the decoder reconstructs(perhaps with a small error) the original input video fromthe low-dimension code vector. The neural network weightparameters (Wi for encoder and Wi′ for decoder) are trainedby minimizing the reconstruction error, that is, minimizingthe difference between the input and the output of the decoder.The smaller the code, the larger the compression factor but106MS-SSIM 0.90MS-SSIM 0.94MS-SSIM 0.97MS-SSIM 0.99Original Imager0High information lossr1r2r3Low information lossFigure 6: Illustrating the residuals (r0 , . . . , r3 ) from an original frame to a series of compressed-then-decoded frames.MS-SSIM [56] is a perceptual measure of image quality. Ahighly compressed frame (lowest MS-SSIM) has more residual information (r0 ).higher the reconstruction error.Our insight in using Autonencoders is that a their loss function can be optimized to encode a video frame into uniquelayered codes by purely encoding residuals from previous layers, unlike the traditional layered coding where it is nontrivialto handcraft each layer to have unique information.33.1SwiftOverviewAutoencoders are already shown to provide similar or betterperformance relative to traditional codecs [32, 43, 60]. Recentwork such as Elf-vc [43] is also able to use Autoencodersto provide flexible-rate video coding to fit a target networkcapacity or achieve a target compression quality. However,current work does not provide a way to encode in the videoin incrementally decodable layers. To do this, we make use ofresiduals to form layered codes. A residual is the differencebetween the input to an encoder and output of the corresponding decoder. Residuals has been used in the past for taskssuch as recognition and compression to improve the application’s efficiency (e.g., classification accuracy or compressionefficiency) [19, 55, 60].Swift uses residuals for designing layered codecs for videostreaming. The idea is to employ a chain of Autoencoders ofidentical structure. Each Autoencoder in the chain encodes theresidual from the previous layer, with the very first Autoencoder in the chain (implementing the base layer) encodingthe input video frames. Figure 6 shows an example, wherethe residuals are shown from an original frame to a seriesof progressively compressed-then-decoded frames. The firstdecoded frame (marked with MS SSIM 0.9) has a relativelyhigh loss from the original frame. As a result, the residualr0 has more information. When this residual information isused for the next layer’s compression, the resulting decodedframe is closer to the original, and in-turn the residual hasless information, and so on.The above ‘iterative’ chaining implicitly represents a layered encoding mechanism. Each iteration (i.e., layer) pro-19th USENIX Symposium on Networked Systems Design and ImplementationUSENIX Association

3.2Iteration 1𝑟0 IEhE0𝑟0EWe first describe how the encoder and the decoder are trainedat the server side. Assume, I t is the image or video frameat time t, for t {0, 1, .}. The encoder (E ) takes each ofthese frames as input and generates a compact code vector(c) for each frame, i.e., ct E (I t ). This code for each frameis constructed by exploiting the redundancy across multipleprevious frames in the video. Therefore, the encoder takes aset of previous frames as reference in order to encode eachframe. The decoder (D ) reconstructs the frame Iˆt given ct ,i.e., Iˆt D (ct ). The optimization problem here is to train Eand D pairs so as to minimize the difference between Iˆt andI t . Since we add our layered coding as a generic extension toany neural codec without changing its internal logic, E andD can be assumed as blackboxes. An example of a neuralcodec is presented in Appendix A.Figure 7 shows the design of our layered encoder-decodernetwork on the server-side. Here, each iteration (or ration 2𝑟1 𝑟0 arizerEBinarizer𝑟L𝑟̂LhDLhD1𝑟̂1hD0 𝑟̂0𝐼 L𝐼 1 𝐼0Layered codes transmitted over networkFigure 7: Deep learning based layered coding: a) iterativeencoding (E ) and decoding (D ): in each iteration, E encodesa residual into a code (ci ) and the decoded output (from D ) isused to generate the residual for the next iteration.encodes a residual ri into a code ci , where residual ri is thedifference between the encoder input and decoder output inthe previous layers. For the very first iteration, the encoderdirectly encodes the the original video frame. Representingthis mathematically: ci E (ri ) and ri ri 1 r̂i 1 with r̂i D (ci ), for i 0, . . . , L, with the exception that for i 0 (baselayer), r0 I.At each iteration, the decoder can enhance the quality ofthe video frame with a plain arithmetic sum of the outputs ofall previous iterations along with the base layer output. Thekey here is that both E & D have separate hidden states (hE and hD ) that get updated iteratively, sharing information between iterations. In fact, this subset of weights shared acrossiterations, allows better reconstruction of residuals. The entropy (i.e., the information) is very high in the initial layers,but progressively decreases due to the presence of the hiddenconnections and thus the code size becomes progressivelysmaller. The training objective for these iterative encoderdecoder pairs is to minimize the L1 reconstruction loss for theresiduals:Lrec Layered Neural EncoderUSENIX AssociationIteration L𝑟L 𝑟L-1 𝑟̂L-1Binarizerduces a compressed version of the original video that we call‘code.’ These codes encode incremental information such thatwith more such codes decoded, the reconstruction becomesprogressively closer to the original. Swift essentially usesthis mechanism of residuals to create the layered codes. Suchiterative minimization of residual also acts as an implicit regularization to guide the reconstruction (at a given bandwidth),instead of closely-following classical compression methodsas in Elf-vc [43].Figure 7 shows the Autoencoder architecture (more detailsin §3.2) on the server side. The architecture jointly learnsboth the encoder and decoder in each layer. As before, theAutoencoder’s weight parameters are trained to minimizethe reconstruction error between the input and output of thedecoder. In this process, the encoder generates a compactcode in each layer which is a compressed version of the inputvideo frames. These codes are transmitted to the client, wherethey can be decoded for progressively better reconstructions.The decoder learnt at the server is then optimized further(§3.3) to be used at the client side. The client decoder initiallyreconstructs the base layer from the first code. Then, if morelayers/codes are downloaded from the server, the decoderreconstructs the residuals from the second layer onward, andcombines with the previous reconstruction(s) to generate theoutput video frame.Overall, Swift has three main components:1. A learning-based layered encoder-decoder pair in a singleneural network to create residual-based layered codes onthe server-side (§3.2).2. A separate learning-based decoder on the client side. Thisdecoder can decode any combination of layered codes ina single-shot for real-time decoding (§3.3).3. Extension of an ABR algorithm that can integrate thecodec into a complete end-to-end system (§4).1 L 1 D (ci ) ri 1L i 0All Autoencoders E & D in the chain share the same network and thus have identical input and output sizes. Theyproduce the same code sizes for all iterations. Swift relieson a separate entropy encoding stage (Figure 7) to create theresidual codes that allocate proportional number of bits tomatch the entropy in each iteration. The fixed length code vector from the output of the encoder E is binarized and passedthrough a traditional entropy encoder similar to CABAC [54].Note that the learned codec can work with a variety ofinput video resolutions, and hence we do not need to traina separate model for each video resolution. This is mainlybecause the Autoencoder here takes one or more video framesas input and extracts the features through convolutions (e.g.,Conv2D [41]). Each convolutional kernel (with a fixed sizeof k k pixels that is much smaller than the input resolution)is applied in a sliding window fashion on k k blocks ofpixels to reduce the dimensions and form the Autoencoder’s19th USENIX Symposium on Networked Systems Design and Implementation107

compact code vector. In our codec we use 4 downsamplingconvolution blocks to reduce the dimensions. This makesany input resolution (w h) to be downsampled to (w/16) (h/16) resolution times the Autoencoder’s bottleneck bits (b)after the encoding stage (see Appendix A). Therefore duringthe testing, the encoder’s output for a 352 288 resolutionwould be 22 18 b, while it is 80 45 b for 1280 720resolution. Similarly the codec scales with other resolutionsduring testing.3.3Layered Neural DecoderLrec EntropyDecoderEntropyDecoder00.The above iterative coding design already includes the decoder (Figure 7) that can reconstruct the video from the layered codes. In principle, the client can use the same decoderalready designed and learned on the server-side. However, theiterative method incurs decoding latency proportional to thenumber of iterations. This is because the residual codes arecreated separately in each iteration and the decoder cannotdecode a code (ci ) unless the previous iteration of encoderencodes ci 1 and the corresponding decoder decodes it toform the residual ri .This latency is acceptable for video servers/CDNs that encode the videos offline and store them ready for on-demandstreaming, but clients need to decode the video in real-time( 30 fps). To address this, we develop a separate design ofsingle-shot decoder to be used at the clients, that can take anycombination of the codes as input and decode the corresponding frames in one shot. See Figure 8.The codes availableCode NotHat the client are fusedAvailablePadand padded with zeros upzerosto a predetermined codelength (corresponding tossH 𝐼"LL levels) to account for𝑐1unavailable codes. Theyare then fed into a ‘multiH𝑐0headed’ decoder (H) asshown in the Figure 8.8: Single-shot decoderIn each head, the padded Figuress ): a variable sized decoder(Dversion of each code ciis separately processed that takes a subset of the codes tothrough individual neural reconstruct a video frame in onenetworks prior to combin- go.ing them into a common network. When higher layers areunavailable, the corresponding heads will have no effect onreconstruction, and when available, generate a desired residual mapping. Essentially, these multiple heads are lightweightand the common network (after combining the residual codesafter multiple-heads) follows the same decoder architectureD from §3.2, but within one model. The heads or the common decoder do not share any parameters, in contrast to theiterative decoder which shares hidden states across iterations.Note that for preparing residual codes on the server-side, we108still need the iterative E -D architecture as described in §3.2.To distinguish from the server-side decoder (D ), we denotethis client-side single-shot decoder as D ss .We train D ss by extending the objective function used atthe server-side. Specifically, in addition to the loss functionat the server-side decoder (D ) which reconstructs a residual,we add a loss function that corresponds to the actual imagereconstruction at client-side decoder (D ss ) using the availablecode layers. Using L1 loss function, both the objectives areas shown below.L 1 L 1 D (ci ) ri 1 D ss ( ik 0 ck ) I 1 L i 0 {z} {z}residual quality lossimage quality lossHere, the server-side decoder (D ) reconstructs the residualimage ri at each iteration based on the code ci , while theclient-side D ss reconstructs the original image I based on thesubset of the codes available at t

Figure 2: Evolution of neural and algorithmic video codecs showing compression efficiency plots across generations. . 2.1 Limitations of Today's Video Streaming Due to Regular Coding Today's video providers predominantly use source rate adap-tation (e.g., MPEG-DASH [50]) where video segments are .