3.1

Structure of the LCEVC encoder

The encoding process to create an LCEVC bitstream is shown in Figure 1 and includes three major steps.

Figure 1 -

Structure of the LCEVC encoder

Base codec: The input sequence is fed into two consecutive downscalers and is processed according to the chosen scaling modes (i.e., any combination of 2-dimensional scaling, 1-dimensional scaling in the horizontal direction or no scaling). The output then invokes the base codec which produces a base bitstream according to its own specification.

Enhancement sub-layer 1. The reconstructed base picture is upscaled and then subtracted from the first-order downscaled input sequence to generate the layer 1 residuals. These form the starting point for the encoding process of the first enhancement sub-layer. A number of coding tools process the input and generate entropy encoded quantized transform coefficients.

Enhancement sub-layer 2. To create layer 2 residuals, coefficients from layer 1 are processed by an in-loop decoder to achieve the reconstructed picture. Since layer 1 might have a different resolution vs. the input sequence, the reconstructed picture is processed by an upscaler. Finally, the residuals are calculated by a subtraction of the input sequence and the upscaled reconstruction. Similar to layer 1, the samples are processed by a few coding tools. Additionally, a temporal prediction can be applied to achieve a better removal of redundant information.

3.2

Key technical features

LCEVC deploys a small number of very specialized coding tools that are well suited for the type of data it processes.

3.3

LCEVC-Enhanced SVT-AV1

When encoding in combination with LCEVC, the overall compression efficiency relies on the efficiency of both the base codec and LCEVC. Since LCEVC uses the base codec at a low resolution and relatively low QPs, the overall coding efficiency relies significantly on suitably calibrating the base codec at low-QP/low-resolution operating points. During the tests, we observed that the current SVT-AV1 encoder has not yet been optimally calibrated for low-QP/low-resolution operating points. The relative per-pixel coding efficiency of SVT-AV1 when used as a lower-resolution base layer codec for LCEVC is lower than that of SVT-AV1 when used natively at full resolution.

To illustrate the above issue, we compared the relative efficiency of SVT-AV1 (preset 11) vs. x264 (preset faster, which according to [1] had a comparable encode time of SVT-AV1 M11) at 1080p and 540p resolutions. At 1080p resolutions, SVT-AV1 outperforms x264 across the tested range, converging only at very high data rates. On the other hand, at 540p resolutions, SVT-AV1 outperforms x264 only at lower qualities, crossing over at around the Constant Rate Factor (CRF) value of 33 [18]. While the high-quality/low-resolution rate-distortion pairs are hardly selected by a convex-hull-optimized encoder, the LCEVC encoder requires that the base encoder operate at lower resolutions and at higher qualities (e.g., CRF values below 33), where SVT-AV1 does not even outperform x264. Therefore, the non-optimized SVT-AV1 base encoder penalizes LCEVC SVT-AV1 when compared to SVT-AV1 at full resolution.

Figure 2 –

VMAF rate-distortion curves for SVT-AV1 M11 vs. x264 faster at 1080p and 540p resolutions

Addressing this issue by improving SVT-AV1 at low-resolution/high-quality operating points improves the performance of LCEVC SVT-AV1 and should be targeted as a future optimization opportunity. Figure 3 below shows the potential impact on LCEVC SVT-AV1 of a patched version [19] with some experimental changes (marked as ‘SPIE_patch’) to preset 11 that optimize low-QP/low-resolution operating points.

Figure 3 –

Impact of the SPIE_patch on BD-rate-computations trade-offs for LCEVC- SVT-AV1 on M11

For the M11 preset, using the patched version generated a BD-rate improvement of over 3% while also reducing encoding time, resulting in a better BD-rate-computation trade-off. Similar improvements for presets below 11 should be possible but could not be completed in time for this paper. Therefore, we used the official release 1.1 of SVT-AV1 for all presets below 11, leaving further improvements of low-QP low-resolution operating points as a potential upside for future releases.

4.1

Selection and preparation of the test video clips

To perform our experiments, we used the open source El Fuente clip [20], which contains 14296 frames with a frame rate of 29.97 fps corresponding to 7 min 52 sec of premium VOD content (at the spatial video resolution of 1920x1080). The partitioning of the video into 140 shots and spatial down-sampling are described in [1].

4.2

Encoders and encoding configurations (native codecs and LCEVC-enhanced)

To give a broader view of LCEVC capabilities, in addition to SVT-AV1, two other open-source encoders representing production grade software implementations of AVC and HEVC video coding standards have been used, both in their native and LCEVC implementations. The tested encoders were the following: x264 (0.163.3060), x265 (3.5), SVT-AV1 (1.1). V-Nova LCEVC SDK 3.7 was used for LCEVC-enhanced encoding. FFMPEG v5.0 was used for all native and enhanced codecs.

4.3

Presets

A representative selection of commonly used presets was tested, excluding the fastest presets (i.e., x264 ultrafast, superfast), and the slowest preset (i.e., SVT-AV1 M0), as they are less relevant for real-life applications:

Note: “lcevc_preset” is similar to the preset configuration of other codecs (i.e., x264, x265, AV1), and provides six discrete combinations of encoding parameters to optimize speed and video quality trade-offs. The options are from 0 to 5, where 0 is the slowest while achieving the maximum compression efficiency, and 5 is the fastest. The following mapping between LCEVC (base codec) preset and lcevc_preset has been applied.

Table 1 -

List of lcevc_preset values corresponding to LCEVC base layer presets

4.4

Command lines

The following command lines have been used for the native encoders:

For the LCEVC-enhanced encoders, we aligned the configurations to those of native encoders, by setting LCEVC-equivalent commands, resulting in the following:

For all tests, only the first frame is encoded as an Intra (Key) frame and all other frames are encoded as Inter frames.

The LCEVC performance heavily relies on the quality of its lower resolution base layer, and it offers a number of scaling mode options for the resolution of the base layer with respect to full resolution. The LCEVC specification allows for the usage of different spatial relations between the base resolution and the full resolution. While LCEVC is typically used with quarter-resolution base layers for full resolutions at or above 720p, at SD and sub-SD resolutions, alternative scaling modes are advisable, such as half-resolution scaling or no scaling.

At particularly low resolutions, such as below 360p, LCEVC is recommended to be used in passthrough mode’ LCEVC commands for the low resolutions are the following:

4.5

Platforms used for the experiments

The encodings for this experiment are performed on AWS EC2 instances, specifically, c6i.32xlarge. All instances ran Ubuntu Server 20.04 Linux OS on a Third Generation Intel® Xeon® Platinum 8375C. Hyper-threading and Turbo frequency are both enabled on these instances allowing the instance to access 128 logical cores (vCPU), with a maximum all-core turbo speed of 3.5 GHz. Instances ran from 3000 IOPS, GP3 EBS mounted disks. Running the list of commands is done by invoking the parallel [22] Linux tool and passing to it the command lines above. The parallel tool then schedules the command lines by executing newer commands when older ones retire, while maintaining 128 command lines running at a time. Each set of encodings per preset per encoder is run independently while capturing the run time using the GNU time command as follows:

4.6

Metrics

In this analysis, we computed visual quality measurements using four objective metrics: Video Multimethod Assessment Fusion (VMAF), Video Multimethod Assessment Fusion No Enhancement Gain (VMAF_NEG), Peak-Signal-to-Noise-Ratio (Y-PSNR), and Structural Similarity Index (Y-SSIM) [23]. Although perceptual-based objective metrics like VMAF were demonstrated to be good predictors of visual quality, they can be influenced by means of pre- or postprocessing operations [24]. For this reason, with respect to [1], VMAF-NEG was added to the set of objective metrics.

4.7

Encoding results

Once all encodings are done, the resulting bitstreams are collected, decoded, and up-sampled to 1080p with the methodology described in [1]. Four performance metrics are then generated using the VMAF software (8b0782c6, based on v2.3.0). In order to select the RD points that would result in the best-possible trade-offs between quality, bit rate and resolution, all resulting elementary bitstream file sizes and quality metrics across all resolutions for each clip are passed to a C sample application [25] that determines the convex hull points. These points are chosen to allow the application to switch between encodings corresponding to different resolutions (based on the available bandwidth) while also maintaining the best possible video quality at a certain bit rate. With respect to the performed simulations, the input to this process is a set of 88 encoding results (8 resolutions * 11 CRF points) per video shot. The BD-rate [26] results for all encodings are generated by comparing the resulting convex hull points for each of the video clips to those generated using libaom AV1 at the slowest preset (i.e., -cpu-used=0, with encoding configurations in line with [1]), which represents the anchor encoder in this comparison.

After obtaining the convex hull for each “shot”, the multiple convex hulls are combined as described in [1] using the constant slope principle. This results in a single rate-distortion curve describing the coding performance over the entire combined sequence. Having two such combined rate-distortion curves, one for the anchor and another for a test encoder, allows the evaluation to be based on a single BD-rate figure that captures the performance over the entire ensemble.

Figure 5 shows the BD-rate vs complexity results of using the combined convex hull approach for VMAF, VMAF_NEG, PSNR, SSIM. Average of (VMAF, PSNR, SSIM), where average curves are calculated by averaging the BD-Rates of multiple metrics for each codec preset, has been added for consistency with [1]. However, in light of the considerations below regarding the reliability of using PSNR to compare single-layer schemes with multi-resolution schemes such as LCEVC-enhanced encoding, the average of (VMAF, PSNR, SSIM) will not be reported any further in the report.

Figure 5 -

AVC/HEVC/SVT-AV1 vs. LCEVC-enhanced BD-rate - Computations trade-offs across presets on El Fuente data set using the combined approach

While VMAF, VMAF_NEG and SSIM show an improved BD-rate, PSNR results are quite different from those of other objective metrics, with strong BD-rate penalties. It is important to note that objective metrics based on mean-squared error (MSE), such as PSNR, known to have some limitations in general ([27], [28] and [29]), are especially problematic when comparing conventional single-layer schemes with multi-resolution schemes such as LCEVC. Whilst PSNR may still be used to compare different LCEVC-enhanced codecs among themselves, when comparing a single-layer codec with LCEVC, the different error distribution profiles produce relevant structural disadvantages for the multi-resolution scheme in the range of scores between 35 and 45 dB ([30]). Therefore, BD-rates based on PSNR are in line with expectations and not representative of subjective visual quality. Subjective metrics (ITU-R BT.500 formal MOS performed during the course of MPEG standardization of LCEVC [31]) highlighted that comparisons of LCEVC vs. native coding performed with objective metrics (including VMAF) underestimated the formal MOS BD-rate benefits of LCEVC. In absence of formal subjective results, some objective results in this paper may thus underestimate the visual quality benefits provided by LCEVC.

6.1

Assessing compute complexity

The actual encodings that are part of the convex hull for a given sequence are only a subset of the total encodings generated, which means we are using a subset of the encodings for BD-rate calculations, while we use all encodings for computational complexity calculations, as we don’t know a priori which of the (resolution, QP) pairs would be suboptimal. The additional encodes that are not part of the convex hull can be considered as a compute tax to achieve optimality. Section 4 in [1] demonstrated an opportunity to greatly reduce computational complexity in the “restricted discrete” case, if we have a good way to generate estimates for the optimal (resolution, QP) pairs. In that case, the complexity of such an optimized bitrate ladder generation would be equal to the complexity of producing only those 8 (resolution, QP) encodings for each clip that correspond to the final 8 selected aggregate discrete points, plus the cost of the predictor of these encoding parameters.

As shown in [32, 33], one option is to use the same encoder in a faster setting to obtain the convex hull. As demonstrated in [1] for SVT-AV1, this introduces only a minor loss in coding efficiency with a very significant improvement in speed.

6.2

Fast encoding parameter selection

At the same target quality level, a faster and a slower encoder preset differ primarily in bitrate, where the bitrate difference is proportional to the content complexity, while distortion stays roughly the same. Thus, a constant multiplicative factor exists for the slopes in the RD domain between a faster and a slower encoder preset, and that enables a faster encoder to be used to encode the shots at multiple operating points and determine the optimal resolutions and target quality levels for each shot, which can then be used to encode the shot again with a slower preset.

The steps involved in fast encoding parameter selection approach are as follows:

In [1], based on SVT-AV1 version (0.8.8-rc), the fastest available speed setting (M8) was roughly at the level of x264 at the “slower” preset, meanwhile the most recent SVT-AV1 releases unlocked faster presets (M9 to M12), with M12 achieving a comparable speed of x264 “veryfast”. In the following experiments, we perform the “analysis” step to determine the optimal encoding parameter pairs starting with M8 (for consistency with [1]) on SVT-AV1 and on the base layer of LCEVC-enhanced SVT-AV1, and then apply them to other slower speed settings from M7 to M1 (slowest).

Figure 7 shows the results of using the M8 preset in the analysis stage on the BD-rate vs complexity trade-offs of the rest of the SVT-AV1 presets. The total encoding time shown on the x-axis is the sum of the encoder parameter estimation cost, plus the cost of the selected X encodings per shot.

Figure 7 –

Fast Encoding approach: BD-rate vs complexity trade-offs when M8 is used for the prediction

To better understand the effect of choosing various presets during the analysis phase, we also repeated the above experiment several times, each time choosing a different preset. In particular, we predicted the encoding parameters (CRF and resolution) of a target preset, M_x, using a reference preset, M_y. We denote the result as M_y | M_x. As the reference preset needs to be faster than the predicted preset, we require that y > x. We then computed the convex hull, constructed after considering all possible combinations M_y | M_x. Figure 7 shows the corresponding convex hull for SVT-AV1 and LCEVC SVT-AV1, as well as the optimal combinations M_y | M_x (from M3 to M11). Please note that, for some presets, the optimal combinations are not part of the convex hull, in which case we took the closest points to it.

Figure 8 -

Fast Encoding approach: BD-rate vs complexity trade-offs, with optimal M_y | M_x preset combinations

Figure 9 shows the comparisons between the discrete DO calculated in Sec. 5 and fast encoding DO for LCEVC SVT-AV1 and SVT-AV1 with M8 and M11 used for prediction. The fast encoding approach, consistently between the two tested codecs, significantly reduces the total encoding time needed while maintaining a limited BD-rate loss.

Figure 9 –

Comparison of fast encoding approach vs discrete approach with M8 and M11 preset for prediction

In Table 2, we compared the fast encoding DO with optimal preset for prediction (as per Figure 7) against the discrete DO for each preset and computed the percentage saving of cycles and the corresponding BD-rate loss. The benefits of fast encoding are confirmed for LCEVC, it enables both AV1 and LCEVC AV1 to achieve a 75-85% saving in encoding time for the five slowest presets, with a limited BD-rate loss, between 1%-2.4% for both codecs.

Table 2 -

Average BD-rate loss (VMAF_NEG) and cycle usage vs discrete convex hull

7.1

Why efficient AV1 decoding in software is important

Increasingly efficient software (SW) and hardware (HW) AV1 decoders are becoming available for playback on mobile devices. However, despite increasing adoption of state-of-the-art SW decoder by streaming operators and the commitment of chipset manufacturers to add AV1 capabilities in their SoC, as of early 2022 there is only a very limited set of devices with hardware accelerated AV1 decoding. It could be many years until the majority of global consumers can decode high-quality HD AV1 content on their mobile devices. Even when this is possible, battery power consumption for SW AV1 decoders on mobile devices is often unsustainable, draining their battery faster than hardware decoding.

For this reason, in addition to the benefits in the encoding quality-computational cycles trade-off, LCEVC’s decoding performance was assessed to determine its capabilities to extend the device reach of AV1 and to improve battery life, with specific focus on Android mobile devices. Due to the LCEVC hierarchical structure, the LCEVC-enabled decoder decodes the AV1 base layer at a quarter resolution (e.g., 540p for a full resolution HD stream) and leverages existing hardware blocks in the device, such as GPU shaders and scalers, to efficiently decode the LCEVC enhancement.

7.2

Decoder performance testing

Two approaches have been used to compare decoder performance of AV1 and LCEVC with AV1.

Method one measured CPU decoding time as a proxy for overall decode complexity, using an approach similar to that of the encoder performance evaluation. It should be noted that for LCEVC this test used CPU-only decoding libraries that – since they are not used in real-world scenarios – have not undergone significant code-optimization.

Method two measured actual playback capability and overall power usage on real-world mobile devices when they playback video content. For this test, AV1 decoding was carried out using the latest available software decoders (both Dav1d and GAV1) and LCEVC AV1 decoding used optimized CPU-GPU libraries, making results more indicative for real-world use cases. For this test, we used a video playback application capable of decoding and playing back both AV1 and LCEVC-enhanced AV1 content, based on ExoPlayer [34] with added support for the latest Dav1d (version 1.0) and GAV1 software decoders as provided by Android’s MediaCodec interface (further details in 7.5). For native AV1 decoding we also performed sanity checks to verify that the performance of ExoPlayer+Dav1d was in line with VLC+Dav1d.

7.3

Decoder performance test content

The test content is a concatenation of the El Fuente shots encoded according to command lines and [resolution, CRF/pCRF] parameters generated using the discrete DO methodology described in Sec. 5, with specific focus on quality levels at 1080p and 720p. Importantly, the fast-decode SVT-AV1 flag was active for both AV1 and LCEVC AV1 encoding of all bitstreams below M11 (fast-decode is not currently supported by M11).

For method one (CPU decoding efficiency) we generated two sets of El Fuente shots concatenations, in both cases based on discrete DO selection according to VMAF_NEG-optimized convex hull, presets M3 to M11:

For method two (playback test on mobile phones), we focused the analysis on high end qualities and resolutions, at VMAF 95 quality level. For the sake of efficiency, two representative presets have been chosen: M3 and M8 for both AV1 and LCEVC AV1. In line with method one, the tests have been executed on “1080p height match” concatenations and on “full set” concatenations, at resolutions selected by the DO and resulting bitrates illustrated in Table 3.

Table 3 –

Average bitrate in Kbps of the El Fuente concatenations used for mobile tests

For both codecs the bitrates of M3 concatenations are between 15% and 37% smaller than the equivalent M8 (for comparable VMAF 95 quality), hence M8 bitstreams are expected to be more challenging for the decoders.

In addition to the El Fuente content at 30 fps used for the encoder tests, in order to test the limits of AV1 playback and test LCEVC’s potential to improve AV1 decoding capabilities, we expanded playback tests to El Fuente content at 60 fps. The 60 fps test sequence has been prepared using the [resolution, crf] points selected by DO at 30 fps (optimized according to VMAF_NEG), focusing on M3 preset and VMAF 95 quality level. The test focused on 1080p60, concatenating sequences for AV1 and LCEVC AV1 using the “1080p height matched” approach described above.

7.4

Decoder CPU efficiency testing

To measure decoder CPU efficiency, the following set of video decoder software was used:

All target software has been built for Ubuntu and Android operating systems. Matching the encoder efficiency methodology, we simultaneously execute FFMPEG, single threaded, for each processor core of the host computer, using Gnu Parallel utility and using/usr/bin/time is used to capture the CPU execution time.

The following command lines have been used to execute the test runs.

CPU utilization measurements were performed on AWS C6gd.16xlarge instance with Ubuntu 20.04.

7.5

Real-life device testing

To relate decoder CPU measurements to real life playback capabilities and battery usage, power measurements and rendered frame count have been captured using Android based mobile devices with a video playback application capable of decoding and playing back both AV1 and LCEVC AV1. Android and ExoPlayer provide several APIs to enable current drain and battery voltage data to be collected, along with information about the number of frames rendered/dropped during video playback. To enable objective comparisons between AV1 and LCEVC AV1 playback sessions, the following metrics are gathered per playback session: total number of frames, average milliwatts of battery power drain, total number of frames decoded during the total duration of play, number of frames dropped during the total duration of play. Each target playback device was put into the same condition before each test run, to mitigate external factors effecting test runs.

The following performance indicators are then calculated and used:

7.5.3

Real life device results – El Fuente (30 fps), ExoPlayer + Dav1d

The chart below illustrates test result for the “1080p height match” approach in terms of playback continuity and power consumption, to decode AV1 and LCEVC AV1 1080p content (M3 and M8 presets).

The left side of Figure 9 shows the percentage of dropped frames vs. total across the full set of tested devices. On less powerful devices (left), the player struggled to playback 1080p content with native AV1 (more so with M8 due to higher bitrates vs. M3 AV1), resulting in stuttering on complex scenes, occasional freezes and in certain cases, such as with Huawei P8, Motorola G4, Sony Experia Z2 and Samsung Galaxy S5, in playback failures. With LCEVC AV1 playback was smooth, with less than 0.1% of dropped frames across all devices.

For devices that played back smoothly AV1 and LCEVC AV1, we recorded power consumption (in mWatts) and reported on the right side of Figure 10 the AV1 power overhead as a percentage increase vs. LCEVC AV1. The power overhead of AV1 vs. LCEVC AV1 indicates the percentage battery life extension of LCEVC AV1 vs. AV1.

Figure 10 -

“1080p matched” approach: percentage of frames dropped and power measurement (Dav1d)

AV1 playback consistently consumed more power than LCEVC AV1 playback, on average 21% more with M3 and 19% more with M8, resulting in faster battery drain. The overall power figures include not only video decoding, but also all other battery-draining features of the device (most notably the screen), which are common for both AV1 and LCEVC AV1 playback and thus dilute the relative power saving of LCEVC on decoding.

Figure 11 shows the same indicators when the full El Fuente test sequence is played, each shot concatenated at the resolution determined by the DO. Notice that LCEVC played on average a higher resolution.

Figure 11 -

“Full set” approach: percentage of frames dropped and power measurement (Dav1d)

While the frame dropped scenario is consistent with the 1080p-only results on less powerful devices, the LCEVC power saving is slightly diluted vs. the 1080p scenario. On average, native AV1 playback consumes 4% more power than LCEVC AV1 on M3 and 11% more with M8.

7.5.4

Real life device results – El Fuente (30 fps), ExoPlayer + GAV1

Figure 12 on the left side illustrates the playback test results for the “1080p height match” approach, using ExoPlayer with GAV1 software decoder on AV1 and LCEVC AV1 (M3 and M8 presets), for a subset of 10 devices capable of running GAV1.

Figure 12 -

Percentage of frames dropped with “1080p matched” approach (GAV1 decoder)

Using GAV1 decoder with ExoPlayer, native AV1 playback at 1080p drops frames to the point of not being watchable on the majority of tested devices. On Redmi 7, Redmi N8T and Galaxy A50 AV1 does not play at all, resulting in a frozen frame a few seconds after playback start; on Pixel 4, Samsung S9, Pixel 4a playback stutters, with over 20% dropped frames. LCEVC AV1 played on all devices, with occasional stuttering only on the three weakest devices and smooth playback on the others.

The right side of Figure 12 shows power figures on devices dropping less than 30% of the frames. On average LCEVC extends battery life by 41% vs. native AV1 on M8, and by 37% on M3, with reduced battery consumption even on devices where it is rendering more frames. LCEVC gains vs. native AV1 are greater with GAV1 than with Dav1d.

7.5.5

Real life device results – El Fuente (60 fps), ExoPlayer + Dav1d

Figure 13 illustrates test result for the “1080p height match” approach, using the ExoPlayer application combined with Dav1d software decoder on AV1 and LCEVC AV1 1080p60 content using M3 presets. As a ‘sanity check’ the same test (on AV1) has been performed using the VLC player combined with Dav1d decoder with similar results.

Figure 13 –

On El Fuente p60 sequence, % frames dropped on full sequence using ExoPlayer and Dav1d decoder

As expected, ExoPlayer (and VLC + Dav1d, for which the sanity check provided consistent results to ExoPlayer + Dav1d) cannot playback 1080p60 AV1 content on most tested devices, resulting in frequent freezes or stuttering. With LCEVC AV1 video playback was smooth, with occasional stuttering only on older devices (Motorola G4 of 2016, and Samsung S5 of 2014).

7.5.6

Battery power drain test results – El Fuente (30 fps)

In addition to instantaneous power measurement, we performed battery drain tests, performing a real-life use case of playing one hour of content, encoded according to DO (“full set” scenario), on a small sample of devices which could playback smoothly both AV1 and LCEVC AV1. We started from device fully charged and measured the battery status at regular five-minute intervals during an hour of uninterrupted playback. Figure 14 illustrates test results using the ExoPlayer application combined with Dav1d and GAV1 respectively.

Figure 14 -

One-hour battery drain test using ExoPLayer + Dav1d and ExoPLayer + GAV1 decoder

The test showed that native AV1 drained the battery 30% to 50% more quickly than LCEVC AV1, both in the case of Dav1d decoder (both phones rendered 100% of the frames), and GAV1 (where in the case of the S9 the phone rendered only 80% of the frames), demonstrating that the LCEVC power savings measured on a short 5-8 minutes sequence are reflected in even greater actual battery life improvements.