Anthony vetro video codec slide

Advanced Video Codecs: What’s on the Horizon? Anthony Vetro Mitsubishi Electric Research Labs, Cambridge, MA avetro@merl.com ATSC Symposium on Next-Generation Broadcast, October 19, 2010, Alexandria, VA Historical Perspective • Today’s DTV broadcast based on MPEG-2 – Huge success, wide deployment, made DTV possible – Technology basis for this standard is 20 years old – Contending with legacy issues Digital Broadcasting DVD Video Vetro: Advanced Video Codecs: What’s on the Horizon? ATSC Symposium Current State-of-the-Art MPEG-4 / H.264 AVC 100 • Half the bit rate of MPEG-2 with same quality • Supported for mobile, but not main program • Mature standard, large scale deployment for HD • Technology basis for this standard is 10 years old • Candidate for a nextgeneration broadcasting system (but depends on requirements…) Mobile (CIF) Bus (CIF) 80 60 40 20 MPEG-2 MPEG-4 ASP H.264/AVC Percentage bit-rate relative to MPEG-2 (at 32 dB) Vetro: Advanced Video Codecs: What’s on the Horizon? ATSC Symposium Video Codec Design • High video coding efficiency satisfies a fundamental need in digital video systems – Reduce bandwidth / better quality – More programming and services • Other features consider in video codec design – Network friendliness (e.g., NAL unit concept in AVC) – Error resilience (concepts such as data partitioning, slices, resynchronization markers have been around for many years) • Extensions – traditionally come later to expand scope of standard and enable additional applications or services – Scalability (temporal, spatial, SNR) – Professional (4:4:4, 10/12-bit) – Multiview (support for stereo/3D) Vetro: Advanced Video Codecs: What’s on the Horizon? ATSC Symposium Outline • Cursory overview of video coding architecture and existing tools • Recent developments – update on a new video coding standardization project • Enabling new services • What’s on the horizon (in terms of video coding) Vetro: Advanced Video Codecs: What’s on the Horizon? ATSC Symposium Video Coding Basics Vetro: Advanced Video Codecs: What’s on the Horizon? ATSC Symposium Exploiting Redundancy Spatially Temporally Decorrelate data Energy packing I B B P B B P Motion compensated prediction Predict within a picture Vetro: Advanced Video Codecs: What’s on the Horizon? ATSC Symposium Typical Video Coding Architecture block partitioning + - Transform/ Quantization Entropy Coder Inv Quant/ Transform + Intra-frame Prediction In-Loop Filter Motion Compensation Motion Estimation Vetro: Advanced Video Codecs: What’s on the Horizon? ATSC Symposium MPEG-2 Coding Tools 16x16 macroblocks + - Transform/ Quantization 8x8 DCT Quant Matrix Entropy Coder Inv Quant/ Transform VLC Huffman tables + Intra-frame Prediction Adaptive Field-Frame Prediction (16x16, 16x8) DC coef prediction Motion Compensation Motion Estimation Half-pel resolution Vetro: Advanced Video Codecs: What’s on the Horizon? Note: no in-loop filter Post-filter may be applied Two reference pictures (max) ATSC Symposium MPEG-4/H.264 AVC Coding Tools 16x16 macroblock + Block and sub-block motion partitions - Transform/ Quantization Entropy Coder 4x4/8x8 Int Transform Inv Quant/ Transform Spatial intra prediction Intra-frame Prediction + In-Loop Filter Motion Compensation Motion Estimation Quarter-pel resolution Vetro: Advanced Video Codecs: What’s on the Horizon? ContextAdaptive VLC or Arithmetic Coding Adaptive deblocking Multi-reference picture ATSC Symposium Enabling New Services Vetro: Advanced Video Codecs: What’s on the Horizon? ATSC Symposium Which New Services? • What will consumers get excited about? • What will make consumers reconsider the way that they receive content in their homes? • Candidates – Extended programming – Higher resolution – 4K×2K, 8K×4K, Mobile HD – Full color and bit depth – 4:4:4, 10/12 bit – Multiview and 3D – stereo to auto-stereo Vetro: Advanced Video Codecs: What’s on the Horizon? ATSC Symposium Higher Resolution Services Ultra High-Definition Television 100 inch TV on the wall >100 degree view angle Up to 7680 x 4320 (8K) Mobile & Portable HD 25cm viewing distance 25 inch flexible display 8K for 20/20 vision Flexible Television Vetro: Advanced Video Codecs: What’s on the Horizon? ATSC Symposium Multiview/3D • Strong interest in 3D delivery – Production of premium content increasing – Numerous devices supporting stereoscopic display available to the consumer – Many standards being developed/amended throughout the chain • Basic delivery options – Upgrade equipment/infrastructure – Utilize capabilities of existing infrastructure • Most activity focused on stereo services Vetro: Advanced Video Codecs: What’s on the Horizon? ATSC Symposium Auto-Stereoscopic Displays • 3D viewing w/out glasses – Pixel colors vary based on viewing direction • Various prototypes using different display technology – Lenticular, parallax barrier – Integral imaging • Challenge: High number of views needed to drive display View dependent pixel Vetro: Advanced Video Codecs: What’s on the Horizon? ATSC Symposium Target of 3D Video (3DV) Format Stereoscopic displays • Variable stereo baseline • Adjust depth perception Left Right Limited Camera Inputs Data Format Data Format Constrained Rate (based on distribution) Auto-stereoscopic N-view displays Vetro: Advanced Video Codecs: What’s on the Horizon? • Wide viewing angle • Large number of output views ATSC Symposium Multiview Video plus Depth (MVD) • MVD is the reference format for 3DV: stereo texture and stereo depth (encoded with MVC) Left Right • Call for Proposals on 3D Video Coding Technology to be issued in January 2011 Vetro: Advanced Video Codecs: What’s on the Horizon? ATSC Symposium 3DV Framework Limited Video Inputs (e.g., or views) Depth Estimation Video/Depth Codec View Synthesis Larger # Output Views 1010001010001 + Binary Representation & Reconstruction Process Vetro: Advanced Video Codecs: What’s on the Horizon? ATSC Symposium 3DV Implications on Transmission • Require compatibility with existing standards for mono and stereo video services – Expect compatibility with future formats as well – Build on existing service or deploy as new service? • Additional channels may be proposed, leading to higher bandwidth requirements – Occlusion Maps – Transparency Data – Reflectance • Strong need for more bandwidth efficient codecs Vetro: Advanced Video Codecs: What’s on the Horizon? ATSC Symposium The Road Ahead Vetro: Advanced Video Codecs: What’s on the Horizon? ATSC Symposium 5-year Assessment • In years: realistic to assume 75-80% lower rate than MPEG-2 and 40-50% lower than AVC • Does AVC address needs or plan for HEVC? – Depends on targets for next-generation services and corresponding timelines Vetro: Advanced Video Codecs: What’s on the Horizon? ATSC Symposium Future Outlook Is it possible to push the video rates even lower? Yes! • Perhaps following same model that has been working – Better prediction and motion modeling – Better entropy coding and reduction of side info – Better transform and decomposition of source signal • More promising: perceptual video coding – – – – Framework would still make use of spatial/temporal prediction Perhaps utilize geometric modeling, e.g., of textures, regions Leverage computer vision, analysis/synthesis techniques New metrics that allow substantial point-by-point variations at the pixel level w/out compromising structural similarity Vetro: Advanced Video Codecs: What’s on the Horizon? ATSC Symposium Structurally Lossless Images [T Pappas, ICIP 2010] Vetro: Advanced Video Codecs: What’s on the Horizon? ATSC Symposium Structurally Lossless Images • More than 20% pixels are different • PSNR = 22.2dB [T Pappas, ICIP 2010] Vetro: Advanced Video Codecs: What’s on the Horizon? ATSC Symposium Concluding Remarks • Video compression technology has made significant advances in last 20 years – AVC is available, new HEVC standard is emerging – Expect that further advances will come • Next step: determine target services, schedule and requirements for next-generation broadcast (with video compression capabilities in mind) • Acknowledgements: – G.J Sullivan (Microsoft) – T Murakami, K Asai, S Sekiguchi (Mitsubishi) Vetro: Advanced Video Codecs: What’s on the Horizon? ATSC Symposium [...]... Right • Call for Proposals on 3D Video Coding Technology to be issued in January 2011 Vetro: Advanced Video Codecs: What’s on the Horizon? ATSC Symposium 3DV Framework Limited Video Inputs (e.g., 2 or 3 views) Depth Estimation Video/ Depth Codec View Synthesis Larger # Output Views 1010001010001 + Binary Representation & Reconstruction Process Vetro: Advanced Video Codecs: What’s on the Horizon? ATSC... coding) Vetro: Advanced Video Codecs: What’s on the Horizon? ATSC Symposium Tool-Level Evaluation Large MC / Transform Block Size [Ref: JCTVC-B065] 10 Coding Gain[%] 8 6 Adaptive Transform Block Size for Intra Adaptive Loop Filter 4 2 0 More than a dozen tools currently under investigation Vetro: Advanced Video Codecs: What’s on the Horizon? ATSC Symposium Enabling New Services Vetro: Advanced Video Codecs:...Coding Efficiency Improvements Vetro: Advanced Video Codecs: What’s on the Horizon? ATSC Symposium Recent Developments New Standardization Project Vetro: Advanced Video Codecs: What’s on the Horizon? ATSC Symposium The new JCT-VC Partnership • Initial groundwork in VCEG and MPEG • New team formed in January 2010 • Joint Call for Proposals issued • Joint Collaborative Team on Video Coding (JCT-VC) • Chairs:... video clips subjectively tested in Classes B-E • Testing for both “random access” (1 sec) and “low delay” (no picture reordering) conditions • Complexity also considered in anchor encodings Vetro: Advanced Video Codecs: What’s on the Horizon? ATSC Symposium Example Subjective Evaluation Anchor at 2.5 Mbps Anchor at 1.6 Mbps Anchor at 1 Mbps Best Performing Proposal at 1 Mbps Vetro: Advanced Video Codecs:... video services – Expect compatibility with future formats as well – Build on existing service or deploy as new service? • Additional channels may be proposed, leading to higher bandwidth requirements – Occlusion Maps – Transparency Data – Reflectance • Strong need for more bandwidth efficient codecs Vetro: Advanced Video Codecs: What’s on the Horizon? ATSC Symposium The Road Ahead Vetro: Advanced Video. .. stereo services Vetro: Advanced Video Codecs: What’s on the Horizon? ATSC Symposium Auto-Stereoscopic Displays • 3D viewing w/out glasses – Pixel colors vary based on viewing direction • Various prototypes using different display technology – Lenticular, parallax barrier – Integral imaging • Challenge: High number of views needed to drive display View dependent pixel Vetro: Advanced Video Codecs: What’s... Horizon? ATSC Symposium Example Subjective Evaluation Anchor at 4.5 Mbps Anchor at 3 Mbps Best Performing Proposal at 2 Mbps Anchor at 1 Mbps Vetro: Advanced Video Codecs: What’s on the Horizon? ATSC Symposium Overall Average Mean Opinion Score Vetro: Advanced Video Codecs: What’s on the Horizon? ATSC Symposium Sample Objective Gains Bit rate savings (%) relative to AVC anchors Class A Class B Class C... Multiview and 3D – stereo to auto-stereo Vetro: Advanced Video Codecs: What’s on the Horizon? ATSC Symposium Higher Resolution Services Ultra High-Definition Television 100 inch TV on the wall >100 degree view angle Up to 7680 x 4320 (8K) Mobile & Portable HD 25cm viewing distance 25 inch flexible display 8K for 20/20 vision Flexible Television Vetro: Advanced Video Codecs: What’s on the Horizon? ATSC Symposium... April 2010 • Project name: High Efficiency Video Coding (HEVC) • Document archives are publicly accessible http://wftp3.itu.int/av-arch/jctvc-site/ or http://phenix.int-evry.fr/jct/ Vetro: Advanced Video Codecs: What’s on the Horizon? ATSC Symposium Call for Proposals Testing • 27 complete proposals submitted • Each proposal was a major package – Lots of encoded video, extensive documentation, extensive... Symposium Target of 3D Video (3DV) Format Stereoscopic displays • Variable stereo baseline • Adjust depth perception Left Right Limited Camera Inputs Data Format Data Format Constrained Rate (based on distribution) Auto-stereoscopic N-view displays Vetro: Advanced Video Codecs: What’s on the Horizon? • Wide viewing angle • Large number of output views ATSC Symposium Multiview Video plus Depth (MVD)