The effective transmission and processing of mobile multimedia

THE EFFECTIVE TRANSMISSION AND PROCESSING OF MOBILE MULTIMEDIA MA HAIYANG B.E., WUHAN UNIVERSITY, CHINA A THESIS SUBMITTED FOR THE DEGREE OF DOCTOR OF PHILOSOPHY Graduate School for Integrative Sciences and Engineering NATIONAL UNIVERSITY OF SINGAPORE 2014 Declaration I hereby declare that this thesis is my original work and it has been written by me in its entirety. I have duly acknowledged all the sources of information which have been used in the thesis. This thesis has also not been submitted for any degree in any university previously. a c 2014 MA Haiyang All Rights Reserved Dedication This thesis is dedicated to my beloved parents, Ma Wenke and Liu Qiaoyun, who raised me and keep supporting me throughout my whole life. c Acknowledgements This thesis is the outcome of five years of research work during which I have been accompanied and supported by many people. Without them, the completion of my thesis would not be possible. I am honored to take this opportunity to thank them. First, I would like to express my sincere gratitude to Prof. Roger Zimmermann for his consistent support and illuminating guidance during my PhD study. His rigorous attitude on research helped me develop a scientific and systematic thinking which is critical to problem-solving. His wholehearted encouragement helps me overcome many obstacles I once felt insurmountable. I feel extremely proud to have started and spent my PhD study under his supervision. My heartfelt thanks go to Dr. Deepak Gangadharan, Dr. Hao Jia and Wang Guanfeng with whom I have collaborated during my PhD research. I have benefited a lot from their technical insights, as they help me to analyze and solve a problem from different perspectives. I would also like to thank NGS, the Graduate School for Integrative Sciences and Engineering of National University of Singapore for providing me the opportunity to doctoral research in a distinguished university with financial support. The PhD study in NUS has opened up a new door in my life. In the end, I want to express my appreciation to the companion from my dear colleagues: Liang Ke, Hao Jia, Ma He, Shen Zhijie, Zhang Ying, Zhang Lingyan, Fang Shunkai, Cui Weiwei, Wang Guanfeng and Yin Yifang in the Media Management Research Lab. d Publications Peer Reviewed • Deepak Gangadharan, Haiyang Ma, Samarjit Chakraborty, Roger Zimmermann. Video Quality Driven Buffer Dimensioning via Prioritized Frame Drops. In IEEE International Conference on Computer Design (ICCD), October 2011. • Haiyang Ma, Deepak Gangadharan, Nalini Venkatasubramanian, Roger Zimmermann. Energy-aware Complexity Adaptation for Mobile Video Calls. In Proceedings of the 19th annual ACM International Conference on Multimedia (ACM MM), November 2011. • Guanfeng Wang, Haiyang Ma, Beomjoo Seo, Roger Zimmermann. Sensor-Assisted Camera Motion Analysis and Motion Estimation Improvement for H.264/AVC Video Encoding. In ACM Workshop on Network and Operating Systems Support for Digital Audio and Video (NOSSDAV), June 2012. • Haiyang Ma, Roger Zimmermann. Adaptive Coding with Energy Conservation for Mobile Video Calls. In IEEE International Conference on Multimedia and Expo (ICME), July 2012. • Haiyang Ma, Roger Zimmermann. Energy Conservation in 802.11 WLAN for Mobile Video Calls. In IEEE International Symposium on Multimedia (ISM), December 2012. • Jia Hao, Roger Zimmermann, Haiyang Ma. GTube: Geo-Predictive Video Streaming over HTTP in Mobile Environments. In the 5th ACM Multimedia Systems Conference (ACM MMSys), March 2014. • Haiyang Ma, Jia Hao, Roger Zimmermann. Access Point Centric Scheduling for DASH Streaming in Multirate 802.11 Wireless Network. In IEEE International Conference on Multimedia and Expo (ICME), July 2014. e CONTENTS Summary vi List of Figures ix List of Tables xii Introduction 1.1 1.2 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . Motivation: An Energy Aware and Bandwidth Efficient Mul- 1.3 timedia System . . . . . . . . . . . . . . . . . . . . . . . . . Research Work and Contributions . . . . . . . . . . . . . . . 1.3.1 Workload Complexity Reduction of MPEG-4 on Mo- 5 bile Platforms . . . . . . . . . . . . . . . . . . . . . . Energy Efficient Mobile Call Framework with Adap- tive Coding of H.264 . . . . . . . . . . . . . . . . . . Adaptive Packet Transmission Scheme for Mobile Video Calls . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.3.2 1.3.3 1.3.4 1.4 Access Point Centric DASH Scheduling in Multirate 802.11 Wireless Networks . . . . . . . . . . . . . . . Organization . . . . . . . . . . . . . . . . . . . . . . . . . . 10 Literature Review 2.1 11 Video Coding Scalability . . . . . . . . . . . . . . . . . . . . 11 i CONTENTS 2.1.1 2.1.2 2.2 2.1.3 Hardware-Assisted Coding . . . . . . . . . . . . . . . 16 2.1.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . 17 Hardware Energy Conservation . . . . . . . . . . . . . . . . 17 2.2.1 2.2.2 2.3 2.4 Decoding Workload Adaptation . . . . . . . . . . . . 13 Encoding Workload Adaptation . . . . . . . . . . . . 14 CPU . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 Network Interface Card . . . . . . . . . . . . . . . . . 19 2.2.3 Graphical Display . . . . . . . . . . . . . . . . . . . . 20 2.2.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . 22 Energy-Optimized Multimedia Systems . . . . . . . . . . . . 22 2.3.1 2.3.2 Cross-Layer Adaptive Coding Framework . . . . . . . 22 Computation Offloading to the Cloud . . . . . . . . . 23 2.3.3 2.3.4 2.3.5 Server and Middleware-Assisted Rate Adaptation . . 24 Error-Resilient Coding and Transmission . . . . . . . 24 Power-Aware 802.11 WLAN Design . . . . . . . . . . 26 2.3.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . 26 Quality Adaptation in HTTP Streaming . . . . . . . . . . . 27 2.4.1 Architecture of DASH Streaming . . . . . . . . . . . 27 2.4.2 2.4.3 Client-side Approaches . . . . . . . . . . . . . . . . . 28 Server-side Approaches . . . . . . . . . . . . . . . . . 30 2.4.4 2.4.5 Intermediary Approaches . . . . . . . . . . . . . . . . 30 Summary . . . . . . . . . . . . . . . . . . . . . . . . 31 Workload Complexity Reduction of MPEG-4 on Mobile 33 Platforms 3.1 3.2 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 33 System Overview . . . . . . . . . . . . . . . . . . . . . . . . 34 3.3 Complexity Scalability of MPEG-4 . . . . . . . . . . . . . . 35 3.3.1 Profiling Environment . . . . . . . . . . . . . . . . . 35 3.3.2 Encoder Adaptation . . . . . . . . . . . . . . . . . . 36 3.4 3.3.3 Decoder Adaptation . . . . . . . . . . . . . . . . . . 40 Metrics for System QoS and Power Model . . . . . . . . . . 42 3.5 3.4.1 Methodology for Real-time Performance Monitoring . 43 3.4.2 Power Model . . . . . . . . . . . . . . . . . . . . . . 45 Algorithms for Adaptive System . . . . . . . . . . . . . . . . 45 ii CONTENTS 3.5.1 3.5.2 3.6 Coding Module . . . . . . . . . . . . . . . . . . . . . 45 Feedback Module . . . . . . . . . . . . . . . . . . . . 46 Experimental Evaluation . . . . . . . . . . . . . . . . . . . . 49 3.6.1 Experimental Setup . . . . . . . . . . . . . . . . . . . 49 3.6.2 Parameter Calculation for Energy Model . . . . . . . 50 3.7 3.6.3 Experimental Result . . . . . . . . . . . . . . . . . . 50 Overhead Computation . . . . . . . . . . . . . . . . . . . . . 53 3.8 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54 An Energy Efficient Mobile Call Framework with Adaptive 55 Coding of H.264 4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 55 4.2 Complexity Scalability of H.264 . . . . . . . . . . . . . . . . 56 4.2.1 Complexity Adaptation of H.264 Encoder . . . . . . 57 4.2.2 Complexity Adaptation of H.264 Decoder . . . . . . 63 4.3 System Design . . . . . . . . . . . . . . . . . . . . . . . . . . 64 4.3.1 Derivation of Buffer Limit . . . . . . . . . . . . . . . 64 4.4 4.3.2 Adaptation Workflow . . . . . . . . . . . . . . . . . . 65 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . 66 4.4.1 Experimental Setup . . . . . . . . . . . . . . . . . . . 66 4.5 4.4.2 Results . . . . . . . . . . . . . . . . . . . . . . . . . . 66 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67 4.6 4.5.1 Hardware-assisted Coding . . . . . . . . . . . . . . . 67 4.5.2 Applicability to Other Codecs . . . . . . . . . . . . . 68 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69 Adaptive Packet Transmission Scheme for Mobile Video Calls 70 5.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 70 5.2 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . 71 5.2.1 Power Save Mode . . . . . . . . . . . . . . . . . . . . 71 5.2.2 IEEE 802.11e . . . . . . . . . . . . . . . . . . . . . . 72 5.3 Transmission Analysis . . . . . . . . . . . . . . . . . . . . . 72 5.3.1 State Transitions under Dynamic PSM . . . . . . . . 72 5.4 5.3.2 Delay In Video Calling . . . . . . . . . . . . . . . . . 74 Transmission Schedule Design . . . . . . . . . . . . . . . . . 77 iii CONTENTS 5.4.1 5.4.2 5.5 5.6 Session Establishment . . . . . . . . . . . . . . . . . 77 Exchange of Execution Condition . . . . . . . . . . . 77 5.4.3 Estimation of Network Latency . . . . . . . . . . . . 78 5.4.4 Making Transmission Decisions . . . . . . . . . . . . 79 Experimental Evaluation . . . . . . . . . . . . . . . . . . . . 82 5.5.1 5.5.2 Experimental Setup . . . . . . . . . . . . . . . . . . . 82 Processing of Video Packets . . . . . . . . . . . . . . 83 5.5.3 5.5.4 5.5.5 Evaluation Criteria . . . . . . . . . . . . . . . . . . . 84 Experimental Results . . . . . . . . . . . . . . . . . . 85 Overhead Measurement . . . . . . . . . . . . . . . . . 89 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89 Access Point Centric Scheduling for HTTP Streaming in 91 Multirate 802.11 Wireless Networks 6.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 91 6.2 6.3 Fair Queuing in Wireless Network . . . . . . . . . . . . . . . 92 System Design . . . . . . . . . . . . . . . . . . . . . . . . . . 93 6.3.1 6.3.2 6.3.3 6.4 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . 100 6.4.1 Experimental Setup . . . . . . . . . . . . . . . . . . . 100 6.4.2 6.4.3 6.4.4 6.5 6.6 Info Collector . . . . . . . . . . . . . . . . . . . . . . 93 Packet Scheduler . . . . . . . . . . . . . . . . . . . . 94 URL Redirector . . . . . . . . . . . . . . . . . . . . . 94 Evaluation Metrics . . . . . . . . . . . . . . . . . . . 102 Type of DASH Clients . . . . . . . . . . . . . . . . . 103 Experimental Results . . . . . . . . . . . . . . . . . . 104 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113 6.5.1 Layering Principle . . . . . . . . . . . . . . . . . . . 113 6.5.2 End-to-end Principle . . . . . . . . . . . . . . . . . . 114 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114 Conclusions 116 7.1 Summary of Research Techniques . . . . . . . . . . . . . . . 116 7.2 7.3 Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . 118 Limitations . . . . . . . . . . . . . . . . . . . . . . . . . . . 119 7.4 Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . 120 iv CHAPTER 7. CONCLUSIONS 7.3 Limitations Through our research work presented in this thesis it has been demonstrated that by combining the context information in multimedia applications, such as processing and transmission, with power saving techniques on hardware components, we can provide the mobile clients with a better user experience by reducing the energy consumption of video calling and fairly and efficiently distribute the available bandwidth for on-demand video streaming. The proposed frameworks can be extended to other applications as well. However, what we have achieved is not an ultimately impeccable solution. The limitations lie in several aspects which we will point out as follows. First, in our experiment we directly manipulated the hardware modules such as the CPU and the network interface card from the application level and such manipulations require root access. While the root access is easy to obtain in a desktop or server oriented Linux OS, it is not easy to obtain on a mobile operating system and might expose some security risks as well. In real implementations, a better solution is to compile the hardware manipulation code into an individual kernel module that can be dynamically loaded into the kernel space when the application program starts running and control the hardware on behalf of the application. The application communicates and exchanges the execution conditions with the loaded module in the kernel space. Second, in our analysis we assumed that WiFi is the default wireless network access for the mobile platforms. All the experiments were also conducted with a WiFi connection. While it is true that WiFi networks are being deployed widely and will continue to play an indispensable role for the expansion of mobile platforms, contemporary cellular technologies such as 3G and 4G bring greater convenience with unmatched universal access to networks. Our experimental results would be more persuasive if they could be conducted and validated under cellular networks as well. Furthermore, for energy conservation on the network interface card and DASH streaming scheduling we make use of some features specific to the WiFi standard such as dynamic PSM and Access Point. As a result, additional work would be needed and the solutions would need to be modified to be ported to other 119 CHAPTER 7. CONCLUSIONS wireless network infrastructures. Third, the proposed DASH streaming scheduler is situated at the AP and intercepts multimedia traffic between the clients and the server. While there are many similar middleware approaches, they break the end-to-end design principle of computer networking. However, as the myopic nature of client-based rate adaptation logic has been demonstrated by more and more work it is very difficult to coordinate these behaviors in a distributed fashion. Thus intermediary solutions like ours can definitely provide an alternatively effective solution. 7.4 Future Work For the processing and transmission of multimedia contents on mobile platforms, energy conservation, together with a fair and efficient bandwidth usage and distribution, have always been two primary targets researchers keep striving for. As has been demonstrated in this thesis, there is definitely a huge space of unsolved research potential deserving further endeavors. For future work, our research can be extended in many aspects. Some of them are listed as follows. • Energy conservation on graphical display. Backlight accounts for a significant percentage of the total energy consumed on a mobile device, especially with the trend that large resolution displays are becoming the mainstream configuration for current mobile phones. Finding an energy-efficient lighting solution for the current mainstream display technologies such as TFT-LCD, AMOLED, etc., will be a fruitful direction that deserves more research efforts. • Combined energy saving effects from multiple hardware components. In the thesis the energy saving effects on CPU and network interface card during the video call were studied and measured separately. It would be helpful to combine these two approaches and study the combined effects on energy saving. • Diversified wireless network environment. As has been stated in Section 7.3, throughout the thesis all of our experiments were conducted assuming a WiFi connection, seriously limiting its portability. 3G 120 CHAPTER 7. CONCLUSIONS and 4G standards define their own power saving features as well. For instance, the Long Term Evolution Advanced (LTE-A) standard has a discontinuous reception (DRX) mechanism to reduce the power consumption of mobile stations [7]. It would be necessary to extend our mobile video call framework with different network interfaces by making use of their unique power saving features at the system level. • Latest video coding standard. The specification for the next generation video coding standard, the High Efficiency Video Coding (HEVC), has been ratified and many open source and commercial implementations have been released as well. As a successor to H.264, HEVC substantially improves the coding efficiency by introducing several new features such as a Coding Tree Unit (CTU) up to 64 × 64 pixels and a built-in support for parallel coding (Wavefront Parallel Processing). It remains an interesting topic to explore the design space of HEVC and provide an optimized implementation for video conference on mobile platforms given the power constraint. • Multiparty video conference. In this thesis we only considered the traditional two-party video call scenario. Nowadays multiparty video calling is gradually gaining popularity and is being supported by many hardware and software vendors. It poses a great challenge to integrate an energy conservation framework into the multi-party scenario. First, a multiparty call can have different network topologies. Some establish a direct connection between each pair of users, while some others select some users as supernodes or use some dedicated servers, referred to as Multipoint Control Units (MCU), for traffic overlay and multiplexing, which complicates the power saving on the network interface card. Second, given the different multimedia traffic generated by the network topologies, the user program can have various policies regarding the scheduling and synchronization between the encoder and the decoder workload, and the decoder may have to process several received media streams simultaneously. This requires a careful design of the coding workload reduction scheme. • Implementation and experiments on real mobile platforms. All the video call experiments were conducted between two laptop computers 121 CHAPTER 7. CONCLUSIONS running the Linux operating system. While the laptop demonstrates a great resemblance to a mobile phone and could be broadly classified as a mobile device, it could be more persuasive if the experiments can be conducted on real mobile platforms, such as on Android or iOS. Such mobile operating systems have slightly modified kernellevel process scheduling policies that prioritize GUI rendering, which could affect the execution of video coding. Moreover, such devices also have different power consumption ratios between each hardware components, worthy of further exploration. • Implementation of DASH scheduling policies on a real AP. For the AP-centric DASH streaming scheduler, our experimental results were obtained through simulation in ns-2 only. Going forward, we plan to implement our scheduling framework on a real wireless router by burning a customized firmware using OpenWrt1 , or a software AP on PC by Hostapd2 , an open source user space daemon software for 802.11 AP management. We will build a real network environment and study its corresponding performances by including various DASH clients with different rate adaptation logics. https://openwrt.org/ http://hostap.epitest.fi/hostapd/ 122 Bibliography [1] Android Open Source. http://source.android.com/source. [2] FFplay Media Player. http://ffmpeg.org/ffplay.html. [3] Hulu. http://www.hulu.com. [4] MythTV Open Source DVR. http://www.mythtv.org/download. [5] Netflix. http://www.netflix.com. [6] ns-2 Network Simulator. http://www.isi.edu/nsnam/ns. [7] Third-Generation Partnership Project, Tech. Spec. 3G TS 36.304 ver 11.6.0. [8] VLC Media Player. http://www.videolan.org/vlc. [9] Real Time Streaming Protocol (RTSP), Apr 1998. Network Working Group. [10] Wireless Medium Access Control (MAC) and Physical Layer (PHY) Specifications, Jun 1999. IEEE 802.11. [11] SIP: Session Initiation Protocol, Jun 2002. Network Working Group. [12] ITU-T Recommendation G.114. One Way Transmission Time, 2003. [13] Real Time Control Protocol (RTCP) Attribute in Session Description Protocol (SDP), Oct 2003. Network Working Group. 123 BIBLIOGRAPHY [14] RTP: A transport protocol for real-time applications, Jul 2003. Network Working Group. [15] ISO/IEC 14496-2:2004 - Information technology – Coding of Audio-Visual Objects – Part 2: Visual, 2004. [16] Wireless Medium Access Control (MAC) and Physical Layer (PHY) Specifications Amendment 8: Medium Access Control (MAC) Quality of Service Enhancement, Nov 2005. IEEE 802.11E. [17] SDP: Session Description Protocol, Jul 2006. Network Working Group. [18] Intel Core2 Duo Processors on 45-nm Processor for Embedded Applications Thermal Design Guide, 2008. [19] Real-Time Messaging Protocol (RTMP) Specification, Jun 2009. Adobe Inc. [20] Mobile Voice & Video Calling: Strategic Opportunities & Business Models 2011-2016, Aug 2011. Juniper Research. [21] Forecast: PCs, Ultramobiles and Mobile Phones, Worldwide, 2010-2017, 4Q13 Update, Dec 2013. Gartner. [22] Cisco Visual Networking Index: Global Mobile Data Traffic Forecast Update, 2013 - 2018, Feb 2014. Cisco. [23] ISO/IEC 23009-1:2014 Information Technology – Dynamic Adaptive Streaming over HTTP (DASH) – Part 1: Media Presentation Description and Segment Formats, May 2014. Information Technology Task Force (ITTF). [24] Y. Agarwal, R. Chandra, A. Wolman, P. Bahl, K. Chin, and R. Gupta. Wireless Wakeups Revisited: Energy Management for VoIP over WiFi Smartphones. In the 5th International Conference on Mobile Systems, Applications and Services, pages 179–191. ACM, 2007. [25] S. Akhshabi, L. Anantakrishnan, C. Dovrolis, and A. C. Begen. ServerBased Traffic Shaping for Stabilizing Oscillating Adaptive Streaming Players. In NOSSDAV, pages 19–24, 2013. [26] B. Anand, K. Thirugnanam, J. Sebastian, P. G. Kannan, A. L. Ananda, M. C. Chan, and R. K. Balan. Adaptive Display Power Management 124 BIBLIOGRAPHY for Mobile Games. In Proceedings of the 9th International Conference on Mobile Systems, Applications, and Services, pages 57–70. ACM, 2011. [27] N. Balasubramanian, A. Balasubramanian, and A. Venkataramani. Energy Consumption in Mobile Phones: A Measurement Study and Implications for Network Applications. In the 9th ACM SIGCOMM Conference on Internet Measurement, pages 280–293, 2009. [28] J. Bankoski, P. Wilkins, and Y. Xu. Technical Overview of VP8, An Open Source Video Codec for the Web. In ICME, pages 1–6. Citeseer, 2011. [29] J. C. R. Bennett and H. Zhang. WF2 Q: Worst-Case Fair Weighted Fair Queueing. In INFOCOM, pages 120–128, 1996. [30] J. C. Bicket. Bit-rate Selection in Wireless Networks. PhD thesis, Massachusetts Institute of Technology (MIT), 2005. [31] T. Brand˜ ao and M. Queluz. No-Reference Image Quality Assessment Based on DCT Domain Statistics. Signal Processing, 88(4):822–833, 2008. [32] T. Burd and R. Brodersen. Processor Design for Portable Systems. The Journal of VLSI Signal Processing, 13(2):203–221, 1996. [33] Z. Cao, B. Foo, L. He, and M. van der Schaar. Optimality and Improvement of Dynamic Voltage Scaling Algorithms for Multimedia Applications. In 45th Design Automation Conference, pages 179–184, 2008. [34] A. Carroll and G. Heiser. An analysis of Power Consumption in a Smartphone. In the USENIX Conference, pages 21–21. USENIX Association, 2010. [35] N. Chang, I. Choi, and H. Shim. DLS: Dynamic Backlight Luminance Scaling of Liquid Crystal Display. IEEE Transactions on Very Large Scale Integration (VLSI) Systems, 12(8):837–846, 2004. [36] X. Chen, S. Jin, and D. Qiao. M-PSM: Mobility-Aware Power Save Mode for IEEE 802.11 WLANs. In 31st International Conference on Distributed Computing Systems (ICDCS), pages 77–86. IEEE, 2011. [37] Z. Chen, J. Xu, Y. He, and J. Zheng. Fast Integer-pel and Fractional-pel Motion Estimation for H.264/AVC. Journal of Visual Communication and Image Representation, 17(2):264–290, 2006. 125 BIBLIOGRAPHY [38] L. Cheng, S. Mohapatra, M. El Zarki, N. Dutt, and N. Venkatasubramanian. Quality-Based Backlight Optimization for Video Playback on Handheld Devices. Advances in Multimedia, 2007(1):4–4, 2007. [39] C. C. Chi, M. Alvarez-Mesa, J. Lucas, B. Juurlink, and T. Schierl. Parallel HEVC Decoding on Multi-and Many-core Architectures. Journal of Signal Processing Systems, 71(3):247–260, 2013. [40] H. Choi, J. Lee, and D. Cho. On the Use of a Power-Saving Mode for Mobile VoIP Devices and Its Performance Evaluation. IEEE Transactions on Consumer Electronics, 55(3):1537–1545, 2009. [41] K. Choi, K. Dantu, W.-C. Cheng, and M. Pedram. Frame-Based Dynamic Voltage and Frequency Scaling for a MPEG Decoder. In Proceedings of the IEEE/ACM International Conference on Computer-Aided Design, pages 732–737. ACM, 2002. [42] I. Crk, M. Bi, and C. Gniady. Interaction-Aware Energy Management for Wireless Network Cards. ACM SIGMETRICS Performance Evaluation Review, 36(1):371–382, 2008. [43] E. Cuervo, A. Balasubramanian, D. Cho, A. Wolman, S. Saroiu, R. Chandra, and P. Bahl. MAUI: Making Smartphones Last Longer With Code Offload. In Proceedings of the 8th International Conference on Mobile Systems, Applications, and Services, pages 49–62. ACM, 2010. [44] L. De Cicco, S. Mascolo, and V. Palmisano. Feedback Control for Adaptive Live Video Streaming. In Proceedings of the 2nd Multimedia Systems Conference (MMSys), pages 145–156. ACM, 2011. [45] L. Deng, W. Gao, M. Z. Hu, and Z. Z. Ji. An Efficient Hardware Implementation for Motion Estimation of AVC Standard. IEEE Transactions on Consumer Electronics, 51(4):1360–1366, 2005. [46] F. Dogar, P. Steenkiste, and K. Papagiannaki. Catnap: Exploiting High Bandwidth Wireless Interfaces to Save Energy for Mobile Devices. In the 8th International Conference on Mobile Systems, Applications, and Services, pages 107–122. ACM, 2010. [47] M. Dong, Y.-S. K. Choi, and L. Zhong. Power Modeling of Graphical User Interfaces on OLED Displays. In Proceedings of the 46th Annual Design Automation Conference, pages 652–657. ACM, 2009. 126 BIBLIOGRAPHY [48] F. Fu and M. van der Schaar. A Systematic Framework for Dynamically Optimizing Multi-User Wireless Video Transmission. IEEE Journal on Selected Areas in Communications, 28(3):308–320, 2010. [49] J. Hao, R. Zimmermann, and H. Ma. GTube: Geo-Predictive Video Streaming over HTTP in Mobile Environments. In Proceedings of the 5th ACM Multimedia Systems Conference, pages 259–270, 2014. [50] Z. He, W. Cheng, and X. Chen. Energy Minimization of Portable Video Communication Devices Based on Power-Rate-Distortion Optimization. IEEE Transactions on Circuits and Systems for Video Technology, 18(5):596–608, 2008. [51] Z. He, Y. Liang, L. Chen, I. Ahmad, and D. Wu. Power-Rate- Distortion Analysis for Wireless Video Communication under Energy Constraints. IEEE Transactions on Circuits and Systems for Video Technology, 15(5):645–658, 2005. [52] M. Heusse, F. Rousseau, G. Berger-Sabbatel, and A. Duda. Performance Anomaly of 802.11b. In INFOCOM, volume 2, pages 836–843, 2003. [53] G. Holland, N. Vaidya, and P. Bahl. A Rate-Adaptive MAC Protocol for Multi-Hop Wireless Networks. In Proceedings of the 7th Annual International Conference on Mobile Computing and Networking (MobiCom), pages 236–251. ACM, 2001. [54] M. A. Hoque, M. Siekkinen, and J. K. Nurminen. On the Energy Efficiency of Proxy-Based Traffic Shaping for Mobile Audio Streaming. In Consumer Communications and Networking Conference (CCNC), pages 891–895. IEEE, 2011. [55] I. Hou, R. Singh, et al. Scheduling of Access Points for Multiple Live Video Streams. In Proceedings of the 14th International Symposium on Mobile Ad Hoc Networking and Computing (MobiHoc), pages 267–270. ACM, 2013. [56] R. Houdaille and S. Gouache. Shaping HTTP Adaptive Streams for a Better User Experience. In Proceedings of the 3rd Multimedia Systems Conference (MMSys), pages 1–9. ACM, 2012. [57] M. Hrarti, H. Saadane, M. Larabi, A. Tamtaoui, and D. Aboutajdine. A Macroblock-Based Perceptually Adaptive Bit Allocation for H264 Rate 127 BIBLIOGRAPHY Control. In 5th International Symposium on I/V Communications and Mobile Network (ISVC), pages 1–4. IEEE, 2010. [58] C. Hsu and M. Hefeeda. Achieving Viewing Time Scalability in Mobile Video Streaming Using Scalable Video Coding. In 1st ACM SIGMM Conference on Multimedia Systems, pages 111–122, 2010. [59] W. Hu and G. Cao. Energy Optimization Through Traffic Aggregation in Wireless Networks. In Proceedings of the 33rd Annual IEEE International Conference on Computer Communications (INFOCOM), 2014. [60] Y. Hu, A. Simpson, K. McAdoo, and J. Cush. A High Definition H.264/AVC Hardware Video Decoder Core for Multimedia SoC’s. In IEEE International Symposium on Consumer Electronics, pages 385–389, 2004. [61] Y. Huang, B. Hsieh, S. Chien, S. Ma, and L. Chen. Analysis and Complexity Reduction of Multiple Reference Frames Motion Estimation in H.264/AVC. IEEE Transactions on Circuits and Systems for Video Technology, 16(4):507–522, 2006. [62] Y. Huang, A. Tran, and Y. Wang. A Compressed Domain Distortion Measure for Fast Video Transcoding. In 15th International Conference on Multimedia, pages 787–790. ACM, 2007. [63] R. Jain, D.-M. Chiu, and W. R. Hawe. A Quantitative Measure of Fairness and Discrimination for Resource Allocation in Shared Computer System. Eastern Research Laboratory, Digital Equipment Corporation, 1984. [64] W. Ji, M. Chen, X. Ge, P. Li, and Y. Chen. A Perceptual Macroblock Layer Power Control for Energy Scalable Video Encoder Based on Just Noticeable Distortion Principle. Journal of Network and Computer Applications, 34(5), 2010. [65] W. Ji, M. Chen, X. Ge, P. Li, and Y. Chen. ESVD: An Integrated Energy Scalable Framework for Low-power Video Decoding Systems. EURASIP Journal on Wireless Communications and Networking, Special Issue on Multimedia Communications over Next Generation Wireless Networks, 2010. [66] J. Jiang, V. Sekar, and H. Zhang. Improving Fairness, Efficiency, and Stability in HTTP-based Adaptive Video Streaming with FESTIVE. In CoNEXT, pages 97–108, 2012. 128 BIBLIOGRAPHY [67] P. S. Kalekar. Time Series Forecasting Using Holt-Winters Exponential Smoothing. Kanwal Rekhi School of Information Technology, 2004. [68] A. Kamerman and L. Monteban. WaveLAN R -II: A High-Performance Wireless LAN for the Unlicensed Band. Bell Labs Technical Journal, 2(3):118–133, 1997. [69] A. K. Katsaggelos, F. Zhai, Y. Eisenberg, and R. Berry. Energy-Efficient Wireless Video Coding and Delivery. Wireless Communications, 12(4):24– 30, 2005. [70] F. Kelly. Charging and Rate Control for Elastic Traffic. European Transactions on Telecommunications, 8(1):33–37, 1997. [71] R. Kemp, N. Palmer, T. Kielmann, and H. Bal. Cuckoo: A Computation Offloading Framework for Smartphones. In Mobile Computing, Applications, and Services, pages 59–79. Springer, 2012. [72] M. Kim, H. Oh, N. Dutt, A. Nicolau, and N. Venkatasubramanian. PBPAIR: An Energy-Efficient Error-Resilient Encoding Using Probability Based Power Aware Intra Refresh. ACM SIGMOBILE Mobile Computing and Communications Review, 10(3):58–69, 2006. [73] R. Krashinsky and H. Balakrishnan. Minimizing Energy for Wireless Web Access with Bounded Slowdown. In the 8th Annual International Conference on Mobile Computing and Networking, pages 119–130. ACM, 2002. [74] Y.-W. Kwon and E. Tilevich. Energy-Efficient and Fault-Tolerant Distributed Mobile Execution. In IEEE 32nd International Conference on Distributed Computing Systems (ICDCS), pages 586–595, 2012. [75] M. Lacage, M. H. Manshaei, and T. Turletti. IEEE 802.11 Rate Adaptation: A Practical Approach. In Proceedings of the 7th International Symposium on Modeling, Analysis and Simulation of Wireless and Mobile Systems (MSWiM), pages 126–134. ACM, 2004. [76] J. Lee, W. Liao, J. Chen, and H. Lee. A Practical QoS Solution to Voice over IP in IEEE 802.11 WLANs. Communications Magazine, IEEE, 47(4):111–117, 2009. [77] F. Li, G. Shi, and F. Wu. An Efficient VLSI Architecture for 4× Intra Prediction in the High Efficiency Video Coding (HEVC) Standard. In 129 BIBLIOGRAPHY IEEE International Conference on Image Processing (ICIP), pages 373– 376, 2011. [78] C. Liu, I. Bouazizi, and M. Gabbouj. Rate Adaptation for Adaptive HTTP Streaming. In Proceedings of the 2nd Multimedia Systems Conference (MMSys), pages 169–174. ACM, 2011. [79] H. Ma, D. Gangadharan, N. Venkatasubramanian, and R. Zimmermann. Energy-Aware Complexity Adaptation for Mobile Video Calls. In Proceedings of the 19th ACM international conference on Multimedia, pages 1313–1316, 2011. [80] H. Ma, H. Jia, and R. Zimmermann. Access Point Centric Scheduling for DASH Streaming for Multirate 802.11 Wireless Network. In IEEE International Conference on Multimedia and Expo (ICME), 2014. [81] H. Ma and R. Zimmermann. Adaptive Coding with CPU Energy Conservation for Mobile Video Calls. In IEEE International Conference on Multimedia and Expo (ICME), pages 717–722, 2012. [82] H. Ma and R. Zimmermann. Energy Conservation in 802.11 WLAN for Mobile Video Calls. In International Symposium on Multimedia (ISM), pages 266–273, 2012. [83] Z. Ma, H. Hu, and Y. Wang. On Complexity Modeling of H.264/AVC Video Decoding and Its Application for Energy Efficient Decoding. IEEE Transactions on Multimedia, 13(6):1240–1255, 2011. [84] J. Manweiler and R. Choudhury. Avoiding the Rush Hours: WiFi Energy Management via Traffic Isolation. In the 9th International Conference on Mobile Systems, Applications, and Services, 2011. [85] K. Miller, E. Quacchio, G. Gennari, and A. Wolisz. Adaptation Algorithm for Adaptive Streaming over HTTP. In 19th International Packet Video Workshop (PV), pages 173–178. IEEE, 2012. [86] S. Mohapatra, N. Dutt, A. Nicolau, and N. Venkatasubramanian. DYNAMO: A Cross-Layer Framework for End-to-End QoS and Energy Optimization in Mobile Handheld Devices. IEEE Journal on Selected Areas in Communications, 25(4):722–737, 2007. 130 BIBLIOGRAPHY [87] R. K. Mok, E. W. Chan, and R. K. Chang. Measuring the Quality of Experience of HTTP Video Streaming. In IFIP/IEEE International Symposium on Integrated Network Management (IM), pages 485–492, 2011. [88] V. Namboodiri and L. Gao. Towards Energy Efficient VoIP over Wireless LANs. In the 9th ACM International Symposium on Mobile Ad Hoc Networking and Computing, pages 169–178, 2008. [89] J. P. Pavon and S. Choi. Link Adaptation Strategy for IEEE 802.11 WLAN via Received Signal Strength Measurement. In International Conference on Communications, volume 2, pages 1108–1113. IEEE, 2003. [90] S. Peng. Complexity Scalable Video Decoding via IDCT Data Pruning. In IEEE International Conference on Consumer Electronics (ICCE), pages 74–75, 2001. [91] X. P´erez-Costa and D. Camps-Mur. IEEE 802.11e QoS and Power Saving Features Overview and Analysis of Combined Performance. Wireless Communications, IEEE, 17(4):88–96, 2010. [92] G. P. Perrucci, F. H. Fitzek, and J. Widmer. Survey on Energy Consumption Entities on the Smartphone Platform. In Vehicular Technology Conference (VTC Spring), pages 1–6. IEEE, 2011. [93] W. Pu, Z. Zou, and C. W. Chen. Video Adaptation Proxy for Wireless Dynamic Adaptive Streaming over HTTP. In 19th International Packet Video Workshop (PV), pages 65–70. IEEE, 2012. [94] A. Pyles, Z. Ren, G. Zhou, and X. Liu. SiFi: Exploiting VoIP Silence for WiFi Energy Savings in Smart Phones. In the 13th International Conference on Ubiquitous Computing, pages 325–334. ACM, 2011. [95] H. Riiser, H. S. Bergsaker, P. Vigmostad, P. Halvorsen, and C. Griwodz. A Comparison of Quality Scheduling in Commercial Adaptive HTTP Streaming Solutions on a 3G Network. In Proceedings of the 4th Workshop on Mobile Video, pages 25–30. ACM, 2012. [96] E. Rozner, V. Navda, R. Ramjee, and S. Rayanchu. Napman: NetworkAssisted Power Management for WiFi Devices. In the 8th International Conference on Mobile Systems, Applications, and Services, pages 91–106. ACM, 2010. 131 BIBLIOGRAPHY [97] C.-T. Sah. Fundamentals of Solid-State Electronics, volume 437. World Scientific Singapore, 1991. [98] M. Shafique, L. Bauer, and J. Henkel. 3-Tier Dynamically Adaptive PowerAware Motion Estimator for H.264/AVC Video Encoding. In 13th International Symposium on Low Power Electronics and Design (ISLPED), pages 147–152, 2008. [99] A. Shye, B. Scholbrock, and G. Memik. Into the Wild: Studying Real User Activity Patterns to Guide Power Pptimizations for Mobile Architectures. In Proceedings of the 42nd Annual IEEE/ACM International Symposium on Microarchitecture, pages 168–178, 2009. [100] W.-Y. So, P. A. Levermore, V. I. Adamovich, K. Rajan, M. S. Weaver, R. Ma, M. Hack, J. J. Brown, M. H. Kang, H. J. Kim, et al. Power Efficient AMOLED Display with Novel Four Sub-Pixel Architecture and Driving Scheme. In SID Symposium Digest of Technical Papers, volume 41, pages 622–625. Wiley Online Library, 2010. [101] M. Tamai, T. Sun, K. Yasumoto, N. Shibata, and M. Ito. Energy-Aware Video Streaming with QoS Control for Portable Computing Devices. In 14th International Workshop on Network and Operating Systems Support for Digital Audio and Video (NOSSDAV), pages 68–73, 2004. [102] T. C. Thang, H. T. Le, A. T. Pham, and Y. M. Ro. An Evaluation of Bitrate Adaptation Methods for HTTP Live Streaming. IEEE Journal on Selected Areas in Communications, 32(4):693–705, 2014. [103] G. Tian and Y. Liu. Towards Agile and Smooth Video Adaptation in Dynamic HTTP Streaming. In CoNEXT, pages 109–120, 2012. [104] A. M. Tourapis. Enhanced Predictive Zonal Search for Single and Multiple Frame Motion Estimation. In Proceedings of Visual Communications and Image Processing (VCIP), pages 1069–1079, 2002. [105] S. Tsao and C. Huang. An Energy-Efficient Transmission Mechanism for VoIP Over IEEE 802.11 WLAN. Wireless Communications and Mobile Computing, 9(12):1629–1644, 2009. [106] G. Van Wallendael, S. Van Leuven, J. De Cock, P. Lambert, R. Van de Walle, J. Barbarien, and A. Munteanu. Improved Intra Mode Signaling for 132 BIBLIOGRAPHY HEVC. In IEEE International Conference on Media and Expo (ICME), pages 1–6, 2011. [107] P. Verkaik, Y. Agarwal, R. Gupta, and A. Snoeren. Softspeak: Making VoIP Play Well in Existing 802.11 Deployments. In the 6th USENIX Symposium on Networked Systems Design and Implementation, pages 409–422. USENIX Association, 2009. [108] B. J. Villa and P. E. Heegaard. Improving Perceived Fairness and QoE for Adaptive Video Streams. In The 8th International Conference on Networking and Services (ICNS), pages 149–158, 2012. [109] G. Wang, H. Ma, B. Seo, and R. Zimmermann. Sensor-Assisted Camera Motion Analysis and Motion Estimation Improvement for H.264/AVC Video Encoding. In Proceedings of the 22nd International Workshop on Network and Operating System Support for Digital Audio and Video (NOSSDAV), pages 89–94. ACM, 2012. [110] X. Wang, L. Song, M. Chen, and J. Yang. Paralleling Variable Block Size Motion Estimation of HEVC on CPU Plus GPU Platform. In IEEE International Conference on Mutimedia and Expo Workshops (ICMEW), pages 1–5, 2013. [111] Y. Wang and S. Chang. Complexity Adaptive H.264 Encoding for Light Weight Streams. In IEEE International Conference on Acoustics, Speech and Signal Processing, pages II25–28, 2006. [112] Y. Wang and M. Claypool. RealTracer-Tools for Measuring the Performance of RealVideo on the Internet. Multimedia Tools and Applications, 27(3):411–430, 2005. [113] M. Wien, R. Cazoulat, A. Graffunder, A. Hutter, and P. Amon. Real-Time System for Adaptive Video Streaming Based on SVC. IEEE Transactions on Circuits and Systems for Video Technology, 17(9):1227–1237, 2007. [114] Z. Yan, J. Xue, and C. W. Chen. QoE Continuum Driven HTTP Adaptive Streaming Over Multi-client Wireless Networks. In International Conference on Multimedia and Expo (ICME). IEEE, 2014. [115] T. Yensen, J. Lariviere, I. Lambadaris, and R. Goubran. HMM Delay Prediction Technique for VoIP. IEEE Transactions on Multimedia, 5(3):444– 457, 2003. 133 BIBLIOGRAPHY [116] X. Yi, J. Zhang, N. Ling, and W. Shang. Improved and Simplified Fast Motion Estimation for JM. In 16th Meeting of Joint Video Team (JVT) of ISO/IEC MPEG & ITU-T VCEG, 2005. [117] W. Yuan and K. Nahrstedt. Energy-Efficient Soft Real-time CPU Scheduling for Mobile Multimedia Systems. SIGOPS Operating Systems Review, 37(5):149–163, 2003. [118] W. Yuan, K. Nahrstedt, S. Adve, D. Jones, and R. Kravets. GRACE1: Cross-layer Adaptation For Multimedia Quality and Battery Energy. IEEE Transactions on Mobile Computing, pages 799–815, 2006. [119] X. Zhang and K. Shin. E-MiLi: Energy-Minimizing Idle Listening in Wireless Networks. In the 17th Annual International Conference on Mobile Computing and Networking, pages 205–216. ACM, 2011. [120] Z. Zhou and M. Sun. Fast Macroblock Inter Mode Decision and Motion Estimation for H.264/MPEG-4 AVC. In International Conference on Image Processing (ICIP), pages 789–792. IEEE, 2004. [121] L. Zhu, Z. Fan, K. Aggelos K, et al. Joint Video Summarization and Transmission Adaptation for Energy-Efficient Wireless Video Streaming. EURASIP Journal on Advances in Signal Processing, 2008. [122] H. K. Zrida, A. Jemai, A. C. Ammari, and M. Abid. High Level H.264/AVC Video Encoder Parallelization for Multiprocessor Implementation. In Design, Automation & Test in Europe (DATE), pages 940–945. IEEE, 2009. 134 [...]... management for the CPU, NIC and graphical display, respectively 2.2.1 CPU The power consumption of the CMOS circuits can be divided into a static component and a dynamic component The static power refers to the power used when the transistor is not in the process of switching and is essentially determined by the supply voltage and the total current flow [97] Dynamic power, on the other hand, is the sum of transient... connected on the go Third, the latest generation of mobile operating systems such as iOS and Android provide consumers with a huge collection of mobile apps These apps give consumers unique experiences unheard of in the PC era, by taking advantage of touch-based interaction and various hardware components integrated into the mobile devices Figure 1.1 illustrates the prediction of global mobile data... LIST OF FIGURES 1.1 Estimation of global mobile traffic per month by Cisco [22] 2 2.1 The processing flow of an H.264 video encoder 11 2.2 2.3 The processing flow of an H.264 video decoder 12 Architecture of the DASH streaming 28 3.1 System framework for mobile video calls The white blocks show the coding module and the solid arrows show its workflow The blue blocks show the. .. so as to extend the servicing time The popularity of HTTP streaming in recent years has been the driving force for the investigation of an effective scheduling solution for the bandwidth allocation and distribution among different types of clients in a wireless network 1.3 Research Work and Contributions In this thesis we focus on the coding and transmission of video streams on wireless mobile platforms... this thesis is therefore aimed at providing a solution, an energy aware and bandwidth efficient multimedia system, given the limited capacity of batteries and the unfair distribution of bandwidth for several different mobile applications Specifically, we target two applications scenarios, video calling and streaming Given the limited battery capacity that powers mobile devices, video calling on mobile. .. result of a limited service duration Mobile platforms also suffer from the scarcity and fast varying quality of the network bandwidth and it is more challenging to ensure a sustained quality of service in a wireless environment, compared to a wired one It has long been a heated research area on the efficient utilization and fair allocation of the limited bandwidth resources in a wireless network, and the. .. playback freezes and quality fluctuations Our work demonstrates that the proposed framework effectively combats the two primary constraints, energy and bandwidth, on mobile platforms It can reduce the power consumption of mobile devices and provide a fair and effective bandwidth allocation scheme in wireless networks Therefore users can achieve a high level of satisfaction for multimedia services on mobile platforms... correlated to the quality instability of the servicing network In particular, mobile video streaming is a severe sufferer of this problem due to the frequent instability of the wireless network environments It has long been a heated research area to improve the efficient utilization and fair allocation of the limited bandwidth resources in a wireless network Another issue worth noting is that, multimedia. .. presence of mobile devices in people’s daily lives The progress in fabrication techniques of mobile chips and the widespread deployment of advanced wireless transmission technologies have successfully transformed a mobile device into a portable multimedia entertainment hub Killer applications such as HD video streaming and video calling, once considered only possible on PCs due to their demanding processing. .. gets frustrated by the limited service duration Recent trends show that more and more mobile devices are designed with a slim body and a 2 CHAPTER 1 INTRODUCTION large display screen This leaves only space for a tiny and compact battery, aggravating the incompatibility between the growing mobile processing demands and the constraint of battery capacity in the coming years On-demand video streaming, . THE EFFECTIVE TRANSMISSION AND PROCESSING OF MOBILE MULTIMEDIA MA HAIYANG B.E., WUHAN UNIVERSITY, CHINA A THESIS SUBMITTED FOR THE DEGREE OF DOCTOR OF PHILOSOPHY Graduate. to the ra pid improvement of the processing capabilities of the mobile devices. What is worse, a battery can drain very fast for mobile video calls, a s they r equire the simultaneous running of. for the bandwidth distribution among different types of clients. To improve the multimedia consumption experience on mobile platforms in spite of energy and bandwidth constraints, in this thesis