Available online at www.sciencedirect.com ScienceDirect Procedia Computer Science 104 (2017) 413 – 420 ICTE 2016, December 2016, Riga, Latvia Solution for Sound Playback Delay on Android Devices Dmitrijs Zigunovsa, Jekaterina Smirnovaa,*, Gatis Vitolsa, Gintautas Stonysb a Latvia University of Agriculture, Liela Street, Jelgava, LV-3001, Latvia b Alytaus kolegija, 17 Studentu Street, Alytus, 62252, Lithuania Abstract Recent reports show that there is a 40-200 millisecond delay for sound playback in Android devices At the same time there is no such reports and identified problem for other platforms Research focuses on unresolved audio playback latency problems in devices with Android operating systems Examination of possible problems, application of audio playback latency recording and measuring methods and audio playback latency decreasing methods are used to achieve a smallest possible audio playback delay As a result with a proposed solution audio playback delay was reduced to 9ms using OpenSL EU API application and uncoded audio data with a device that support digitization and most high performance © 2017 by by Elsevier B.V.B.V This is an open access article under the CC BY-NC-ND license © 2016The TheAuthors Authors.Published Published Elsevier (http://creativecommons.org/licenses/by-nc-nd/4.0/) Peer-review under responsibility of organizing committee of the scientific committee of the international conference; ICTE 2016 Peer-review under responsibility of organizing committee of the scientific committee of the international conference; ICTE 2016 Keywords: Audio playback latency; Android devices; Mobile application development Introduction Audio playback impact all layers of application processing, initially with giving the command in the software application framework to the audio playback, for example in Linux kernel Use of Android Software Development Kit (SDK) is appropriate way to implement audio file playback to devices with the Android Operation System (OS)1 Literature2,3 confirm that there is on average between 40-200 milisecond (ms) sound delay on Android devices Sometimes delay exceeds 200ms However such problem has not been identified in devices with other OS4 It could be that this issue is Android OS related Usually a person feels sound delay around 10-15 ms and above5 Therefore any developed product sound playback delay should not exceed this level * Corresponding author Tel.: +371 63005701; fax: +371 63023095 E-mail address: jekaterina.smirnova@llu.lv 1877-0509 © 2017 The Authors Published by Elsevier B.V This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/) Peer-review under responsibility of organizing committee of the scientific committee of the international conference; ICTE 2016 doi:10.1016/j.procs.2017.01.154 414 Dmitrijs Zigunovs et al / Procedia Computer Science 104 (2017) 413 – 420 Android OS has the following playback methods6: x Android.media.MediaPlayer object through the Android SDK (from Application framework layer) x EU OpenSL API via Android Native Development Kit (NDK) (Android service layer) To playback a simple audio file, for example when developing audio players, MediaPlayer object typically is used This object provides easy control to playback process using recommended parameters, such as the size of the buffer and the decoding algorithm Programmers usually need to configure playback process in a local Android environment using Application Programming Interface (API) such as Open Sound Library for Embedded Systems (OpenSL ES) This gives more control to playback process, but also increases development complexity Use of this API is based on multimedia application development requiring the smallest possible sound delay time, or necessity to create cross-platform code The aim of this research is to ensure a smallest possible audio playback delay in devices with Android operating system The experimental part of the research includes testing audio playback during application development with various audio data types, audio playback delay measurement and data analysis Materials and methods There are several measurement options with different levels of measuring accuracy and application complexity But all of these measurement methods are based on the idea that it is necessary to register time of the command giving moment and the time of the moment when the sound starts to play Moreover, these time moments must be registered in one time metering system Therefore, commands giving process must be conveyed as observation result For this research light-emitting diode can be used The smallest delay (less than 0.5 milliseconds) between giving command and monitoring can be recorded with the switch of the LED indicator If simultaneously commands of switching on LED and audio file playback through dynamics or TRS connector are given, delay between LED indicator and sound playback can be recorded Signal must be passed to dual-channel oscilloscope that records LED indicator state and the playback state LED indicator activation can be converted into a signal using a photoelectric cell, which must be placed on opposite side to the LED indicator The photoelectric signal channel must be connected to one of the oscilloscope channels LED start-up time takes about ms This time depends on the LED specification7 Also additional photoelectric response time must be taken into consideration which usually is between ms and 100 ms This time have correlation with light brightness When light is brighter, photoelectric response time is faster As modern LED devices are able to produce up to 100 lux and more, the photoelectric response time will be minimal However, for this experimental setting hardware combination adds totally to about ms to playback command signal fixing time It should be taken into consideration when processing the recordings If there is an access to the Printed Circuit Board (PCB) it is possible to use it to connect LED to the oscilloscope channel This will remove the created delay and will increase the accuracy of the result (a) (b) Fig (a) Typical scheme for light switch observation; (b) Actual scheme for light switch observation 415 Dmitrijs Zigunovs et al / Procedia Computer Science 104 (2017) 413 – 420 Photoelectric cell voltage is observed using voltmeter (see Fig 1(a).) where R1 – resistor, Rf – a photoelectric cell, RV – voltmeter resistance, U– voltage source, V– voltmeter, GND – grounding As circuit elements are connected in series, R1 and U are constants, then the voltage of the photoelectric cell varies with the resistance, which in turn varies with the incident light intensity This voltage (Vf) is calculated using the formula (1) Vf U Rf R f R1 (1) In reality, the voltage measuring devices have their constant resistance Therefore, the real light switching on observation scheme is showed in Fig 1(b) With the known voltmeter resistance voltage (Vf) is calculated using the formula (2) Vf U U R1 Rv R f R1 Rv R f (2) Knowing the photoelectric cell resistance dependence on light intensity and tested device LED light intensity, voltage to the photoelectric cell can be calculated when the LED is turned on and off Using these values, the indicator, which indicates when LED is turned on, can be identified An indicator can take a situation where the voltage of the photoelectric cell falls in range from ʹΤ͵ ή ሺ୫ୟ୶ െ ܸ ሻ and ͳΤ͵ ή ሺܸ௫ െ ܸ ሻ having background voltage into consideration The voltage in circuit should not be too low Experimental setting and testing application To get the experimental results it was necessary to connect oscilloscope channel to the sound TRS audio output channel The developed scheme was used for final testing (see Fig 2) Fig Common playback delay observation scheme To get objective results of audio playback delay, measurements were performed on multiple devices: Sony Xperia V, LG G3, Just5 Blaster These devices vary with Android OS version, branding, and technical support and cover majority of Android OS modifications at the moment of the experiment An oscilloscope with a time digitization and functionality of measuring time delay in milliseconds is needed to delay time measuring in testing smartphones Therefore, selected oscilloscope must be able to perform measurements with a frequency of kHz or faster The requirement is also two-channel availability and the ability to record the direct current SEINTEK S2405 digital oscilloscope was used for delay time measuring in testing smartphones 416 Dmitrijs Zigunovs et al / Procedia Computer Science 104 (2017) 413 – 420 For testing purpose Android application was developed Following audio file playback methods were implemented into the developed application: x Android.media.MediaPlayer object via Android SDK x OpenSL ES via Android NDK x Superpowered Audio API via Android NDK Immediately after application launcher the necessary objects are initialized, configured and prepared for work The application provides the control using graphical interface (see Fig 3) Each method complies with a button, which provides playback start command In this way there is no activity related to the playback start command giving and finally no linked playback delay Fig Developed application interface for research purposes Immediately after the sound playback, application calls the LED command Both these commands are given in a separate threads Choosing formats for the sound testing MP3 is the most popular and more distributed file format for sounds storage Audio data in this format is encoded with losses The data must be encoded at first to playback MP3 type audio files In this situation playback delay occurs due to decoding process To observe the audio data storage format impact on the playback delay time, it was necessary to make tests using open-label uncompressed audio data formats The way to play such data with all the methods is use of Waveform Audio File Format (WAV) container with Pulse-code modulation (PCM) format data To know exactly the audio data storage format impact on the play amount of delay, the third format was used, which is supported by all the used testing methods As the third format Advanced Audio Coding (AAC) format was chosen because it is the modernized version of the standard MP3 format This audio data encoding compresses them over the same MP3 sound quality which in result produces ACC format sound with better quality than MP3 with the same size of data file 417 Dmitrijs Zigunovs et al / Procedia Computer Science 104 (2017) 413 – 420 Results and discussion The results (see Table 1) were grouped by playbacks methods, devices, and audio data Table Playback delay measurements (milliseconds) LGG3 Playback method Data format 44.1kHzfrequency MediaPlayer AAC 50,520 OpenSLES Superpowered 37,406 48kHzfrequency 47,032 52,744 60,482 23,222 48,338 49,275 49,210 57,664 MP3 89,476 43,086 42,180 47,351 45,174 37,484 44,824 47,400 99,758 24,741 PCM 37,940 44,693 49,478 54,898 50,262 19,870 22,850 12,407 54,939 54,334 AAC 114,697 72,160 69,408 105,153 74,201 87,186 49,856 57,843 47,267 72,054 MP3 104,881 47,596 60,379 49,935 50,224 51,831 47,904 48,222 49,137 47,532 PCM 55,331 49,435 17,592 42,233 39,931 11,977 12,517 9,722 62,674 60,141 AAC 91,660 97,369 107,431 107,961 107,408 102,802 100,761 94,804 61,731 94,358 MP3 99,411 98,255 92,090 99,723 98,084 47,068 45,041 99,317 34,989 72,280 PCM 36,832 45,170 87,571 62,896 62,765 44,775 47,544 47,501 50,050 48,335 200,207 197,478 203,003 Just5Blaster2 Playback method Data format 44.1kHzfrequency MediaPlayer AAC 179,262 OpenSLES Superpowered 180,890 48kHzfrequency 180,657 175,183 185,006 194,880 187,522 MP3 145,209 162,081 154,208 152,509 154,991 174,368 170,122 171,817 177,812 180,330 PCM 122,716 137,532 149,284 134,625 127,305 135,235 140,048 155,536 150,212 152,562 AAC 149,647 180,472 177,886 154,768 167,945 174,802 162,083 167,842 174,833 177,727 MP3 150,036 155,080 152,041 162,496 164,885 160,130 162,858 171,875 169,142 169,648 PCM 134,924 138,285 129,221 141,744 145,129 149,856 152,923 148,242 150,070 149,583 AAC 424,409 436,802 427,528 430,195 421,945 435,638 432,307 449,796 429,680 434,906 MP3 396,587 402,905 385,593 407,501 422,529 412,185 400,079 385,011 402,853 424,950 PCM 327,069 323,266 306,958 342,307 347,202 330,404 353,262 345,459 351,866 355,504 SonyXperiaV Playback method MediaPlayer OpenSLES Superpowered Data format 44.1kHzfrequency 48kHzfrequency AAC 344,937 345,528 349,963 239,148 350,154 194,384 344,442 350,226 210,266 374,209 MP3 220,183 294,427 285,063 299,701 294,648 190,531 290,439 299,856 199,985 284,579 PCM 179,978 300,593 300,179 289,395 319,614 199,702 274,986 279,500 274,813 290,062 AAC 254,576 220,343 225,385 285,587 239,145 244,978 194,906 219,565 234,992 205,443 MP3 295,111 290,134 190,373 240,180 184,318 229,485 194,965 185,331 194,552 184,499 PCM 224,890 169,821 180,473 189,682 179,990 245,073 169,782 185,558 170,019 165,815 AAC 499,633 500,038 489,949 510,057 519,792 480,806 489,226 490,046 520,111 479,388 MP3 510,070 480,146 479,481 499,809 489,322 470,017 499,686 499,905 489,546 480,226 PCM 430,132 440,744 440,734 420,485 430,132 440,248 439,735 419,542 409,376 440,005 418 Dmitrijs Zigunovs et al / Procedia Computer Science 104 (2017) 413 – 420 Because all measurements were made directly, measurement error was calculated using formula (3) where Ɂ – relative mean error, %; – measuring device absolute error, V; ୶ – the mean value of the root mean square error (4), V; – measurements count; Ƚ – importance rate, Ƚ אሾͲǢ ͳሿ; ǡ୬ିଵ – student's t-distribution; ୟ୴ – arithmetic average of the n measurements, V ) 12,13 ߜൌ ටሺௌೣ ௧ഀǡషభ ሻమ ାమ ܵ௫ ൌ ට (3) ௫ೌೡ మ σ సభሺ௫ೌೡ ି௫ ሻ ሺିଵሻ (4) Generally relative measurement error is accepted as broad, because each research which measure different data, allow special maximum relative error rate This rate permits to determine the reliability of measurement results Performed playback delay measurement can raise a subjective agreement that the results with a relative error of more than 50% are not reliable The maximum relative error is accepted with the agreement, that the used method of measurement is accurate But background processes which may be affected to the result of the obtained values are included in solution with tested devices Fig Relative errors measurement As a result, measurement errors are different (see Fig 4) There is no assurance that the results reflect the real situation One factor could be that device loading is impacted by the background running software In addition, the device load increases with use of Wi-Fi, Global System for Mobile Communications (GSM), Near-field communication (NFC), Global Positioning System (GPS), Bluetooth The greater is the device loading level, the more load it takes between sound processing cycles Since the test device LG G3 was under increased load measurements, it could show a greater distinction than measurements of other devices In addition, testing using Just5 Blaster device gave lowest relative error measurement because it was used only by default included software Dmitrijs Zigunovs et al / Procedia Computer Science 104 (2017) 413 – 420 Playback delay average values by used data encoding type are different in test devices (see Fig 5) Device specification impact to the audio playback delay is undeniable When comparing average playback delay value used by the data encoding, it can be concluded that the PCM-format data is played back the fastest This speed has been triggered with no need to decode the data MP3 playback data decoding due delay to increases for 27.64 ms comparing with PCM audio data playback testing Tested AAC data decoding takes about 17.56 ms more time than the MP3 data decoding because AAC encoding algorithm execution takes more time Fig Data format impact to audio playback delay Fig Supported and unsupported audio digitization playback delay difference Sounds discretization also impacts performance time and computing resource This factor also increases playback delay time This conclusion is illustrated in the test results (see Fig 6) 419 420 Dmitrijs Zigunovs et al / Procedia Computer Science 104 (2017) 413 – 420 It can be observed that test devices playback delay difference is small for data with different discretization Conclusion Audio playback delay was reduced using OpenSL EU API application, uncoded audio data in the device supported digitization Using developed application it is possible to reduce the sound delay time to 9ms that does not interfere to human perception Uncoded data playback is faster than the coded data playback Data is processed faster using digitization that is supported by devices rather than using other digitization method High performance devices have smallest playback delay time Testing process can be improved by reducing measurement errors using factory setting Android operation system which is free from additionally installed and running software, turned off communication modules and full battery connected to the charging network Such conditions are close to the ideal test conditions, however real living situation such conditions are not present References 6 Android Open Source Project Audio Available: https://source.android.com/devices/audio/ BlackBerry Developer Blog Tone Generatorௗ: How to add Low Latency Audio to your Native or Cascades App July 2013 Available: http://devblog.blackberry.com/2013/07/tone-generator/ Superpowered.com Android Audio Latency and iOS Audio Latency Test App Available: http://superpowered.com/latency/#table Lobao M Android Audio Latency In-Depth: It’s Getting Better, Especially With The Nexus 5X And 6P International Telecommunication Union Recommendation BT.1359: Relative Timing of Sound and Vision for Broadcasting Geneva, Switzerland; 1998 Huang CC, Nie R Design and realization of music player based on android system In: Int Conf Mechatronics Inf Technol Vol 462; 2013 p 1123-1126 Khan LU Visible light communication: Applications, architecture, standardization and research challenges Digit Commun Networks; 2016 Zheng H, Chen J, Yu C, Gurusamy M Inverse design of LED arrangement for visible light communication systems Opt Commun.; 2017 Bolton JR University of Western Ontario Photochemistry Unit., Boston University Department of Chemistry Solar Power and Fuelsௗ: In: Proceedings of the First International Conference on the Photochemical Conversion and Storage of Solar Energy London, Canada Academic Press; 1977 Henderson R Samsung’s new LED components could improve Galaxy S5 flash and hint at April launch; 2014 Available: http://www.pocket-lint.com/news/127328-samsung-s-new-led-components-could-improve-galaxy-s5-flash-and-hint-at-april-launch Rumsey F, McCormick T Sound and Recording Elsevier/Focal; 2009 Myttenaere A, Golden B, Le Grand B, Rossi F Mean Absolute Percentage Error for regression models Neurocomputing 192; 2016 p 38-48 Nakajima J, Omori Y Stochastic volatility model with leverage and asymmetrically heavy-tailed error using GH skew Student’s tdistribution Comput Stat Data Anal 56(11); 2012 p 3690-3704 Gatis Vitols received PhD in Information technologies at Latvia University of Agriculture Primary research fields include cross-cultural information systems development, usability and database technologies Author works as a leading researcher and associate professor at Faculty of Information Technologies in Latvia University of Agriculture Contact him at gatis.vitols@llu.lv ... execution takes more time Fig Data format impact to audio playback delay Fig Supported and unsupported audio digitization playback delay difference Sounds discretization also impacts performance... observed that test devices playback delay difference is small for data with different discretization Conclusion Audio playback delay was reduced using OpenSL EU API application, uncoded audio... configure playback process in a local Android environment using Application Programming Interface (API) such as Open Sound Library for Embedded Systems (OpenSL ES) This gives more control to playback