Evaluation of real time performance of embedded linux

EVALUATION OF REAL TIME PERFORMANCE OF EMBEDDED LINUX by Truong Quang Vinh A thesis submitted in partial fulfillment of requirements for the degree of Master of Engineering Examination Committee: Nationality: Previous degree: Scholarship Donor: Dr Nitin V Afzulpurkar (Chairman) Dr Lertsak Lekawat Prof Phan Minh Dung Prof Helmuth Gesch Mr Markus Waidelich (External) Vietnamese Bachelor of Electronics Engineering, Ho Chi Minh University of Technology, Vietnam Ministry of Education and Training (Vietnam) Asian Institute of Technology School of Advanced Technologies Thailand December 2003 i ACKNOWLEDGMENT Firstly, I would like to thank my advisor Dr Nitin for his encouraging and supporting my research Therefore, I finish this thesis confidently I would like to send special thanks to Dr Helmuth and Mr Markus who give me a chance to study and research in Microfuzzy GmbH and also support on finance and technique for me to accomplish my thesis I would also like to thank all technicians Ramiro, Xia, Yon, Su and Jani in MicroFuzzy GmbH for their constant advice and assistance Without their help, I would never have come close to finishing Finally, I would like to thank all of my family and friends for putting up with me over the years ii ABSTRACT Nowadays, embedded systems with Linux OS are commonly applied in automotive industry Some of applications require strict time response, and others need to be exactly scheduled to execute a period task All of these are called time-sensitive applications Measuring and evaluating time parameters of an embedded system for time-sensitive applications is very necessary for developers to guarantee that it works functionally For this purpose, signal stimulating and signal analyzing hardware are required In this thesis, modular signal generating and signal analyzing means will be designed The prototype of the evaluation system will be implemented to test and verify an embedded system Improving the performance of the system is the next part of this thesis The research focuses deeper on the possibility of turning Linux into a real-time operating system Particularly, it investigates available real-time solutions for Linux, but also looks into the soft variants for completeness A number of performance tests are conducted during the improvement to make sure that the implementation meets the demands of a real-time operating system iii Table of Contents Chapter Title Page Acknowledgment Abstract Table of Contents List of Figures List of Tables ii iii iv v vi Introduction Literature Review 2.1 Embedded Linux OS 2.2 Real-time embedded Linux OS 2.3 The survey of real-time operating systems found in the market 2.4 The scheduler of Linux 2.5 The target system _ Triton Starter Kit 2.6 Intel PXA255 processor 2.7 Previous work 3 3 10 11 Evaluation of Real-Time System 3.1 Evaluation methods 3.2 Metrics of Characterization 3.3 Experiments 13 13 13 15 Implementation of Testing System 4.1 Jitter Test Module 4.2 Interrupt Latency Test Module 4.3 Test Control Software 18 18 23 28 Result and Analysis 5.1 Jitter 5.2 Interrupt latency 31 31 32 Improvement 6.1 OS timer 6.2 FIQ _ Fast Interrupt Request 35 35 38 Conclusion and Future Work 45 References Appendix A: Pictures of experiment hardware Appendix B: Test program files 47 48 49 iv List of Figures Figure Title Page 2-1 2-2 2-3 Scheduling algorithm Triton Starter Kit Block diagram of Intel PXA255 10 11 3-1 3-2 3-3 3-4 3-5 3-6 Real-Time Jitter Interrupt Latency Hardware setup for test Output wave form Hardware setup for test Timing for Interrupt Response Latency 14 14 16 16 17 17 4-1 4-2 4-3 4-4 4-5 4-6 4-7 4-8 4-9 4-10 4-11 4-12 4-13 4-14 4-15 4-16 Block diagram of jitter test module Control unit ASM chart of Control Unit Timing diagram for the operation of state machine Block diagram of Clk_div unit Block diagram of Timer Counter Unit Block diagram of Mux Unit Block diagram of UART Unit Block diagram of Interrupt Latency Test Module IRQ_generator unit Control unit Block diagram of Timer Counter Unit Block diagram of Multiplexer unit User interface during jitter testing progress User interface during Interrupt latency testing progress User interface for setting the Serial port 18 19 20 21 21 21 22 22 24 24 25 27 27 28 29 30 5-1 5-2 Histogram for Jitter of the target system Histogram of interrupt latency using IRQ 31 33 6-1 6-2 6-3 Histogram of jitter of the periodic task using OS timer interrupt Histogram of jitter of the periodic task using FIQ Histograms of Interrupt latency using FIQ 37 41 43 v List of Tables Table Title Page 2-1 2-2 2-3 Class of processes Alternative class of processes List of system calls 4-1 4-2 4-3 4-4 4-5 4-6 Pin description of control unit Pin description of Timer Counter Unit Pin description of UART Unit Pin description of IRQ_generator Unit Pin description of Control Unit Pin description of Timer Counter Unit 19 21 22 25 25 27 5-1 5-2 Statistic data of jitter of the periodic task using sleep function Statistic data of interrupt latency using IRQ 32 33 6-1 6-2 6-3 6-4 6-5 OS timer register addresses Statistic data of jitter of the periodic task using OS timer interrupt Type of exception Statistic data of jitter of the periodic task using FIQ Statistic data of Interrupt latency using FIQ 35 38 39 41 43 vi 5 CHAPTER INTRODUCTION 1.1 Overview Nowadays, embedded systems have become a buzz word in the last five years, but embedded systems and processors have been around for much longer than that One only needs to look around to see embedded systems everywhere: cell phones, alarm clocks, personal data assistants (PDAs), automobile subsystems such as ABS and cruise control, etc [1] Embedded systems now are more and more modern and complex so that they require a powerful and effective operating system Following the development of embedded systems, the popularity of Linux is increasing rapidly the world Linux is used not only in general computer, but also in embedded system That is called embedded Linux It can be ported to many microprocessors such as ARM, PowerPC, MIPS… Because of its versatile and cost effective characteristic, many device manufacturers turn Linux for embedded systems Therefore, the number of embedded electronic system supported by Linux used in automobiles, tractors, and other control systems, increases explosively Performance of Linux OS impact on operation effect of embedded systems For this reason, the evaluation of performance of embedded Linux running on these systems is an important task in period of product development All parameters about timing need to be investigated in order to ensure that systems response enough for a certain application However, the standard Linux kernel cannot support real-time processing, which may be required by certain embedded applications There exist a number of extensions to Linux that provide support for real-time tasks 1.2 Objectives of the thesis The purpose of this thesis is to evaluate the real-time performance of the embedded Linux on a specific platform The target system chosen in this thesis is Intel PXA255 processor platform The evaluations are realized by hardware experiments with FPGA The major timing parameters in the Linux kernel are statistically tested in various conditions The goals of the evaluation are to verify the functionality of the system as well as measure its real-time performance on the PXA255 platform With these evaluations, we can foretell exactly the timing problem for time-sensitive application such as jitter, interrupt latency, deadline of priority tasks…etc After the timing parameters have been investigated, the next challenging part of this thesis is to improve the real-time performance of the Linux on PXA255 processor In order to safely implement the extension module for Linux kernel, it is important to understand how the kernel ports in the PXA255 processor, how Linux OS manages the hardware A number of performance tests are conducted during the improvement to make sure that the implementation meets the demands of a real-time operating system 1.3 Approach To evaluate real-time performance, a hardware instrument is designed to measured jitter interrupt latency, cycle times, etc For this purpose, signal stimulating and signal analyzing means are required In this thesis, modular signal generating and signal analyzing means will be designed and implemented The evaluation system is required to be high accuracy and able to collect data stream in real-time Capability to generate synchronous stimulating signal and analyzing response signal in real-time is important for the evaluation system To obtain greater accuracy in pinpointing timing problems of a real-time system, the analysis and prediction of tasks schedulability was done using actual measured data from the system The measurements are performed by the way that some tasks are created to run in various condition of system load; each scheduling mechanism is applied; then, execution times and periods of all executing tasks are logged We then utilize a FPGA to timestamp, and collect the data while the system is executing The collected data is then exported to a host PC, where the test results will be shown The extracted data is analyzed to determine whether the tasks are satisfactorily scheduled, and whether any timing problems are encountered The results from the analysis can be sent to the predictor and modified to determine the effect of changes in the timing parameters of a system Evaluating an embedded system is premier step to conduct the next: improvement This part is an extension of the thesis Features affecting the performance need to be brought out the meaning Understanding how embedded Linux schedule the tasks is very important to enhance the performance of the system Evaluation step will be repeated to verify the improvement The loop of these tasks is carried out as a development cycle CHAPTER LITERATURE REVIEW This section offers a brief background into the areas that are related to the research performed in the report as well as the areas that support the reasons for performing this research 2.1 Embedded Linux OS Since Linux was initially developed by Linus Torvalds in 1991 as an operating system for IBM compatible personal computers based on the Intel 80386 microprocessor, Linux today is available for many embedded systems with others various microprocessors Over the years, Linus and many coordinating developers around the world have worked to make Linux available on other architectures, including Intel Xscale, Alpha, SPARC, Motorola MC680x0, PowerPC, and IBM System/390 [1] Linux is a versatile and cost effective operating system for embedded systems It can be embedded in a surprisingly small system to handle simple tasks and scaled up to handle more complex tasks Linux can run on most microprocessors with a wide range of peripherals and has a ready inventory of off the shelf applications Development cycles are shortened through the use of mature tools, open source code, substantial documentation and available support services Due to all those advantages, Linux is a good choice for embedded systems [2] 2.2 Real-time embedded Linux OS Since the evaluation system needs to be process all analyzing signals at strict time, we need a real-time operating system for the processor Linux, with the real time extension, provides this kind of precise control over interrupt handling Essentially, there is an interrupt manager that handles all interrupts It does a good job of making sure that critical interrupts get run when needed The hardness of this approach depends mostly on the CPU interrupt structure and context switch hardware support This approach is sufficient for a large range of real time requirements Even without the real time extensions, Linux is pretty good at keeping up with multiple streams of events [6] [7] 2.3 The survey of real-time operating systems found in the market This section describes theoretical study of real-time operating system in the market The study is focused on their scheduling policies and how these policies are related to the microprocessor-specific hardware In particular, scheduler of Linux is presented to understand its operating manner before the software and hardware structure is developed for testing the performance of Linux running on PXA255 processor Nowadays, there are many real-time operating systems developed by different companies such as RTEM by OAR (On-Line Applications Research-Corporation), uCLinux by Arcturus Networks Inc., Timesys Linux by TimeSys Corporation, RTLinux by FSMLabs (Finite Machine Labs) and RTAI by Dipartimento di … This survey provides some general background features about scheduling of real-time operating systems in the market [2] 2.3.1 RTEM RTEMS, Real-time Executive for Multiprocessor System, is a real-time kernel which supports a high performance environment for embedded application RTEM provides four mechanisms which allow the user to impact the task scheduling process: user-selectable task priority level, task preemption control, task times-slicing control, and manual round-robin selection RTEM also provides facilities to manage the execution of periodic task This manager uses Rate Monotonic Algorithms 2.3.2 uCLinux uClinux’s stands for “microcontroller Linux.” This is a derivative of the Linux kernel originally intended for operation on microcontrollers The variations from Linux are: No fork() A process can’t fork and have both parent and child in the runnable state Instead it can vfork(), which blocks the parent until the child either exec()s another executable or exits, whereupon the parent returns to runnable No malloc() Processes allocate memory to themselves via mmap() This requires a bit of care in handling pointers, but isn’t really an issue since the Linux ports of most C libraries implement malloc() via mmap() anyway, since it’s faster No memory protection Applications have unhindered access to the entirety of system RAM, including kernel space 2.3.3 TimeSys Linux The highlights of TimeSys linux are as follows: The TimeSys Linux kernel is fully preemptable A scheduling decision is made whenever an interrupt occurs, regardless of whether the processor was executing within the kernel or user space at the time of the interrupt Interrupt Handlers are run as real-time threads in TimeSys Linux This provides a prioritized, preemptable interrupt hierarchy SoftIRQ processing, which executes interrupt bottom halves and various other kernel functions, is performed by a real-time thread The TimeSys Linux scheduler makes a scheduling decision in constant time 2.3.4 RTLinux RTLinux is one of the best-known real-time Linux implementations; the development was sponsored by Finite State Machine Labs (FSMLabs) RTLinux claims genuine hard real-time, via an RT kernel which runs the real-time processes, and runs the Linux kernel as its least-priority thread Processes not requiring real-time response are written as standard Linux apps Real-time processes are implemented either as real-time threads or interrupt handlers, and are controlled by the RT kernel The RTLinux project claims a typical 15usec response time to an interrupt assertion on the x86 platforms, with an admonishment to compare the worst-case response time (less than 30ms on a 486-33, “near the hardware limit”) Figure 6-2: Histogram of jitter of the periodic task using FIQ Table 6-4: Statistic data of jitter of the periodic task using FIQ Type of Load Mean Max Min Standard Sample size Deviation (us) (us) (us) (us) No load 6.29 10 1.1 1000 Background 5.55 11 1.13 1000 Calculation 7.52 12 1.21 1000 Disk 7.09 13 1.27 1000 Network 6.64 12 1.2 1000 Full 8.55 14 1.59 1000 The result shows that the standard deviation values not change so much when the load of system increases The maximum value on jitter is 14 microsecond This value is much less than one from the previous experiment that used sleep function The method using FIQ improved significantly jitter value on the target system 41 6.2.3 External interrupt using FIQ 6.2.3.1 Implementation To demonstrate an external interrupt using FIQ, GPIO FIQ is implemented GPIO58 is chosen for interrupt source When the processor detects a rising edge on GPIO58, the FIQ handler will be executed The handle function generates a pulse on GPIO11, so that the Interrupt Test System can measure interrupt latency of the FIQ interrupt The value of FIQ latency is the time interval between the rising edge of GPIO58 and the rising edge of GPIO11 output The procedure for enabling GPIO FIQ is the code below: set_irq_type(IRQ_GPIO58, IRQT_RISING); GPDR1 &= ~0x04000000; //set GPIO58 as input ICLR |= 0x00000400; //set GPIO IRQ to FIQ level In order to register an interrupt handler, set_fiq_handler and set_fiq_regs functions are used set_fiq_handler(fiqhandler_start, fiqhandler_length); set_fiq_regs(&FIQ_regs); enable_fiq(IRQ_GPIO58); The handler function is written by Assembly language and is located in program memory from FIQ vector address .text global fiq_handler_end ENTRY(fiq_handler_start) loop: str ip, [r10, #0x18] ldr r9, =0x000000050 mov r13,#1 str r8, [r10, #0x4c] @ clear status bit GEDR1 sub r9, r9,r13 @ Delay cmp r9,#0 bne loop str ip, [r10, #0x24] subs pc, lr, #4 @ GPSR0 = GPIO_GPIO11,set GPIO11 @ GPCR0 = GPIO_GPIO11,clear GPIO11 @ return from FIQ fiq_handler_end: 6.2.3.2 Test result The result of FIQ latency testing is shown in histograms below 42 Figure 6-3: Histograms of Interrupt latency using FIQ Table 6-5: Statistic data of Interrupt latency using FIQ Type of Load Mean Max Min (us) (us) (us) No load 2.01 0.02 1000 Background 2.02 0.03 1000 Calculation 2.01 0.02 1000 43 Standard Sample size Deviation (us) Disk 2.62 10 0.93 1000 Network 2.14 0.25 1000 Full 4.31 12 1.04 1000 According to the test result, we can recognize that the FIQ latency is much less than the IRQ latency from the previous test The worst case when the system has full load is 12 microsecond interrupt latency Almost all of real-time systems accept this latency 44 CHAPTER CONCLUSION AND FUTURE WORK 7.1 Summary This thesis has presented a method of using FPGA-based emulation to evaluate the real-time performance of an embedded system with Linux OS A number of tests were designed and implemented in chapter and chapter Two parameters of real-time behavior jitter and interrupt latency have been measured and evaluated Jitter is relative to scheduling precision of en operating system Jitter is a measurement of how well the scheduler is able to maintain a desired fixed timeperiod for a task Normal technique to implement a periodic task on Linux system is using sleep function But the actual sleep time is not the same as requested The result of jitter test shows that the load of system affects remarkably the precision of scheduler The maximum value of jitter goes up from 184 microseconds to approximate 9.8 milliseconds when the load increases The second parameter relative to the response of the system is interrupt latency It is also a very important measurement in a real-time system because it affects many other performance aspects such as scheduling precision and the interrupt task latency The worst case interrupt latency yields for example a lower boundary for the worst case interrupt task latency The most interesting tests are those with load as the worst latencies show up under different levels of load The maximum value of interrupt latency in full load case goes up 400 microseconds The statistic data from the test show that the latency is affected much by other processes with interrupt request for peripheral devices Throughout this evaluation, users can estimate how well the time-sensitive applications run on the target system The maximum values of latency are investigated and analyzed to ensure time constrains of applications Some solutions have been proposed to reduce the latency in chapter By utilizing the advanced features of ARM architecture, real-time performance of the embedded system can be improved significantly Two these features are OS timer and Fast Interrupt Request (FIQ) OS timer is an integral part of any processor and Linux need the timer to control the system time and schedule processes PXA255 processor has four timers that can be utilized for periodic tasks Result of timer test shows that jitter can be reduced remarkably by OS timer The maximum value of jitter in worst case is about 300 microseconds It is good performance for applications that not require very strict real-time The next feature that can improve the real-time performance is FIQ This technique is for fast, low-latency interrupt handling, and interrupt request (IRQ) is for general interrupts The interrupt latency is also reduced for FIQ interrupts because the extra banked registers may be used to maximum efficiency by preventing the need for a context save The FIQ exception shall disable all IRQs and FIQs and the handler should be hand crafted to execute as quickly as possible The best performance of FIQ is proved by the test The interrupt latency measures 12 microseconds It is near the hardware limit, and is a very good result for hard real-time applications 45 7.2 Future work After evaluation of embedded system has been done, some improvements have been proposed to gain better real-timer performance of the target system with OS Linux All these methods depend on the hardware of ARM architecture Real-time applications running on the target system can meet their requirement by using some special techniques relating hardware control But, actually Linux kernel on the target system is still original standard Linux and generally is not dedicated for real-time systems In the market, there are many Linux OS products for this kind of systems; however, until now there is no real-time Linux product for Intel PXA255 embedded processor In the future, in order to apply Linux OS into real-time application effectively, the Linux kernel needs to be modified, adding some more real-time functions about the timer, interrupt, scheduling… Test conditions also need to be extended for various levels of load 46 References [1] John Lombardo, 2001, “Embedded Linux”, New Riders [2] Blue Mug, Inc., December 2002, “Embedded Linux Survey”, www.bluemug.com [3] Intel PXA255 and PXA210 application Processors, 2003, “Developer’s Manual”, http://www.intel.com [4] Daniel P Bovet and Marco Cesati, 2000, “Understanding the Linux Kernel”, O’Reilly [5] Tigran Aivazian, 2001, “Linux Kernel Internal”, http://www.moses.uklinux.net/patches/lki.sgml [6] Bill Weinberg, MontaVista Software Inc., 2001, “Embedded Linux – Ready for realtime”, white paper, http://www.mvista.com [7] Bill Weinberg, MontaVista Software Inc., 2001, “Moving from a Proprietary RTOS To Embedded Linux”, white paper, http://www.mvista.com [8] Blue Mug Inc., 2002, “Embedded Linux Performance Study”, http://www.bluemug.com [9] Marco A Sanvido, Vaclav Cechticky Walter Schaufelberger, “Testing embedded control systems using hardware-in-the-loop simulation and temporal logic”, 15th Triennial World Congress, Barcelona, Spain [10] Andrei V Gurtov, 1999, “Technical Issues of Real-Time Network Simulation in Linux”, FDPW99 Volume 2, University of Helsinki [11] Steve Babin, March 2003, “Diagnostics for design validation”, article at http://www.embedded.com/story/OEG20030325S0033 [12] Luca Abeni, Ashvin Goel, Charles Krasic, Jim Snow, Jonathan Walpole, 2002, “A measurement-base analysis of real-time performance of Linux”, scientific report at Oregon Graduate Institute, Portland [13] Nicholas Mc Guire, 2001, “MiniRTL - Hard Real Time Linux for Embedded Systems”, FSMLab [14] Clark Williams, Red Hat, Inc., March 2002, “Linux Scheduler Latency”, www.linuxdevices.com/articles/AT8906594941.html [15] Neil Matthew and Richard Stones, 1997, “Linux Programming”, Wrox Press [16] Alessandro Rubini and Jonatthan Corbet, “Linux Device Driver”, O’Reilly [17] ARM Limited, 1999 , “ARM Architecture Reference Manual”, www.arm.com [18] Douglas L Perry, 1999, “VHDL”, McGraw Hill [19] Stephen Brown and Zvonko Vranesic, 2000, “Fundamentals of Digital Logic with VHDL design”, McGraw Hill 47 Appendix A: Pictures of experiment hardware Experiments are implemented in MicroFuzzy Company GmbH, Munich, Germany Experiment hardware including Triton PXA255 board and Xilinx Virtex board 48 Appendix B: Test program files Load program Calculation Load #include main() { double a,b,c,f; a =1.12345; b=2.23456; c=5.23461; printf("starting to calculate \n"); while(1) { f=a*b/c; //printf("c = %f\n",c); } } Disk load cd /proc while true cat cmdline cat cpuinfo cat devices cat apm cat execdomains cat filesystems cat interrupts cat iomem cat ioports cat ksyms cat loadavg cat locks cat meminfo cat misc cat modules cat mtd cat partitions cat stat cat swaps cat uptime cat version done Disk load while true ls –lR / done Disk load while true cp diskload3 temp 49 rm temp done Network Load Ping –f 192.168.0.21 Full Load Calload & Diskload & Netload & Test program Jitter test #include #include #include #include #include #include #include #include #include #include #include #include #define FATAL { fprintf(stderr, "Error at line %d, file %s (%d) [%s]\n", \ LINE , FILE , errno, strerror(errno)); exit(1); } while(0) #define MAP_SIZE 4096UL #define MAP_MASK (MAP_SIZE - 1) int main() { int fd,i; void *map_base, *virt_addr; unsigned long read_result, writeval; off_t target; if((fd = open("/dev/mem", O_RDWR | O_SYNC)) == -1) FATAL; printf("/dev/mem opened.\n"); fflush(stdout); target=(unsigned long)GPSR0; map_base = mmap(0, MAP_SIZE, PROT_READ | PROT_WRITE, MAP_SHARED, fd, target & ~MAP_MASK); if(map_base == (void *) -1) FATAL; fflush(stdout); //LED start to twinkle while(1) { virt_addr = map_base + (GPCR0 & MAP_MASK); *((unsigned long *) virt_addr) |= 0x800; usleep(20000); // virt_addr = map_base + (GPSR0 & MAP_MASK); *((unsigned long *) virt_addr) |= 0x800; 50 usleep(20000); } if(munmap(map_base, MAP_SIZE) == -1) FATAL; close(fd); return 0; } Interrupt latency test #define KERNEL #define MODULE #include #include #include #include #include #include #include #define irq_in_pin 25 #define irq_out_pin 11 #define IRQ GPIO_2_80_TO_IRQ(irq_in_pin) static void test_irq(int irq, void *dev_id, struct pt_regs *regs) { unsigned long i; gpio_set_bit(irq_out_pin); //printk("interrupt # %d \n ", irq_in_pin); for (i=0;i