Nicolescu/Model-Based Design for Embedded Systems 67842_C012 Finals Page 366 2009-10-1 366 Model-Based Design for Embedded Systems PLBv46 Slave PLBv46 master burst Slave buffer interface LocalLink write buffer LocalLink read buffer Reset logic Interrupt generation Bridge control logic Bridge status signals Interrupt request Reset Control bus Control bus Control bus DCR slave PLB bus PLB bus DCR bus Reconfigured region Bus macro enable Reconfigurable socket Reset request FIGURE 12.7 Reconfigurable socket abstraction based on the “PLBv46 PLBv46 bridge” architecture. The “PLBv46 slave” and “PLBv46 master burst” blocks are stan- dard IP components and all blocks except the DCR slave block are part of the bridge. Bus macros are implicitly present on all signals crossing the bound- ary of the reconfigured region. An alternative is to architect the interface around a bus bridge, with inde- pendent busses in the static region and in the reconfigurable region. The design of the socket is based on partitioning the Xilinx “PLBv46 PLBv46 bridge” IP [23], as shown in the block diagram in Figure 12.7. Internally this core is based around 32-bit fixed-width data FIFOs and a small number of control signals. Most of the bridge is treated as part of the static region, with only a small amount of logic required in the reconfigurable region to com- plete the bridge. In addition to the bus interface, which is primarily used to interface to the reconfigured region, the socket core also contains a control interface (based on the DCR protocol [7]) which is used to generate an inde- pendent reset signal to the reconfigurable region and to force signals driven by the reconfigurable module to stable values during reconfiguration. 12.5.3 Direct Memory Access Interfaces The bus interface above is a generic and flexible interface, which can be used to communicate with the reconfigured portion of the system in different ways. For instance, it may be used by the processor to both send and receive data from the reconfigured region or as a control interface to set parame- ter values of IP cores executing in the reconfigured region. However, it does have several disadvantages. Primarily, the bandwidth of data to or from the Nicolescu/Model-Based Design for Embedded Systems 67842_C012 Finals Page 367 2009-10-1 FPGA Platforms for Embedded Systems 367 processor is limited because of the overhead of bus arbitration and the fact that the memory range is treated as uncached I/O transactions. Although performance could be improved somewhat for large transactions by using DMA engines or treating data transfer regions as cached and manually man- aging cache coherency, this would significantly increase the complexity of the processor software. Secondly, many FPGA algorithms require access to external memory for buffering data until it can be processed. For instance, in a network router, packet data may need to be stored until a routing decision can be made, or in a streaming video system, several frames of video data may need to be stored to analyze object motion between frames. Because of these limitations, it is best to consider the bus interface above as primarily an interface used for low-bandwidth control and configuration information. In systems that require higher bandwidth communication, or direct access to external memory, the control interface can be augmented with additional interfaces to memory. Although it may seem straightforward to include a complementary bus bridge that can be driven by the reconfig- ured region to provide this functionality, this tends not to be the highest bandwidth option since performance can be limited by the arbitration logic of the PLB bus. This logic is heavily pipelined in order to maximize the bus throughput under a wide variety of usage, typically incurring three cycles of latency before a slave can respond to a bus access. One solution is to provide an interface connected directly to the native port interface (NPI) of the Xilinx MPMC IP core, as shown in Figure 12.8. External memory (e.g., DDR/DDR2) Arbiter Multiported memory controller Physical interface FIFO FIFO PIM PIM FIGURE 12.8 Architecture of the Xilinx MPMC. Nicolescu/Model-Based Design for Embedded Systems 67842_C012 Finals Page 368 2009-10-1 368 Model-Based Design for Embedded Systems Typically, this interface exhibits both lower latency and higher bandwidth than the PLB bus. Although the MPMC must still arbitrate between different ports attempting to use the memory controller, this arbitration can be per- formed locally within the memory controller and concurrently with the data being provided. The only disadvantage of connecting directly to the mem- ory controller is that other IP cores in the static region cannot be accessed from the reconfigured region. However, since in the SRP usage model these IP cores are likely being managed by device drivers in the operating system of the processor, it is questionable whether such access should be allowed anyway. 12.5.4 External Interfaces In addition to communicating with the static region, a reconfigurable mod- ule may also communicate with other interfaces external to the FPGA. In order to accomplish this, a reconfigurable region may include external I/O pins and/or high-speed serial transceivers. For the most part, these resources can be treated as any other FPGA primitives and can be placed and routed as usual. However, there is some complexity with regard to external I/O pins, since in many FPGA designs, the input/output buffer (IOB) primitives rep- resenting external I/O pins are not explicitly instantiated in a user design but are inferred in the synthesis process. Normally in a hierarchical design, the netlist can be synthesized using a special option to disable inference of these primitives, since they will be inferred or instantiated during synthesis of the toplevel design. However, when building a generic FPGA platform, relying on this may not be desirable, since the reconfigured region may require more control over the configuration of these primitives. In other cases, exactly which IOB primitives are explicitly instantiated in a reconfigurable module and which ones are not may not be known when the static design is synthe- sized and implemented. One way to solve this is to not expose any I/O pins of the reconfigurable region as external signals of the static region, implying that synthesis of the static design will never include IOB primitives for these pins. When a reconfigurable module is synthesized, signals interfacing with the static region are individually tagged with the constraint BUFFER_TYPE set to NONE, indicating that no IOB primitives should be inferred for those signals. High-speed serial transceivers also have additional design complexity, since each transceiver is associated with specialized clock resources in the FPGA. These clock resources typically include phase-locked loops for clock synchronization and dedicated clock distribution paths and may be shared between transceivers. From the perspective of building FPGA platforms, this resource sharing combined with how transceivers are grouped into configu- ration frames may need to be considered during the floorplanning stage in order to gain maximum usage of the available transceivers. Nicolescu/Model-Based Design for Embedded Systems 67842_C012 Finals Page 369 2009-10-1 FPGA Platforms for Embedded Systems 369 Static design flow EDK base system builder + hand design EDK BSP generator Linux design flow .elf EDK platgen .mhs .ucf Floor- planning .ngc .dts PR-enabled NGDBuild, Map, and PAR PR-enabled bitgen EDK genace.tcl static.ace .bit .ncd static.used UCF merge .ucf EDK platgen .mhs EDK hand design Module design flow Hand design .ucf .ngc PR-enabled NGDBuild, Map, and PAR .ncd PRMergeDesign + PR-enabled bitgen Meta-information C code gcc + objcopy EDK genace.tcl partial.bit .bit merged.ace configure.elf .ucf FIGURE 12.9 Design flow for PR systems based on EDK. 12.5.5 Implementation Flow The implementation flow for the system is shown in Figure 12.9. The static design is implemented first, as shown in the left-hand side of the figure, using the EA PR tools. During this sequence, no netlist for the reconfigurable region is present, and the place and route tools only implement logic for the static region. Design constraints are provided in a .ucf file and must include the required floorplanning constraints for the PR flow. After routing is com- pleted, the routing resources used by the static logic are saved in the file static.used for later use. Since by default the interface with the reconfigured region is driven to an idle state, the resulting bitstream can be used in a sys- tem without programming the remainder of the FPGA. The device tree for a particular design is generated from the EDK design, and after being con- verted to a binary device tree blob, can be included in the Linux kernel image, or stored as the initial value of a BRAM in the bitstream. Lastly, EDK is used to package the FPGA bitstream with the Linux kernel binary in a bootable image that can be used with Xilinx SystemAce [24] to boot the kernel. The right-hand side of Figure 12.9 shows a second pass for the imple- mentation of a reconfigurable module. During this pass, the logic of the Nicolescu/Model-Based Design for Embedded Systems 67842_C012 Finals Page 370 2009-10-1 370 Model-Based Design for Embedded Systems reconfigurable module is implemented together with a small portion of the static logic called the “context logic.” The context logic is necessary to pro- vide the context of the reconfigurable module, so that hierarchical names in the design and location constraints for clock signals and bus macros can be preserved. The design constraints for implementation are created by merging the design constraints from the static design with any additional design constraints specific to the reconfigurable module, such as pin loca- tion constraints. During this pass, the routing resources in the file static.used are excluded from use, since these resources are already used in the static design. The final bitstream for the reconfigurable module is generated by first merging the design database (contained in an .ncd file) from both passes, ensuring that the configuration bits used in the static design are pro- grammed correctly. In addition, design rule checks and timing analysis can be applied to the merged design database, to ensure that individual passes were implemented correctly. From the merged design database, it is possi- ble to generate both a partial bitstream that can be used after configuration with the static bitstream and a merged bitstream which can be used as an ini- tial configuration bitstream, with the reconfigurable module already loaded. To enable reconfiguration in a Linux system, the partial bitstream is encapsu- lated with the Linux code for performing PR and the meta-information about the reconfigurable module, to generate a Linux executable, as described in Section 12.6. 12.6 Managing Partial Reconfiguration in Linux Two device drivers are used to manage the reconfiguration process. Primar- ily, the device driver for the ICAP device performs the actual reconfigura- tion. When a partial bitstream is written to this device (for instance, using the cp command or the write() system call), the bytes are transferred to the ICAP. Since the device driver does not inspect or modify the stream of bytes, the data being written must include the appropriate control words, as expected by the configuration interface [26]. The device driver also includes simple locking of the ICAP resource, in order to prevent different processes from unexpectedly interleaving accesses to the ICAP. Readback is also possi- ble using this device driver by writing the correct readback request bitstream to the ICAP and subsequently reading data (using the read() system call). The second device driver used to manage reconfiguration is associated with the reconfigurable socket core. This driver exports a character interface to which meta-information about a reconfigurable module can be written. A simple way of representing this meta-information is in the form of an array of struct platform_device, a data structure which is used internally by Linux to represent devices. A more complex, but perhaps more robust Nicolescu/Model-Based Design for Embedded Systems 67842_C012 Finals Page 371 2009-10-1 FPGA Platforms for Embedded Systems 371 Reconfigure FPGA Notify kernel of devices Load kernel modules Enable bus macros Reset reconfigurable module Processing Unload kernel modules Release devices Disable bus macros FIGURE 12.10 The reconfiguration process. representation of meta-information could be an additional device tree blob. This meta-information is parsed and checksummed and, if valid, is used to notify the Linux kernel of the presence of new devices, which can then be bound to other device drivers. An invalid checksum is interpreted as an indi- cation to unbind any previously loaded devices and release ownership of the reconfigured region. Secondarily, this device driver also enables and disables the bus macros between the static region and the reconfigured region, and controls the reset of the reconfigured region. As with the ICAP device driver, the socket device driver includes a simple locking mechanism in order to prevent a process from unexpectedly reconfiguring an active region in use by another process. The complete process of reconfiguration is shown in Figure 12.10. In the initial state, we assume no module is loaded in the reconfigured region. Next, a reconfigurable module is loaded into the FPGA through the ICAP device driver. Next, meta-information about the reconfigurable module issent to the socket device driver, which registers the presence of any new devices, resets the newly loaded module, and enables the interface between the static region and the reconfigurable module. At this point, although Linux is aware of the presence of the reconfigured devices, it may not have device drivers appro- priate to those devices. Next, device drivers for new devices are provided by loading the appropriate kernel modules and the Linux kernel binds those device drivers to the reconfigured devices. At this point, application code may use the device drivers to communicate with the reconfigured region. A similar sequence of steps in reverse order occurs to unbind the device drivers and release the reconfigured region so that different processing may occur. Since the ICAP device and the control interface of the socket are exposed through device drivers, it is relatively straightforward to implement recon- figuration through a regular user process. One possibility for implementing Nicolescu/Model-Based Design for Embedded Systems 67842_C012 Finals Page 372 2009-10-1 372 Model-Based Design for Embedded Systems this involves linking the bitstream and meta-information into a single exe- cutable along with the code for reconfiguration. The process created when this executable is executed can be controlled through any operating system mechanism (such as POSIX signals) to manage the life cycle of the module loaded in the FPGA. The executable can also be linked together with other application code, resulting in a familiar processor-centric usage model for the FPGA fabric. This approach is similar in spirit, but greatly different in implementation from that proposed in [18], which performs essentially the same processes using the Linux kernel’s ability to implement new executable formats. It is important to recognize that although the reconfiguration process is managed by a user process, it must be treated as a privileged opera- tion executed as the root user, since there are many places where both unintended errors and malicious attacks may result in unintended behav- ior. Some of these places are not specific to the PR process, such as loading kernel modules, whereas others are more subtle vulnerabilities. For instance, as noted before, partial bitstreams have significant constraints on how they are constructed and are specific to a particular implementation of the static system. More directly, it is possible to trigger reconfiguration of the FPGA through the ICAP interface, resulting in the loss of the current state of the system. If the bus macros are enabled during PR, then it is likely that glitch- ing on the interface signals will result in unintended behavior of the static system. One particularly common usage error is simply attempting to load a par- tial bitstream that does not correspond to the current implementation of the static design. This may happen during development when a modification is made to the static region, but a designer neglects to reimplement a recon- figured module. One way of avoiding such errors is to prepend each partial bitstream with a hash generated from the static design. This hash can also be stored in the static design, possibly in the device tree blob, and checked before being loaded into the FPGA. If the partial bitstream is not signed properly, then the reconfiguration process can be halted without affecting the operation of the static design. This technique can be simply applied to prevent unintended errors, or adapted using more cryptographically secure techniques to prevent malicious attacks [2,4]. 12.7 Putting It All Together This section illustrates a SRP design targeted at a variant of the WARP Software-defined Radio hardware built by Rice University [12]. Since the original hardware is based on an older Virtex 2 Pro FPGA, we present a design based on an updated Virtex 4 FX 100 device in order to better Nicolescu/Model-Based Design for Embedded Systems 67842_C012 Finals Page 373 2009-10-1 FPGA Platforms for Embedded Systems 373 PPC405 (ppc_virtex4 v2.00.b) Interrupt controller (xps_intc v1.00a) Multiported memory controller (mpmc v3.00b) Ethernet MAC (xps_ll_temac v1.01a) plb plb sdma Reconfigurable socket RS232 Uart (xps_uartlite v1.00a) ICAP Interface (opb_hwicap v1.00b) (using bridge) BRAM interface (xps_bram_if_cntlr v1.00a) npi dcr (using bridge) PLB bus Reconfigured region FIGURE 12.11 Architecture of a reconfigurable platform. Some signals and standard cores have not been shown. represent the PR capabilities of newer FPGA architectures. In particular, we focus on a MIMO OFDM reference design for this board, which implements a bridge from Wired Ethernet to a two-radio MIMO system. The design uses a processor to manage the packet headers and to perform configuration man- agement of the radios, while packet payloads are communicated directly between the wired and wireless network interfaces using direct memory access to a processor-managed memory buffer. In the reference design, the packed payload buffer is implemened in BRAM and communicated through a PLB bus. In the reconfigurable design, we assume that the packet payload buffer is implemented in external DRAM, which must be accessed from the reconfigurable region through a separate port of the memory controller. As a nonreconfigurable system, this design uses approximately 50% of the device (21294 of 42176 slices). The design of the static subsystem is shown in Figure 12.11. This design is architected around the PowerPC 405 processor core and was largely gener- ated using the Base System Builder capability in Xilinx EDK. Standard serial port and ethernet IP cores provide external connectivity. Access to external 64 bit wide DDR2 SDRAM, including DMA access for the ethernet core, is provided by the Xilinx MPMC IP core. In this system, the processor, memory bus, and memory controller are designed to be “quasi-synchronous,” mean- ing that clocks must be edge-aligned. Based on the speeds of the individ- ual components, a design point was chosen targeting a slow speed grade FPGA (−10) with the memory bus clocked at 83.3 MHz, the memory con- troller clocked twice as fast (166.6 MHz), and the processor clocked three times as fast (250 MHz). Nicolescu/Model-Based Design for Embedded Systems 67842_C012 Finals Page 374 2009-10-1 374 Model-Based Design for Embedded Systems Reconfigured region ICAP interface Control interface bus macros Memory interface bus macros Utilized powerPC core Static region FIGURE 12.12 Placed and routed design of an FPGA processor platform, targeting a Virtex 4 FX 100. The FPGA layout of the design is shown in Figure 12.12, overlaid with the PR floorplanning constraints. The static region is at the south of the chip, and is exactly two configuration frames tall. This layout provides approximately 8600 slices and 128 external I/O pins, which accommodates both the logic Nicolescu/Model-Based Design for Embedded Systems 67842_C012 Finals Page 375 2009-10-1 FPGA Platforms for Embedded Systems 375 requirements of a simple processor design, and the I/O pins requirements of a 64-bit DDR2 memory interface. A significantly smaller region would fail to provide enough logic cells for the static design, while a larger region would allocate too many pins to the static region, which would be difficult to access from the reconfigurable region. Note that the majority of the routed signals are contained within the floor- planned area for the static region. The routes entering the top region connect primarily to external I/O pins and FPGA resources, such as clock buffers and the ICAP, located in the center column of the FPGA. Some routes into the top region also connect to the PowerPC cores. Although only one PowerPC is actually used in the static design, current versions of the EA PR tools do not allow PowerPC cores to be part of the reconfigured portion of the design. Hence, this design instantiates both PowerPC cores in the static region, in order to enable use of the JTAG chain, which is assumed to connect through both cores. The device tree for this design is shown in Figure 12.13. Since the targeted board includes Xilinx SystemACE, this is used to configure the FPGA and initialize external memory with the kernel image. The compressed device tree blob is initialized in the BRAM at address 0xfffff800 and decom- pressed by the Linux bootwrapper executing out of external memory. The root filesystem is stored on an external file server and loaded over the net- work interface using the NFS protocol. 12.8 Conclusion Although high-level algorithmic modeling offers significant promise for increasing design productivity, a common problem with many approaches is representing the environment in which a model exists in a system. A solu- tion to this problem is often to provide platforms that abstract lower level details, provide standardized interfaces, and can be targeted by a high-level design tool. Although this difficulty exists in any embedded system, it is par- ticularly apparent in FPGA systems, which include complex IP blocks, such as processor cores, and where physical interfaces to the rest of the system are highly flexible and incorporate many features that cannot be easily modeled even at the circuit and gate level. However, using the architectural features of some FPGAs, such as PR, higher level platforms can be constructed that abstract many of these details and are more appropriate for mapping from a high-level design tool. This chapter has particularly shown how this technique can abstract the complex- ities associated with including a control processor and operating system as part of an FPGA platform. . transceivers. Nicolescu /Model-Based Design for Embedded Systems 67842_C012 Finals Page 369 2009-10-1 FPGA Platforms for Embedded Systems 369 Static design flow EDK base system builder + hand design EDK BSP generator Linux design flow .elf EDK platgen .mhs .ucf Floor- planning .ngc .dts PR-enabled NGDBuild, Map,. 12.8 Architecture of the Xilinx MPMC. Nicolescu /Model-Based Design for Embedded Systems 67842_C012 Finals Page 368 2009-10-1 368 Model-Based Design for Embedded Systems Typically, this interface exhibits. Nicolescu /Model-Based Design for Embedded Systems 67842_C012 Finals Page 366 2009-10-1 366 Model-Based Design for Embedded Systems PLBv46 Slave PLBv46 master burst Slave buffer interface LocalLink write