More than 300 million mobile devices today have Graphical User Interfaces (GUIs) based on Adobe Flash, and that number is projected to exceed a billion by 2010, according to Adobe Systems. Those who design embedded systems for the industrial, medical, and automotive industries are beginning to embrace Flash for one simple reason: it can reduce GUI development time by up to 50 percent. In the past, software teams had to translate their GUI prototypes into C, C++, or Java code, a laborious process that took many weeks or months. Now, teams can design and build GUI components with high-level Flash tools and run those components directly on embedded Flash players without writing graphics code.

Flash is gaining momentum among embedded developers for several reasons, such as:

• More than a million graphics designers worldwide use Flash authoring tools, providing an immense pool of graphics expertise that developers can draw upon. Moreover, developers can easily integrate thousands of Flash components already built for the desktop and handset markets.
• Compared to desktop Flash players, embedded Flash players from Adobe (for instance, Flash Lite 3) use less memory and provide faster rendering with less CPU overhead.
• CPUs and graphics chips for embedded designs now support the frame rates needed for a pleasing Flash experience on VGA and larger displays. For instance, to achieve a smooth 10 frames per second animation, a system needs a CPU with a minimum of 100 MIPS, well below the typical 300-plus MIPS offered by most embedded CPUs today.

To migrate to Flash, embedded developers can choose from an array of tools, many of which they already use. For instance, developers can use CAD tools and desktop word processors to generate Flash content and employ conversion utilities that translate a variety of presentation formats directly into Flash format. Developers also can leverage Flash components that integrate Flash content with ActiveX controls. This breadth of support for Flash content creation and screen control makes for a simple transition to Flash interfaces.

Endless possibilities

Unlike general-purpose programming languages and tools, Flash provides a domain-specific environment for graphics and multimedia, offering almost endless possibilities for building user experiences. Using Flash, developers can create animations and special effects in a fraction of the time traditionally required. Also, Adobe Flash player certification ensures that Flash-based applications work the same across hardware platforms. As a result, developers can create GUI components once, then deploy them on a range of systems that target different markets or price points.

Nonetheless, to satisfy all the requirements of an embedded system, a Flash implementation must address several issues as to how developers can:

• Combine Flash content with other graphics programs such as Web browsers or 3D visualization applications. Can a single graphics display render both Flash-based and native 2D/3D graphics simultaneously even though they use different draw models?
• Make a Flash-based user interface perform consistently under all load conditions. The GUIs for most embedded systems must respond quickly to user input at all times, which requires a level of priority control and real-time performance.
• Make a Flash-based user interface reliable. Can the system monitor for user interface failures and recover from them gracefully? Can the Flash content coexist reliably with critical processes?
• Control how Flash content interacts with Operating System (OS) services, such as audio output, touch screens, time-critical device drivers, file systems, and networking stacks.

Integrating Flash programs with other graphical applications

Traditionally, a Flash player runs within a Web browser or is launched from a windowing system. However, GUI development can be greatly simplified by turning this model on its head and making Flash the main environment that launches all graphical applications, regardless of whether they are based on Flash. Flash excels as a screen manager, allowing the GUI designer to intimately control menu transitions and audio effects; it also simplifies customization by allowing designers and developers to freely position, resize, and configure graphical components.

Figure 1 shows an example of using Flash as a screen manager. The program on the left draws 3D visualizations in OpenGL ES, a standards-based 3D API for embedded systems. The program has loaded three components directly into its application space: a 2D graphics library, the OpenGL ES 3D library, and a graphics driver that controls the graphics hardware. Loading the driver in this way allows the program to control the graphics chip directly and thereby increase performance. The program on the right is a Flash player. Like the OpenGL ES program, it also directly controls the graphics hardware, ensuring high performance.

(click graphic to zoom by 2.2x)

Many graphics chips for embedded systems now support multilayering, which allows Flash-based programs to blend seamlessly with other graphics applications on the same display. In Figure 1, the Flash player draws on a foreground layer and controls the drawing of the 3D visualization on a background layer. To make the 3D canvas visible, the developer used a chroma key technique on the foreground layer. Because the 3D rendering and Flash rendering take place on independent layers, the graphics controller can refresh the 3D visualization without redrawing Flash content, thus eliminating flicker and reducing the load on the CPU.

Developers also can use techniques such as alpha blending and chroma key to make menu buttons and other Flash components semitransparent and then place the semitransparent components directly over other content. Figure 2, which shows a semitransparent warning displayed over an animated control console, demonstrates how this approach can help pack more information onto a small screen.

(click graphic to zoom by 2.1x)

Ensuring predictable response times

An embedded GUI should respond promptly to user commands at all times, even when the system is running CPU-intensive tasks. There is no room for a desktop user experience. The challenge becomes more complicated when multiple graphics programs compete for CPU time. One solution is to create a central display manager that uses thread priorities to determine when each graphics program gets control of the CPU.

In this approach, a program (for instance, a Flash-based player for displaying documentation videos) that needs to join the graphics environment sends the display manager a request. The manager responds with a yes or no, depending on whether the program has sufficient permissions to join. Upon joining, the program gains access to a mutual exclusion lock (mutex).

When the program wants to draw something to the screen, it will wait on the mutex, acquire it, draw directly to the graphics chip, and then release the mutex. Every graphical program competes for this mutex based on its individual priority. Because the highest-priority graphics program will always acquire the mutex first, this approach ensures a level of real-time performance and a consistently fast user experience.

Managing failure conditions

To prevent system downtime, many embedded systems require some level of dynamic fault recovery. Using fault-notification mechanisms provided by the underlying OS, the aforementioned display manager can learn about graphical applications that fail.

If a Flash-based or graphical program fails while holding the mutex, the display manager can release the mutex and give it to the next program in the priority queue. The manager also can recover any resources the failed program used and restart the program.

Interacting with Flash

To integrate Flash successfully, developers must manage two types of interactions:

• Using Flash content to launch and control other Flash content, which requires a mechanism that lets a master Flash application load, position, run, and release secondary Flash applications
• Enabling communication between Flash content and OS services, which requires a mechanism for relaying data requests

Figure 3 provides an example of how to implement these interactions. An HMI engine allows a master Flash application to manage secondary Flash applications (data point viewer, system configuration manager, and more) while an HMI server provides a gateway to native OS services.

(click graphic to zoom by 2.0x)

Consider how this design can work in a video player model. As depicted in Figure 4, a user interface consists of the following:

• A viewing area that displays the video (top half of the screen)
• A Flash video player that provides controls for play, next, previous, and so on (middle of the screen)
• An HMI engine written in Flash that controls the main menu buttons (bottom of the screen)

(click graphic to zoom)

In this example, the HMI engine launches and controls the Flash video player (Flash controlling Flash). The video Flash player then communicates with the video playback OS service through the HMI server (Flash content communicating with native OS services).

The HMI server provides asynchronous communications to OS services. This approach prevents the HMI engine, which runs in the context of the Flash player, from potentially causing the Flash-based GUI to stop accepting input or drawing to the screen.

Moving Flash forward in embedded

By combining high-level Flash tools with an appropriate software architecture, embedded teams can leverage Adobe Flash’s power and time-to-market advantages while preserving real-time performance and reliability. Moreover, developers can employ component-based HMI frameworks that seamlessly blend 2D/3D applications, Flash applications, and multimedia.

Paul Leroux is a technology analyst at Ottawa, Canada-based QNX Software Systems, with interests in usability, high availability design, multiprocessing systems, and OS kernel architecture.

Bill Graham is a senior product line manager at QNX Software Systems. He has more than 18 years of experience in the software industry, including embedded and real-time systems development, UML modeling, and object-oriented design. Prior to joining QNX, Bill held software development, consulting, and product management positions at IBM, Rational, Klocwork, and ObjecTime. Bill holds BS and MS degrees in Electrical Engineering from Carleton University in Ottawa, Canada.

QNX Software Systems
613-591-0931
paull@qnx.com
bgraham@qnx.com
www.qnx.com