Seven years ago I was working in a small company where we based our entire existence (and our investors’ money) on one single fact; that 2004 would be the year when hardware accelerated graphics took off in mobile phones. Every major handset vendor would ship devices with performance & features going beyond current handheld consoles. Accelerated mobile gaming, ranging from high-end 3D action to 2D casual games, was going to be the main driving factor, and would soon be followed by animated greeting cards, 3D mascots and real-time advertisements.

Six years after that important breakthrough, which, by the way, didn't quite arrive as planned, I'm now rather happy that we adjusted our direction a couple of times during the early 00's. Not just because I still have a company to work for, although that is nice, but also because the market is now finally ready for the kind of GPU capabilities we were planning for back then.

Different forms of graphics acceleration have already been around for some time, but with a disappointing lack of utilization. In some cases the GPU has mostly been a "checkbox OpenGLES 1.1 support" feature, in many others it has only been utilized by one or two builtin apps. This has sometimes been the result of restrictive vendor/operator application policies, but in other cases it has simply been because of the product quality (performance, driver stability) being too low for general use.

Of course, far from everything has been bad, and graphics acceleration has started to gain some real momentum in not just phones but also DTV, set top boxes, navigation and mobile internet devices. The change that is happening now is the result of several different factors, including the user interface evolution and the idea of internet connectivity & browsing "everywhere". Both of these drive graphics requirements by defining real use cases for GPUs rather than just having a checkbox feature. The need for deploying the same content on different platforms and porting legacy applications also means that the GPU interface is opened up and made available to an infinitely larger developer base, which again results in a lot more content (both good and.. bizarre) becoming available to everyone.

GPU requirements are constantly increasing, and we are now at the point where hardware acceleration is a must-have for a lot of applications. Whereas GPUs have previously been an image quality upgrade and a way to improve performance without weeks and months of software optimizations, there are now real use cases which are simply not possible without a GPU. As a direct consequence of this a large number of ARM’s Partners are now producing and shipping chips including Mali GPUs for use in phones, navigation, media players, internet devices, televisions and set top boxes.

At CES earlier in the month ARM and partners showed what can be achieved with Mali graphics for a variety of form factors, from ST showing their STi7108 HD set top box technology based on Mali-400 to Movial showing Qt UI for mobile applications on top of ST Ericsson’s U8500.

ARM together with a number of Partners, released several new graphics demos & applications to show off some of the capabilities of the new generation of application processors. One of these, developed by our internal graphics demo team together with Lapland Studios, was the HD user interface concept "Canvas"


(Trouble viewing this video on a mobile device? Click here to open the mobile version)

Canvas is rendered at 720p resolution with 4x multi-sampled anti-aliasing. It uses advanced shaders and 3D rendering to play with the idea of what a touch-screen UI might look like if all processing power was dedicated to this task alone. Or, as someone described it, "what a very energetic kid would want if his cell phone had a 40 inch touch screen". The platform it is running on is an ARM11 and Mali-200 based system-on-chip of the kind which goes into phones, navigation & internet devices, media players and similar products.



Canvas consists of a 3D world where the user freely can create, modify and arrange a large number of objects; including such household necessities as cubes, skulls, sofas, post-it notes, bowling pins and holiday photographs. A physics engine running on the CPU ensures that all the objects interact with each other, something that is best demonstrated by starting a game of desktop 3D bowling bowling, or by watching the ball bounce off the sofa and then crash into a tall stack of cubes, spreading them everywhere.

Everything on the 720p screen is textured, some of the objects are multi-textured and envmapped, whereas others (like the always important sofa) are per-pixel lit and bumpmapped. Mali-200s multi-threaded shader core can perform varying interpolation, a texture lookup and more than 10 programmable FLOPS in a single cycle, meaning that the frame rate is good even though objects with heavy shaders sometimes fill large areas of the HD screen. Additionally, the frame buffer blending which is used by the soft shadows is handled in dedicated hardware, meaning that the shader core is fully dedicated to the shaders supplied by the user application.

An icon-based menu pops up when touching the screen, allowing you to create new objects, modify existing ones or maybe write a note (well, finger-paint a note). Photographs can be spread out on the ground, rescaled, rotated, or replaced with dynamic "video textures" displaying movie clips while interacting with the rest of the 3D visuals.

Every object added to the scene also casts shadows using an approach which enables soft shadowing without requiring multiple render passes, needing just a single-cycle shader for each shadow pixel. This is done by generating two separate shadow geometries for each object, both based on the silhouette of the object as seen from the light. The shadow interior geometry is drawn with a constant alpha value, giving a “hard” shadow center, whereas the exterior shadow geometry (extruded from the silhouette and outwards/away from the object) is rendered with an alpha gradient giving soft shadow edges. In addition to performing the extrusion and projection of the shadow geometry in the vertex shader the engine also performs batching so that the shadows for all the different objects are handled with a minimum of software overhead, freeing the CPU up for other tasks (realtime physics in this case).



Does this mean that Canvas represents a sensible use case for embedded systems? No, absolutely not, but by focusing all processing power on a single task it gives a good idea of what capabilities are required in a modern application processor. To cope with high resolution user interfaces, composition of applications & HD video, web browsing with the same level of Flash support as your pc and much more, this is the level of performance that you must have, and it all has to work within the power & memory constraints of an embedded system.

So, going back to what I wrote in the beginning, it's good to see these things now being at a level where they add real value and become a make-or-break thing for a product. Even though it is being driven less by fast-paced, shiny, space-racing games, and more by multi-dimensional user interfaces, the technology is basically the same and developers will keep finding additional uses for it. We’ll get our space games, and mobile 3D mascots will get to kick back in bump-mapped sofas on our desktops.

Try out the Canvas demo yourself – come along to the ARM booth at GDC in San Francisco where you can show off your 3D bowling skills.

Remi Pedersen, Graphics Product Manager - Media Processing Division, ARM, joined ARM in June 2006 with the acquisition of Norwegian graphics specialist Falanx Microsystems. Remi currently product manages the Mali range of GPUs within the Media Processing Division at ARM. He started his professional career as one of the first employees of Falanx Microsystems in Norway, where he progressed to Chief Software Architect, leading the design and development of the drivers for graphics processors. Remi has been a graphics enthusiast for many years, having started graphics coding around the age of 13 and never really been able to put it down since.

Shortlink to this post: http://bit.ly/7LtgFj