ARM Community: Is Image Processing the Killer App driving GPUs? - ARM Community

In my first blog, I discussed the technologies that enable GPU computing and the standards that arose to encourage that development. Now, what about the use-cases for GPU computing in mobile? It's often the first question I am asked when I visit our silicon partners and our OEMs and this is a subject that rouses a number of passions. Some people are suggesting that a killer use-case will be Augmented Reality...

Battery-powered GPU Computing
On the desktop, the killer applications have largely come from the High Performance Computing (HPC) community first: scientific analysis, financial predictions, weather forecasting, and structural analysis etc. EDA tools are starting to accelerate algorithms like place and route on GPUs as well. There are companies building supercomputers by building racks and racks of graphics add-in cards. I predict (quite safely, I believe) that little of this will make its way onto mobile phones. Even connecting clouds of millions of phones together to harness that computational ability will have issues, as the communications overheads/latencies would be high, and the history of parallel computing tells us that the communications latency/bandwidth between computational elements dominates these setups. So, what will the killer mobile OpenCL applications be?

Image Processing
What is happening on the desktop is image analysis and image processing - accelerating programs like Adobe Photoshop using the GPU. That, I believe, will move to the mobile. If I wish to hand-polish every pixel before publishing an image, I might well move the image to a larger format device (particularly a large screen) and work there. If all I want to do is mash two photos together or tweak the red-eye removal, I want to do that on my mobile device. A trend in Asia, for example, is face-beautification on digital cameras: taking the image, finding the face in the image, and removing any pores, blemishes etc. Today, this is being done on ISPs, DSPs etc. but a GPU running OpenCL would make a very good target for this computation. What we see in Digital Still Cameras (DSCs) today will move seamlessly to mobile phones tomorrow. I would venture to suggest that the image manipulation on mobile phones might need even more heavyweight computation than on a DSC, because the lens is usually smaller to start with. Of course, some algorithms will need to stay on the ISP, because they must be run "close" in time domain terms to the sensor/lens mechanism (choosing between multiple image captures for blink detection, focusing etc.). However, some features such as face beautification would execute very efficiently on a GPU.

For those who haven't come across face beautification yet, have a look at this research done, published in a SIGGRAPH paper
and this IP being sold.

The Killer App or Killer Apps?
I don't think it's just Augmented Reality. A number of phone OEMs' R&D departments are looking very closely at image processing to see what can be achieved by moving computation to GPUs, through OpenCL. I am seeing some common themes running through their work: face recognition and indexing, face processing, including beautification, and also wider computer vision work. Let's say you have GPS, a compass and an accelerometer in your phone, you point the camera. It knows where you are and which direction you are pointing; it identifies the buildings from the images and overlays information onto the screen to tell you about them. It identifies any people in the picture and tells you about them as well (what his name is, who he works for and when you last saw him). Obviously in shops, you can read bar codes to look up more information on the web, but if you can recognize objects without having to scan bar codes, it's all more seamless and user friendly. I think a whole range of image processing killer apps will come out of these themes.

I'll stick my neck out - the Killer App for me (actually a whole range of Apps) is an amalgam of image processing, computational photography and computer vision that will allow mobile phones with tiny lenses to take pictures that look as good as those from full-size digital cameras and then allow those images to be used intelligently in a whole host of add-on applications. That will take a lot of image processing horsepower and GPGPU looks great for that. However, as I say later, I'm prepared for my prediction to be wrong...

A standard is needed
Now of course you can do some of this today without GPGPU or OpenCL. You can see some great examples here, but GPUs will be very good at this, and OpenCL will provide a standardized API for doing it. I believe a standard, almost any standard, will be good for the market - there seems to be too much proprietary code out there. I have certainly had off-the-record conversations where developers are banging the table about the fact that they can't move code from one DSP to another because of differences between them, and between their toolchains. It will be interesting to move to an environment where people will be more freely able to move code around to the most suitable (in performance or energy terms) computational device, rather than "I have to do it there, because they provide the algorithm and it only runs on that". I think that could be the catalyst for a number of important changes in the market. Once we have momentum, we will see people porting existing libraries of image processing code (such as OpenCV, perhaps?) to OpenCL.

Build it and they will come
While image processing is one of the killer mobile applications for GPUs in mobile phones, I believe that if we build open systems – platforms on to which users can download applications from sources such as an online store, more use-cases for GPU computing will appear, and I am sure that some of them will surprise us.

It is not necessary for us to know exactly what will capture the imagination of users. No, that’s not just a cop-out. If you look at the market for applications on phone platforms onto which people can download applications I think we can agree that some of the most successful applications have been surprising at first sight. I believe the same will happen for GPU computing. The killer app for GPU computing may not have been written yet, perhaps not even thought of yet, but if we provide the capabilities, they will appear.

One note of caution - we will see some applications ported to GPU computing and some will perform badly (see my previous blog describing the things that work well and what won't work so well). We should not be disheartened by that. Not all code will accelerate well on GPUs - we know that. What we should concentrate on is the new apps that become possible through the acceleration of suitable algorithms.

Where does this lead to?
Now, I might be in the minority that still own a separate digital camera, but I also miss a lot of good pictures because I don’t always have my camera with me, but my phone is nearly always in my pocket! If the developers can make exciting, cool, easy-to-use applications that enable them to attract lots of people to buy their mobile devices, you can believe GPU computing will take off. We in the industry must be ready to provide them with the technology they need. I am desperate to tell you about where ARM is going next with GPU technology, but I'd get in trouble now and it will have to wait to a later post where I talk about the best embedded GPU in the world and great tools to go with it to help developers tap into this potential.

What do you think the killer mobile apps are that will drive the adoption of GPUs in mobile devices and the introduction of GPU computing as an energy-efficient processing method?

Jem Davies, ARM Fellow. Jem likes to think of himself as "The Godfather" to technical talent in ARM. After spending some time in his youth writing software for satellites and traffic-lights among other fascinating things, Jem spotted the technical inflection point of the mobile industry: graphics, video and other visual processing. As VP of technology in the Media Processing Division of ARM, Jem is busy with a lot of projects involving the future of cool ARM technology, which will revolutionise how people experience and interact with digital devices.

Shortlink to this post: http://bit.ly/cEHnAg