Thank you everyone for joining our call this afternoon. I'm Phil Sparks from Arm's investor relations team, and I'm joined here today by Jem Davies, Arm Fellow and VP Technology, and Ambrish Srivastava, Managing Director and Equity Analyst at BMO Capital Markets. Today, Jem and Ambrish are going to discuss machine learning in Arm powered client devices. Earlier this week, Jem published a 20-minute presentation on this topic on Arm's YouTube channel. Today's call will discuss that presentation in more detail. If you haven't seen it yet, don't worry. You can find the presentation by typing Machine Learning on Arm Powered Client Devices into YouTube's search bar. This will be a listen-only call. If you have any specific questions you would like answered during the call, please submit them using the questions box on the webinar control panel.
If we don't get a chance to answer your questions on today's call, we will follow up with you afterwards. Without further ado, I will now hand over to Ambrish.
All right. Thank you, Phil. This is Ambrish with BMO. Jem, thank you for making the time, and pleasure to talk to you again. This is a pretty timely piece you put up on the Arm channel. As we think about artificial intelligence, AI, machine learning, deep learning, a lot of terms that are now topics du jour, your experience over the years is great to help us gain some insights and perspectives. First of all, let me start off by a question on, what is Arm seeing in the marketplace in terms of whether it's a greater adoption of AI and machine learning. If you could please share with us a perspective on over the last year or so, what is Arm seeing in terms of the inbound requests as well as working with your partners, whether there is a greater adoption, greater engagement around this topic?
Yes, thank you very much. It's perhaps helpful if I just start off by defining a few terms, make sure we're all on the same page. Artificial intelligence, is a sort of an all-encompassing term. It's trying to get computers to apparently be smart or to think. Of course, it's interesting to observe that once that actually starts working, people stop calling it AI, they just call it computing. Machine learning is a similarly large field and a very old one. There's been lots of work being done in machine learning over the years. What is perhaps most interesting over the last few years is the application of so-called deep learning and neural networks.
We've seen a couple of very high-profile things like the computer vision, teaching computers to recognize images, where the application of deep learning to that has got computers better at recognizing objects in pictures than people now. Better correct detection rates, lower false positive rates, and a couple of other very high-profile public things like the application of deep learning to voice controlled services on handsets and even the successful beating of chess grandmasters and Go masters by computers. This has caused everybody to get a lot of attention in this area, but of course like all sudden overnight successes, there's usually been the 25 years of hard sweat and labor before that comes up to that point.
In terms of Arm's marketplace, it's really all about our partners saying very much, "Hey, about this machine learning thing, what does this mean for us?" They're very keen to be educated, very keen to be led by Arm, and they want to know what this is going to mean for Arm's R&D efforts, what products we're going to be producing which will be affected by the trends that we're seeing for deep learning.
Okay. Kind of related to that, what is Arm doing different, from a development perspective to address these new opportunities, these changes, in a qualitative way or any metric you could provide, where is the shift, if you will, on the R&D GBP to address what seems to be a pretty large opportunity over the next few years?
Yeah, we agree. We do think this is going to be a very large opportunity. I do think that machine learning altogether is probably going to be one of the biggest shifts in computing that we'll see in quite a few years. I'm reluctant to put a number on it like, "Oh, the biggest thing in 25 years," or whatever, but this is going to be big. It is going to affect all of us. It affects quite a lot of Arm, in fact. The CPU group will be looking at the workloads, the neural network compute frameworks, and using that as examples to our performance analysis and benchmarking efforts to see, well, what could we do to our CPUs to make them better at performing on those workloads.
Equally, of course, once you step out of the CPU domain into something like the GPU area, for us it's all about non-generic workloads. GPUs, graphics processing units, and of course there's a clue in the name there, are specific very good at graphics. It turns out that the way that graphics is done efficiently is we have very parallelized workflows. We're able to do lots of things at once. Graphics is in fact very computationally intensive. There's lots and lots of arithmetic being performed in graphics. It turns out that GPUs are actually very efficient at performing these sorts of workloads. For years, we've had very highly developed benchmarking, content analysis, performance analysis, and virtuous feedback loops from that into GPU design because we are trying to produce a processor here that is specifically tailored to a type of computation.
Graphics, it is a digital world. Pictures on screens get generated from GPUs. It's all ones and zeros. There's lots of computation in graphics, and what we're trying to do is to produce GPUs that are very good at executing. By good, of course, I mean efficient, high performance, low power. It's an extension of our work in that area, which is getting us to look at the neural network and machine learning workloads, to analyze those on our GPUs and to inform our future roadmap. What should we be doing to the design of our GPUs to make them even better at running machine learning workloads? Indeed, is there a place in our roadmap for future products, which are more specifically tailored to this?
Okay. In your presentation, that you provided us with, you went through and you talked about certain use cases where a device might be the right place to use machine learning than on the server side. Most of the discussion on machine learning has really centered around the server side. If you could please help us understand under what cases is it more useful to deploy or use machine learning on the device side as opposed to on the server side?
Again, with your permission, what I'd like to do is just sort of step back from the question slightly and set some background. Machine learning has been very successful on the server side, initially because the server side tends to be where the data is. If you looked in the presentation that I talked about, the difference between training and inference. The process of training neural networks, in particular, is best done where people have vast quantities of good quality and annotated data. If you are one of the big service providers who has access to vast amounts of our data, then you are in a good position to do this training of those networks, to teach the networks to perform the task that you're trying to do.
Of course, having trained a network to do a thing, it's actually quite a separate process to do the inference. You say, "Well, okay, I've now trained this thing. Now I want to run this thing and have it identify pictures of cats or perhaps something more useful." The place that thing might be done, might well be more convenient to users if it was on devices that they have in their hands. More than convenience, there are some very fundamental technical reasons why some of the things that you would want to do would be best done locally on device without reference to servers. If we look at automotive, for example, if you look at ADAS, the safety critical systems, we can't have those being reliant upon a connection over the Internet to a server.
Your mobile phone signal might drop, Wi-Fi might drop, whatever it is, whatever your connection mechanism is. We know there are cases when that connectivity may drop, and we can't have a car's ability to recognize another car in front of it, being dependent upon that connection. Equally, even if the connection is there are latency issues. If I were, for example, to collect all my data, send it all the way up to a server somewhere, in another continent, and have it processed there and then come back, that is all going to take a finite length of time. Even with radio waves traveling at the speed of light, the distances involved are not your friend here. We need to respond to certain things, particularly safety critical things, but also even for user experience criteria, we need to respond to them pretty quickly.
Latency is another issue. When you talk about a future in which such capabilities have been scaled out to billions, literally billions of devices, not just mobile phones, but also devices in the Internet of things, the bandwidth available for these communications comes into play here. That bandwidth may not be available simply because there's so much device data being thrown around the place that we've broken the Internet, or it may manifest itself as power, because the power it takes to transmit data is often greater than the power it would take to process it locally. Again, there would be strong reasons why you would want to process it locally on device, rather than send it up. Finally, there's security. As machine learning becomes more pervasive in devices, we will find it being involved in things that we care about.
It may be related to our personal data, perhaps related to our health data. If you had a server that is trying to detect whether you are in fact having either a panic attack or a heart attack, you probably don't want your personal health data being transmitted around the world. If you did, you'd be very concerned about the security associated with that. There are a multitude of examples there where security, you'd feel an awful lot better about if it stayed locally on device and weren't transmitted around the world up to a server. What we find, particularly in summary then, is to divide the problem up into machine learning training and machine learning inference is useful. It helps explain the problem space.
Looking at the use cases where you might really want that inference being done on device side, not on server side. It's not to say it's a battle. There are clearly use cases for both, but we think that there will be a predominance of use cases for inference where you will want that done on client side, locally on device.
Good point. I think to your point on the ADAS opportunity, it's hard to imagine that, oh, wait a minute, we don't have connectivity, and we need to recognize what's right in front of us. Let me just bring it back to Arm. When you think about machine learning and the billions of devices that are out there, the install base that's out there with Arm technology, how does ML change the opportunity for Arm, taking it beyond handsets?
Yeah.
We start with handsets, how do you see it expanding beyond that?
Well, we see, of course, that handsets is a huge area because everybody's got one, and every year they get more capable, and it's often the place where new capabilities are first shown. Because it's actually relatively easier to put a new capability into an already very complex and capable device and sell it as next year's model. Everybody's going to start carrying it around in their pocket, as opposed to introducing a new form factor device, and they say, "Oh, I've got to carry two things now, have I? Oh, maybe I'll think about that before I'm really convinced I need it." Handsets is the most amazing market for introducing new technologies. We've seen mobile being the first area in which so many things are showcased. Machine learning changes the opportunity for Arm, I think in several ways.
The capability of machine learning, even inference only on device, is going to require more compute capability. It's going to require compute capability of a particular type to be done in a power efficient fashion. That's going to require us to produce more capable processors. We have an opportunity there to create new products and more capable products in that space. Then as we see that fan out, as I say, we see a start of these capabilities in handsets, but that's going to fan out because we've seen what becomes first groundbreaking in mobile phone handsets, we see progressively spread out into the world of other consumer electronics devices, and indeed then the creation of new factors of consumer electronics devices, the so-called smart connected devices that you're seeing in your home. What does this mean for us in market terms?
Of course, it means we have an opportunity to sell into more devices. We have an opportunity to sell more capable products into those more devices, and we have an opportunity to add value to the products that our partners create. We look to increasing royalties there. If you look at the end markets, it's not just going to mobile. Of course, there is no market in the world like mobile, as we've already talked about. But self-driving cars have had a lot of press. Autonomous self-flying drones is an area that is almost entirely powered by Arm at the moment.
Security cameras is becoming an area that is basically All of the security cameras in the world are basically now being replaced by smart cameras, by cameras that are already doing analysis of the images that they collect, and looking at being more smart with what they transmit, what they record, and in safety terms, picking out features that are relevant to public safety or whatever. One of the problems with cameras, well, depending if you're a disk drive manufacturer, of course, it's great. A high-def camera will transmit enough data to fill up a modern standard one terabyte hard disk drive in about 24 hours. Of course, that's not really sustainable.
What we need to do is to replace all of those surveillance cameras in the world with smart cameras that only record that which is important in a very similar way to the way that human beings do. If you say to me, "What changed in this office in the last half hour?" I would say, "Well, not very much. Oh, yes, Ian got up, walked out, came back in again." That's a very small amount of data compared to recording a half hour of 4K video. We think security cameras is going to be a huge market for the addition of computer vision and machine learning capabilities.
What we're seeing with new use cases for new format devices, such as Amazon's Echo, but also, of course, there are lots of other devices like that, is that we're seeing the digital assistant taking a new format, a new user interface, a new method of interacting with people that is much more natural to them. Kids, for example, kids love Amazon Echo, the Alexa personal assistant. I've literally heard people say, "Oh, yes, Alexa is a member of our family." They talk about it as one of them. Kids just align with it very quickly. They have different set of expectations to us. I think you'll see a huge change with machine learning capabilities being added to devices to kind of break down that glass barrier between ourselves in the real world and the digital world.
Okay. I know Alexa is smart and kids are smarter because now I see in my inbox orders that were placed by Alexa, and it were an example of my kids placing those orders.
I didn't have the big trolls.
Let me just stay with smartphones, then we can go a little bit more technical that I wanted to go into. Arm has made some acquisitions and many, some of the-- I don't want you to name any customers, but some of your Arm's partners have tremendous capability themselves. Is the model we should be thinking about is smartphone makers are assuming 100% of them are using Arm CPUs and graphics close to 80%, 90%? Some smartphone makers are going to be differentiating via their own machine learning capabilities. For those that don't have that, what is Arm bringing to the table on the software side?
Yeah, sure. As you will understand, Arm's traditional business model is that of working to supply the partnership with the things that the partnership requires in order to make great, compelling devices that they then sell and make money out of. That traditional market economics doesn't change. Machine learning doesn't break any of that. It provides something of an inflection. It means that we all have to be providing devices with these new capabilities. Obviously, Arm is spending a lot of its R&D people on the work that I talked about earlier, adding these capabilities to our CPUs, our GPUs, and other devices like special-purpose computer vision processors and others yet to be produced. The traditional make versus buy economics apply just as much for this as it ever did.
What we find is that this is actually a significant piece of engineering effort and research that we're doing that costs a lot of money. We have to employ a lot of smart people to do it. If you're not shipping a lot of devices, then it's not going to make sense to do your own. Even if you are shipping a small number and are buying from us, you're going to be pushing us, you're going to be saying, "Well, look, I need these capabilities, and I can buy this from you, or I can buy this from anybody else." It is a competitive marketplace. It always has been. We are continually pressured extremely hard by our partners to provide more capabilities, more efficiency, and to provide things more cheaply. None of that's going to change.
I'm confident it's always going to be a tough marketplace for us to have economic pressures on us to which we respond, and produce great products. If we don't do that we'll fail. Obviously, I'm confident that we are doing that right, and we are supplying our partners with what they need. Some of them clearly will have specific requirements, that they think that they can better provide by their own internal engineering efforts. Of course, that is the case today, and that will continue in the future.
Okay. Let me turn a little bit into the technical side of things, starting with, I say a relatively straightforward one, precision. Help us understand the difference between floating point 32, 16, in your presentation, you had talked about INT8. Specifically, you mentioned that going from 32 to 8-bit integer code, there's a power consumption saving that you entail, which helps on the client side. Part A, a quick primer on precision. Part B, what kind of power savings do you get, to support the claim that from 32 to 8, there is a big power saving hence easier to use on the client side as opposed to on the server side?
Integers are whole numbers. One, two, three, four. FP, floating point, 3.4, 5.6, et cetera. The precision, also called the width of the numbers, illustrates the scale of those numbers. An 8-bit integer can code the numbers from 1-2 56. A 16-bit integer will go from 1- 65,000. Floating point numbers have a huge range of numbers that they can represent. The width of that floating point number usually affects most directly the accuracy with which it can express that number. If you think the number of places after the decimal point is what comes up in the difference between the lengths of the floating point numbers. Standard precision in floating point is generally considered to be 32 bits, with FP 16-bit floating point numbers generally referred to as half precision.
Why do we care so much about this? First is storage. Obviously, you can store twice as many 16-bit numbers as you can 32-bit numbers, and memory is still an issue. It is less of an issue than it used to be, but it is still relevant. It does very definitely affect bandwidth. Bandwidth equals power, as we talked about earlier. If you are transmitting or storing your numbers as smaller numbers, then you are actually using less power to store those numbers. Internally, in the very insides of our processors, the ALU, the arithmetic logic unit, it is the square of the width that affects the amount of power. For me to do a 16-bit floating point calculation is probably a quarter of the power than a 32-bit floating point number, and similarly down with integers as well. Why is this all relevant?
What tended to happen with the initial development of these algorithms is, as I said, they were developed by the people who had access to vast quantities of data. The people who have vast quantities of data tend to have vast quantities of hyperscale servers. They all have double precision, single precision floating point as standard, and nobody thinks too much about it. When you come to transfer those trained neural networks down onto devices, several things come into play. First, the size of the code and the size of the storage of those neural networks, and then the actual efficiency with which they are executed and the power they consume whilst being executed. At this point, when the device guys get into play, they say, "Well, hang on, I care a lot about this. Do I really need full 32-bit floating point precision?
Can I recast this algorithm to run in a smaller width of data type?" The answer turned out to be yes, with a surprisingly small reduction in the accuracy of the algorithms. If, for example, nothing is ever perfect, an image recognition algorithm might only be 98% accurate at recognizing polar bears. If you reduce the widths of everything and compress the neural networks, you might end up with something that is 97.5% accurate or something close to that.
A lot of the work that is gone on in the last year, including most publicly from Google with their TensorFlow framework, is production of a whole set of tools which take a neural network which has been trained on a server, and then compressing the network itself and reducing the precision of the data types used in that network to make it more amenable to running on devices where memory is short, bandwidth is power, and power equals battery life.
Okay. Turning over to compute power, what is the right way to think about compute power, either in terms of GFLOPS, INT8, or otherwise in training and, more importantly, in inference, where speed and efficiency are more important? For example, what I am trying to get to is, how does an Arm CPU and a Mali combo compare with the x86 CPU and a GPU combo? Are there any benchmarks you can provide us which would help us understand where an Arm combination, for the lack of a better word, would stack up against a competitive instruction set and a GPU?
Sure. GFLOPS is 1,000 million floating point operations per second, which is a number that often gets bandied around when trying to measure the capability for computation. As I just talked about, actually, an awful lot of those operations on inference on device side are now being moved from 32-bit floating point operations to smaller width floating point operations and increasingly to integer operations. Increasingly, the GFLOPS metric is getting a little bit confusing, and what we now need to do is knowledge of the algorithm you are trying to run and the type of data that you are trying to run it on, how many operations per second can your device perform? We now see GOPS, giga operations per second, or even tera operations per second. The operation in that number, of course, is not immediately defined.
You have to know that your tera ops per second are the same tera ops per second as you are comparing to your competitor's tera ops per second. Actually here, the comparison, unfortunately, does get a little more involved. You have to know what it is you are comparing like with like. In terms of benchmarks, there are some fairly commonly used ones on the image processing side. One very commonly used example of neural networks and machine learning is to do scene recognition, to recognize the objects in a scene. I showed a demo of an application doing that in the slide deck. There are several academic datasets, and therefore benchmarks in that, and tera ops per second doing that, or image recognitions per second doing that, have become something of a common benchmark to do that.
Of course, once you have something like operations per second or recognitions per second, the next thing people care about is operations per second per watt, or recognitions per second per watt. Because in nearly all client devices, the capabilities of that device will be thermally limited. With the biggest battery in the world, it is still only going to be able to consume a certain number of watts. If you are moving towards always-on devices, which are waiting to be woken up by you saying, "Hey, Google," or "Alexa," or whatever, then that always-on capability is going to be one where power consumption is absolutely the defining critical factor.
Great. You answered my related question that I was going to ask is, where does the power come into play?
Yeah. As I say, your high-end handsets are universally thermally limited. They cannot get used more than a certain power. Even in lower spec devices and the IoT devices that we are talking about, battery life, of course, will be critical.
Let me turn to another one, and we have talked about it already a few times. On the server side, there is a lot of debate, and we tried to address that in a piece that we had written a few weeks ago about inferencing versus training. What is the right way to think about the ratio of compute power that is required in the cloud, and then how do you translate that to the mobility side? What I am referring to is the ratio of compute power between inference and training.
I'm going to answer your question slightly strangely. The compute power on device is the maximum that you can provide in that 2-W budget. That is the defining factor. The definition is you'll get as much as you can afford within that power budget. On server, of course, it's different, and really there it's as much as you can afford in terms of money. You will throw racks and racks and racks and racks of server CPUs at the problem until you've spent all your money. The two are not directly comparable in that sense. Certainly, we see training taking a huge amount of time. You might be training a neural network for days and weeks, whereas, of course, typically an inference on a handheld device, ideally you'd have the answer back in 20 milliseconds or something within human reaction time.
The total amount of power consumed is in those cases quite radically different.
So far we have touched on, we've talked a great deal about servers, and we talked about mobility. Machine learning and industrial market. Does that open up more opportunity for Arm as a part of an acceleration co-processor, working either independently or more importantly, with an FPGA supplier such as Xilinx, which has been very public about using Arm since their Zynq introduction a few years ago. What's the right way to think about Arm machine learning in industrial?
Machine learning's been applied in industry actually for many, many years in the wider sense of machine learning. We're now seeing neural networks being applied in industy. There's a couple of use cases that are driving that, one of which is intelligent monitoring and intelligent maintenance. In the old days, you had a man in a brown coat who knew that that 2-GW generator should sound like that. When it sounds like this, when it sort of starts getting that slight screech, he knew there was something wrong with it. What we're finding is that actually you can train a machine to do that. Intelligent monitoring, both acoustic and other forms of vibration, is proving to be extremely important because the last thing you want to do is take one of these big machines down for maintenance when you don't need to.
The last thing you want to do is have it go catastrophically wrong because you failed to maintain it. We find industrial to be a very interesting application space, particularly the proliferation of connected devices, relatively cheap devices that are doing the sort of monitoring of machine health that is now becoming very easily available to big plant. I was at a conference the other day where one of the panel speakers-- I was on a panel, one of the panel speakers was talking about precisely this application, where they were monitoring the power distribution network of the entire U.S., they were feeding this into machine learning systems and extracting all sorts of data from it that they previously didn't know was there. That data is becoming to be an asset. It is being monetized by them in various ways.
Not just heavy machinery, but also robots, of course, being extensively used in industry these days. Most of them still bolted to the floor but increasingly now more autonomous. You'll have seen pictures of robots maneuvering around Amazon distribution centers and things like that are called one example. There's a lot of machine learning going on into those devices. They need to be capable. They need to react to the things not being properly straight on the shelves. They need to react to people walking in front of them, all of the sort of things you can easily imagine. They need to be trained to do that. As we are present in so many of the microcontrollers and the processing devices contained within these sorts of new devices, that's becoming a big area for us.
Okay. In the interest of time, I have two more questions. One referring to an example you had, keyword spotting on phones on slide seventeen of your presentation. You mentioned looking at a use case for future IP products. How is keyword spotting done today on flagship phones with an always-on capability? If this were to be implemented using Arm IP, would that require a separate hardware for that always-on, or what would the implementation look like?
Usually, the always-on electronics is kept in a separate power domain. When you're trying to control power and energy consumption, you always want to turn as much off as you can possibly get away with. Regardless of the processor that it's actually running on, such as a Cortex-M3 Arm-powered microcontroller, you would have that running in this separate power domain. What you might have is in something that's analogous to the big.LITTLE CPU thing, is you might have a little microcontroller saying, "Well, I think I heard some speech. It might have had a G in it.
Tell you what, I'll record it, I'll wake up the bigger one, and I'll pass it on to the bigger CPU for him to determine whether somebody said 'Hey, Google' or not. Regardless of whether that's using Arm hardware IP or not, I would say you're always going to have them as somewhat functionally separate, even if it's actually in the same chip. You're probably going to want a separate processing unit doing that work because you're going to want it always powered on when everything else is powered off.
Okay. Putting it all together, and coming back to the title of this presentation that we had, which was: When Does My Smartphone Become Smarter Than I? When do we see all this coming to fruition, whether it's a smartphone smarter than I or my car that I can be lazy and not touch anything, and I have a fully automated Level 5 ADAS system powered by machine learning?
Well, of course, today, some things are already smarter than you. Sorry, smarter than me.
Smarter than me and a lot of people.
For example, you would never, these days, have your eyes operated on by a human being wielding a scalpel, or mostly wouldn't, except in very rare cases. You would actually have a machine doing that, which is obviously incredibly more precise. It doesn't have bad days, and its hands never shake, which has been taught by the experience of lots and lots of humans. Even if you don't have a completely autonomous car. Nobody would think about having a car these days that didn't have an airbag that exploded when it detected a force of more than whatever it is G, and applied the anti-lock braking system when it sensed that the wheels had locked up. I think what we'll see is the introduction of increasing levels of smartness in increasing areas of capability. My smartphone today, pretty smart.
I can use it to connect to services like Amazon's Alexa or Google or Apple's, and I can ask it questions that there's no way I know the answer to. I already have access to considerable levels of smartness more than I have. What I think you're going to find over the years, and I'm not going to make a prediction of the absolute sort of date of the singularity when my smartphone is smarter than me. I think you will see huge amounts of capabilities being added into the devices around us. I think there is no limit to what these things can do for us to make our lives better and easier and safer, and more enjoyable. Machine learning has the capacity to affect pretty much every area of computing around us. I think it's an incredibly exciting time to be involved.
Okay. Great, Jem. Thank you. Thank you very much. This was very, very helpful, and I'm sure listeners who have logged in would find it very useful as well. Phil?
Yeah. Thank you, Ambrish, for leading the call.
Yes. Thank you very much.
Thanks to everyone for listening. As a reminder, if you'd like to learn more about the topics discussed today, you can find Jem's online presentation by typing Machine Learning on Arm Powered Client Devices into the search bar on YouTube. If you have any follow-up questions for Jem, you can email us at investor.relations@arm.com. We look forward to seeing you in 2017. Thanks again, and goodbye.
Bye.