Arm Ethos-U65 Innovation in new AI world – Machine Learning IP blog – Processors – Arm Community

There is an explosion of edge and endpoint artificial intelligence (AI) happening in the world today, having a significant effect on our everyday lives. AI technologies are being used in public safety, improving the retail experience, transportation and a large and rapidly growing list of other scenarios. Consumer applications from smart door locks to home electronics are revolutionized by AI. In infrastructure AI is being used in a growing number of areas such as data plane optimization and power management. AI truly is, going everywhere.

Earlier this year, we announced new additions to Arm’s Machine Learning (ML) portfolio to enable extremely low-power machine learning inference at the endpoint. The combination of the Arm Cortex-M55 CPU and the world’s first microNPU (Neural Processing Unit), the Arm Ethos-U55 NPU, provides an enormous 480x increase in ML performance over previous generation Cortex-M CPUs alone. This combination enables devices to run neural network inference on endpoint devices and not send massive amounts of data to the cloud. Keeping the data on device not only makes these systems more responsive but also more reliable, secure, and private. The success of the Ethos-U55 makes me confident that we are rapidly moving towards an exciting future with unprecedented AI developments on devices.

Expanding AI into new devices with the new Arm Ethos-U65

To enable even more innovation and expand AI applications into more devices, we have announced the latest addition to our Ethos product line. The Arm Ethos-U65 microNPU which provides neural network acceleration in high-performance embedded devices and subsystems. The Ethos-U65 maintains the power efficiency of the Ethos-U55, while extending its applicability to Arm Cortex-A and Arm Neoverse based systems. Although Ethos-U65 can be used alongside any Cortex-A or Neoverse CPU, it is particularly well suited to Arm CPUs with advanced vector capabilities, like the Arm Cortex-A55.

Micro size, mega efficiency

The Ethos-U55 introduced our first microNPU architecture, which allows acceleration of neural networks in extremely low-area and with low-power consumption. Adding to the success of the Ethos-U55, and maintaining a focus on power efficiency, the Ethos-U65 extends its applicability to Arm Cortex-A and Neoverse based systems. The Ethos-U65 provides a new, higher performance point for more demanding applications, achieving 1 TOPs. This enables new capabilities in devices such as high-resolution smart cameras, smart home solutions, and even infrastructure applications such as bandwidth and power management subsystems.

Extended network support

Fundamental to the Ethos-U product line’s design is its native support for ML operators, giving it the ability to execute the most popular neural networks completely on the NPU with no operator fallback to the CPU. For Ethos-U65, this operator support has been further updated and expanded. Nonetheless, where fallback to a CPU is still necessary, these operators are typically still accelerated through highly optimized software from Arm such as Arm Compute Library or CMSIS-NN.

Ethos-U65 provides two different configurations of 256 and 512 MACs/cycle. It includes a dual AXI which delivers better bandwidth for weight bound networks.

For these more complex systems, this results in an average increase in network performance (infs/sec) of 150% over Ethos-U55.

Choose the right system for your ML requirements

Ethos-U65 is designed for use with DRAM based systems, which leads to higher bandwidth availability. This allows Ethos-U65 to be used with more classes of embedded systems:

While using Ethos-U65 in a Cortex-A or Neoverse based system, the software flow remains the same as Ethos-U55, using TFLmicro runtime. The TFLmicro stack runs on a companion Cortex-M processor next to the NPU and handles any offload operators that cannot be executed on the microNPU itself.

A significant advantage of Ethos-U65 is enabling the execution of networks of any size. With the Ethos-U65. it is possible to efficiently accelerate much larger networks, enabling applications like real time object detection, classification, and recognition, to name a few.

Additionally, with Ethos-U65, any investments made building ML applications on Arm processors is not lost and remains reusable, as Ethos-U65 uses the same software and tools as Ethos-U55. Along with unified software and tools, Arm Ethos-U microNPUs are supported by a rich and vibrant ecosystem of partners providing a wide variety of solutions across audio and vision use-cases including speech recognition, image classification and object detection.

Accelerating AI capability into edge and endpoint devices

Providing more AI capability into edge and endpoint devices opens the door to un-imaginable innovation, creativity and efficiency, leading to amazing future products. A recent article I read claimed, ‘AI will be as transformative as electricity’. For me, AI is a once in a generation change in computing that transforms everything from cloud servers, to smart home solutions through to the tiniest IoT devices. The opportunities are massive, the market is open to everyone and I cannot wait to see what applications are enabled by the Ethos-U65 NPUs.