My Progress wih Local AI - Failing to Get NPU to "Just Work"
This is a follow-up to the previous article in which I talked about using KoboldCpp to run AI Models locally. For AMD APUs, this framework mainly uses CPU and GPU via Vulkan. I noticed that Koboldcpp doesn't use NPU.
The reason I even bought this specifc device was that it's cheap and has NPU chips, so not utilizing it didn't sit right with me. I tried to look for alternatives...
NOTE: An NPU (Neural Processor Unit) is a chip inside the computer that does neural operations, which makes it perfect for generative AI.**
My research made me discover that the GGUF library does not use the NPU. It may do in the future, but not right now. If I want to make use of the NPU in my system, I had to use ONNX library with Ryzen AI Software!
The app-server I found that does that was Lemonade-server, though when I installed it, it couldn't recognize my NPU hardware.
I realized that I need to install Ryzen AI Software first. I installed it, after installing its requirements, Conda (Miniforge3) and VS Code 2022 (I installed 2026 instead which is backwards compatible.) It wasn't easy to set up, but I finally managed to get it working:
Still, even with Ryzen AI Software, Lemonade couldn't recognize my NPU hardware... Apparently, Lemonade only works with Ryzen AI 300 series or newer.
My hardware isn't supported by Lemonade!
But I still have the Ryzen AI Software, I ran the QuickTest python application, and it worked... I tried to install a small ONNX-format model, but I found out they're HUGE!
I've been using the GGUF library format up until now, and only the quantized models at that, so I didn't realize how big ONNX models could be. What's worse, to use them, I'll have to pull a whole repositery from HuggingFace, unlike GGUF models which I know which files to pick and download, the ONNX seem to need the whole folder.
So, I'm going to stop looking into the ONNX format for now, but I'll keep RyzenAI installed just in case.
What do you think?
Related Threads
https://inleo.io/threads/view/ahmadmanga/re-leothreads-wbayqgrf?referral=ahmadmanga
https://inleo.io/threads/view/ahmadmanga/re-leothreads-cgkgwugx?referral=ahmadmanga
Posted Using INLEO
Congratulations @ahmadmanga! You received a personal badge!
Participate in the next Power Up Day and try to power-up more HIVE to get a bigger Power-Bee.
May the Hive Power be with you!
You can view your badges on your board and compare yourself to others in the Ranking
Check out our last posts:
Thanks for your contribution to the STEMsocial community. Feel free to join us on discord to get to know the rest of us!
Please consider delegating to the @stemsocial account (85% of the curation rewards are returned).
Consider setting @stemsocial as a beneficiary of this post's rewards if you would like to support the community and contribute to its mission of promoting science and education on Hive.