Forum's subscription has expired

Unable to load 70B LLM into the 96Gb VRAM of my GMKtec Nuc_Box Evo-X2


  • As i wrote in an earlier post i am absolutely satisfied about the performance of this little box. But there is one thing i am unable to achieve. The configuration is set in AMD Adrenaline to 32Gb RAM and 96Gb VRAM under Windows 11 Enterprise. Every 70B LLM i am trying to load into VRAM using LM Studio leads to an error message between 75% and 85% of loading progress. The file size of those models is between 45Gb and 55Gb and the description in LM Studios library tells it should fit completely in my VRAM. What am i doing wrong? For instance, loading the Qwen 32B LLM with a file size of 20Gb works perfect. When i watch the GPU Performance Monitor in AMD Adrenaline while a bigger model is loading i notice the use of VRAM increases up to around 32Gb and then suddenly stops and i get the error message in LM Studio. Every hint is welcome :-) Thank you.



  • Since there is a good chance that you won't get a support from these guyes, I am writing my experience. I have tested quite a bit of models when I had Evo-X2, which is returned for a defect and their customer support went radio silent with my (alomost) 2,000USD.

    I first had to install 1.04 BIOS iirc, which has native option in the BIOS to select 96B GPU memory allocation. I used Ollama and most 6~70B models in Ollama's website run without issues except the couple latest models including llama4:scout. That seems to be a bug in the engine. When something is claimed to be supported by a model runner(LM Studio, Ollam, etc), that usually means it is supported on CUDA(Nvidia) architecture, and AMD's driver for ROCm is also kind of garbage at this point for this chipset. Everything worked much better in Linux btw but not much else to say.

    When I contacted them through the email for instability issues I was having, they asked me to use 64GB because that is the most balanced setting, whatever that means. I told them that I wouldn't have bought it knowing 64GB is the maximum I can use for GPU and asked for refund if they cannot guarantee that it will work with 96GB. Then they started a refund processing. Not a very confidence inspiring move if you ask me.

    In any case, good luck. When the machine was working, it was pretty cool but I just had so many issues, and now figthing for my refund with the customer support after I did everything they asked for. Now I am out of both my money and the machine that I returned, which they claimed not received, even after I have provided a document from UPS that someone signed at their return location for the delievery. They may just be slow on responding so I am just frustrated at the moment, but that was enough for me to decide that I will not recommend anyone buying from them based on my experience.

     


  • Thanks a lot JackeT for sharing your own experience with that device and the GMKtec support. I am sorry for that. GMKtec is a young company so i guess they are still learning how to deal with customers by a professional way :-) My own machine is now running without any problems since almost 2 weeks now, so i hope i got a reliable one. As i am just starting playing around with AI and LLMs your info is very valuable for me. My machine came with BIOS 1.04 already installed. In the BIOS options i do not see an option to choose CPU/GPU RAM sharing ratio but i can do this by AMDs Adrenaline Software. The only strange thing no matter what i configure in Adrenaline it seems (according Adrenalines monitoring figures) there is a 32Gb limit somehow. I hope your returned machine shows up soon and you get your money refunded.


  • It's me again willing to share some more experiences i have made after reading JackeTs comments. My machine came with a BIOS 1.04 already installed. But i thought i give it a try and install GMKtecs BIOS 1.04 all over the existing one and tadaaa after that there is a lot more options showing up for finetuning. So now i have found the 96G setting for VRAM and keep trying whether 70B models are possible now. Again big thanks to JackeT for sharing his knowledge.


Please login to reply this topic!