Afaik most LLMs run purely on the GPU, dont they?

So if I have an Nvidia Titan X with 12GB of RAM, could I plug this into my laptop and offload the load?

I am using Fedora, so getting the NVIDIA drivers would be… fun and already probably a dealbreaker (wouldnt want to run proprietary drivers on my daily system).

I know that using ExpressPort adapters people where able to use GPUs externally, and this is possible with thunderbolt too, isnt it?

The question is, how well does this work?

Or would using a small SOC to host a webserver for the interface and do all the computing on the GPU make more sense?

I am curious about the difficulties here, ARM SOC and proprietary drivers? Laptop over USB-c (maybe not thunderbolt?) and a GPU just for the AI tasks…

  • h3ndrik@feddit.deEnglish
    2·
    8 months ago

    I don’t know of any technical problems preventing that per se. But the thunderbolt gpu docks seem quite pricey. You can get half a PC for that money. Or a whole used one. These old ExpressPort adapters used to be cheaper if I remember correctly (the ones which got some PCIE lanes out of an old ThinkPad with some flexible flat cable to a dedicated pcb.) The downside was, it’s just a few PCIE lanes. And that’s probably also a downside with the thunderbolt version of that. It’d take some time to transfer the model into the GPU’s memory this way. But after that it should be fast.

    I mean if the drivers etc work with that setup. I’m not an expert. And it’s been a while since I last saw people using adapters like that. I wouldn’t spend 250€ on that. It’s too much compared to the price of that NVidia card and also too much compared to other solutions. I can get a refurbished office PC for that.