# NVIDIA GPUs on Linux for local AI: which cards to buy and how to set them up

> Buy NVIDIA Founders Edition, MSI or PNY cards if you can. They're the brands I trust for fan control on Linux, while two Gigabyte RTX 4090s I owned ran with their fans stuck at idle until they overheated. On Ubuntu, install the driver with ubuntu-drivers, use the open kernel modules on RTX 20 series and newer (they're required on RTX 50 series cards), check it with nvidia-smi, then install Ollama or build llama.cpp. You only need the CUDA toolkit to compile code, and a power limit set with nvidia-smi keeps long runs cooler.

If you're going to run models on Linux for hours at a time, buy a card whose fans the NVIDIA driver can actually control. In my experience that means NVIDIA's own Founders Edition cards, MSI and PNY. I learned this the hard way with two Gigabyte RTX 4090s whose fans sat at idle under Ubuntu until the cards got hot enough to shut the whole system down.

The software side is much easier than it used to be. On Ubuntu, one `ubuntu-drivers` command installs a working driver, NVIDIA's open kernel modules are now the default for RTX 20 series and newer cards, and a few more commands get you to a model running in Ollama or llama.cpp. The second half of this guide walks through that setup step by step, with links to the official docs for each part.

## Why Linux for local AI

Almost every GPU server running AI workloads runs Linux, so a Linux machine at home matches the environment that models, drivers and tools are built and tested on: the same software stack, CUDA libraries and performance assumptions.

When I first wrote this post I recommended Ubuntu 22.04 LTS, which nearly everyone running serious AI work used, from a single RTX 3060 12GB up to racks of B200s. It's still supported, but for a new build in 2026 I'd start with 24.04 LTS. NVIDIA's [CUDA installation guide](https://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html) lists Ubuntu 22.04, 24.04 and 26.04 LTS as supported for CUDA 13.4.

I also said back then that I only used NVIDIA's closed-source drivers. NVIDIA now ships open-source kernel modules as the default for modern cards (more on that below), while the user-space libraries that CUDA uses are still NVIDIA's own. The community nouveau driver still isn't an option for AI, because CUDA doesn't run on it.

## AI workloads are hard on cards

Gaming is a burst workload: GPU utilization swings between about 25% and 90%, with idle moments where temperatures spike and then cool off. Inference and training are sustained workloads:

- Utilization sits around 80 to 100%, with very little idle time
- Temperatures stay high for hours
- The whole card heat-soaks over long runs, which is why blower cards suit server-style builds

LLM inference also works the memory harder than most games, because every generated token reads the model's weights out of VRAM. Memory cooling matters more than the "it's just a chatbot" load suggests.

## The fan-control problem

This is the issue that started this post. I bought two Gigabyte RTX 4090s that worked fine on Windows. On every version of Ubuntu I tried, I could use the GPUs but couldn't control their fans. The fans sat at idle, and once the temperature climbed high enough, the system turned itself off. `nvidia-smi` saw both cards, but the fans seemed to be run by a secondary controller that needed Windows-only software.

One correction to what I wrote back then: `nvidia-smi` reports fan speed but doesn't set it. Manual fan control on Linux goes through NVIDIA's management library, NVML, which can set fan speed on Maxwell and newer cards with fans ([NVML reference](https://nvidia.github.io/cuda-python/cuda-bindings/latest/module/generated/cuda.bindings.nvml.device_set_fan_speed_v2.html)), or through `nvidia-settings` on a desktop. On a reference-design card, the driver's automatic fan curve just works. On some partner cards, the driver can't reach the fans at all, and no setting fixes that.

Before trusting a new card with long jobs, run a model for 20 to 30 minutes and watch `nvidia-smi`. If the fan percentage rises along with the temperature, you're fine.

## Which brands to buy

This list comes from my own experience, not a test of every model each brand makes, and partner cards can change between production runs. For anything not listed, search for reports of Linux fan control working on that exact model.

### NVIDIA Founders Edition

NVIDIA's own reference cards have the best driver support and the most reliable fan control on Linux: the [RTX 3090](/gpus/rtx-3090), [RTX 4090](/gpus/rtx-4090) and [RTX 5090](/gpus/rtx-5090) Founders Editions. If you can get one, start here.

### MSI, air-cooled and liquid-cooled

MSI is the partner brand I'd pick first: on the air-cooled side, the RTX 3090 Suprim X, RTX 4090 Gaming Slim and RTX 5090 Suprim SOC, and on the liquid-cooled side, the Suprim Liquid RTX 4090 and RTX 5090 if you have room for the radiator. I'd still avoid a used liquid-cooled 3090, because its pump is several years old by now.

### PNY consumer cards

PNY's GeForce cards make the list too: the XLR8 RTX 3090, the Verto RTX 4090 and the Epic-X ARGB RTX 5090.

### PNY professional cards

This is where the original post was cut off. PNY builds and sells NVIDIA's professional RTX cards in North America. They use NVIDIA's own board and cooler designs, so they get the same first-party driver support as Founders Edition cards, and they're built for sustained loads. The ones you'll see in local AI builds:

- [RTX A6000](/gpus/rtx-a6000): 48GB on Ampere with a blower cooler, the workstation sibling of the 3090.
- [RTX 6000 Ada](/gpus/rtx-6000-ada): 48GB on Ada, also a blower.
- [RTX 4000 Ada](/gpus/rtx-4000-ada): 20GB in a single slot. [PewDiePie's rig](/builds/pewdiepie-ai-gpu-rig-nvidia-rtx-4000) uses a stack of them.
- [RTX PRO 6000 Blackwell](/gpus/rtx-pro-6000-blackwell): 96GB. The 300W Max-Q version uses a blower and is what [a16z's four-GPU workstation](/builds/a16zs-personal-ai-workstation-with-four-nvidia-rtx-6000-pro-blackwell-max-q-gpus) runs.

They cost a lot more per gigabyte than GeForce cards. An RTX PRO 6000 Blackwell sold for about $16,000 in October 2026, and street prices for every card here have been volatile because of the memory shortage.

### What I avoid

- **Gigabyte**, after my 4090 experience. Your card may work fine, but I haven't gone back.
- **OEM pulls of consumer cards**, like the Dell version of the RTX 3090, which are built to the PC maker's spec and don't always behave like retail cards.
- **Used cards with an AIO cooler** that has years of pump wear on it.

## Blower, open-air or liquid

| Cooler | Good for | Watch out for |
| --- | --- | --- |
| Blower | Packing two to four cards side by side; heat goes out the back of the case | Loud under load, and rare on GeForce cards |
| Open-air | Single-card desktops; quieter | Dumps heat into the case, and stacked cards starve each other of air |
| AIO liquid | Quiet one or two card builds with room for radiators | Radiator mounting, and pumps wear out over the years |
| Passive (datacenter) | Server chassis with strong front-to-back airflow | Needs a fan shroud or server fans in a desktop case |

## Power and connectors

AI loads hold a card near its power limit for hours.

- **RTX 3090:** 350W. The Founders Edition uses NVIDIA's 12-pin connector with an adapter; most partner cards use two or three 8-pin PCIe plugs. It's known for sharp transient spikes.
- **RTX 4090:** 450W through a 16-pin 12VHPWR connector.
- **RTX 5090:** 575W through the updated 16-pin 12V-2x6 connector.

With the 16-pin connectors, push the plug all the way in and avoid bending the cable sharply at the connector. Size the power supply for every GPU's full rating plus the CPU, with headroom for spikes; big rigs like [The Machine](/builds/the-machine-12x-rtx-3090), with twelve 3090s, need more than one supply. A power limit, covered in the last setup step, takes pressure off all of this.

## Ubuntu setup, step by step

This works the same on a fresh Ubuntu LTS install with one card or several.

### 1. Install the driver

Ubuntu's [NVIDIA driver docs](https://ubuntu.com/server/docs/how-to/graphics/install-nvidia-drivers/) recommend the `ubuntu-drivers` tool, partly because it installs prebuilt, signed kernel modules that work with Secure Boot. Update and reboot first, then:

```bash
sudo ubuntu-drivers list
sudo ubuntu-drivers install
sudo reboot
```

That installs the driver Ubuntu considers the best match for your card. On a headless server, add `--gpgpu` to both commands to get the `-server` branch, and install the matching utilities package, for example `nvidia-utils-580-server`. `cat /proc/driver/nvidia/version` shows the running driver, and a "driver/library version mismatch" error after an update goes away with a reboot.

### 2. Open or proprietary kernel modules

NVIDIA [made its open-source kernel modules the default](https://developer.nvidia.com/blog/nvidia-transitions-fully-towards-open-source-gpu-kernel-modules/) starting with the R560 driver. In short:

- **Blackwell cards (RTX 50 series, RTX PRO Blackwell) require the open modules.**
- **Turing, Ampere, Ada and Hopper** (RTX 20, 30 and 40 series and their workstation siblings) should use the open modules.
- **Maxwell, Pascal and Volta** cards, like the [Tesla P40](/gpus/tesla-p40) or a V100, need the proprietary modules. If you mix one of those with a newer card in the same machine, you're on the proprietary modules too. NVIDIA has said [R580 is the last driver branch](https://nvidia.custhelp.com/app/answers/detail/a_id/5706) for these older architectures.

In the `ubuntu-drivers list` output, packages ending in `-open`, like `nvidia-driver-580-open`, are the open modules, and the first line of `/proc/driver/nvidia/version` mentions "Open Kernel Module" when they're loaded. NVIDIA's own apt repository calls the open driver package `nvidia-open`. Pick one source for drivers and stick with it, because mixing the two causes conflicts.

### 3. Check it with nvidia-smi

Run `nvidia-smi`. You should see every card with its memory, temperature, power draw and fan speed. `nvidia-smi -L` lists the GPUs with their UUIDs.

The "CUDA Version" in the header confuses a lot of people. According to [NVIDIA's nvidia-smi docs](https://docs.nvidia.com/deploy/nvidia-smi/index.html), it's the newest CUDA version your driver supports, not a toolkit you've installed. For long runs, this one-liner prints the numbers that matter every five seconds:

```bash
nvidia-smi --query-gpu=index,name,temperature.gpu,fan.speed,power.draw,memory.used --format=csv -l 5
```

### 4. Decide whether you need the CUDA toolkit

The driver is what runs models. The CUDA toolkit is the compiler (`nvcc`), headers and developer libraries, and you only need it to compile CUDA code, like building llama.cpp from source. [Ollama's Linux docs](https://docs.ollama.com/linux) list the CUDA install as optional.

As of CUDA 13.4, the toolkit packages no longer install or depend on the driver on Linux, so the two are managed separately. CUDA 13.x needs a driver from the R580 branch or newer, per the [CUDA release notes](https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/index.html). The [installation guide](https://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html) uses NVIDIA's network repository; on Ubuntu 24.04 that looks like this:

```bash
wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64/cuda-keyring_1.1-1_all.deb
sudo dpkg -i cuda-keyring_1.1-1_all.deb
sudo apt update
sudo apt install cuda-toolkit
echo 'export PATH=/usr/local/cuda-13.4/bin${PATH:+:${PATH}}' >> ~/.bashrc
```

Change `ubuntu2404` to match your release and the version in the path to match the toolkit you installed, then open a new shell and check with `nvcc --version`.

> **Careful with apt autoremove:** If your driver was originally installed as a dependency of the `cuda` meta package (CUDA 13.3 or earlier), upgrading to CUDA 13.4 leaves it marked as no longer needed, and `apt autoremove` can remove the whole driver. NVIDIA's guide says to check with `dpkg -l nvidia-open cuda-drivers`, mark the installed package with `sudo apt-mark manual`, and confirm with `sudo apt autoremove --dry-run` first.

### 5. Run a model

The quickest route is Ollama, using its official install script:

```bash
curl -fsSL https://ollama.com/install.sh | sh
ollama run gpt-oss:20b
```

[gpt-oss-20b](/models/gpt-oss-20b) needs about 15GB at 8K context, so it fits tightly on a 16GB card and comfortably on a 24GB one. In a second terminal, `ollama ps` should show "100% GPU"; anything else means part of the model is running from system RAM. Ollama defaults to a 4,096-token context. To raise it for the background service, run `sudo systemctl edit ollama` and add a line like `Environment="OLLAMA_CONTEXT_LENGTH=32768"` under `[Service]`, as described in [Ollama's FAQ](https://docs.ollama.com/faq).

For the latest features and full control, build llama.cpp with CUDA, following the [llama.cpp build docs](https://github.com/ggml-org/llama.cpp/blob/master/docs/build.md). This needs the CUDA toolkit from step 4:

```bash
sudo apt install build-essential cmake git libcurl4-openssl-dev
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release -j
./build/bin/llama-server -hf ggml-org/gpt-oss-20b-GGUF -c 32768 --port 8080
```

Then open `http://localhost:8080` for the built-in chat page. Per the [llama-server docs](https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md), it listens on 127.0.0.1 by default, works out GPU layers automatically, and splits the model by layer across all your cards. Set `CUDA_VISIBLE_DEVICES=0,1` to choose which cards it uses, and check the [VRAM calculator](/tools/vram-calculator) to see which models fit.

### 6. Set a power limit

This is the step I'd never skip on a machine that runs for hours. The [nvidia-smi docs](https://docs.nvidia.com/deploy/nvidia-smi/index.html) say the `-pl` power limit is set in watts, needs root, and has to fall between the card's minimum and maximum limits:

```bash
nvidia-smi -q -d POWER
sudo nvidia-smi -pm 1
sudo nvidia-smi -i 0 -pl 280
```

The first command shows the current, default, minimum and maximum limits. The second turns on persistence mode, which keeps the driver loaded between jobs. The third caps GPU 0 at 280W; leave out `-i 0` to apply it to every card. For a 3090, 280 to 300W is a sensible starting point; the [dual 3090 build](/builds/portable-25l-nvlinked-dual-3090-llm-rig) holds its cards at 300W in a 25-liter case. Generation is mostly limited by memory bandwidth, so a moderate cap usually costs less speed than you'd expect, but measure it with your own model.

Neither setting survives a reboot (NVIDIA's docs note persistence mode resets to disabled every time), so put them in a small systemd service at `/etc/systemd/system/nvidia-power.service`:

```ini
[Unit]
Description=Set NVIDIA persistence mode and power limit

[Service]
Type=oneshot
ExecStart=/usr/bin/nvidia-smi -pm 1
ExecStart=/usr/bin/nvidia-smi -pl 280

[Install]
WantedBy=multi-user.target
```

Enable it with `sudo systemctl enable --now nvidia-power.service`.

## Where to go from here

If you're still choosing hardware, my [RTX 3090 guide](/guides/rtx-3090-for-local-ai) covers the card I'd start most people on, and the [GPU buying guide](/best/gpus-for-local-llms) covers everything else. For finished Linux builds, the [sleeper RTX 3090](/builds/sleeper-rtx-3090) and the [GPUter RTX 5060 Ti](/builds/gputer-rtx-5060-ti) are good single-card examples.

## Frequently asked questions

### Which Ubuntu version should I use for local AI?

Use an LTS release. NVIDIA's CUDA 13.4 officially supports Ubuntu 22.04, 24.04 and 26.04 LTS on x86_64 machines. The original version of this guide recommended 22.04; for a new build in 2026 I'd start with 24.04 LTS.

### Should I use the open or proprietary NVIDIA kernel modules?

Use the open modules on Turing (RTX 20 series) and newer cards. NVIDIA made them the default with driver R560 and requires them on Blackwell cards like the RTX 5090. Maxwell, Pascal and Volta cards such as the Tesla P40 still need the proprietary modules, and R580 is the last driver branch that supports them.

### Do I need the CUDA toolkit to run Ollama?

No. Ollama only needs the NVIDIA driver, and its Linux docs list the CUDA install as optional. You need the CUDA toolkit, which includes the nvcc compiler, only to compile CUDA code, such as building llama.cpp from source.

### Why does nvidia-smi show a different CUDA version than nvcc?

The CUDA version in nvidia-smi is the newest CUDA version your driver supports, while the nvcc compiler reports the version of the toolkit you installed. A toolkit at or below the driver's version works fine. CUDA 13.x needs a driver from the R580 branch or newer.

### How do I lower the power limit of an NVIDIA GPU on Linux?

Run sudo nvidia-smi -pm 1, then sudo nvidia-smi -i 0 -pl 280 to cap GPU 0 at 280W. The value has to fall between the minimum and maximum limits shown by nvidia-smi -q -d POWER, and the setting resets at reboot, so put it in a systemd service.

### Why can't I control my GPU fans on Linux?

Some partner cards run their fans through a separate controller that the Linux driver can't reach, so fan settings do nothing. nvidia-smi only reports fan speed; setting it goes through NVIDIA's management library or nvidia-settings, and on those cards neither works. That's why I stick to Founders Edition, MSI and PNY cards.

---
Source: https://llamabuilds.ai/guides/nvidia-gpus-on-linux · Updated October 14, 2026 · Llama Builds. Speeds marked ~ are estimates from our bandwidth model (https://llamabuilds.ai/methodology); measured results are labeled as such.