LLM@home - privacy-first local inference in practice

CPU (AMD, Intel, RISC-V)

Pros:

• Cheap
• All RAM available
• Open architectures available (RISC-V)

Cons:

• Slow
• Takes resources away from other tasks

GPU (Nvidia, AMD, Intel)

Pros:

• Fast
• Straightforward
• Lots of software available
• Good parallelism
• Open architectures available (AMD)

Cons:

• Expensive
• Limited vRAM

NPU

Pros:

• Most efficient hardware

Cons:

• Small range available for purchase
• Limited software
• Usually low power

RAM

Pros:

• Can install up to 4TB on a server

Cons:

• Slow (DDR4-DDR5)

Unified Memory

Pros:

• Cheap way to give GPU lots of memory

Cons:

• Slow

vRAM

Pros:

• Very fast (DDR6-DDR7)

Cons:

• Expensive
• Limited (32GB / 80GB) per chip

Alpine Linux

Pros:

• Small attack surface due to minimal service set
• Musl instead of libc (security-hardened)

Cons:

• One step left one step right and it doesn't work
• Building software from source - common practice
• No Nvidia driver

Debian-based Linux

Pros:

• Everything works out of the box

Cons:

• Larger attack surface due to more unnecessary or unclear services

OpenBSD

Pros:

• Very small kernel (7M LoC vs 40M LoC for linux)
• Security-focused

Cons:

• Lacks optimizations
• No Nvidia drivers
• No NUMA

llama.cpp

When to use:

CPU inference

Pros:

• 0 dependencies
• No issues using quantized models

Cons:

• Batching on CPU practically non-existent

vLLM

When to use:

GPU inference

Pros:

• Can batch requests

Cons:

• Needs expensive GPUs

Ollama / LM Studio

When to use:

Never done anything before, want quick hello world

Pros:

• Easy to use

Cons:

• Low performance
• Lots of unnecessary stuff

KV cache

Pros:

• Speeds up performance

Cons:

• Requires RAM

Memory map

Pros:

• Can avoid using RAM for model weights

Cons:

• Slower if RAM is the bottleneck
• Only suitable for CPU inference

Batch/queues

Batch pros:

• Processes faster on average since multiple at once

Queue pros:

• Predictable latency
• Works better on CPU