Fine-tune any LLM on a 4GB GPU with layer streaming
Visit WebsiteFreeVisit Website
Tracked since2026
0 reviews trackedThe Bottom Line
Entry price
Free, no paid tier
Biggest pro
Enables fine-tuning of large models on low-cost consumer GPUs that would otherwise be impossible
Biggest con
Layer streaming is still in BETA with known edge cases and limitations (text-only, plain LoRA, specific architectures)
TL;DR - Soup CLI
- Fine-tune 8B models on a 4GB GPU using layer streaming instead of loading the entire frozen base into VRAM.
- Automatically writes training configs, validates data, and gates saves with a SHIP/DON'T-SHIP verdict.
- Open source, free, and integrates with the existing ML ecosystem including HuggingFace, vLLM, and DeepSpeed.
Pricing: Free forever
Best for: Individuals & startups
What is Soup CLI?
Soup CLI is an open-source tool for LLM post-training that eliminates the hardware barrier of fine-tuning. It uses a patented technique called layer streaming, which keeps the frozen base model in CPU RAM or NVMe and streams it one decoder layer at a time into VRAM, allowing models like Llama-3.1-8B to be fine-tuned on a 4GB GPU. The tool automatically writes training configurations, validates data, derives evaluations from your own data, gates every save with a ship/don't-ship verdict, and self-corrects reward hacking mid-run. It supports 23 training methods (SFT, DPO, ORPO, SimPO, KTO, etc.), 142 recipes, 17 quantization formats, and integrates with HuggingFace, Ollama, vLLM, DeepSpeed, Unsloth, and others. A migration command converts existing configs from LLaMA-Factory, Axolotl, or Unsloth in seconds. Layer streaming is in BETA with support for nine architectures (Llama, Qwen, Mistral, Gemma, Phi).
Pros & Cons
Pros
- Enables fine-tuning of large models on low-cost consumer GPUs that would otherwise be impossible
- Fully open source and free with no paid tier or vendor lock-in
- Comprehensive automation: from data validation to config writing to model evaluation
Cons
- Layer streaming is still in BETA with known edge cases and limitations (text-only, plain LoRA, specific architectures)
- Requires Python 3.10 to 3.12 and is primarily designed for CUDA-based GPUs
Key Features
Layer streaming: fine-tune models up to 8B on a 4GB GPU by streaming decoder layers from CPU/NVMe23 training methods: SFT, DPO, ORPO, SimPO, KTO, and moreAutomatic config generation: writes task, quantization, LR, and epochs based on rules, not searchPre-flight data validation: refuses runs that won't fit on hardwareSelf-correcting: catches reward hacking mid-run rather than just haltingMigration command: converts configs from LLaMA-Factory, Axolotl, and UnslothSupports 9 architectures: Llama, Qwen, Mistral, Gemma, Phi17 quantization formats including NF4 for 4x compression
Pricing
Free
Soup CLI is completely free to use with no hidden costs.
Reviews
$99Free with your review
Write a reviewReview Soup CLI, get a free AI guide
Share your experience and we will send you Improve Your Thinking Patterns Using ChatGPT, free.
Best Soup CLI Alternatives
Top alternatives based on features, pricing, and user needs.
Google Vertex AIPaid
Unified AI platform for ML development
H2O.aiPaid
Open-source and enterprise AI platforms for machine learning
AnyscalePaid
Platform for scaling Ray and Python AI applications
ReplicatePaid
Run, fine-tune, and deploy open-source ML models via API
Llama.cppFree
Run LLMs efficiently on consumer hardware
Grid AIFreemium
Accelerate machine learning development by abstracting infrastructure complexities.
Still deciding?
Most buyers shortlist 2 or 3 tools before committing. Pull a side-by-side comparison or browse the full alternatives shortlist below.
Explore More
Soup CLI FAQ
How does layer streaming work technically?
The frozen base model is rewritten into one shard per decoder layer, stored in CPU RAM or NVMe. Each layer is copied into a small pool of pre-allocated VRAM buffers on a dedicated CUDA stream, so the load of the next layer overlaps the compute of the current one. Only the LoRA adapters stay resident in VRAM, bounding peak memory by a single layer instead of the entire model.
Which training methods support layer streaming?
SFT, DPO, ORPO, SimPO, and KTO all support streaming. For DPO, the reference model is the same streamed base with its adapters switched off, so it costs no extra weights.
Can I migrate my existing fine-tuning configuration from another tool?
Yes, Soup provides a
soup migrate command that auto-converts configs from LLaMA-Factory, Axolotl, and Unsloth notebooks in seconds. You don't need to rewrite anything manually.What hardware is required to run Soup CLI?
Soup runs on any system with a CUDA-capable GPU, even a 4GB RTX 3050 laptop GPU can fine-tune Llama-3.1-8B. The layer streaming technique bounds VRAM usage by a single decoder layer, making it possible to train models that are larger than your GPU's VRAM.
Does Soup support quantization during training?
Yes, Soup supports 17 quantization formats including NF4, which compresses the streamed base model about fourfold. This is what enables an 8B model to fit on a 4GB GPU.
Is Soup CLI production-ready?
Layer streaming is currently in BETA. The streamed forward pass has been verified bit-exact against resident checkpoints from 8B up to 72B on 8x H100 GPUs, but certain edge cases—such as the gradient defect found in earlier releases—have been corrected. The documentation lists known limitations: transformers only, text only, plain LoRA, and specific architectures.
How does Soup prevent reward hacking during alignment training?
Soup self-corrects reward hacking mid-run instead of just halting. It monitors the reward signal and can adjust the training trajectory automatically, without requiring manual intervention or separate reward model checks.
Does Soup work with Apple Silicon via MLX?
Yes, Soup includes an MLX and Apple adapter, enabling fine-tuning on Apple Silicon devices. The tool supports 17 quant formats and 23 training methods on Apple hardware as well.
Source: trysoup.dev