Build a Local RAG Pipeline in 30 Lines of Python (No Docker, No API Keys)

Stop sending your sensitive documents to ChatGPT. Build a completely local Retrieval-Augmented Generation (RAG) system that runs on your laptop, processes your private data securely, and costs zero dollars per query.

Why This Matters

Privacy-conscious developers and organizations need AI capabilities without data leaving their infrastructure. Traditional RAG solutions require expensive API calls to OpenAI, complex Docker setups, or cloud dependencies that leak sensitive information. A local RAG pipeline gives you:

• **Complete data sovereignty** — your documents never touch external servers
• **Zero ongoing costs** — no per-token charges or subscription fees
• **Offline capability** — works without internet after initial model download
• **Customizable retrieval** — fine-tune exactly what context gets fed to your model
• **Compliance friendly** — meets strict data residency requirements

How To Build It

This implementation uses Ollama for the language model and basic Python libraries for document processing and vector search. No GPU required, though it helps with speed.

Step 1: Install Dependencies

# Install Ollama (handles local LLM inference)
curl -fsSL https://ollama.ai/install.sh | sh

# Pull a lightweight model (3.8GB download)
ollama pull llama3.2:3b

# Install Python dependencies  
pip install sentence-transformers faiss-cpu PyPDF2

Step 2: The Complete RAG Pipeline (30 lines)

import faiss
import numpy as np
from sentence_transformers import SentenceTransformer
import requests
import json
import PyPDF2
import os

class LocalRAG:
    def __init__(self):
        self.encoder = SentenceTransformer('all-MiniLM-L6-v2')
        self.index = faiss.IndexFlatIP(384)  # 384 is embedding dimension
        self.chunks = []
    
    def add_document(self, filepath):
        with open(filepath, 'rb') as f:
            reader = PyPDF2.PdfReader(f)
            text = ''.join(page.extract_text() for page in reader.pages)
        
        # Split into chunks (simple sentence splitting)
        chunks = [s.strip() for s in text.split('.') if len(s.strip()) > 20]
        embeddings = self.encoder.encode(chunks)
        
        self.index.add(embeddings.astype('float32'))
        self.chunks.extend(chunks)
    
    def query(self, question, k=3):
        question_embedding = self.encoder.encode([question])
        scores, indices = self.index.search(question_embedding.astype('float32'), k)
        
        context = '\n'.join([self.chunks[i] for i in indices[0]])
        prompt = f"Context: {context}\n\nQuestion: {question}\nAnswer:"
        
        response = requests.post('http://localhost:11434/api/generate', 
                               json={'model': 'llama3.2:3b', 'prompt': prompt, 'stream': False})
        return json.loads(response.text)['response']

# Usage
rag = LocalRAG()
rag.add_document('your_document.pdf')
answer = rag.query('What is the main topic of this document?')
print(answer)

Step 3: Test Your Pipeline

# Start Ollama server (if not already running)
ollama serve &

# Run your RAG script
python rag_pipeline.py

Best Practices

• **Chunk size optimization** — Experiment with 200-500 word chunks instead of sentence splitting for better context preservation
• **Embedding model selection** — Use `all-mpnet-base-v2` for better accuracy at the cost of speed, or `all-MiniLM-L6-v2` for faster inference
• **Index persistence** — Save your FAISS index to disk with `faiss.write_index()` to avoid reprocessing documents
• **Multi-document support** — Track document sources in your chunks list to enable source attribution
• **Query preprocessing** — Clean and normalize questions before embedding to improve retrieval accuracy

Common Pitfalls

Memory issues with large documents — The simple chunking strategy loads entire PDFs into memory. For large document sets, implement streaming chunk processing or use document databases like ChromaDB.

Poor retrieval relevance — Sentence-splitting creates uneven chunk sizes and breaks context. Use libraries like langchain text splitters or implement semantic chunking based on topic boundaries.

Model hallucination — Even with context, local models may generate plausible-sounding but incorrect answers. Always implement confidence scoring and source citation in production systems.

Ollama connection errors — Ensure Ollama server is running on port 11434. Check with curl http://localhost:11434/api/tags before running your RAG pipeline.

Related DevToolKit Tools

Enhance your RAG development workflow with these complementary tools:

• **[API Request Builder](/tools/api-tester)** — Test your RAG pipeline endpoints and debug Ollama API responses with a visual interface
• **[JSON Formatter](/tools/json-formatter)** — Clean up and validate the JSON responses from your local LLM before processing
• **[Git Command Generator](/tools/git-command)** — Version control your RAG experiments with proper branching strategies for model iterations
• **[Docker Compose Generator](/tools/docker-compose)** — When ready to productionize, containerize your RAG stack with Ollama, vector databases, and your Python service

This local RAG setup gives you complete control over your AI pipeline while maintaining privacy and avoiding vendor lock-in. Start with this foundation and iterate based on your specific use case requirements.

Related DevToolKit Tools