Building a Local RAG Knowledge Manager with Ollama and Qdrant

I wanted a way to ask questions about my own documents without sending their contents to a hosted AI service. That led me to build a local retrieval-augmented generation (RAG) application: a small knowledge manager that ingests documents, indexes their chunks, retrieves relevant passages and gives those passages to a language model as context.

The project uses Streamlit for the interface, Ollama for local embedding and language models, Qdrant for vector search, and SQLite for application-side metadata and ingestion bookkeeping. It is a local learning project, not a claim that a vector database alone makes an answer reliable. Retrieval quality depends on parsing, chunking, embeddings, metadata and evaluation.

1. The request flow

The main path looks like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


Document
   |
   v
Extract text -> Split into chunks -> Create embeddings (Ollama)
                                      |
                                      v
                              Store points in Qdrant

Question -> Embed question -> Search Qdrant -> Retrieved chunks
                                               |
                                               v
                                   Prompt local LLM (Ollama)
                                               |
                                               v
                                            Answer

Each component has a separate job:

Streamlit provides the upload, collection and question-answering interface.
Ollama runs the embedding model and the generation model locally.
Qdrant stores vectors with payload metadata and performs similarity search.
SQLite stores application metadata such as document records and ingestion state. It is not a replacement for Qdrant’s vector index.

2. Why chunk documents?

Embedding an entire book or a long PDF as one vector loses useful detail. I split extracted text into smaller, overlapping chunks. A chunk should be large enough to preserve context but small enough that retrieval can select a relevant passage instead of returning an entire document.

A simplified example:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


def chunk_text(text: str, size: int = 900, overlap: int = 120) -> list[str]:
    if size <= 0 or overlap < 0 or overlap >= size:
        raise ValueError("Require size > overlap >= 0")

    chunks = []
    start = 0
    while start < len(text):
        end = min(start + size, len(text))
        chunk = text[start:end].strip()
        if chunk:
            chunks.append(chunk)
        if end == len(text):
            break
        start = end - overlap
    return chunks

This is a deliberately small character-based example. A production chunker should respect document structure where possible—paragraphs, headings, pages, code blocks—and should record source information such as filename, page and chunk number in metadata. Character count is not the same as token count.

3. What is a Qdrant point?

A Qdrant collection contains points. A point has an identifier, a vector (or vectors) and optional payload. The vector represents the chunk in embedding space; the payload stores the text and metadata needed to show the retrieved result.

For a document chunk, a point might look conceptually like this:

1
2
3
4
5
6
7
8
9


{
  "id": 42,
  "vector": [0.012, -0.084, 0.031],
  "payload": {
    "text": "A thread does not inherit another thread's local context...",
    "source": "java-concurrency.md",
    "chunk": 42
  }
}

The three numbers above are illustrative, not a real embedding. A Qdrant point ID is an unsigned integer or UUID; the actual vector contains as many dimensions as the selected embedding model outputs. The collection’s vector size must match that dimension.

A simplified Python insert using qdrant-client looks like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22


from qdrant_client import QdrantClient, models

client = QdrantClient(url="http://localhost:6333")

client.create_collection(
    collection_name="knowledge",
    vectors_config=models.VectorParams(
        size=768,  # Must match the embedding model output
        distance=models.Distance.COSINE,
    ),
)

client.upsert(
    collection_name="knowledge",
    points=[
        models.PointStruct(
            id=42,
            vector=chunk_embedding,
            payload={"text": chunk_text, "source": source_name},
        )
    ],
)

This is a simplified example; chunk_embedding, chunk_text and source_name are values produced by the ingestion pipeline. In the application, collection creation should be idempotent and the configured vector dimension should come from the embedding model rather than being guessed.

4. Retrieval before generation

At ingestion time, Ollama returns an embedding for each chunk. At question time, the application embeds the question using the same embedding model and compatible configuration. It then asks Qdrant for the nearest vectors and reads the text from each result’s payload.

1
2
3
4
5
6
7


import ollama

EMBED_MODEL = "your-embedding-model"

def embed_text(text: str) -> list[float]:
    response = ollama.embed(model=EMBED_MODEL, input=text)
    return response.embeddings[0]

The model name is a configuration value; choose an embedding model you have pulled into Ollama and use the same model for ingestion and queries.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


question_vector = embed_text(question)  # Same embedding model as ingestion

result = client.query_points(
    collection_name="knowledge",
    query=question_vector,
    limit=5,
    with_payload=True,
)

context = [point.payload["text"] for point in result.points]

The retrieved passages are then added to the prompt sent to the local language model. A simplified generation call looks like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16


import ollama

prompt = f"""Answer using only the context below.
If the context does not contain the answer, say you do not know.

Context:
{chr(10).join(context)}

Question: {question}
"""

response = ollama.chat(
    model="your-chat-model",
    messages=[{"role": "user", "content": prompt}],
)
answer = response.message.content

The model name is a configuration value. Retrieval is not a guarantee of correctness: inspect the returned passages, preserve source references, and test with questions whose answers you already know.

5. Why Qdrant?

I chose Qdrant because it gives the project a purpose-built vector-search service with collections, payload metadata, similarity queries and filtering. I did not want to treat a general-purpose relational table as a substitute for an approximate-nearest-neighbour index. SQLite and Qdrant have different responsibilities in this application.

Qdrant organizes data into collections, and collection data is split into segments. Segments hold vector and payload storage, indexes and ID mapping. Qdrant uses HNSW for approximate nearest-neighbour search and supports payload indexes so metadata filters can be applied efficiently. Its write-ahead log (WAL) records updates so operations can be recovered after an abnormal shutdown. Current Qdrant documentation describes dense vectors as memory-mapped files, with memory tiers controlling whether they are cached in RAM. The exact storage and indexing configuration depends on the deployment and collection settings.

6. Why is Qdrant written in Rust?

Qdrant’s core is written in Rust. That is relevant for a database that performs concurrent work, manages memory and handles disk I/O. Rust provides low-level control and native performance while its ownership and type systems help prevent many classes of memory-safety bugs without requiring a garbage collector.

That does not mean Rust automatically makes every operation fast or bug-free. Data layout, indexing algorithms, disk access, caching, batching and workload shape still determine real performance. The language is one engineering choice in the system, not the whole explanation.

7. RocksDB, and what changed with Gridstore?

This is where a lot of older Qdrant explanations become outdated. Qdrant initially used RocksDB as an embedded key-value store. The Qdrant team later described operational and architectural trade-offs with that choice, including compaction-related latency spikes and the complexity of tuning a general-purpose key-value engine for Qdrant’s access patterns.

Starting with Qdrant 1.13, Qdrant introduced Gridstore, its own Rust key-value storage component, for payload and sparse-vector storage. It was designed around Qdrant’s needs rather than as a general-purpose replacement for every database. Do not summarize this as “Qdrant uses RocksDB internally today” or “RocksDB disappeared from every code path.” The accurate statement is that RocksDB was used historically, and Gridstore replaced it for the documented payload and sparse-vector storage paths.

Also, Qdrant’s dense vector storage has its own storage behavior: current documentation describes vectors stored in memory-mapped files, with memory tiers controlling how they are cached. That is separate from the payload key-value store.

8. What I learned

RAG is a pipeline, not just a prompt sent to an LLM.
Chunk boundaries and metadata affect retrieval quality as much as the choice of vector database.
A Qdrant point combines an ID, vector and optional payload; the payload connects a search hit back to readable source text.
Embedding dimensions and model choice must stay consistent between ingestion and query time.
Qdrant’s storage architecture has evolved. Older RocksDB explanations need to be read with their version context.
Local execution improves control over where documents are processed, but it does not remove the need for access controls, backups, validation and careful handling of sensitive files.