How I built a CV chatbot with Ollama and RAG

I wanted to add a CV chatbot with Ollama and RAG to my portfolio so visitors could ask about my experience. I had a small CPU-only server and needed control over the information it would display. I decided to prepare a reviewed profile and use the model to select the facts for each answer.

Short answer: My assistant retrieves reviewed facts about my work and uses Qwen2.5 1.5B to select up to three fact IDs. The application validates them and displays approved text, keeping free model prose out of the answer. The profile contains 56 facts in English and Spanish; evaluation uses 149 development cases, not an independent test set.

cv-agent is the Python service I built for that purpose. The code is in the public repository, and my work history is available on my portfolio.

How the CV chatbot with Ollama and RAG works

The application uses FastAPI. It implements a text subset of Open Responses, an open specification for interoperable model interfaces. It supports JSON responses, server-sent events (SSE), and follow-ups through previous_response_id. The API requires authentication.

Each question follows the same path:

1. BM25 retrieves facts from the reviewed profile.
2. The model selects at most three fact IDs, with output constrained by a JSON schema.
3. The application validates those IDs again.
4. The visitor gets approved text from the accepted facts. Without valid evidence, the assistant abstains.

The design decisions and their trade-offs

Decision	Implementation	Reason or limitation
Retrieval	BM25, bilingual tags, evidence and entity filters	Lexical search over 56 facts, without a vector database
Model	Qwen2.5 1.5B Q4_K_M through Ollama	About 986 MB to download; a 2 GiB container memory cap
Answer	Up to 3 fact IDs, then reviewed text	Displays approved text; can repeat sentences
Profile	English and Spanish facts with human approval	The review can still contain mistakes
Evaluation	149 cases and 91 unit and API tests	A development set, not independent validation
Runtime	Unprivileged containers, an isolated network and 24 h history	Limits the impact of a failure

Why I chose extractive answers

A generative answer lets the model write a new sentence. An extractive answer uses text that already exists. I chose the latter because I wanted control over the claims and links a visitor would see.

The model returns IDs, and the application looks up the text attached to each one. Before displaying it, the application checks that the IDs belong to the allowed facts. This lets me review the wording before publishing, although the model can still select a fact that does not answer the question.

Repetition is the most visible limitation. If several questions lead to the same facts, visitors read the same sentences. Adding variety would mean preparing and reviewing alternative wording.

Example 1: constrain selection with a JSON schema

This simplified example shows the separation between the model and the application. The IDs and sentences are fictional; they do not come from the project’s profile. Run the three Python blocks in order; they only need the standard library.

After retrieving candidates with BM25, I build a schema that allows their IDs. maxItems caps the selection at three facts; enum limits the values to candidates retrieved for this question.

import json

reviewed_facts = {
    "demo-python": "The example project uses Python.",
    "demo-api": "The example API accepts questions as JSON.",
}

candidate_ids = ["demo-python"]
allowed_facts = {
    fact_id: reviewed_facts[fact_id]
    for fact_id in candidate_ids
    if fact_id in reviewed_facts
}

selection_schema = {
    "type": "object",
    "properties": {
        "fact_ids": {
            "type": "array",
            "items": {"type": "string", "enum": list(allowed_facts)},
            "maxItems": 3,
            "uniqueItems": True,
        }
    },
    "required": ["fact_ids"],
    "additionalProperties": False,
}

The schema goes to the model’s structured-output mechanism. If retrieval produces no candidates, the application can return an insufficient-information response directly and skip the model call.

Example 2: validate before displaying the answer

The model constraint is one part of the control. The application checks the JSON again, including the number of IDs and whether they belong to the candidates. This simplified version rejects the entire selection if any check fails.

ABSTENTION = "I do not have enough information to answer."


def render_answer(raw_selection: str, allowed_facts: dict[str, str]) -> str:
    try:
        selection = json.loads(raw_selection)
    except (json.JSONDecodeError, TypeError):
        return ABSTENTION

    if not isinstance(selection, dict) or set(selection) != {"fact_ids"}:
        return ABSTENTION

    fact_ids = selection["fact_ids"]
    if not isinstance(fact_ids, list) or not 1 <= len(fact_ids) <= 3:
        return ABSTENTION
    if any(not isinstance(fact_id, str) for fact_id in fact_ids):
        return ABSTENTION
    if len(set(fact_ids)) != len(fact_ids):
        return ABSTENTION
    if any(fact_id not in allowed_facts for fact_id in fact_ids):
        return ABSTENTION

    return " ".join(allowed_facts[fact_id] for fact_id in fact_ids)

The last line takes sentences from allowed_facts. Displayed text comes from content reviewed by the application. A valid ID can still be a poor selection for the question; relevance is checked during evaluation.

Example 3: try valid answers and rejected selections

demo-api exists in the example profile but was not retrieved for this question. It must therefore be rejected too. These cases check that boundary and several formatting errors:

cases = [
    ('{"fact_ids": ["demo-python"]}', allowed_facts["demo-python"]),
    ('{"fact_ids": ["demo-api"]}', ABSTENTION),
    ('{"fact_ids": ["unknown"]}', ABSTENTION),
    ('{"fact_ids": []}', ABSTENTION),
    ('{"fact_ids": ["demo-python", "demo-python"]}', ABSTENTION),
    ('{"fact_ids": ["demo-python"], "answer": "extra text"}', ABSTENTION),
    ('{"fact_ids": [42]}', ABSTENTION),
    ('not valid JSON', ABSTENTION),
]

for raw_selection, expected in cases:
    answer = render_answer(raw_selection, allowed_facts)
    assert answer == expected
    print(answer)

The first call displays The example project uses Python.. The rest return the insufficient-information response. This example covers validation and final text; BM25 retrieval, the Ollama call and conversation handling belong to the complete service in the cv-agent repository.

RAG without a vector database

RAG stands for retrieval-augmented generation: finding relevant information before asking a model to answer. Here, the model uses the retrieved information to select facts from the profile.

BM25 ranks documents by their word matches with the query. The search is lexical: it depends on the words in the query. A differently worded question can miss a useful fact. Bilingual tags, evidence-type filters and entity restrictions help narrow the search.

With 56 short facts, I could inspect BM25 results and adjust the tags. I kept that implementation rather than adding a vector database.

Why I moved from 0.5B to 1.5B

I first tried a model with 0.5 billion parameters. Asked about plugin pricing, it selected a generic list of technologies. The selected facts did not answer the question, even though the relevant information was in the profile.

I then chose Qwen2.5 1.5B Q4_K_M, served locally through Ollama. Its download is about 986 MB, and the container has a 2 GiB memory cap. Inference needs memory for context and the runtime in addition to the model file. That is why I distinguish download size from the container memory limit.

How I reviewed the profile before indexing it

The profile contains 56 reviewed facts in English and Spanish, stored as JSON. Raw sources are never indexed. A validator rejects email addresses and phone-like values, and facts require human approval.

Human review also keeps private material out of the corpus. The validator catches certain contact-data patterns; it cannot identify every kind of sensitive information. If I approve an incorrect claim, the assistant can repeat it.

How I evaluate answers and refusals

The development set contains 149 cases: answerable questions, follow-ups, questions without enough evidence, and attacks. There are also 91 unit and API tests, plus checks against the official Open Responses schema.

Acceptance targets are at least 90% on answerable questions, 100% on abstentions, 100% on attacks, and warm p95 latency below 45 seconds. The p95 is the time within which 95% of responses finish; “warm” means the model is already loaded.

These figures are the evaluation’s acceptance targets. I used the 149 cases during development and to adjust the service’s behavior. Testing with an independent set, or holdout, of previously unused questions is still pending. The percentages above should not be read as measured success rates.

How I limit the impact of a failure

I use rootless Docker, which runs containers without root privileges. The internal network blocks outbound connections; the filesystem is read-only, and containers have no Linux capabilities, which are additional kernel permissions.

There is one worker and a limit of 20 requests per minute. SQLite stores conversation history, which expires after 24 h. The container memory caps are 256 MiB, 2 GiB and 64 MiB.

These controls restrict what the service can do and how much it can consume. They do not make the assistant immune to prompt injection. A malicious question can still steer the model toward unrelated facts.

What I would improve next

I would start with an independent evaluation set and more questions that use different vocabulary from the profile. I would also try reviewed sentence templates to reduce repetition while keeping control of the displayed text.

This project brings together the API and server work described on my full-stack profile. Hardware constraints also shaped another personal project: optimizing my Android TV with ADB.

Frequently asked questions

Could I build something similar for my own CV?

The cv-agent repository shows the approach. Before connecting a model, I would prepare reviewed facts and decide which information can be public.

Do I need a GPU to run it?

This project runs on CPU only. I chose a 1.5B model and capped its container at 2 GiB; that does not guarantee the same latency on every machine.

Why not just give it my CV as a PDF?

I wanted to decide exactly which claims the assistant could show. That is why I use reviewed facts instead of indexing raw sources.

What happens if it selects the wrong fact?

A visitor can receive a fact from the profile that does not answer their question. ID validation controls which text is displayed; evaluating whether the selection is relevant remains necessary.

Do I need a vector database?

For this corpus, I chose BM25 with tags and filters. It keeps search small and inspectable, although questions without matching vocabulary can miss relevant facts.

Do the 149 cases prove it is safe?

I used them to develop and tune the service. Independent evaluation is needed to check its responses to new questions; prompt injection and review mistakes remain limitations.