Giving JARVIS a memory

The first version of JARVIS was impressive for about ten minutes. It answered questions, wrote code and was polite. Then I closed the terminal, opened it again, and it had no idea who I was.

That's not a bug. A language model has no memory between runs. Every conversation starts from zero, and the only thing it "knows" about you is what you put in the prompt right now. So the question becomes: what do I put in the prompt?

The idea: remember facts, recall the relevant ones

You can't paste your entire life into every prompt. Models have a limited context window, and long prompts are slow on a laptop. What you want is:

1. When something worth remembering happens, save it.
2. Before every answer, find the few memories that matter for this question.
3. Add only those to the prompt.

Step 2 is the interesting part. "When is my exam?" should find the memory "my exam is on Friday" even though the words barely overlap. Keyword search can't do that. Embeddings can.

Embeddings in one paragraph

An embedding model turns a sentence into a list of numbers, a point in space, so that sentences with similar meaning land close together. "My exam is on Friday" and "when is my test?" end up near each other. Searching memory becomes: turn the question into a point, then find the nearest stored points.

I run the embedding model locally too, through Ollama:

ollama pull nomic-embed-text

The whole memory system

import chromadb, ollama

db = chromadb.PersistentClient(path="jarvis_memory")
mem = db.get_or_create_collection("memories")

def embed(text):
    return ollama.embeddings(model="nomic-embed-text", prompt=text)["embedding"]

def remember(fact):
    mem.add(ids=[str(mem.count())], documents=[fact], embeddings=[embed(fact)])

def recall(question, k=3):
    if mem.count() == 0:
        return []
    hits = mem.query(query_embeddings=[embed(question)], n_results=min(k, mem.count()))
    return hits["documents"][0]

That's it. ChromaDB stores the vectors on disk in a folder, so memories survive restarts. Before each answer, JARVIS calls recall(question) and prepends the results:

context = "\n".join(recall(q))
prompt = f"Things you remember:\n{context}\n\nQuestion: {q}"

Now remember my exam is on Friday, restart, when is my exam? works.

What I learned

• Retrieval beats a bigger model. A small 3B model with the right three memories answers better than a big model with none.
• Bad memories are worse than no memories. If you save every line of chat, recall fills up with noise. Saving only facts worth keeping matters more than the database you pick.
• It all runs offline. No API bill and nothing leaves the laptop, which matters for an assistant that's supposed to know personal things.

Next I'm teaching it to decide by itself what's worth remembering, instead of me typing remember.

The full step-by-step setup, with the chat loop and folder structure, is in the free Build your own JARVIS guide.