Every AI Session Starts With Amnesia, and I'm the One Carrying the Context

I have noticed a recurring problem in the way I work with AI tools. The models are getting better, but I am spending more time carrying context between them.

I might start with a coding agent inside a repository, move to a chat model to think through a product decision, and then use another agent to analyse a spreadsheet or a set of research documents. The work is connected, but the tools usually are not. I end up uploading the same requirements again, pointing another agent at an architecture document, explaining which spreadsheet is current, and repeating which decisions have already been made.

The project itself already exists. The code, requirements, research, datasets, screenshots and design decisions are all somewhere on my machine. What is missing is a durable way for different AI systems to access that material without making me reconstruct the project inside every new session.

This is not just a token-efficiency problem. Imagine a pricing spreadsheet changes the annual plan from $199 to $249, while a requirement document adds that upgrades must be prorated from the next billing cycle. One agent saw the old spreadsheet yesterday. Another agent sees the new requirement today but not the pricing change. A third agent is asked to update the entitlement logic from a compacted conversation summary. All three can appear competent while reasoning from different versions of the same project.

That is where the annoyance becomes risk. The question is no longer only “how do I give the model enough context?” It becomes “which project state did this agent actually use?”

The project is larger than the repository

A real software project rarely fits neatly inside Git. The repository contains the implementation, but much of the context required to understand that implementation lives elsewhere.

Requirements arrive as PDF or DOCX files. Commercial assumptions live in spreadsheets. Research accumulates in reports and notes. Screenshots capture flows that were never formally documented. Architecture decisions may sit in an old design document, a meeting summary or a thread that nobody has looked at for months.

Over time, a person develops a mental model of how these things relate. We remember that a particular pricing workbook affects subscription logic, that an architecture document explains why a service boundary exists, and that the latest requirement replaces an older one.

An AI agent does not automatically inherit that mental model. A coding agent can inspect the repository, but it may not know about the spreadsheet in another directory. A web chat can read an uploaded PDF, but that understanding normally belongs to that conversation. When the session ends, gets compacted or moves to another tool, much of the useful context has to be loaded again.

This is inconvenient when the material is static. It becomes a reliability problem when the project is changing.

A requirement is updated. A CSV receives new data. A pricing assumption changes. The code moves forward while the architecture document stays behind. One agent has seen the latest files while another is still working from a previous version. At that point, it becomes difficult to answer basic questions such as which sources the model used, whether it saw the current requirement, what changed since the previous session, and whether two agents are operating from the same understanding of the project.

In most current workflows, the answer is approximate.

We are using the context window as storage

A model’s context window should mainly hold the information required for the work it is doing now: the task, the plan, the relevant implementation details, important constraints and recent tool results.

Instead, we often use it as a temporary project database.

We load a repository overview, several documents, spreadsheet exports, previous decisions, conversation history and anything else the model might conceivably need. The model may only require three paragraphs from a hundred-page report or four rows from a large spreadsheet, but we frequently give it the whole artifact and expect it to find the useful part.

Larger context windows reduce some of the immediate friction, but they do not change the underlying model. The material still needs to be selected, loaded, interpreted and often reloaded in a later session. Long conversations are eventually compacted, at which point exact values, exceptions and the reasoning behind earlier decisions can be reduced to an imperfect summary.

The problem is not that models have no access to information. The problem is that project knowledge is being moved into temporary working memory over and over again.

Terraform for AI Context

My rough idea is to keep the project outside the conversation and let the model retrieve only what it needs for the current task.

The workflow would look less like uploading files into a chat and more like managing infrastructure state. You declare the sources, compile the current project state, inspect the diff, activate a known-good build and let connected tools work from that state. That is the Terraform mental model applied to AI context.

I imagine pointing a local tool at a repository and the surrounding project artifacts once. It would process them into a persistent representation of the project. Coding agents, chat models and custom agents could access that representation through interfaces they already understand, such as MCP, a local API or a generated context file.

The model would begin with a small map of the project rather than the full corpus. When given a task, it could request a focused context package containing the relevant requirements, data, decisions, code and tests. If it needed exact wording from a document or specific rows from a spreadsheet, it could fetch those separately.

For example, a coding agent asked to implement subscription upgrades should not need to reread every product document. It should be able to request the context for that task and receive the current subscription requirement, the relevant pricing rows, the architecture decision governing entitlements, the associated source files and the existing tests. If the price changed since the last build, that should be visible. If the requirement supersedes an older one, that should be explicit.

This is not only a search problem. Search can return passages that resemble a query. The more useful capability is assembling the set of information required to complete a task correctly against the current version of the project.

That distinction is the center of the product. RAG asks, “Which passages match this question?” A project context runtime should ask, “What does this agent need to know to perform this task against the active project build?”

“Can’t I already do this with X?”

That is the obvious response, and it is reasonable. Most of the individual components already exist.

Each of these solves part of the problem, but none of them necessarily provides a consistent project state across changing repositories, external documents, structured data and multiple AI clients.

For this idea to justify another tool, it cannot be another document index with an MCP endpoint. Search, embeddings and protocol support are necessary parts of the implementation, but they are not enough to distinguish the product.

The stronger idea is to treat context as explicit project state. Retrieval finds information. A context runtime should define the project state an agent is allowed to reason from.

That also means Lorepack would not replace Git. Git should remain the source of truth for repository files and code history. Lorepack would record the Git commit a context build corresponds to, then add the external sources Git does not usually know about: requirements, spreadsheets, research, screenshots and decisions.

Context as a build artifact

Most document systems follow a relatively simple lifecycle:

files -> index -> search -> passages

I am thinking about something closer to:

repository + requirements + data + decisions
                      |
                      v
             reproducible context build
                      |
                      v
        task-specific context for an agent

A context build would represent an exact combination of a Git commit, external document versions, dataset fingerprints, source metadata and any declared authority or supersession rules.

That would make it possible to say that an agent worked against a specific project-context version rather than vaguely saying it had access to some files. If the agent produces a different result after a requirement changes, there is a concrete state difference to inspect.

This is where the Terraform comparison becomes more than a metaphor. Lorepack would turn project context into a build artifact AI agents can inspect, diff, activate and roll back.

If context becomes trackable, should an agent’s work be tied to a specific context state identifier? Agent X finished task Y based on context version Z. If the result is wrong, you can inspect exactly what Z contained: the repository commit, the requirement version, the pricing CSV, the decision record and the sources the agent was allowed to use. I am not sure yet whether this belongs in the first version, but it opens an interesting possibility: something like AI git blame, where failures can be traced back to the project state an agent actually reasoned from.

When one source changes, the system should identify the affected parts rather than rebuilding or rereading everything. A changed spreadsheet might update one typed table and invalidate a small number of task views. A revised requirements document might add a new constraint while leaving the rest of the project representation untouched.

The lifecycle could look roughly like this:

source change -> incremental build -> diff -> validation -> activation

If the new state is incorrect, the previous context build should still be available. The goal is not to force a manual approval process for every file edit. The goal is to make changes visible and reversible when they affect agent behaviour.

Preserve the shape of the source

The internal representation also matters. Treating every source as anonymous text loses useful information too early.

A PDF or DOCX file should retain its document hierarchy, headings and provenance. A CSV or Excel workbook should retain schemas, column types and queryable rows. Code should remain connected to symbols, dependencies, tests, branches and commits. Decisions should retain their rationale and indicate what they superseded. Screenshots should remain accessible as visual artifacts rather than being reduced solely to an OCR transcript.

Retrieval can then use the appropriate mechanism for each source. A model can search a document, query a table, inspect a source symbol or compare two versions of a requirement. It does not need to treat all of those operations as variations of vector similarity.

The interface exposed to the agent can remain simple, but it should not depend on vague magic. The user or project should declare sources, precedence and important relationships. The runtime can then expose explicit operations such as project overview, task context, source fetch, table query and context diff.

Instead of asking for search results related to “subscription upgrade,” the agent should be able to ask for the context required to implement subscription upgrades. The runtime can assemble a bounded package and provide references to exact sources when further detail is needed.

That package might include the authoritative requirement, relevant rows from the pricing table, the active architecture decision, related code symbols, existing tests and any known ambiguity. The agent receives a task-oriented view of the project rather than a loosely related set of retrieved passages.

A small CLI-first experience

There may be substantial machinery underneath this: parsing, structured-data imports, source fingerprinting, provenance, incremental compilation, indexing, version activation and protocol adapters. The normal workflow should hide most of it.

Using Lorepack as the working name for this implementation, an initial setup could look like this:

cd project-x

lore use . ~/Documents/project-x
lore connect claude-code

The first command would discover the repository and related artifacts, create the initial project-context build and start a local runtime. The second would connect that runtime to Claude Code through MCP.

There would still be a configuration step for anything that should not be inferred. If a contract document overrides an older product brief, that precedence should be declared. If a spreadsheet column is the authoritative source for price, that should be visible. The tool should make the common path simple without pretending source authority can be guessed reliably.

Normal use should require very little ceremony:

lore dev
claude

When an artifact changes, Lorepack would rebuild the affected representation and activate a new valid context version. The connected agent would not need to reconnect or read the entire project again.

For a web chat that cannot connect to a local runtime, the same project state could produce a bounded context export:

lore export \
  --task "Review the new pricing strategy" \
  --profile chat

That export would contain the material selected for the task, source references and a record of what was omitted. It would still involve uploading a file, but it would replace a manually assembled collection of unrelated artifacts with a reproducible task-specific package.

Project knowledge and agent memory are different things

Long-running agents will continue to compact their conversation histories. That is unlikely to disappear.

A persistent project context would make compaction less damaging because the canonical project state would remain outside the conversation. After compaction, the agent could request the current context for its task instead of relying entirely on a lossy summary of everything it had previously read.

The agent’s own work state should be stored separately. Its plan, observations, changed files and unresolved questions are temporary task memory. The repository, requirements, data and formal decisions are canonical project knowledge. Agent memory can refer to project knowledge, but it should not silently modify it.

This separation would also make it easier for another agent to continue the work. The second agent could load the current project state and a compact task checkpoint without being given the entire previous conversation.

What would make it worth building

The architecture is not enough. The tool would need to improve real work in ways that can be measured.

The relevant test is not whether it can answer a question about a PDF. Many tools already do that. The useful test is whether an agent can understand a mixed project faster, use less context, recover accurately after compaction, recognise when an external requirement changes and remain consistent with another agent using the same project state.

A convincing first demonstration should be deliberately narrow. Combine a repository, one requirement document and one spreadsheet. Ask a coding agent to implement subscription upgrade behaviour using a bounded task context. Then change the requirement from “upgrade immediately” to “prorate from the next billing cycle” and change the annual price in the spreadsheet. The system should show the resulting context diff, create a new build and let a fresh agent work from the updated state without rereading the entire project. Rolling back the context build should restore the previous project state.

That is the v0.1 proof I care about: can a coding agent implement a feature from a bounded context package, then correctly adapt when an external requirement changes, without rebuilding project understanding from scratch?

If ordinary filesystem access and repository search produce the same result with less complexity, then this does not need to exist. The value has to come from making changing, multi-source project context reliable across sessions and agents, not from packaging existing retrieval components under another name.

The idea I am exploring is therefore not another document chatbot or local RAG server. It is a local-first project context runtime that turns repositories, documents, structured data and decisions into a reproducible project state, then gives each AI system only the information required for its current task.

That is the problem I would like to solve.