‘Private AI’ usually means the language model runs on your own infrastructure. That matters, but a document can leave a system by more routes than the model. This piece walks through those routes, using a client prototype as the worked example: a document-analysis system built so that no document content is sent to external AI services.
- The model is one route out. Embeddings, storage, logs and other users’ sessions are others.
- Every component that touches document text has to run where the documents live.
- Privacy is also about accuracy: if the AI never generates the numbers, it cannot invent them.
Route 1: the language model
The obvious one. Any question that reaches a cloud model carries document text with it. In the prototype, the language model (Qwen 2.5 14B) runs on a dedicated GPU server, so no data is sent to OpenAI, Google or any external API.
The check trace every call the application makes to a model and confirm each endpoint is on your own infrastructure.
Route 2: the embedding model
Retrieval-augmented generation turns every document passage into an embedding, and that conversion needs a model too. It is easy to run the language model locally and still send every passage to a hosted embedding service. In the prototype, the embedding model (nomic-embed-text) runs locally as well.
The check check the embedding step separately. It processes every word of every document, not just the passages a question touches.
Route 3: storage and other users
Once documents are indexed, the index is a copy of their content. In the prototype, files stay on the server, and each user has their own ChromaDB index, so each user’s questions are answered from their own documents only. Connections are encrypted with HTTPS.
The check ask where the index lives, who can query it, and whether one user’s search could ever return another user’s passages.
Route 4: what the AI is allowed to generate
Privacy and accuracy meet here. When numbers and narrative come from the same model, the model can invent statistics. In the prototype, counts, charts and tables are produced by Python code; the AI writes only the qualitative analysis from retrieved passages, and the two are combined only at presentation. Most data questions never reach the AI at all: a deterministic first tier with 16 patterns and a schema engine handle them, and those two tiers were validated against 96 test queries.
The check list which outputs the model generates and which come from code. Anything that must be exact should come from code.
From the project: Private Document Analysis AI
Questions to ask of any ‘private’ AI system
- Where does the language model run?
- Where does the embedding model run?
- Where are the index and the original files stored, and who can query them?
- Are users’ documents isolated from each other?
- Which outputs are generated by the model, and which by code?
- What leaves the server at all, and can you show it?
RJ Data Voyage builds privacy-first applied-AI systems that run entirely on infrastructure you control. If you are weighing up a private document-AI project, get in touch.