'Talk to your data': RAG over business data, done responsibly

"Can't we just ask the data questions in plain English?" is now the most common request I get from business users. The honest answer is: yes — but only if you build it the right way. The right way has a name, Retrieval-Augmented Generation, and most of the work has nothing to do with the model.

The instinct behind "talk to your data" is reasonable. People don't want to file a request and wait two days for a chart; they want to type a question and get an answer. The naive way to deliver that is to point a chatbot at a powerful model and hope it "knows" your business. It doesn't. A language model knows language — patterns from its training — not your last-quarter revenue, your contract terms, or which SKU shipped late. Ask it directly and it will happily produce a confident, plausible, and wrong number. That failure mode has a name too: hallucination, where a model generates fluent text untethered from any real source (overview here).

What RAG actually is

Retrieval-Augmented Generation flips the order. Instead of hoping the model already knows, you retrieve the relevant pieces of your own data first, then hand them to the model as context, and ask it to answer from that context. The model stops being the source of truth and becomes the thing that reads your source of truth and explains it in words. The term comes from a 2020 paper by Lewis and colleagues at what was then Facebook AI (arXiv:2005.11401), and it has quietly become the default pattern for putting AI on top of business data.

The mental model I give people is simple: it's an open-book exam, not a memory test. You're not asking the model to recall your figures. You're putting the right page in front of it and asking it to read.

Why this is the sane way to "ask the data"

Once you internalise the open-book framing, the benefits fall out naturally. Because the answer is assembled from retrieved passages, you can show which passages — that's grounding, and it's what makes citations possible. A business user doesn't have to trust a black box; they can click through to the contract clause or the report row the answer came from. You also stop needing to retrain anything when the data changes. New figures land in your warehouse, retrieval picks them up, and the next answer is current. The model stays frozen; your data does the moving.

It also scales to the boring-but-real shape of company knowledge: policies in PDFs, definitions in a wiki, numbers in tables. Retrieval can span all of them and bring back the three paragraphs and two rows that actually matter, instead of stuffing everything into the model and hoping. Microsoft frames the same idea well in its RAG overview: the retrieval system owns relevance and access, the model owns phrasing.

The responsible part starts here

RAG makes "talk to your data" possible. It does not make it automatically safe. The same machinery that retrieves the right document will cheerfully retrieve a document the asker was never meant to see, and the same model that summarises faithfully will, on a thin retrieval, fill the gap with a confident guess. Responsibility is mostly about closing those two doors.

• Enforce access control at retrieval, not in the prompt. The bot must inherit the asker's permissions. If a salesperson can't open the finance folder normally, the assistant must not fetch it on their behalf. Trim the candidate set by identity before anything reaches the model — never ask the model to "please don't reveal" restricted rows.
• Ground every answer and show the citations. Each claim should trace to a retrieved source the user can open. No source, no claim. This is what turns "trust me" into "check me".
• Guard against confident-wrong answers. When retrieval comes back weak or empty, the right behaviour is "I don't have enough to answer," not a fluent invention. Make abstaining a first-class outcome.
• Keep a human check on decision-grade output. For anything that moves money, people, or contracts, the assistant drafts and a person decides. Speed up the human; don't replace the judgement.
• Log questions and sources. Being able to answer "why did it say that, and to whom?" later is the difference between a governed tool and a quiet incident.

Where RAG quality actually comes from

Here is the part teams underestimate. When a "talk to your data" pilot disappoints, the instinct is to blame the model and shop for a bigger one. In my experience it is almost never the model. It is retrieval and data quality. If the right document isn't indexed, or the table column is named amt_2 with no definition, or two systems disagree on what "active customer" means, no model on earth will rescue the answer. Garbage retrieval in, confident garbage out — just faster.

This is why I keep saying it to anyone who'll listen: most of "improving the AI" is improving the inputs. Better-curated sources, clearer definitions, fresher indexes, and honest evaluation of whether retrieval is actually surfacing the right passages will outperform a model upgrade nearly every time.

Your data foundations decide your AI ceiling

If there's one line I want to leave you with, it's the one I keep coming back to in everything I build: your data foundations decide your AI ceiling. RAG is the clearest proof of it I've found. The technique is elegant and the models are remarkable, but the quality of a "talk to your data" experience is set almost entirely by the quality, governance, and accessibility of the data underneath it. Do the unglamorous work — clean definitions, documented tables, proper access control, fresh indexes — and RAG turns your data into something a non-technical colleague can genuinely converse with. Skip it, and you've just built a very articulate way to be wrong.

That's the trade I'd take every time. The model is the easy part. The foundation is the moat.