|
EXPLAINER |
AI FOR ACCOUNTANTS · PART 4 OF 16
Retrieval vs. training: what an LLM actually knows about your books
One mechanism reads your file at runtime; the other baked in the world as of a cutoff - and mixing them up causes real errors in practice.
There are two ways an LLM can appear to know something about your firm. The first is training: the model absorbed enormous amounts of text before it was released, and that knowledge is now fixed inside its weights. The second is retrieval: at the moment you ask a question, the system pulls a relevant document - a policy, a workpaper, a chart of accounts - and feeds it into the prompt alongside your question. The model then reasons over what it just received, not over what it memorized months ago.
Training is why a general-purpose model can explain the matching principle or describe how a bank reconciliation works without you providing a textbook. That knowledge is baked in. But your client's chart of accounts is not. Account 4210 - Rental Income does not exist in any training corpus; it exists in a file on your server. If you ask the model to categorize a transaction against your chart without giving it the chart, it will invent plausible-sounding account names or silently use a generic structure it absorbed during training. The output will look reasonable and be wrong in ways that are easy to miss on a first pass.
Retrieval-augmented generation - RAG, in the shorthand - solves this by attaching the relevant content at question time. A well-built RAG workflow for a workpaper review would pull the specific trial balance, the prior-period lead schedule, and the relevant policy excerpt, then present all three to the model before asking it to identify unusual variances. The model is reasoning over your actual numbers, not over some general notion of what numbers look like. The limitation is that retrieval is only as good as what gets pulled: if the wrong version of a schedule is retrieved, or a key document is not in the index at all, the model works confidently with incomplete context.
For practitioners, the practical distinction is this: use the model's trained knowledge for tasks that are general by nature - drafting an explanatory memo, summarizing an accounting standard, structuring a checklist. For anything that requires your client's specifics - account mappings, entity structure, prior-year balances, engagement-specific policies - assume the model does not have that information and hand it the relevant pages explicitly. Pasting in a chart of accounts or a reconciliation template before asking a question is not a workaround; it is how the mechanism is supposed to work.
A firm building more systematic workflows around this distinction will find that the architecture question - what gets indexed, how retrieval is triggered, how context windows are managed - matters as much as which model is chosen. PrimeGlobal's technology advisory partnership with Sikich, announced to help member firms build AI and automation capabilities, is one sign that this kind of infrastructure thinking is moving from early adopters into broader firm practice.
WORKED EXAMPLE
In practice
A senior accountant is reviewing a bank reconciliation workpaper for a mid-size client. The reconciliation has 14 outstanding items and uses account codes specific to the client's chart. She wants the model to flag items that look inconsistent with the account descriptions.
What came back. The model returned a list of four flagged items: two with account codes that did not match their descriptions, one that had carried forward from the prior month with no change in amount, and one described only as 'misc transfer.' It missed a fifth item where the account code was correct but the amount differed from the supporting deposit slip by a small rounding figure - that item required a manual check against the source document.
How it was checked. The accountant traced each flagged item back to the underlying bank statement and the client's general ledger export, confirming the two mismatched codes and the stale item before clearing the others.
A constructed example. The prompt is usable as written; the figures show the shape of a result, not a measured one.
WHEN TO USE IT
| WHEN NOT TO
|
WHAT TO TAKE FROM THIS
| Never assume the model knows your chart of accounts - paste it in explicitly. | |
| Retrieval errors are silent: wrong document version in the index, plausible output on screen. | |
| Use trained knowledge for general tasks; retrieval context for anything client-specific. |
SPONSORED
QUESTIONS THIS ANSWERS
What is retrieval-augmented generation in accounting?
RAG is a technique where relevant documents - a chart of accounts, a prior-period schedule, an engagement policy - are pulled and attached to the prompt at question time, so the model reasons over your actual content rather than its frozen training data.
Can an LLM learn my firm's chart of accounts through training?
Not through standard training. A general-purpose model's weights are fixed at a past cutoff and contain no client-specific data. Your chart of accounts has to be provided at runtime, either pasted into the prompt or retrieved from an indexed source.
What goes wrong if I skip retrieval and rely on the model's training?
The model fills gaps with plausible-sounding detail drawn from generic accounting knowledge. It may invent account numbers, use a standard chart structure instead of yours, or reference policy language that does not match your engagement - without flagging any of it as uncertain.
SOURCES
Where this comes from
What the accounting job market is actually asking for.
GO DEEPER
Go deeper
IN THIS SERIES
Previously: What an MCP server is and why vendors ship one
Next: What human in the loop means in an audit file (coming)
An explainer, not a study: it carries no statistics on purpose. Examples are illustrative.
How accountants are using AI, automation and smarter workflows to close faster, audit cleaner, and free up time for real work.
Accounting Stack · Audit Friendly Data · Remote A&F Jobs
Audit Friendly · modernaccounting.ai