|
EXPLAINER |
AI FOR ACCOUNTANTS · PART 14 OF 16
Why you probably do not need to fine-tune a model
The firms chasing custom models usually have a retrieval problem or a prompt problem - both of which are cheaper to fix.
Fine-tuning an AI model means taking a general-purpose model and retraining it on your own data so it behaves differently. It sounds like exactly what a firm would want: a model that knows your chart of accounts, your workpaper style, your clients. The problem is that fine-tuning is expensive, slow to update, and solves a different problem than the one most firms actually have. The more common issue is that the model never sees the right information in the first place, or the instructions given to it are vague enough that a new hire would also struggle.
Think of it like onboarding a staff accountant. You would not rewire their brain before letting them touch a file - you would hand them the engagement letter, the prior-year workpapers, and a clear set of instructions. What most firms call 'the AI doesn't know our process' is really 'we handed the AI a blank desk and a vague question.' Retrieval-augmented generation (RAG) is the structural fix: the model is given the relevant documents at the moment it needs them, the same way you would pull a client's depreciation schedule before asking a question about it. The model's knowledge does not change, but its answer does, because the context does.
Fine-tuning makes sense in a narrower set of circumstances. If you need a model to consistently produce output in a specific structured format - say, a particular workpaper template with fixed section headers and cross-reference codes - and prompt engineering alone cannot hold that format reliably across thousands of runs, fine-tuning can lock it in. It also helps when the terminology in your work is genuinely specialized enough that the base model consistently misreads it. A firm doing a lot of cost segregation work, for instance, might find the model conflates certain asset classifications in ways that retrieval alone does not cure. But those are edge cases, not the starting point.
The more common failure mode looks like this: a senior manager asks the AI to draft a memo summarizing the revenue recognition treatment for a specific client contract. The AI produces something plausible but wrong in the details, because it never saw the contract. The instinct is to conclude the model needs to be trained on revenue recognition. It does not. It needs the contract in the context window and a prompt that says exactly what sections matter. Retrieval handles the first part; a well-written system prompt handles the second. Neither requires a machine learning engineer or a training run.
The practical sequence for most firms is: write clearer prompts, add the relevant documents to context, and test systematically before concluding the base model is the problem. If output quality is still poor after those steps, the next question is whether the failure pattern is consistent enough to justify fine-tuning's overhead - including the fact that a fine-tuned model needs to be retrained every time the underlying guidance changes. For a profession where standards shift regularly, that maintenance cost is real. On the governance side, the AICPA and former IRS Commissioner Danny Werfel have launched the Council on AI Risk in Tax (CART) to develop shared guidance on responsible AI use in tax practice, which is the kind of framework that will matter more as firms move from experimentation to production.
The takeaway is not that fine-tuning is never appropriate. It is that most firms have not yet exhausted the simpler options. Get retrieval working. Write instructions that would make sense to a competent staff accountant seeing the task for the first time. Test on real files. If the model still fails in ways that are systematic and structural, then the conversation about fine-tuning is worth having - with someone who can estimate what it actually costs to maintain.
WORKED EXAMPLE
In practice
A senior manager needs to draft a client memo on the revenue recognition treatment for a multi-element software contract. The contract, the engagement letter, and the prior-period workpaper are all available as PDFs.
What came back. The model returned a structured memo that correctly identified three performance obligations and flagged a usage-based royalty clause as variable consideration requiring constraint analysis. It incorrectly described the implementation services as recognized over time using an input method when the contract specified a fixed milestone - a distinction visible in paragraph 7 of the contract but not caught without a follow-up review.
How it was checked. The reviewer compared each performance obligation and its recognition pattern against the contract's delivery and payment milestone schedule before the memo left the file.
A constructed example. The prompt is usable as written; the figures show the shape of a result, not a measured one.
WHEN TO USE IT
| WHEN NOT TO
|
WHAT TO TAKE FROM THIS
| Before considering fine-tuning, confirm the model has access to the actual documents it needs. | |
| Check whether your prompt specifies format, scope, and relevant standards as explicitly as a review note would. | |
| Fine-tuned models need retraining when guidance changes - factor that maintenance cost in. |
SPONSORED
QUESTIONS THIS ANSWERS
What is the difference between fine-tuning and retrieval-augmented generation?
Fine-tuning changes the model's weights through additional training on your data. RAG leaves the model unchanged but feeds it relevant documents at query time - like handing a staff accountant the right workpapers before asking a question.
When does fine-tuning actually make sense for an accounting firm?
When a specific structured output format cannot be held reliably through prompting alone, or when specialized terminology is consistently misread by the base model in ways that retrieval does not correct.
What should a firm do before deciding a model needs to be customized?
Write more explicit prompts, ensure relevant documents are in the context window, and test systematically on real files. If failures remain consistent and structural after those steps, then fine-tuning is worth evaluating.
SOURCES
Where this comes from
What the accounting job market is actually asking for.
GO DEEPER
Go deeper
IN THIS SERIES
Previously: What an AI audit trail should contain
Next: Where client data goes when the assistant answers (coming)
An explainer, not a study: it carries no statistics on purpose. Examples are illustrative.
How accountants are using AI, automation and smarter workflows to close faster, audit cleaner, and free up time for real work.
Accounting Stack · Audit Friendly Data · Accounting & Finance Jobs
Audit Friendly · modernaccounting.ai