Cleo’s quick replies only help if they arrive before users start typing, so we fine-tuned a specialized small model to reduce latency.
Quick replies (QRs) are the suggested response buttons that appear after Cleo replies to a user’s message, offering plausible next steps without requiring the user to type anything from scratch. When QRs work well, they reduce friction and make common actions easier to discover.
The usefulness of QRs, however, depends on their timing as much as their quality. As Cleo’s chat architecture evolved, QRs became part of a broader LLM-powered generative UI system. That architectural shift made the suggestions more relevant and expressive, but it also made latency a bigger concern. Even when the model produced reasonable suggestions, we often saw users begin typing before the QR buttons appeared.
That raised the engineering question behind this project: For a narrow task like this, did we actually need a large, general-purpose model, or would a smaller, fine-tuned one be a better production choice? We hypothesized that a small model trained on the right data could match or exceed the quality of the baseline (a larger-parameter, general-purpose model), while being fast enough to reach users before they started to type.
Quick replies appear to the user as suggested responses to a message from Cleo.
Why quick replies were a good candidate for fine-tuning
In general, QR generation is a simpler problem than freeform chat. The QR generation model sees recent conversational context together with application state, and its job is to return QR candidates in a fixed structure.
In practice, the model outputs fall into two categories: text QRs, which help users continue the conversation without typing, and app navigation responses, which send them to a specific app screen (e.g., AccountsScreen for connecting and managing linked bank accounts, or SpendScreen for transactions and card management). Because those outputs are relatively constrained, the task is easier to evaluate and train against. Rather than producing the best possible continuation of a conversation, we’re asking the model to produce QRs that are useful in context and appear before the user starts typing. The model’s value here depends less on versatility than on how well it fits a particular moment in the interaction.
The data available to us reinforced that view. The existing system already generated QRs in production, which meant we had access to a scalable, behaviorally grounded training signal in the form of real user interactions that told us which suggestions were contextually useful.
Training approach and data
Our baseline was GPT-4.1-mini, the model Cleo had integrated for chat shortly after its release in mid-2025. It generated reasonable QRs, but its latency was high enough that suggestions often arrived after users had already started typing, which made even good suggestions irrelevant. For the fine-tuned model, we moved to GPT-4.1-nano. On its own, GPT-4.1-nano isn’t adequate for this task (early checks put its accuracy below the GPT-4.1-mini baseline), but we expected that fine-tuning on Cleo data would close the quality gap while preserving the smaller model’s speed advantage.
Our training strategy relied on real product behavior rather than hand-labeled examples. Instead of building a human annotation pipeline to decide what the “best” QR should look like, we used the existing system’s traces to look for signals of success already present in the data.
We pulled roughly 2,000 conversations from our LLM logs and filtered to the Quick Replies step. We only kept cases where users had engagement with QRs, as engagement is a signal that the suggestion was useful in context. Where engagement data wasn’t available, we used an LLM judge to filter for good QRs. We then balanced the training set across routes so that rare destinations were not underrepresented relative to more common ones.
For the QR feature, a clicked QR is close to the actual product objective, since it indicates that the model produced a suggestion useful enough that the user preferred it to typing. Implicit behavioral data was both cheaper than annotation and better aligned with that objective. The pipeline itself was the simplest plausible version of this idea: pull traces, filter for higher-signal cases, build a balanced training set, translate into the required fine-tuning format, and validate before investing in more elaborate changes.
Training pipeline: Where users had engaged with a QR in production, we used that engagement as the training signal, while an LLM judge stepped in for cases where engagement data wasn’t available.
Validation framework
Before shipping the fine-tuned model, we evaluated it offline against the existing system along two axes:
Guardrail-style comparison. We measured how often the fine-tuned model agreed with the production model. While this measure can’t fully capture improvement on its own, high agreement means the new model is not wildly off distribution, which is a useful offline safety check. We applied it to both the response type (text vs. navigation) and the specific app navigation route.
Improvement signal. We measured how often the fine-tuned model agreed with a stronger oracle. For routing cases, where correctness can be judged more directly, we used GPT-4.1 (the highest-capability model from the same model family) as an LLM judge. The clearest offline gains appeared here. On AccountsScreen examples, oracle agreement improved from 39% to 78% (+39 percentage points). On SpendScreen, it improved from 58% to 72% (+14 points).
A qualitative review also found a pattern the numbers obscured. The fine-tuned model was sometimes more conservative in its navigation behavior, and several of the apparent offline regressions turned out to be product improvements.
In ambiguous cases where the baseline confidently routed the user to a screen that didn’t match their intent, the fine-tuned model would instead stay in the conversation and offer text QRs. The user could then clarify what they wanted before being pushed to another screen. The “regression” in the offline metric was, in practice, a softer and more appropriate response than the baseline’s.
Production results
That evidence justified a production test, and the A/B results confirmed our hypothesis:
QR engagement increased by 13.5% at the event level. The lift was driven mainly by button QRs, where engagement rose by 18%. Because buttons make up more than 90% of the relevant engagement volume, that alone accounted for most of the overall gain.
P50 latency dropped by 53%, from 1,310 ms to 631 ms with the fine-tuned model.
The latency change is what enabled the engagement gains; halving response time moved QRs back inside the window in which users are deciding how to continue the interaction.
Downstream effects pointed in the same direction: users were finding what they needed with fewer conversation turns. That also showed up as a 5% reduction in token cost for generating messages and calling tools.
Sensitivity to schema changes
The gains held up in production, but follow-up research surfaced an operational cost. Fine-tuning made the model more specialized, which also made it more sensitive to changes in the shape of its inputs and outputs, particularly the response schema.
A few weeks after launch, we noticed an increase in the rate at which the model generated app navigation buttons. Users were prompted to navigate away from chat at points that didn’t make sense, and occasionally the navigation button text was off, too. We hypothesized that the cause was a change to the model’s response schema, added to support a new product feature, that had been deployed without retraining the fine-tuned model on the new format.
To test that hypothesis, we ran a controlled comparison on the fine-tuned model, varying one input at a time:
Prompt changes. We updated the prompt to incorporate a group of features for the next product launch. Prediction distributions were roughly unchanged. Prompt edits didn’t appear to degrade the fine-tuned model meaningfully as long as the underlying task stayed the same.
Response format changes. We updated the output schema to include a new field for our upcoming launch. This change alone, with the prompt held constant and no additional fine-tuning, caused a 65% increase in UI button predictions along with several hallucinated outputs. The experiment reproduced the pattern we'd seen in production.
The asymmetry comes from how fine-tuning interacts with structured output. To keep a model’s JSON outputs valid, providers use a technique called constrained decoding, where at each generation step, the model is restricted to tokens that keep the output conforming to the schema. Fine-tuning on a fixed schema trains the model to produce outputs that fit that schema naturally, so the constraint rarely overrules the model. If the schema later changes, the constraint has to overrule the model’s learned behavior more often, introducing drift and hallucinations. A prompt, by contrast, is just input the model reads; there’s no equivalent enforcement on what tokens the model is allowed to produce.
In practice, the schema and the fine-tuned model have to stay in sync. Schema changes should trigger re-fine-tuning, and distribution-level monitoring on output types can catch drift when that retraining doesn’t happen in time.
What we learned
Model choice should match the workflow. A large general-purpose model could generate perfectly reasonable QRs, but if they arrive after the user is already typing, most of their value disappears. A smaller fine-tuned model only needs to match the baseline on quality and beat it on timing.
The tradeoff is that specialization introduces brittleness that doesn’t show up in offline metrics. A fine-tuned model is trained against a specific prompt and schema, and changes to either can produce drift and hallucinations. The right fit is a narrow, high-frequency task where the prompt and schema stay stable. Schema changes need to be paired with regenerating training data and re-fine-tuning, which is an operational cost that only makes sense in some scenarios. For QRs, the latency improvement was worth it.
