If you've spent any time with Claude Code, Cursor or any modern coding agent, you've probably noticed something frustating, they burn through tokens surprisingly quickly. Much of this cost comes from redundant reasoning, repeated context retrieval, and inefficient planning. So I had a plan, to take a large language model and somehow to make it optimise prompts to improve token efficiency and response accuracy.
Wanting to understand whether this problem had already been studied, I explored prior work on RLHF and prompt optimization. While my approach differs from existing methods, papers such as InstructGPT (Ouyang et al., 2022), PRewrite (Kong et al., 2024),\ and StablePrompt (2024) provided valuable inspiration.
// the setup
The policy model is Qwen2.5-1.5B-Instruct fine-tuned using LoRA adapters on all attention projections (r=16, alpha=32). On top of the base model I attached a small linear value head that maps from hidden states to scalar value estimates, trained jointly with the policy. This follows the standard actor-critic formulation used in PPO-based reinforcement learning.
To train the model, I generated 60 synthetic noisy restaurant orders containing filler words, typos, corrections, and conversational phrasing—for example: "hey um can i get a burger with no onions and maybe a large soda? actually make that diet. and freis too thanks". The desired output is a compact, normalized representation such as "burger (no onions) | soda (large, diet) | fries". Although the task itself is simple, the noisy inputs require the model to learn semantic compression, typo correction, and structure extraction.
Note: Restaurant order parsing was only chosen as a proof of concept, because it provides a simple, intuitive and easily evaluable environment for experimentation. The underlying approach however, is domain-agnostic. By replacing the training data and reward function, the same framework can be applied to prompt rewriting, query optimisation, or any setting where a model must learn to generate more useful or efficient intermediate representations.
// the reward
This is arguably the most interesting part. Instead of designing a hand-written reward function, I used a Groq-hosted Llama-3.1-8B model as an AI reward model. Each training sample requires two LLM calls:
1. The model's structured output is passed to Llama as a restaurant order to process and confirm. The result is a natural-language response such as: "Got it - one burger with no onions, one large diet soda, and fries."
2. The original messy order, the structured output, and the generated confirmation are passed to a second Llama call, which scores how well the final answer satisfies the original request, on a scale of 0 to 10, which is normalised to [0, 1] and used as the PPO reward.
This reward signal is inherently noisy: the same output may receive slightly different scores across evaluations. However, over many training iterations the signal remains useful. The model gradually learns that vague or partially structured outputs tend to score lower than clean, normalized representations.
In effect, this turns the system into a lightweight form of Reinforcement Learning from AI Feedback (RLAIF), where another language model acts as the evaluator instead of a human.
// what worked
The model does learn. By the second epoch, the outputs are noticeably cleaner than at the start. The pipe-separated format emerges consistently, filler words such as "um", "hey", and "actually" mostly disappear, and modifiers tend to attach to the correct items.
Using a real LLM as the reward model worked better than I expected. It captures mistakes that a regex-based scorer would likely miss, such as dropped modifiers, incorrect associations between items and attributes, or subtle semantic mismatches.
// what didn't
60 samples is simply too few. The model appears to quickly overfits the phrasing patterns present in the synthetic data and struggles on inputs that deviate from those templates. Increasing the diversity of the data generator would likely improve generalization considerably.
The reward is expensive. At the default settings, a full run requires 3 epochs * 20 steps * 4 samples * 2 LLM calls = 480 Groq API calls. Even with a fast inference provider, rate limits and network latency become noticeable. Fine for a weekend project—less so for larger-scale experiments.
Training on CPU is slow but surprisingly workable. A 1.5B parameter model with LoRA adapters fits comfortably in float32, though each PPO step takes noticeably longer than on GPU.
// code
Full code is available on GitHub: github.com/AngadBasandrai/tune-ai