# OPAL: How Reactive Optimizers Respond to Overfitting During AI Training A model can improve at predicting its training labels while becoming less reliable on new examples. That problem becomes especially clear when some labels are wrong: continued training can turn an image classifier into an effective memorizer of mistakes. **A falling training loss does not necessarily mean the model is learning something useful.** [OPAL](https://github.com/xingjian-bai/opal), short for OPtimizer with an Agent in the Loop, explores a different way to manage this problem. It lets a coding agent inspect a running training job and rewrite the optimizer's behavior. In a Better Stack experiment with deliberately mislabeled images, the agent-assisted model reached **79.1% test accuracy versus 74.4% for a fixed AdamW recipe**, while repeating far fewer incorrect training labels. The result is interesting because the model architecture and training data stayed the same. What changed was the software deciding how the model's weights should move. Understanding that distinction also explains the limits: this was one controlled experiment, several interventions failed, and a familiar weight-averaging technique accounts for part of the improvement. ## Why an optimizer can keep learning the wrong thing Training an image classifier involves comparing its predictions with the labels supplied in a dataset. A loss function measures the disagreement, and backpropagation computes gradients that indicate how the model's parameters affect that loss. The optimizer uses those gradients to update the parameters, also called weights. The learning rate controls the size of those updates. Other choices affect how gradients are accumulated, how previous updates influence the next one, and whether different parameter groups receive different treatment. Together with the learning-rate schedule, these choices form part of a training recipe. [AdamW](https://docs.pytorch.org/docs/stable/generated/torch.optim.AdamW.html) is a widely used optimizer that combines adaptive updates with decoupled weight decay. Its updates already depend on information collected during training. Calling a recipe *fixed* therefore does not mean every update is identical or that the learning rate never changes. It means the governing algorithm and schedule were selected beforehand. That distinction matters when the dataset contains incorrect labels. A cat labeled as a truck still contributes a training signal encouraging the classifier to predict “truck.” The optimizer receives that signal through the loss function; it does not have an independent understanding of the image's correct class. A sufficiently flexible model may initially learn useful visual patterns and later fit the mislabeled examples too. Training accuracy rises, but performance on clean, unseen examples can deteriorate. A validation set helps expose this divergence because its examples do not participate in gradient updates. An ordinary training pipeline can already respond through predefined mechanisms such as early stopping or learning-rate adjustments. OPAL extends the range of possible responses by allowing an agent to write a different update procedure after examining the run. ## How OPAL connects an agent to a training run OPAL wraps an existing PyTorch optimizer rather than replacing the model with a language model. The neural network still performs the classification task. The coding agent operates around the training process, inspecting measurements and deciding whether the optimization code needs to change. Two editable programs form the feedback loop. `analysis.py` gathers information about training, while `updater.py` determines how gradients become parameter updates. The agent can change what it measures as well as how it reacts. The [Reactive Optimizers research](https://arxiv.org/abs/2610.33826) studies this design across several training settings. ![OPAL's feedback loop, with analysis.py carrying training measurements to an agent and updater.py carrying revised update logic back to training](https://imagedelivery.net/xZXo0QFi-1_4Zimer-T0XQ/319ba99a-edc4-483b-dfec-a5675dbbc700/lg2x =1920x1080) The arrows represent different responsibilities. Measurements help the agent form a hypothesis, while executable update code gives that hypothesis an effect on subsequent training. An explanation in the agent's notes is not enough by itself: its usefulness depends on what the resulting code does and what happens afterward. The project's documented integration interface illustrates the wrapping mechanism. This excerpt assumes an existing `model` and `total_steps`; it represents the connection between the original optimizer and OPAL, rather than a complete training script: ```python [label training.py] import torch import opal optimizer = torch.optim.AdamW(model.parameters(), lr=3e-4) [highlight] optimizer = opal.wrap( optimizer, model, total_steps=total_steps, goal="An image classifier judged by held-out accuracy after training", backend="keep", ) [/highlight] ``` Here, `backend="keep"` leaves the programs unchanged and makes no agent calls. The project uses this mode to check that wrapping the optimizer preserves the original run before enabling agent decisions. The training loop still computes a loss, backpropagates, and calls the wrapped optimizer. The [repository README](https://github.com/xingjian-bai/opal) describes that interface and its agent-enabled configuration. The intended scope is optimizer behavior: update rules, schedules, and related state. Keeping the architecture, dataset, and evaluation procedure unchanged makes it possible to ask whether the optimizer intervention helped, rather than comparing two entirely different learning systems. ## A controlled experiment with incorrect image labels The Better Stack experiment used ResNet-18, a residual neural network for image classification, with [CIFAR-10](https://cave.cs.toronto.edu/kriz/cifar.html). CIFAR-10 contains small color images from ten classes, including cats, dogs, airplanes, and trucks. Its official split provides 50,000 training images and 10,000 test images. The experiment divided the official training portion again, reserving 5,000 clean images for validation. Of the remaining 45,000 training images, 4,500 received an incorrect label. Those deliberately corrupted examples create a direct way to measure memorization: after training, does the model repeat the assigned wrong class or recover the original class? | Component | Role in the experiment | | --- | --- | | 45,000 training images | Used for weight updates, including 4,500 incorrect labels | | 5,000 clean validation images | Provided feedback about performance beyond the training examples | | 10,000 test images | Used for the final assessment, withheld from the agent's feedback | | ResNet-18 | Same model architecture in both runs | | Fixed AdamW and OPAL | Two approaches to updating the model's weights | Both runs used a single rented NVIDIA H100 graphics processing unit (GPU), with Claude Opus 5.5 acting as the OPAL agent. The experiment ran for 120 epochs, where an epoch represents a pass through the training set. The agent was not told that labels had been deliberately corrupted. Validation feedback is an important qualification. The agent did not infer the problem from training loss alone: it could compare training behavior with clean validation performance. The test set served a separate purpose, assessing the resulting model rather than helping the agent choose interventions. This arrangement evaluates adaptation within a deliberately difficult training setting. It does not establish how much an agent would help on a clean dataset, against every tuned recipe, or across repeated runs. ## What changed when the model started overfitting The agent's first intervention changed the update rule for much of the model. That alone did not resolve the eventual problem. Around epoch 24, training performance improved while clean validation behavior worsened, and the agent's notes explicitly identified overfitting. ![Agent notes identify overfitting around epoch 24 as training loss falls while validation behavior deteriorates](https://imagedelivery.net/xZXo0QFi-1_4Zimer-T0XQ/63df4b71-8abb-4ec5-a437-51f24f95aa00/public =1920x1080) The useful evidence is the relationship between measurements. A lower training loss is expected when the network fits the supplied labels more closely. It becomes concerning when the same changes do not improve clean validation performance. The agent's diagnosis was a hypothesis based on that divergence, not proof that it knew exactly which labels were incorrect. ### Why the early interventions failed One attempted response was flooding, a regularization idea that resists pushing training loss below a chosen level. In this run, the intervention worsened performance. Adding noise to the weights also failed to deliver the desired improvement. A later attempt to apply flooding only to the final layer drove validation accuracy down to roughly 52%. ![The experiment dashboard records unsuccessful flooding and weight-noise interventions before an attempt to apply flooding only to the classifier weights](https://imagedelivery.net/xZXo0QFi-1_4Zimer-T0XQ/7f379967-c5ac-4f39-5699-b90bbbfd2b00/md1x =1920x1080) These failures are part of the result. The agent was not supplying a reliably correct adjustment at every decision. It was testing ideas against feedback, sometimes degrading the model before finding a more useful approach. That behavior has practical implications for long training jobs. An intervention can be plausible and still be unsuitable for the current run. Useful oversight therefore depends on retaining measurements and model states, rather than assuming an agent's explanation is enough to justify a change. ### Why averaging helped without immediately removing memorization The more successful approach combined larger updates with averaging of model weights over time. Instead of relying on one instantaneous set of parameters, a weight average combines information from multiple points in the training trajectory. The agent initially proposed that averaging would cancel memorized labels. Clean validation accuracy improved to about 78%, but the accompanying measurements still showed substantial memorization. Its notes acknowledged that the proposed mechanism was wrong. ![Agent notes report that averaging improved validation accuracy but did not cancel memorization as originally predicted](https://imagedelivery.net/xZXo0QFi-1_4Zimer-T0XQ/ce0d0162-0a8d-40f2-cc62-a54221bb8500/md1x =1920x1080) This is a useful distinction between an improvement and an explanation for that improvement. The averaged model performed better, but that did not establish the agent's initial account of why it worked. Further increases in update size were followed by a sharp decline in the wrong-label memorization measure. A later intervention overshot, after which the agent selected a previously saved merged weight state and held it for the end of training. The final result therefore reflects both adaptation and selection, including recovery from an unsuccessful late change. ## What the final measurements show The held-out test results favored the agent-assisted run. Accuracy increased from 74.4% to 79.1%, a gain of 4.7 percentage points. Test loss fell from 2.39 to 0.70. ![Final held-out test results show fixed AdamW at 74.4 percent accuracy and 2.39 loss, compared with the agent-assisted run at 79.1 percent and 0.70](https://imagedelivery.net/xZXo0QFi-1_4Zimer-T0XQ/15676754-be20-48f4-348c-cd02a55e1300/md2x =1920x1080) Accuracy and loss describe different aspects of the predictions. Accuracy counts whether the selected class is correct. Cross-entropy loss also responds to the probability assigned to the correct answer, so a confidently incorrect prediction can be much more costly than a less certain mistake. The lower loss supports a broader improvement in this experiment, without by itself proving perfect calibration. The deliberately incorrect labels provide another view. The fixed-recipe model repeated all 4,500 wrong training labels. The agent-assisted model repeated 250, about 5.6%, and assigned the true class to 3,447, about 76.6%. ![The memorization evaluation shows 4,500 wrong labels repeated by fixed AdamW, 250 repeated by the agent-assisted model, and 3,447 true classes recovered](https://imagedelivery.net/xZXo0QFi-1_4Zimer-T0XQ/4bfe8f83-023f-447f-f07e-606bd27bd600/lg2x =1920x1080) The cat-versus-truck example makes this concrete. A cat image carried a truck label in the training data. The fixed model predicted truck, while the agent-assisted model predicted cat. Individual examples are selected illustrations; the counts across all 4,500 corrupted images provide the more useful summary. Recovering many original classes also does not mean the agent repaired the dataset. The labels supplied for training remained incorrect. What changed was how the resulting model behaved when evaluated on those images. The final count should also be distinguished from intermediate dashboard values. Memorization varied substantially during training, and a temporary low point is not the same as the measurement for the final selected weights. The intervention timeline separates detecting a problem, testing an unsuccessful change, and finding an approach that improves validation. Those are different kinds of progress. ![Timeline distinguishes the early overfitting diagnosis, failed flooding, revised mechanism, and successful averaging](https://imagedelivery.net/xZXo0QFi-1_4Zimer-T0XQ/f31866e1-af8a-4be2-1ad3-75ec4851d500/lg2x =1920x1080) ## How much of the gain comes from weight averaging? Weight averaging predates OPAL. The 2018 paper [Averaging Weights Leads to Wider Optima and Better Generalization](https://arxiv.org/abs/1803.05407) introduced stochastic weight averaging, which combines weights collected along a suitable optimization trajectory. Its purpose is to improve generalization rather than simply select the checkpoint with the lowest training loss. That history changes how the OPAL result should be interpreted. The interesting contribution is adapting and assembling techniques during a run, not inventing averaging from scratch. The researchers' separately reported noisy-label experiment provides a stronger comparison than the fixed recipe alone. Their table includes an AdamW baseline with an exponential moving average (EMA) strategy adapted from OPAL. EMA gives different influence to different points in time; it is related to, but not identical to, the original stochastic weight averaging method. ![The researchers' separate comparison includes final AdamW, early stopping, an EMA baseline, and OPAL, with different accuracy and memorization results](https://imagedelivery.net/xZXo0QFi-1_4Zimer-T0XQ/924cc4e7-4294-497e-42f5-82a6f8efc200/lg2x =1920x1080) In that [research experiment](https://xingjianbai.com/reactive-optimizers/), final AdamW accuracy was 73.5%, the adapted EMA baseline reached 77.5%, and OPAL reached 78.6%. Early stopping on validation produced 72.9%. These are separate measurements from Better Stack's 74.4% and 79.1% results. The EMA comparison shows that a substantial portion of the improvement is available without an agent continuously rewriting code. It also leaves a narrower question: what additional value comes from choosing and modifying the strategy during training? A single run cannot establish a dependable margin over a carefully tuned averaging baseline. For evaluation, the relevant comparison is therefore wider than “agent versus no agent.” It includes well-chosen static recipes, averaging, stopping criteria, repeated seeds, and the time spent obtaining each result. ## The cost of reasoning while training waits The Better Stack agent-assisted experiment took about 45 minutes and reported approximately $7.60 in agent tokens. Its dashboard showed $7.62. The agent-assisted run took more than three times as long as the fixed run, largely because training paused for agent responses. Token spend and elapsed time are separate costs. A relatively small language-model bill can accompany a significant increase in GPU rental time. For a fast training job, waiting for repeated coding-agent decisions can dominate the work performed by the network itself. OPAL's [examples documentation](https://github.com/xingjian-bai/opal/blob/main/examples/README.md) also describes nonblocking operation through `blocking=False`. Training continues with the current programs while the agent works, and an edit takes effect after its response arrives. This can reduce waiting, but it changes when an intervention reaches the training process. The blocking result here should not be presented as a benchmark for that alternative. Longer jobs may offer more opportunity to absorb the cost of occasional decisions. Conversely, an already well-tuned recipe may leave little room for improvement. Both factors matter when deciding whether reactive optimization is useful for a particular workload. The length of the training run affects both the opportunity to adapt and the fraction of time spent waiting for decisions. ![Research discussion highlights reduced gains when a training run is short and its recipe is already tuned](https://imagedelivery.net/xZXo0QFi-1_4Zimer-T0XQ/5c6aa951-4f5f-47e1-5d45-17d95494e100/public =1920x1080) ## The boundary between a helpful agent and trusted training code Allowing generated Python to influence training introduces a trust question beyond model accuracy. The [OPAL guide](https://github.com/xingjian-bai/opal/blob/main/docs/GUIDE.md) distinguishes the agent's restricted shell from the generated programs, which execute inside the training process with its permissions. Instructions limiting changes to the optimizer are not a security boundary. The guide also describes information sent to the model provider, including training reports and, by default, source-code references. That matters when a run involves proprietary architectures or sensitive information. The generated code and its accessible resources require review alongside the numerical results. There is a related experimental concern: an agent can optimize toward any feedback it receives. Validation data can guide decisions, while final test data must remain a separate assessment. Code inspection and evaluation isolation help establish what a reported improvement actually measures. ## Final thoughts OPAL treats optimization as software that can be revised while a model trains. In the controlled Better Stack experiment, that approach produced higher test accuracy, lower test loss, and much less repetition of deliberately incorrect training labels. **The strongest result is the change in the final model's behavior, not the confidence of the agent's explanations.** The limitations are equally concrete. Several interventions failed, the run was substantially slower, and the researchers' averaging baseline recovered much of the accuracy gain without continuous agent control. One small-model experiment does not establish a universal remedy for overfitting. **Reactive optimization is most useful to evaluate where a training recipe still needs adaptation and the value of that adaptation can justify its overhead.** OPAL provides a way to explore that idea while preserving a record of changes. Reliable conclusions still depend on strong baselines, separate evaluation data, and scrutiny of the code producing the result.