AnyJev: Turn Any Open-Weight LLM into a Structured Decision Model

Ask a generative LLM whether a news headline is satire or real, and it will usually give you a paragraph explaining its reasoning. That is useful for a human reader, but awkward for an automated pipeline.

Your application then has to parse free text, interpret hedging such as “most likely” or “hard to be certain,” and convert that language into a value it can actually use.

Decision models take a different approach. Instead of generating prose, they return a probability distribution over a predefined set of options. The same classification task might return:

{"satire": 0.90, "real_news": 0.10}

That output can go straight into application logic without another parsing or interpretation step.

AnyJev is a Python library from Nokia’s applied research team that turns open-weight LLMs into decision models like this. It does not require fine-tuning or labeled training data, and because it runs on models you host yourself, your inputs do not need to be sent to a third-party API.

The result is a way to use existing LLMs for structured, probabilistic decisions rather than free-form text generation.

How AnyJev extracts a decision

The mechanism is a one-pass approach called next-token probability readout.

A standard generation request asks the model to produce tokens one by one until it produces a complete response. AnyJev stops after the first token. It formats the prompt to end with something like Answer:, then intercepts the raw logit scores the model assigns to every token in its vocabulary at that position. It isolates only the tokens corresponding to the option labels (A, B, C, or the option text itself), normalizes those probabilities to sum to 1, and returns the result.

This is fast because it's a single forward pass. It produces structured output because only the option-label tokens are examined. No generation loop, no text parsing.

The bias problem: why a raw LLM isn't enough

Running this approach on an unmodified model exposes two systematic biases that make results unreliable.

Position bias: The model's answer changes based on the order options appear in the prompt. Tested on the headline "The world bank's role in a bloody land war": with options presented as satire or real news, the raw Qwen3-8B model says 99% real news (correct). Swap the order to real news or satire and the same model says 99.8% satire (wrong). The content hasn't changed. The position of the options has.

Label bias: The model has an intrinsic preference for certain words based on training data. Qwen3-8B has a strong lean toward the word "satire" regardless of headline content, which inflates its satire predictions across the board.

AnyJev's L0 (Level 0) mode corrects both without any labeled examples.

L0 correction: how it works

For position bias: AnyJev runs the same query twice, rotating the option order between runs, then averages the results. If the model said satire: 0.99, real_news: 0.01 with order A, and satire: 0.02, real_news: 0.98 with order B, averaging produces satire: 0.505, real_news: 0.495, which reflects genuine uncertainty rather than a positional artifact. This doubles inference time but eliminates position sensitivity entirely.

For label bias: AnyJev offers two prior-estimation methods.

The batch prior is used when processing many inputs at once. It observes the model's answer distribution across the full batch, identifies systematic over-prediction of certain labels, and subtracts that lean from each individual decision. If the model answered "satire" 69% of the time across 500 headlines, it knows to discount satire predictions proportionally.

The content-free prior is used for single decisions. It feeds the model a neutral placeholder input (like "N/A") with the same question, observes the baseline distribution, and uses that as the bias estimate to correct the actual decision.

Installation and usage

AnyJev supports Hugging Face Transformers, vLLM, and llama.cpp backends. For local CPU or Metal GPU inference on GGUF models:

pip install anyjev llama-cpp-python

Download a Qwen3-8B GGUF model from Hugging Face, then:

from anyjev.backends.llamacpp import LlamaCppBackend
from anyjev import Decider, Question

# Load the model
backend = LlamaCppBackend("path/to/your/Qwen3-8B.gguf")

# Create a decider with L0 bias correction
# Use prior="content_free" for single decisions, prior="batch" for batches
decider = Decider(backend, level="L0", prior="content_free")

# Define the question and options
q = Question.choice(
    "Is this headline satire from The Onion, or real news?",
    ["satire", "real news"],
    name="verdict"
)

# Run the decision
headline = "Malaysian in hyrox bangkok disqualified for late start because he went to the toilet first"
d = decider.decide(headline, [q])["verdict"]

print(d.distribution)
# {'satire': 0.0, 'real_news': 1.0}

Benchmark: AnyJev vs. Jev vs. raw Qwen3-8B

The benchmark ran all three approaches against 500 headlines (250 satire, 250 real) from a 2019 dataset.

Accuracy: Jev leads by 12 points. AnyJev L0's bias correction adds only 0.2% over raw Qwen on this dataset. The bias correction didn't close the gap between open-weight and purpose-trained models here.

Flip rate: AnyJev L0 achieves a perfect 0.0%. The rotation method fully eliminates position bias. Raw Qwen flipped 4% of the time; Jev flipped 3.6%.

Calibration (ECE): Jev's ECE of 0.074 means its confidence scores are meaningful. When Jev says 80%, it's right about 80% of the time.

Both local models have ECE around 0.30. The root cause is overconfidence: 87% of Qwen's answers came with over 99% confidence, but nearly a third of those maximally-confident answers were wrong. AnyJev's L0 mode corrects bias but doesn't fix overconfidence.

Latency: Jev at 79ms beats both local options. AnyJev is roughly twice as slow as raw Qwen because of the double inference pass for position debiasing.

The harder test: tricky recent headlines

The 500-headline benchmark used 2019 data that models may have seen in training. A second test used 10 recent, deliberately tricky headlines: seven real news stories designed to sound fake, three recent satire pieces.

In this single-decision setting (using content-free prior), the results reversed:

AnyJev's correction of Qwen's strong pro-satire bias made the difference. By suppressing that lean, it correctly identified more of the tricky real-news headlines. The bias correction that barely moved the needle on 500 familiar headlines produced a 30-point improvement on 10 adversarial ones.

When to use each

Jev is the better choice when you need high accuracy on a consistent, known-distribution dataset, reliable calibrated confidence scores you can use to set auto-approval thresholds, or the lowest possible latency.

AnyJev is the better choice when data privacy means inputs can't leave your infrastructure, cost at scale matters, your data distribution is unusual or your model has known label biases that you want to suppress, or you want to experiment with different base models without paying per-decision.

The two aren't fully interchangeable. AnyJev doesn't solve the calibration problem, and on large standard datasets Jev's accuracy advantage is real. But for privacy-first deployments or adversarially-distributed inputs, AnyJev's bias correction can meaningfully outperform a hosted specialist model.

Got an article suggestion? Let us know

This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.