Underdog Saluki 27B: How a 7.9 GB Model Competes with Full-Size Qwen on Tool Calling
A 27-billion-parameter language model usually requires substantial memory, especially when you want to run it locally. Underdog Saluki 27B takes a different approach: it compresses Qwen3.8-27B from roughly 54 GB to 7.89 GB, while focusing on one particularly useful capability for AI agents: tool calling.
In Underdog's 120-task tool-calling benchmark, Saluki completed 88 tasks, compared with 84 for the full-size model. That result is intriguing, but it does not mean the smaller model is generally smarter. Saluki loses ground on mathematics and reasoning, and the four-task lead comes from a relatively small test suite.
The interesting question is what that trade-off means for local AI agents. Saluki offers a concrete example of how aggressive quantization can make a large model more accessible without preserving every capability equally.
Why local AI agents need reliable tool calling
A language model can answer questions using its existing knowledge, but an agent often needs to interact with something outside the conversation. It might look up information, retrieve records from a database, or request that another program perform an action. That interaction is called tool calling.
The model does not execute these tools by itself. Instead, it selects a tool and produces structured arguments. The application hosting the model validates and executes the request, then returns the result to the model. For example, a weather question might produce a tool call resembling this one:
The useful behavior is not simply producing valid JSON. The model must choose the correct tool, provide the right arguments, and avoid making tool calls when a normal answer is enough. Mistakes in any of those decisions can break an otherwise capable agent.
Running this kind of model locally also presents a hardware challenge. The source article compares three versions of the Qwen-based model:
| Model representation | Approximate file size | Practical implication |
|---|---|---|
| Full-size Qwen3.8-27B | 54 to 55.6 GB | Requires substantial GPU or unified memory |
| Conventional 4-bit quantization in the source comparison | 16.5 GB | Much smaller, but still demanding on limited-memory machines |
| Underdog Saluki 27B | 7.89 GB | Can fit on more modest local systems, with additional memory needed for runtime state |
File size is not the same as total memory consumption. The inference engine also needs space for the context cache and other runtime allocations, so available memory and context length still matter.
How quantization makes a 27B model smaller
An AI model stores learned parameters as numerical values called weights. Large models commonly use 16-bit formats such as BF16 for their weights. Quantization reduces how many bits are needed to represent those values, substantially shrinking the model file.
With a conventional 4-bit representation, a weight has far fewer possible values than it does at 16-bit precision. At 2 bits, only four discrete values can be represented before accounting for the quantization scheme's scales and other metadata.
Saluki uses an aggressively compressed mixed-precision representation averaging approximately 2.35 bits per weight. Different parts of the model can receive different precision budgets rather than being compressed identically.
A smaller representation reduces weight storage and can lower memory requirements, but it also changes the model's numerical calculations. Some capabilities are more sensitive to these changes than others. The challenge is determining which losses matter for the intended workload.
It is also worth distinguishing file compression from quantization. Saluki is not a zip archive that expands back into the original 54 GB weights during inference. The quantized values are used as part of the model's actual computation.
Why Saluki prioritizes tool calling
General-purpose model evaluations combine many abilities: mathematical reasoning, coding, language understanding, instruction following, and tool use. A model designed for an agentic application does not necessarily need to maximize every score.
Saluki's optimization places particular emphasis on preserving tool-calling behavior under aggressive compression. This is useful when the language model mainly acts as a coordinator that chooses functions and formats their inputs, rather than solving difficult problems entirely on its own.
That focus should not be mistaken for a guarantee that Saluki is always the better agent model. Real agents also need to interpret ambiguous requests, handle failed tools, reason over results, and decide whether an action is safe. Tool-call accuracy measures an important part of the workflow, but not all of it.
What the benchmarks actually show
Underdog reports its results on a 120-task subset of the Berkeley Function Calling Leaderboard (BFCL v4), with thinking disabled and temperature set to zero. In that test, Saluki completed 88 tasks and the full-size Qwen model completed 84.
The model card also reports a separate parallel tool-calling result of 42/100 for Saluki versus 35/100 for the full model. These are promising results, but the main test has only 120 tasks. A difference of four passes is not enough on its own to establish that Saluki will outperform the full model across applications and repeated runs.
Other benchmarks reveal the cost of aggressive compression.
| Benchmark | Saluki 27B | Full-size Qwen3.8-27B | What it evaluates |
|---|---|---|---|
| Underdog Bench | 88/120 | 84/120 | Tool-call selection and formatting |
| Parallel tool calling | 42/100 | 35/100 | Multiple tool calls |
| AIME 2025 | 79.2 | 96.7 | Competition mathematics |
| MuSR | 67.5 | 79.6 | Multi-step reasoning |
| SWE-bench Verified subset | 30/50 | 33/50 | Software issue resolution |
The values come from Underdog's published model card. Its documentation notes that some public baseline results use a different evaluation harness, so not every comparison in the table has the same level of experimental control.
The main finding is narrower than “small beats large.” Saluki performs competitively on the tool-calling workload highlighted by its developers, while the full model remains noticeably stronger in some reasoning-heavy evaluations.
What tool-calling behavior looks like in practice
Two illustrative prompts make the distinction between using a tool correctly and knowing when not to use one easier to understand.
Consider the question:
What's the weather in Salt Lake City, Dubai, and Saigon?
An application exposing a get_weather function might expect the model to produce three independent calls, one for each city. A simplified response could look like this:
In the source comparison, both Saluki and the conventional 4-bit Qwen model produced appropriate calls for the three locations. This illustrates a useful limit of the example: both models succeeded, so it does not by itself establish a performance difference.
A second prompt asks:
How do you build a house?
The weather function has nothing to contribute to that request. Both models answered directly instead of calling get_weather. This is as important as choosing the right function when one is needed. An agent that calls tools indiscriminately can create unnecessary work or unintended side effects.
Running the compressed model with llama.cpp
Saluki is distributed as a GGUF model compatible with llama.cpp, an inference runtime commonly used for local models. The following commands illustrate how two separate model servers can be configured for comparison. They are examples of the runtime's interface, not a required setup procedure.
A conventional 4-bit Qwen variant can run on one port:
Saluki can run on another:
The -m option identifies the model file, -c 32768 sets a 32,768-token context, and --port specifies the listening port. The -ngl 99 setting requests GPU offloading for up to 99 layers, subject to hardware support. The --jinja option enables chat-template processing, which matters for models that represent tool calls through a structured conversation template.
Actual executable names and supported flags depend on the llama.cpp build. The Saluki model card uses llama-server and documents additional recommended settings. Neither command guarantees that all model data will fit in available GPU memory.
How Saluki was developed
The source material describes Saluki as a refinement of an already compressed Qwen model, rather than a direct one-step reduction from full precision.
Its starting point was work from ISTA-DASLab using a constrained quantization approach associated with Riemannian optimization. That earlier representation was approximately 8.4 GB and scored 76/120 on the tool-calling benchmark cited in the source.
Underdog then developed its Saluki variant, reducing the file to 7.89 GB and reporting 88/120 on the same type of tool-calling test. The precise details of its additional optimization process are not fully documented in the source material, so the score improvement should not be attributed to an unverified training technique.
The result is interesting because it shows that compression is not necessarily a single fixed transformation. Model developers can evaluate compressed checkpoints against particular tasks and make further trade-offs within a limited size budget.
Where Saluki fits and where it falls short
Saluki makes the most sense when memory is constrained and the model's primary job is to interpret requests and select tools. A local assistant that routes requests to a search function, database query, or application API is closer to its advertised strengths than an application centered on advanced mathematical reasoning.
There are still important limitations. Its mathematical and some reasoning benchmark results trail the larger model. Parallel tool calls are not perfectly reliable, and the model card reports formatting slips in some responses. Long contexts increase runtime memory use even when the weight file is small. Also, the main GGUF file is text-focused; image input requires an additional vision projector file.
This is why evaluation should cover the entire application, not just the model file size or one benchmark. Correct tool choice, valid arguments, sensible behavior when tools fail, latency, and memory pressure all influence whether a smaller model actually improves the user experience.
Final thoughts
Underdog Saluki 27B demonstrates that a much smaller model can remain highly effective at a carefully chosen task. At 7.89 GB, it is far easier to accommodate on local hardware than the roughly 54 GB full-size Qwen model, and its published tool-calling results are competitive.
The trade-off is equally important. Saluki is not a general replacement for full-size Qwen. Its stronger result on a small tool-calling test comes alongside weaker mathematics and reasoning performance, which may matter in more demanding agent workflows.
For local AI applications built around structured tool calls, Saluki is worth evaluating. Its broader lesson is that the best model is the one that handles your actual workload reliably within your hardware limits, not necessarily the one with the most parameters.
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.