Gemini Trapped in 26M Parameters — Needle and the Opening Shot of the Router-Specialist Era

Is tool calling really reasoning, or is it a well-disguised structured output? If the latter, have we spent the last two years asking too-big models to do too-simple work?

Opening — A Fourteen-Megabyte Finger

On May 8, 2026, a small startup called cactus-compute posted to Hacker News under the title “Show HN: Needle — We Distilled Gemini Tool Calling into a 26M Model.” Within 24 hours the post had gathered 415 points and 144 comments. The attached GitHub repository collected more than 700 stars in three days. As of the launch, the model size is 26,000,000 parameters, the FP16 weight file is about 14 megabytes, the license is MIT, and the training starting point is Google Gemini 3.1.

If you look only at the numbers, the announcement is strange. Frontier LLMs in the spring of 2026 have been operating at the trillion-parameter scale for over a year. GPT-5.5 and Claude Opus 4.6, both shipped the same month, foreground tool-call accuracy and backtracking on long-horizon agent workflows as core metrics. In the middle of that landscape, someone claims to have made a 26M-parameter model — roughly 1/150th of Phi-3 Mini, roughly 1/300th of Llama 3 8B — “mimic Gemini’s tool calling exactly.”

What looks at first glance like a joke was taken seriously for two reasons. First, cactus-compute has been building mobile/edge inference runtimes for some time, and Needle arrives atop their own SDK with phone-friendly performance numbers: 6,000 tokens/sec prefill, 1,200 tokens/sec decode. Second, the announcement aims squarely at the wall the Small Language Model camp has been running into for two years — “small models hallucinate absurdly on tool calls.”

The implications are not simple. “An LLM runs on a mobile device” has been a hackneyed topic since the 2024 Apple Intelligence announcement, but “an agent runs on a mobile device” is different. What Needle does is, in the end, one thing: take the user’s natural-language utterance and emit the appropriate function name and arguments as JSON. If that simple work really fits in 26M parameters, then a large share of what we have been making 100B and 1T models do is the result of inefficiency.

Section 1 — What Needle Does, and What It Does Not

Start with the facts. Per the Needle README on GitHub, the model uses a transformer variant called the “Simple Attention Network.” Twelve encoder layers and eight decoder layers, eight attention heads and four KV (key-value) heads per decoder layer, embedding dimension 512, vocabulary size 8,192 BPE tokens. The most striking feature is the absence of an FFN (feed-forward network) in the encoder layers. The authors put the reasoning directly in the README: “Experiments at Cactus showed that MLPs can be completely dropped from transformer networks, as long as the model relies on external knowledge source.”

The implications of this sentence are large. In transformers, the FFN is conventionally said to serve as the “knowledge store.” If attention builds the relationships between tokens, the FFN lays factual information on top of those relationships. Needle targets the fact that tool calling is a task in which “all the information (the names and signatures of the available tools) is already inside the prompt,” and rips the FFN out entirely. This is the core trick behind the 26M number.

A user named fizza_pizza pointed precisely to this in the comments. “The ‘no FFN for retrieval tasks’ observation is interesting — essentially arguing that if the knowledge is in the context, FFN weights are redundant for that task. Curious whether this generalizes to multi-turn tool calling where the model needs to track state across several calls, or if it breaks down there.” This question goes straight to the core. And cactus-compute’s answer is already in the README. Needle is explicitly specialized for “single-shot function call.” That is, the scope of the promise covers picking a tool and filling its arguments for a single user utterance.

Performance comparisons are presented only in relative terms. The authors write only that Needle “outperforms FunctionGemma-270m, Qwen-0.6B, Granite-350m, and LFM2.5-350m on single-shot function call.” Standardized benchmark scores like BFCL (Berkeley Function Calling Leaderboard) were not published at launch. This is the weakness meander_water flagged in the comments. “Gemma4 edge models were promised to be good for agentic use, but in my tests they failed even on the most basic tool-calling scenarios. Have you run any tool-use benchmarks for Needle, or do you plan to? Would be great if you could add results to the repo.” A comment that exposes the time gap between the excitement an announcement produces and the academic verification that follows.

Another interesting comment is from varenc. “Aren’t you worried about how Google will react to this? Google reportedly reacts to distillation attempts with real-time proactive defenses that can degrade student model performance. So if they detected you, they may have intentionally fed you a dumber but plausible-seeming variant of Gemini.” That remark quotes Google’s own threat-intelligence blog directly. cactus-compute did not respond to this concern directly, but two points can be made. First, Needle is a distillation confined to the narrow task of tool calling and does not threaten Google’s business core. Second, the expected output of tool calling is a JSON schema, which admits external verification. If the training data had been “plausible but subtly broken,” the student model’s outputs would not have compiled in the first place. Tool calling, unlike a natural-language summary, is a task in which wrongness reveals itself immediately.

A subtle gray zone remains around training-data generation and licensing. The authors do not state under which API terms they consumed Gemini’s outputs, and as of this writing Google API terms of service include clauses prohibiting “use of outputs to train models that compete with Google models.” How Needle fits that clause is unclear from the announcement alone. This is an unresolved problem for the distillation camp as a whole, and Needle is positioned as the first meaningful tool-calling model to ship on top of that gray zone.

To summarize: Needle redefines “tool calling” as “prompt-dependent structured output” and then carves into the transformer architecture itself to fit that definition, dragging model size down into the two-digit-megabyte range. It does not promise multi-turn reasoning or generalization to new schemas. But the meaning carried by that narrow promise is not small.

Section 2 — Why Tool Calling Distills and Reasoning Does Not

The biggest mental barrier to absorbing the Needle announcement is that for two years we have been assuming “tool calling is also part of reasoning.” The ReAct pattern, ChatGPT’s function calling, Anthropic’s tool use, OpenAI Assistants API — every one of these interfaces planted tool calling deep inside the model’s thought flow. The result was an entrenched intuition that a model good at calling tools is a model good at reasoning, and vice versa.

Needle inverts this intuition. Given the user’s utterance “What’s the weather in San Francisco?” and the tool signature get_weather(location: string), producing the output {"name":"get_weather","arguments":{"location":"San Francisco"}} is not reasoning so much as pattern matching. More precisely, it is the mapping from natural-language slots into function-argument slots. What the model has to solve here is not “will the weather in San Francisco be cloudy today” but the recognition that the token sequence “San Francisco” is the appropriate value for the location argument.

A second feature of tool calling combines with this. Its output format is strictly defined. The JSON schema provides an external verification function over model output. Bad output gets caught at the parse step. Unlike a natural-language response that can come out “plausible but wrong,” a tool call receives a binary verdict: syntactically correct or not. For tasks like this, the reward signal for RLHF is cleanly defined, and learning efficiency leaps. This is the second factor that allows enough capability to be crammed into a 26M-parameter budget.

The HN commenter jumploops touched the same point from another angle. “Sonnet often makes quick tool calls to gather more context, while Opus tends to reason longer trying to solve the problem with the context it has. (…) My takeaway was that ‘dumber’ (i.e. smaller) models might be better as an agentic harness, or at least feasibly cheaper/faster to run for a large swath of problems.” This observation lines up exactly with Needle’s hypothesis. Large models think too long before calling a tool, and as a result tend to define the same function twice, recall a tool they already invoked, or perform side tasks the user did not request.

Architecturally, this split takes the form of a router-specialist structure. A small model — the router — takes the user’s utterance and decides “handle this with the code-search tool” or “handle this with the calendar-add tool,” and a separate large model takes the tool’s result and generates a natural-language reply. Or, in a more extreme variant, the tool’s output is returned directly to the user. In this division of labor the router need not reason. And if the router need not reason, the router can live on a mobile device.

This is where the Needle announcement is genuinely shocking. The number 26M is not merely a small model but the first empirical demonstration of the proposition “the router can actually run on a mobile phone’s NPU.” 6,000 tokens/sec of prefill means processing 1,000 tokens of tool-signature context in 0.2 seconds; 1,200 tokens/sec of decode means emitting 100 tokens of JSON in 0.1 seconds. If the latency between a user finishing their utterance and the function call beginning is under 0.3 seconds, that is latency the human will not perceive.

But the picture comes with a decisive caveat. When the router picks the wrong tool, the meta-cognitive ability to notice the mistake does not live inside a small model. A commenter named nl put it precisely. “My question is how effective is it at handling ambiguity. Can I send it something like a text message ‘lets catch up at coffee tomorrow 10:00’ and a command like ‘save this’ and have it choose an ‘add appointment’ action from hundreds (or even tens) of possible tools?” The honest answer is “Needle does not know.” The larger the tool pool grows, and the more ambiguous the user’s intent, the more the limits of a distilled model are exposed.

This is a fundamental weakness of distillation in general. The student model is reliable only inside the distribution the teacher model saw. New tool signatures, new argument types, new call patterns — every one of these requires additional fine-tuning. When cactus-compute writes in the README that “we recommend fine-tuning on your own tools after testing,” that is not just a user guide but an honest disclosure of the distilled model’s nature.

Section 3 — The Future of the Edge Agent and the Division of Labor It Implies

What changes when a model like Needle settles in? Three scenarios.

First, a real change in the meaning of the on-device assistant. Mobile assistants (Siri, Google Assistant, Bixby) up to now have in fact been cloud assistants. Voice recognition and intent classification could be done on-device, but the reasoning step that mapped that intent to the right action happened in the cloud. The Needle class of model brings that mapping step itself onto the device. The result is “an assistant that works in airplane mode,” “an assistant that adds a calendar appointment in a zero-bandwidth environment.” This is the future Apple Intelligence promised in 2024 but only partially delivered.

Second, a new interface layer for CLI tools. The commenter ilaksh framed this interestingly. “This might make it feasible to build something like a command-line program where you can optionally just specify the arguments in natural language. E.g. ’> toolcli what can you do’ runs ‘toolcli —help summary’, and ‘toolcli add tom to teamfutz group’ becomes ‘toolcli —gadd teamfutz tom’.” This scenario marks the first point at which it becomes rational to bundle a 14-megabyte model with every CLI binary. The same commenter’s added concern — “if every CLI starts doing that, it could get pretty bad” — is also legitimate. But that concern is itself a signal that the model has crossed a threshold where it might plausibly be embedded in many places.

Third, the reconciliation of privacy with agents. Up to now the biggest adoption barrier for agents has been data exposure. A coding agent seeing the code meant the code was uploaded to the cloud; a personal assistant seeing your messages meant the messages were processed on someone else’s server. The router-specialist structure partially solves this. If the router lives on the device, the decision “which tool to call” itself never leaves the device. If the tool itself must hit a cloud API, the data leaves at that point, but at least the user’s intent stays on the device.

Honesty also demands skepticism about the picture as a whole. A single line from the commenter binyang_qiu captures this cleanly. “How does this behave once workflows become multi-step and state starts accumulating across calls?” This question marks the decisive boundary between domains where a Needle-style model fits and where it does not. The fitting domain is the short loop: “single utterance → single tool call → return result to user.” The unfit domain is the long loop: “call several tools in sequence, with one tool’s output feeding the next.” The latter remains the territory of large models.

Restated technically: Needle is a model that goes at the very front of an agent system — the first hop that maps user intent into tool space. What comes after that is up to the system designer. It might be another small model, or a large cloud model. What matters is that the first hop has moved onto the device, and as a result the latency, cost, and privacy curves of the whole system are redrawn.

Conclusion

Is tool calling really reasoning? The Needle announcement returns a negative answer. At least in its most basic form, single-shot function calling, tool calling is not reasoning but a well-disguised structured output. It is the mapping of natural-language slots into function-argument slots, and its reward function is cleanly defined as a binary signal of JSON-parser pass/fail. For a task like this, 26M parameters is a sufficient budget.

The division of labor this implies is clear. Domains that require reasoning — multi-step planning, ambiguous intent interpretation, environments where the tool pool changes dynamically — remain the large model’s share. But the first mapping of user intent, the single-hop tool call, argument extraction — these tasks can now execute on the device. We have spent two years asking too-big models to do too-simple work, and Needle finally makes the bill from that inefficiency visible.

cactus-compute will not single-handedly build this future. The next versions of Apple Intelligence, Google’s next Gemini Nano, and follow-on distillation models from the open-source camp will likely follow the same path. But it is worth remembering that the first meaningful coordinate on this trajectory was set by an announcement of 26M parameters, 14 megabytes, MIT license. It is proof that without massive capital, by approaching a narrow task narrowly enough, you can carry one capability of a frontier model into a phone. And it is also the first moment in the LLM era when a task is explicitly separated as “one for which scale is not the answer.”