Dev Update #4 - Routing prompts without asking an LLM

What was milestone 4 about

The 4th milestone was about implementing the core component of smista: the router.

smista-router exposes a REST API that the user must use to interact with LLMs and sessions.

The requests are then forwarded to the Orchestrator, which is responsible for executing all steps to determine the model to use based on the user's input and routing policy, and finally forwarding the required context and prompt to the LLM. The LLM response is then processed to update the user's session and provide a response to the user, which can either request approval to execute commands or be a plain response.

If I had to sum up the whole milestone in one sentence, it would be this:

Routing is deterministic and never depends on an LLM.

This is the principle the entire router is built around, and you'll see it come back in every section below. The router classifies the task, matches it against the user's policy, picks a model, and assembles the context, all according to fixed rules. No model is ever asked, "Which model should run this?".

How the Router is structured

The router is not a daemon you have to babysit. It's a library crate (smista-router) that the CLI embeds and runs locally by default. There is a single entry point, run(), which takes an already validated configuration and a cancellation token, initialises storage, and spawns the services. That's it. The host process (the CLI) owns loading the config and layering its own observability on top of it, which keeps the router itself dumb about where it's running.

Originally, we wanted to keep two binaries, one for router and one for the cli, but eventually we decided to opt for a single binary to make installation and usage simpler.

Internally, it's split into a handful of modules, each with one job:

• web: The Axum HTTP API: routes, middleware, streaming, error format.
• orchestrator: The run loop that drives a prompt to completion, turn by turn.
• router: The deterministic resolver: classify, match policy, select model.

The path a request travels through them is always the same. An HTTP request comes in, gets authenticated by middleware, reaches a route handler, and is handed to the orchestrator. The orchestrator asks the resolver to make the routing decision, calls the provider, mediates any tool calls back through the client, persists the result and a trace, and returns the response.

The important thing here is the separation of concerns. The CLI never decides anything about which model runs your task. It only expresses preferences and renders results. Every routing decision belongs to the router, and that's not a stylistic choice; it's what makes the whole thing explainable.

Resolving prompts against user-defined routing policies

Now to the heart of it. How does the router choose a model without ever asking an LLM to decide?

It runs a fixed pipeline of stages, and each one is a pure function of observable inputs. Same inputs, same decision, every single time. The stages are: classify the task, match a policy rule, select the model, select the context, and apply privacy.

Classification maps your prompt to a single intent: chat, edit, review, plan, summarize, prompt or skill. If you named a command explicitly, that wins, and we're done.

Otherwise, the router evaluates your classification rules in order of priority, and the first match wins. The interesting part is the keyword matching: it's not a raw substring scan, it's token-based and typo-tolerant, using Optimal String Alignment edit distance with a static, length-bucketed cap. So impelment still matches implement, but short words like edit require an exact match so they don't collide with audit.

Policy matching turns the intent plus the candidate file paths into exactly one rule. Rules are evaluated in ascending priority order, then by specificity (path+intent beats path beats intent beats nothing), then by config order. If you set an explicit model override, matching is bypassed entirely. If nothing matches, the router uses your default route, and if you have no default route either, it raises a no_route error. It never guesses.

Model selection takes the matched route's model reference and turns it into a model that can actually serve the turn. It resolves the reference against the router's own catalogue, checks availability (does it need an API key, and was one supplied in the request headers?), validates capabilities and context window, and walks the fallback chain if the first choice can't serve. An explicit override has no fallback chain because it's your deliberate choice: if it can't be served, the run fails with an error rather than silently routing elsewhere.

Context selection is filter-only. The router ranks and trims the candidates it already holds in storage; it never goes to the filesystem to gather more. The client ships the files; the router decides what makes it into the prompt.

Privacy is the last and most important constraint, and it's a routing input, not a redaction step. If a required piece of context is locality-constrained (it matches your restricted paths, your remote-blocked paths, or a local_only rule), then only local models are eligible, and that dominates everything, including an explicit override. The bad pairing, sensitive content on a remote model, is never selected by construction. It's not a state the router has to clean up afterwards; it simply can't happen.

Why go to all this trouble to stay deterministic? Because it makes routing explainable, reproducible and safe. You can ask the router, "What would you do with this prompt?" via the /preview endpoint, and it'll tell you the model, the matched rule, the context, and the estimated cost, without calling a single model.

Orchestrating tasks with the LLM

The resolver makes a decision. The orchestrator runs the entire task based on those decisions.

A run is one user prompt carried all the way to a terminal state, and a session has at most one run in flight at a time. A run is a loop of turns, where a turn is one model invocation. Here's the part I like most: the router re-classifies and re-routes before every turn. The model that serves turn two can be different from the one that served turn one. A single run can be planned with a reasoning model, edited with a coding model, and reviewed with a third, each chosen deterministically as the work drifts. That is the most powerful feature of smista.ai: every single task is routed to the best model capable of executing it.

And there's no phase engine doing this. The router never sequences "first plan, then edit, then review" on its own. The phases emerge from the loop because the state grows every turn, and the classifier reads the new state. That's a much simpler design than hardcoding a workflow, and it adapts on its own.

The model never touches your machine directly. Every tool call (reading a file, running a shell command, hitting the network) is mediated by the router based on your tool permissions.

• A deny is synthesized server-side and fed straight back to the model, it never even reaches the client.
• An allow is returned for the client to execute.
• An ask is returned flagged for approval. And approvals are folded into the tool result: the same machine that approves a command is the one that runs it, so there's no separate approval round trip. A standalone approval only exists for decisions with no tool to run, like disclosing context to a remote provider, a cost limit, or accepting a generated plan.

All these states live in storage, not in the router's memory. That's deliberate. A dropped connection, or a router running on another host, never loses the run. It also gives us one clean rule that unifies aborting, mid-run input and reconnection:

A new client request for a run cancels any in-flight turn, persists the partial output, cancels pending tool calls, and starts the next turn.

So when you hit Esc, or type "stop, do this instead" while the model is still generating, it's the same mechanism. The injected input is just more state at the head of the loop, picked up for free by the next re-classification.

The last piece is end-to-end encryption. For encrypted sessions, the router never stores plaintext and never holds the encryption key. It relies on the client to decrypt sealed history when context is needed and to seal new output before it is persisted. The router is therefore blind at rest by construction, while still being able to orchestrate the active run.

Building a strong API with Axum

All of this is exposed over an HTTP JSON API built on Axum. The surface is small and deliberate:

Group	Endpoints
Auth	bootstrap, sign-in, sign-out, me
Sessions	create, get, list, update, delete
Execution	execute, continue, preview
Insight	traces, usage
Models	llm/providers, llm/models

Authentication is a bearer middleware that uses short-lived session tokens, which you get by signing in with your API key. There's separate middleware for provider credentials: the request never tells the router which providers exist or what to trust; it just carries the keys in headers, and the router reads them from the headers it cares about. The client is never trusted to decide availability. There's also rate limiting with governor sitting in front.

Execution can be buffered or streamed. When the client asks for text/event-stream, a turn is delivered as server-sent events: text deltas, reasoning deltas, tool-call events, usage, and exactly one turn_end that carries the final status.

One convention I'm strict about: the API schema is generated from the Rust types via utoipa, behind a feature flag, by a test rather than a build script. The committed openapi.json is the source of truth for clients, and CI fails when it detects drift. The types and the documented contract can't diverge, because the moment they do, the build goes red.

Telemetry is crucial

There's a project rule that business-logic code must emit tracing events. Not "should", must. Every decision, every branch, every failure path on a meaningful code path has to be observable, and crucially, it has to log not just what was chosen but why each alternative was rejected.

That rule isn't busywork. It's the natural consequence of deterministic routing. If routing is deterministic, then every decision has a reason, and if every decision has a reason, then you'd better be able to read it back. A trace is how you see the "why" behind a route. Without it, "deterministic" would just be a word.

In addition to plain logging, the router now supports OpenTelemetry. The CLI provides observability for the locally started router: it builds an OTLP exporter from the resolved config (gRPC or HTTP) and returns a tracing layer that composes alongside the formatting layer.

The one hard line, here as everywhere: secrets are never logged, traced, printed or persisted. You get full observability into routing decisions and zero leakage of what should never leave your machine.

What's next

With the router done, the next milestone (M5) is about getting that API into people's hands: the smista-router-client (an async Rust client for the HTTP API), the Rust smista-sdk facade on top of it, and the TypeScript SDK. After that comes M6, the actual CLI experience with clap and ratatui, slash commands, diff review, skills and plans, and finally M7 for distribution.

The hard part is behind us. The router is the brain, and it's deterministic, explainable and private by construction. Everything from here is about making it pleasant to talk to.