Dev Update #3 - A common interface for LLMs with Rig

What was Milestone 3 about

On June 13th, we closed our 3rd milestone.

This is a huge achievement! Not only have we completed half of the milestones needed to land our first release, but we are also finally ready to work on the two binaries: smista-cli and smista-router. 🎉

Milestone 3 was about implementing the smista-providers crate.

This crate defines the common interface used by smista.ai to interact with language models, regardless of whether they come from Anthropic, OpenAI-compatible APIs, Gemini, Ollama, or local deployments.

Supporting different models from different providers

One of the earliest goals for smista.ai was to support models from multiple providers.

Initially, the target was to support Anthropic, OpenAI (and any compatible provider), and Ollama. Later, Gemini was added as well.

Why Rig

We wanted the integration with all these providers to go through a single interface (trait) with common logic. Luckily for us, a crate which does this already exists: rig.

With rig, we could implement a common Agent wrapper around Rig's provider-specific completion clients:

pub struct Agent<C>
where
    C: CompletionClient,
{
    agent: RigAgent<C::CompletionModel>,
}

Rig already provides provider-specific clients that can be configured with the required settings, such as API keys, base URLs, and provider-specific options.

That allowed us to keep our own abstraction small.

At the smista.ai level, models only need to implement two core operations: a full completion request and a streamed completion request.

///
/// Sends a request and awaits the full completion.
/// Returns the model's final content, any tool calls it issued for the
/// router to mediate, usage totals and the reason generation stopped.
async fn complete(&self, request: CompletionRequest) -> ProviderResult<CompletionResponse>;
 
/// Sends a request and returns a stream of response events.
///
/// The streaming counterpart to [`Self::complete`]: events carry partial or
/// final content, tool-call activity, usage updates and a terminal marker as
/// the model produces them.
async fn stream(&self, request: CompletionRequest) -> ProviderResult<ResponseStream>;

With these two methods set and implemented in our Agent, we could easily define the Model trait to include exactly those two methods, plus a couple more for metadata about the model.

From static to dynamic model catalogues

Initially, the idea was to create a static catalogue of available models for each provider.

In smista.ai, each model must provide several pieces of information:

• the ID to invoke it
• the capabilities it offers
• context size and limits
• cost per input MTok and per output MTok (million tokens)

With how quickly model catalogues change, keeping this data hardcoded would become a maintenance problem.

For that reason, we have decided to add the possibility to fetch it dynamically.

Many providers offer a callable REST API endpoint to retrieve available models. So, after sorting that out, smista.ai can now always provide the latest available models without waiting for a new release.

Some providers expose only partial information. OpenAI, for example, exposes model IDs, but not enough structured metadata for smista.ai to build a complete catalogue from the API alone.

There is one final limitation to this: these endpoints never expose costs. So we currently still have a static pricing table that needs monthly updates.

Local and remote providers

One of our first goals was indeed to support local models through Ollama. For that reason, we initially set up the Ollama provider to be local-only. A minor complication is that Ollama is no longer necessarily local; it can also be used via a hosted API. For that reason, smista-router treats local Ollama and hosted Ollama as two different provider configurations.

Furthermore, with the advent of OpenAI-compatible providers, there is a need to specify whether each provider is local.

Local does not always mean localhost

In smista.ai, "local" means requests are not sent to an internet-exposed model provider. The model may run on the same machine, on a LAN machine, or behind private infrastructure.

Hitting hard the ToS wall

There is a provider which has actually been deleted from our plans. Unfortunately, GitHub Copilot won't land on smista.

The reason is quite simple: contrary to what we thought, because rig.rs exposes it, we can't really provide support for GitHub Copilot.

pub enum CopilotAuth {
    ApiKey(String),
    GitHubAccessToken(String),
    OAuth,
}

Unfortunately, though, Copilot doesn't provide api keys (like Anthropic or Google) at all.

So how does that work? This relies on accessing Copilot via authentication mechanisms intended for GitHub’s own clients, rather than through a public API key-based provider interface.

Rig.rs exposes that, but doesn't do it for you. Of course, you can use it at your own risk, but that would violate GitHub.com's ToS.

Why don't we provide the methods on the user's shoulders?

We do not want smista.ai to depend on unofficial or ToS-sensitive integrations, especially because the project may eventually have a SaaS component.

This led us to formalise a simple project policy: smista.ai will only support providers through official, documented, ToS-compliant interfaces.

The same principle applies to subscription-based products such as Claude plans or ChatGPT/Codex access: unless there is an official API-key-based integration, smista.ai will not try to route through them.

What's next

The next milestone is the full implementation of smista-router. Since this milestone is larger than the previous ones, it will probably be split across multiple dev updates.

Once we finish this milestone, we'll have a fully working router for task routing.