Dev Update #2 - Local-first storage for AI sessions

What was Milestone 2 about

Milestone 2 consisted of implementing the smista-storage crate. This crate provides both the entities that must be stored and the interface for interacting with the database.

Finding a database for a dynamic software

smista.ai is not just a stateless CLI wrapper around LLM APIs. The router is indeed meant to work either as a local instance or as a scalable SaaS service for a large user base.

This means that we had particular requirements for the database, which led us to immediately reject several engines.

In particular, running smista.ai locally means that we don't expect the user to set up other services to run smista-router. Even if a full local infrastructure could be provided via docker-compose, it would still require users to install and run Docker just to use smista-router. That is not the local-first experience we want. For this reason, the first requirement was that the database engine be embeddable within smista-router. Of course, for SaaS, the database must scale well and be runnable as a standalone service on a dedicated cluster.

This immediately narrowed the options to just a few. Furthermore, we needed to establish relationships because the user's session involves many entities that are related to each other.

At this point, the best option left was SurrealDB.

Why SurrealDB

The first time I heard about SurrealDB was at the RustNation'23 conference in London.

At the time, it sounded like one of those projects that are either extremely promising or dangerously ambitious: a database trying to combine documents, graphs, queries, permissions, and multiple storage backends under the same roof.

Since then, the project has grown significantly and has begun to look like a serious option for this kind of architecture.

For smista.ai, it was exactly the kind of database we were looking for.

The router is not just storing flat data. A session is not a single row. It is a small graph of related entities: users, sessions, messages, skills, routing decisions, model executions, token usage, cost estimates, provider metadata, and eventually audit information.

Of course, all of this could be modelled in a traditional relational database. PostgreSQL would have been the safest choice for a SaaS product, and SQLite the obvious choice for a local-first tool.

The problem is that smista.ai needs both.

SurrealDB offers a more interesting path: the same database model can run embedded within the Rust router for local use, and later run as a standalone service when the SaaS version needs it.

This matters because the local version should feel like installing a developer tool, not operating a miniature cloud platform.

At the same time, the data model should not be a toy model that we later throw away when moving to a hosted service.

That was the real reason SurrealDB became interesting: it lets us design the product as dynamic software from day one, without forcing users to run dynamic infrastructure on day one.

So the key behind SurrealDB is local-first, but not local-only.

Designing the entity model

The entity model for smista.ai is focused on users' sessions.

This sentence already conveys the core identity of the data model: there are users, and each user can have many sessions.

User as a simple concept

The top-level entity is the User. smista.ai is designed as a local-first service, so the concept of the user is simpler than you might expect. In smista-storage, the user has just a unique ID (UUIDv7) and an API key. No additional information, such as the email or name.

The reason behind this decision is that smista-router is not meant to handle public users. This means that whenever we or others implement a SaaS for smista.ai, there will need to be another service in front of smista-router that handles public users, including API keys, pricing, registration, OAuth, etc.

Local - but without unauthorised access

smista.ai is local-first, but that doesn't mean we don't need any credentials. For that reason, users have an API key and sign in to receive an Auth Token for any subsequent interaction with the smista-router.

smista.ai, even on a local network, is not designed for use by a single user. Many users can work with smista.ai in the same network, but we don't want other users interacting with each other's sessions. For that reason, authentication is required for every interaction with the router.

Session as more than a chat history

A session is an exchange of messages between the user and the LLM agent, with context enclosed.

This exchange of messages is not as simple as it seems; it indeed includes simple chat messages, routing decisions, tool calls, approvals, planning, file diffs to show the user, and more.

And all of this also has to be recorded as trace events so that the user can check what has been done during a session.

For this reason, the Session entity has been designed with a major entity called just Session, having many routing decisions, context references, tool calls, approvals, plans, diffs, and trace events.

smista-router must be able to access these sub-entities quickly to make decisions and provide users with a trace.

User and Context memory

Finally, one of the most important things added to milestone 2 is the memory entities.

Agents' tools usually have two kinds of memories:

• User memory: basically, preferences that the user has specified for themselves and that must be part of the context of every session, no matter what. Some examples are Always call me Chris or Never use git stash commands.
• Context memory: information or preferences that are valid just for this session. Some examples are Do not make any commits during the task or Don't check lint, I know there are some unused imports, I will fix those later.

Since we need to support two types of memory for different entities, we have designed two separate tables: user_memory and context_memory. User memory is scoped to the user, while context memory is scoped to a single session.

What's next

The next two weeks will be spent on the third milestone, one of the most important before v1.

This milestone is about implementing the model abstraction, the rig integration, provider adapters, and streaming.

In particular, the work will be focused on the smista-providers crate.

The deadline is set for June 26th, but we expect to finish earlier since we are ahead of schedule.