Skip to main content
Everything else in this section describes the canvas as a user experiences it. This page is for developers: how the data layer and the visualization layer are implemented inside the open-source unifyai/unify repo, how they connect to the hosted backend, and where to hook in if you’re extending them. Two packages own this territory:

​
The big picture

DataManager architecture: the Actor calls primitives.data.*, which lands on DataManager (contract in base.py, implementations in ops/, schemas in types/); ContextRegistry supplies destinations; SimulatedDataManager mirrors the same contract in memory; persistence flows through unisdk HTTP to Orchestra, where contexts hold log events across personal Data/… and Teams/{id}/Data/… roots with federated reads between them. Both packages follow the same layered pattern, shared with every state manager in the repo:
Three properties define the design:
  1. Pure primitives. Neither manager has ask/update LLM tool loops. From base.py: “DataManager exposes pure primitives with no ask/update tool loops. High-level orchestration is handled by Actor composing these primitives.” The Actor writes Python that calls them directly.
  2. Docstrings are the API docs. The @abstractmethod docstrings on the base classes are what the Actor (an LLM) reads to learn the API — concrete classes inherit them via functools.wraps. If you change behavior, the docstring is the interface; keep it truthful.
  3. Contract-first discovery. The primitives registry (unify/function_manager/primitives/registry.py) introspects @abstractmethod definitions on the base class to decide what the Actor sees as primitives.data.* and primitives.canvas.*. Add an abstract method with a docstring and it becomes an Actor-callable primitive with no registry edits.
Both managers are synchronous internally, listed in _SYNC_MANAGERS in unify/function_manager/primitives/runtime.py; the runtime wraps them in _AsyncPrimitiveWrapper (dispatching via asyncio.to_thread) so Actor code can uniformly await primitives.data.filter(...). Implementation selection is env-driven: UNITY_DATA_IMPL and UNITY_CANVAS_IMPL ("real" | "simulated"), read by DataSettings and CanvasSettings and resolved through ManagerRegistry — unify/manager_registry.py.

​
DataManager

​
Package anatomy

The layering rule is strict: base.py holds contract and docstrings only; data_manager.py orchestrates and resolves contexts; ops/ talks to the backend. The *_impl functions are internal — always go through the manager.

​
The contract at a glance

BaseDataManager groups its 26 abstract methods into five families: A defining property, from the class docstring: the primitives “work on ANY Unify context”. DataManager semantically owns the Data/* namespace, but executes analytical operations against foreign namespaces too — Files/*, Knowledge/*, FileRecords/* — which is exactly how FileManager and KnowledgeManager use it (both delegate their query/join execution to a DataManager internally; see KnowledgeManager._data_manager and the convenience wrappers like filter_files in unify/file_manager/file_manager.py).

​
Storage model: a table is a context of log events

There is no bespoke table storage. A table is an Orchestra context (a hierarchical path like Data/Sales/Monthly) and each row is a log event in that context, written and read through unisdk. Row identity is the Orchestra log ID — insert_rows returns them, filter can return them (return_ids_only=True), and delete_rows accepts them (log_ids=). Context paths resolve through three private helpers in DataManager:
  • _resolve_context (reads): strips a leading /, passes through any path starting with a known absolute prefix (Data/, Files/, Knowledge/, Teams/, …), and prepends the assistant’s base context ({org}/{assistant_id}/Data) for relative names.
  • _resolve_context_for_write(context, destination=): for Data-owned paths, routes through ContextRegistry.write_root(self, "Data", destination=...) — "personal" (default) or "team:<id>", the latter landing under Teams/{id}/Data/... and requiring live team membership.
  • _resolve_contexts_for_read: for Data-owned, non-exact paths, fans out across ContextRegistry.read_roots(...) — the personal root plus every accessible team root — merging results via federated_filter, federated_ranked_search, and federated_reduce from unify/common/federated_search.py.
Destination scopes: write_root routes each write to exactly one destination — the personal root or a team root, each holding Data and Canvas contexts — while read_roots federates reads across personal plus all teams. ContextRegistry (unify/common/context_registry.py) is the single choke point for scope: write_root provisions and returns exactly one destination, read_roots returns the ordered fan-out list, and invalid destinations (unknown team, non-member team) raise ToolErrorException with error_kind: "invalid_destination". Two Data-specific conveniences applied at creation time: tables under Data/* default to unique_keys={"row_id": "int"} and auto_counting={"row_id": None} when the caller doesn’t specify them (_resolve_unique_keys_and_auto_counting).

​
The query engine

Filter. filter takes a Python-like boolean expression over column names — "amount > 1000 and status == 'open'". The string passes through normalize_filter_expr (unify/common/filter_utils.py, currently a passthrough) and is evaluated server-side by Orchestra, which compiles the expression against each row’s JSON. Private columns (leading _) are excluded from results unless explicitly requested. limit is capped at 1000. Search. Semantic search rides on derived embedding columns. A column text gets a private sibling _text_emb whose value is a derived equation embed({lg:text}, model='text-embedding-3-small'); ensure_vector_column / vectorize_rows (delegating to ensure_derived_column in unify/common/embed_utils.py) create and backfill these. A search call maps column → reference text (references={"description": "billing complaints"}), ranks by cosine similarity server-side, and averages across terms when several columns are given. Expression sources hash into derived columns (_expr_<hash>) before embedding. Reduce. Aggregations (count, sum, mean, var, std, min, max, median, mode) with optional group_by, executed by Orchestra’s metrics endpoint. Note for extenders: DataManager.reduce calls federated_reduce (not reduce_impl directly) so decomposable metrics merge correctly across personal and team roots. Joins. Two execution shapes:
  • join_tables materializes a joined table into a destination context (unisdk.join_logs) — use when the join result is itself a dataset.
  • filter_join / reduce_join are ephemeral, fused queries (unisdk.join_query) — a single server round-trip returning rows or aggregates, no temp context. search_join joins into a temp context, searches semantically, and cleans up. The *_multi_join variants chain steps, with $prev referencing the previous step’s result.
Join expressions namespace columns by full context path ("Data/orders.customer_id == Data/customers.id"), and DataManager rewrites those paths per root group when federating across personal and team scopes (rewrite_join_paths in unify/common/join_utils.py).

​
The ingest pipeline

ingest is the preferred bulk-load path and the most mechanically interesting op. run_ingest in ops/ingest_ops.py builds a TaskGraph and hands it to PipelineExecutor (utils/pipeline.py), a small thread-pooled DAG engine with retries, backoff, and a scheduling policy that prioritizes downstream tasks (so embedding can run along inserts instead of degenerating to after them):
  1. Type prescan — ops/type_prescan.py infers column types from a stratified sample (prescan_column_types → TypeMap) and coerce_batch cleans each chunk (empty strings → None, type mismatches → None, tallied in CoercionStats).
  2. Create + chunked insert — table created if needed, rows inserted in chunks (default 1000), serialized when auto_counting demands it.
  3. Optional embedding — vector columns ensured and backfilled, batched.
  4. Post-ingest derived columns — PostIngestConfig rules (ExplicitDerivedColumn with an equation like "{unit_price} * {quantity}", or AutoDerivedColumn by source type).
The whole run returns an IngestResult (rows_inserted, rows_embedded, chunks_processed, coercion_stats, duration_ms, …), and IngestExecutionConfig exposes the pipeline knobs (max_workers, insert_parallelism, embedding_batch_size, fail_fast).

​
Mutation semantics worth knowing

  • update_rows is implemented as delete + re-insert in ops/mutation_ops.py — not an atomic field update.
  • update_rows / delete_rows require a filter (or explicit log_ids); destructive table ops require dangerous_ok=True.
  • Writes into shared team contexts strip authorship fields via is_shared_authored_context (unify/common/authorship.py).
  • describe_table deliberately omits row_count (expensive); use reduce(metric="count").

​
The simulated backend

SimulatedDataManager keeps everything in dicts (_tables, _schemas, _embeddings, …) and is honest about its shortcuts: filters run through local eval, search fakes ranking with word overlap, joins are simplified merges. It exists so Actor evals and unit tests run with zero backend — don’t use it to validate join-expression correctness. Fixtures live in tests/data_manager/ (simulated_dm, seeded_dm), and tests/data_manager/ doubles as the best map of behavioral guarantees: context resolution, destination routing, streaming ingest, type prescan, pipeline mechanics.
Two orphaned modules — ops/plot_ops.py and ops/table_view_ops.py — remain from the era when DataManager rendered visuals. plot() and table_view() were removed from the public contract; all visual output now goes through CanvasManager.

​
CanvasManager

​
Design in one paragraph

The Actor writes a real React module — TSX importing only react and @unity/canvas-kit — and hands it to create_view together with declared query bindings and actions. The manager lints, typechecks and bundles the module, dry-runs every binding through the live DataManager, verifies every action target exists, renders the view headlessly and critiques the result before anything is published. The stored row carries the source, the compiled bundle and its sha256; a 12-character routing token is registered with the backend, and the Console serves the view in a genuinely isolated frame. There is no raw-HTML path and no unauthenticated token-is-access URL — both were properties of the dashboard tiles this layer replaced, and both were retired with them.

​
Package anatomy

Rows live in three registered contexts — Canvas/Views, Canvas/Actions, Canvas/Invocations — written through the data layer like every other manager. There is no separate layout table: React composes, so a multi-tile board is simply one view, and the dangling-reference bug class of a layout pointing at a deleted tile cannot exist.

​
The binding type system

A view declares its data as PrimitiveBinding entries — an alias the canvas reads via canvas.data[alias], the owning manager and table, and one of four read-only query shapes mirroring the DataManager families (filter, reduce, join, join_reduce). Every binding is dry-run at author time, and at view time the server executes only the stored binding a client names by alias: no context path, filter expression or row limit is ever accepted from the viewer, which is the security inversion that retired the old tile bridge.

​
Actions and invocations

A view may declare CanvasActions: named operations with JSON-Schema input (every string and array bounded), an execution lane (stored function, task trigger, or assistant request), optional confirmation text rendered by the Console outside the frame, and per-hour rate limits. A viewer’s invocation is validated server-side, recorded as a Canvas/Invocations row, then executed by the assistant — the frame never learns which function or task sits behind an action.

​
destination and visibility

  • destination — where the view’s row lives ("personal" or "team:<id>"), which controls who sees it in listings.
  • visibility — who may read it (private, team, public_link), enforced server-side on every read path; publishing and quarantining flip status without reissuing the URL.

​
Extending

A new binding shape touches the full stack — in this repo: a new args model with a unique operation literal in canvas_manager/types/binding.py; resolution and a real DataManager dry-run in ops/binding_ops.py; and documentation in the create_view docstring (remember: docstrings are the Actor’s API). Orchestra then needs the matching server-side execution in its canvas query route. A new DataManager operation is simpler: abstract method + docstring on BaseDataManager, an *_impl in the right ops/ module, implementations in both the real and simulated managers — the primitives registry picks it up automatically. A new backend for either manager: subclass the base, implement the contract, register via ManagerRegistry.register_class, and extend the settings enum.

​
Where to start reading