# 0.39.1-M1 - Workspaces, Copilot, secrets and validation

> The first milestone of 0.39, the biggest change Orbital has had. Multiple workspaces per instance, organisations, a secrets manager with five backends, a data validation framework, an MCP server, and a new multi-file code editor.

Source: https://orbitalhq.com/changelog/2026-08-24-release-announcement-0.39.0
Date: 2026-08-24
Version: 0.39.1-M1

_Available since 0.39.1-M1_

0.39.1-M1 is the first milestone of 0.39 — the largest change Orbital has had. It has been in development since
April, and it changes the shape of the product in a few significant ways.

The headline is **workspaces**: a single Orbital instance can now run many independent schemas, each with its
own projects, its own compiled schema, its own connections and its own query context. Everything downstream of
that — auth, secrets, search, history, caching, the language server — was reworked to be workspace-aware.

Alongside that: a **secrets manager** with five backends, a **data validation framework**, **Copilot**
(chat-driven querying and schema building), an **MCP server** so external agents can query your data mesh, and
a **new code editor** that edits whole projects rather than single queries.

Two features we announced over the summer also ship here for the first time: **collection options**
(`limit`, `offset`, `orderBy`, `uniqueBy` on any query, pushed down to the source where it can take them), and
**`@DeleteOperation` / `@SqlQuery`** for the database connectors.

As a milestone, 0.39.1-M1 is for early adopters and teams who want to test ahead of the full 0.39 release.
Expect rough edges, and tell us about them.

**Warning: Read the breaking changes first**
This milestone requires a license, renames a feature toggle, migrates the query history schema, and removes
  the legacy Jet pipeline server. Start with [Breaking changes](#breaking-changes) before you upgrade.

---

## Breaking changes

### Orbital now requires a license
Orbital checks for a `license.json` on startup. When you first sign in, Orbital contacts the license server at
`https://account.orbitalhq.app`, downloads a **free** license and installs it. In most environments this needs
no setup.

If no license is found, Orbital issues a short-lived fallback license so the server still starts — but that
fallback only lasts **30 minutes**, after which the UI shows a blocking overlay.

Orbital searches these paths in order, and the first valid license wins:

1. `--vyne.license.path` (if set)
2. `${vyne.app.data.path}/license.json` — by default `./orbital_data/license.json`
3. `~/.orbital/license.json`
4. `/opt/var/orbital/license/license.json`

For air-gapped deployments that can't reach the license server, Orbital supports an air-gapped mode that
disables usage reporting. [Get in touch](mailto:hello@orbitalhq.com) if you need it.

See [configuring your license](/docs/deploying/configuring-your-license) for the full detail.

One improvement worth calling out: an expired license no longer terminates the JVM. The old `LicenseMonitor`
called `exitProcess(0)` on expiry. That's gone — expiry now surfaces in the UI and is governed by the license
policy instead of killing the process.

### The workspaces-enabled toggle is replaced by workspace-mode
The boolean toggle is gone, replaced by a three-state enum:

```yaml
vyne:
  toggles:
    workspace-mode: None    # None | Workspace | OrgAndWorkspace
```

| Mode | Behaviour |
| --- | --- |
| `None` (default) | Single workspace. No workspace prefix in UI routes, no selector. Behaves as Orbital always has. |
| `Workspace` | Workspace routing. UI routes are prefixed `/{workspaceId}/...` |
| `OrgAndWorkspace` | Org and workspace routing: `/{orgId}/{workspaceId}/...` |

If you were setting `workspacesEnabled: true`, that key is now ignored and you will silently fall back to
`None`. Set `workspace-mode` instead.

**Note: Use the kebab-case spelling**
Some of the code reads this property through a `@Value` placeholder, which does not get Spring's relaxed
  binding. Always spell it `vyne.toggles.workspace-mode`, not `workspaceMode`.

### Query history tables gain non-null orgId and workspace_slug columns
A Flyway migration adds a `workspace_slug` column across eight history tables, adds `orgId` to
`query_error_event` and `trace_event`, backfills both with `default`, and enforces `NOT NULL`. Indexes on
`(orgId, workspace_slug)` are created alongside.

The migration runs automatically. On instances with a large query history, the `SET NOT NULL` statements take
a table lock, so budget for some downtime on the first boot after upgrading.

The same migration drops the Postgres row-level-security policies on those tables. RLS was keyed on
`current_user`, which never worked with a shared connection pool. Org and workspace isolation is now enforced
in the application layer via org-membership checks and explicit `orgId` / `workspace_slug` filters on every
query.

Rows written before this milestone cannot be attributed to a workspace — they were never stamped — so they stay
as `default`/`default`.

### The legacy Jet pipeline server has been removed ::breaking
`pipelines/pipeline-jet` is gone from the build, along with the pipeline transports that only it used (S3,
SQS, Cask, file-watcher, HTTP listener, JDBC, Kafka, Redshift, logging sink, polling query input) and the
transform stage.

This is the standalone pipeline runner, which has been unmaintained and undocumented for a long time.
**Streaming queries are unaffected** — `stream { }` queries run through `pipelines/stream-engine`, which stays.

### H2 is no longer a shipped JDBC driver ::breaking
`H2DatabaseSupport` has moved to test sources. The drivers shipped to end users are now Postgres, MSSQL,
Oracle, Redshift, Snowflake and Databricks. If you had an H2 connection defined in `connections.conf`, it will
fail to resolve.

### Spring Boot 4 and Jackson 3
Orbital is built on Spring Boot 4.0 and Jackson 3. If you write custom Orbital extensions — custom taxi
functions, custom connectors, anything compiled against Orbital's jars — you will need to migrate. The main
things that bite:

- Jackson 2 → 3: the mapper and builder are immutable, and several APIs were renamed. The annotation package
  is still `com.fasterxml`.
- Spring 7 / Security 7 JSpecify nullability, and `: Any` generic bounds in Kotlin.
- The Spring Boot 4 modular auto-config split (`spring-boot-health`, `-jdbc`, `-hibernate`, `-mongodb`, and the
  matching test slices).
- Testcontainers 1.x → 2.x, if your tests use it.

### Numeric comparison is now type- and scale-insensitive ::breaking
Comparisons between numbers of different JVM types now coerce both sides to `BigDecimal` and compare with
`compareTo`. Two consequences:

- Cross-type numeric comparisons work. Previously only a narrow set of pairings (such as `Int` vs
  `BigDecimal`) compared correctly; others silently failed.
- `Equal` and `NotEqual` are now scale-insensitive. `2.50 == 2.5` is now `true` where it was previously
  `false`, because `BigDecimal.equals` treats different scales as different values.

Taxi's `Long` is backed by `BigInteger`, which previously had no branch in the arithmetic calculator at all
and errored with *"Unsupported number type"*. `Long` arithmetic now works, and division on `Long` truncates —
the same semantics as `Int`.

### The right() function now reads from the right ::breaking
`right(input, n)` was returning characters counted from the *left* of the string. It now returns the last `n`
characters, as the name and documentation always said. If you worked around the old behaviour, remove the
workaround.

### Operation parameters are no longer matched to raw primitives by type ::breaking
Given `operation confirm(OrderId, note : String)` and an `OrderId` in context, the old code would bind the
`OrderId`'s value to `note` — because everything is assignable to `String`, so a raw primitive parameter
matched whatever value happened to be nearby.

Parameters typed as raw primitives (`String`, `Int`, ...) or `Any` are now populated only from an **exact**
type match already in context — never an inherited match, never a graph query. Otherwise they resolve to null
and normal nullability handling applies.

**A default declared in the signature is unaffected**, because it names the parameter it belongs to rather
than being matched to it. That remains the supported way to populate a raw primitive.

If a schema was relying on the old behaviour, the operation will now be invoked with a null (or not invoked at
all, if the parameter is required). Give the parameter a semantic type, or a default.

### Query responses honour the Accept header ::breaking
HTTP query responses now follow standard content negotiation, in this order:

1. An explicit, specific `Accept` header wins.
2. Otherwise the response type's declared format (`@Xml`, `@Csv`) is used.
3. Otherwise JSON.

Previously the model's declared format always won, so a model annotated `@Xml` requested with
`Accept: application/json` still came back as XML. If you have a client relying on that — sending an `Accept`
header it didn't mean and getting the model's format regardless — it will now get what it asked for.

An absent `Accept` header is treated as "no preference" rather than being forced to JSON, so it defers to the
model's declared format. This applies uniformly to the `/api/taxiql` endpoints and to routed saved queries.
Streaming responses (SSE and WebSocket) negotiate separately and are unchanged.

### API routes are workspace-prefixed when workspace mode is enabled
In `Workspace` or `OrgAndWorkspace` mode, all workspace-scoped APIs move under `/api/{orgId}/{workspaceId}/...`,
and published query endpoints move to `/api/{orgId}/{workspaceId}/q/...`.

In `None` mode — the default — legacy routes such as `/api/schema/types` and `/api/q/myQuery` continue to work
exactly as before. This includes the anonymous-access security matcher, which now picks the pattern matching
the active mode.

### Smaller breaks
- **The auth-token listing API changed shape.** `GET /tokens` returned a flattened
  `Map>`. It now returns a row-shaped listing that preserves the package each
  entry came from, plus a sibling field carrying per-package parse errors.
- **Git credentials in `workspace.conf` are now secret references.** Inline `GitCredentials` / `GitSshAuth`
  are replaced by a `GitAuthentication` reference that points at a secret name resolved through the workspace's
  secrets manager at fetch time.
- **SOAP responses are parsed by JAXB element names.** Responses are read using `@XmlElement` metadata and
  field access rather than by introspecting CXF's generated Java getters. This fixes acronym and casing
  mangling (`getSISOCode` was becoming `SISOCode` rather than `sISOCode`), but if you had types hand-shaped to
  match the old mangled names, they now need to match the WSDL names.
- **UI routes renamed**, with redirects in place: `/stubs` → `/local-environment`, `/catalog/diagram` →
  `/mesh`, `/workspace` → `/workspaces`. The old query editor at `/query/editor` is replaced by
  `/code-editor`.
- **New privileges.** `LIST_SECRETS` and `EDIT_SECRETS` were added and are granted to the admin and platform
  manager roles in the bundled default role definitions. Deployments that already have a `roles.conf` on disk
  will not pick these up automatically — Orbital only writes the defaults when the file is absent. Add them by
  hand if you want the secrets UI available to existing roles.

---

## Workspaces and organisations
A workspace is a named context that compiles to a schema. Everything schema-driven in Orbital — queries,
streams, published endpoints, the catalog, the search index — operates within one.

Typical uses:

- **Team isolation** — each team owns a workspace with its own services and types.
- **Environment separation** — production, staging and development in one cluster.
- **Safe experimentation** — test a new version of a service in a workspace without touching the others.

An **organisation** is the tenant above that. A cluster serves one or more organisations; on-prem deployments
typically have one, and a default org is created on first boot if none exists.

### Turning it on

```yaml
vyne:
  toggles:
    workspace-mode: OrgAndWorkspace
```

When enabled, a combined org-and-workspace selector appears in the header. Collapsed, it shows the active
workspace above the active organisation; expanded, it lists the workspaces in the org, with a nested
*Switch organisation* submenu and management actions. In `Workspace` mode the org half is hidden.

Every page — catalog, code editor, projects, activity — is scoped to the active workspace, and the URL
reflects it.

### Where workspace config lives

Two things need storing: which orgs and workspaces exist, and what projects make up each workspace. Both are
configured under `vyne.orgs`:

```yaml
vyne:
  orgs:
    store: FILE                 # FILE | IN_MEMORY | DATABASE
    config-path: ./orbital_data
```

| Backend | Best for | Writable from the UI? |
| --- | --- | --- |
| `FILE` (default) | Local dev, single-node on-prem | Yes |
| `IN_MEMORY` | Ephemeral environments, tests | Yes, until restart |
| `DATABASE` | Multi-node and cloud deployments | Yes |

The file backend reads an `organisations.conf` from `config-path`, with one `workspace.conf` per workspace
underneath:

```
orbital_data/
  organisations.conf
  workspaces/
    acme-bank/
      production/workspace.conf
      analytics/workspace.conf
```

The database backend stores the same HOCON content as text blobs in `org_config`, `workspace_config_entry` and
`workspace_config`, using Spring Data repositories over Orbital's existing datasource. Because the stored
content is byte-identical to the file format, you can move between backends without transforming anything.

Workspace config also materialises back to disk on boot, so a cloud deployment with a populated database and a
fresh disk rehydrates correctly. The file backend leaves hand-edited content alone.

**Note: Git-backed org config**
A `GIT` store type is declared in the enum but is not implemented in this milestone. Use `FILE` or `DATABASE`.

### What became workspace-aware

The bulk of the work in 0.39 was making everything downstream of the schema respect workspace boundaries. Things
that used to be global singletons and are now per-workspace:

- **Connections and invokers.** Every connection-based invoker (JDBC, Kafka, MongoDB, AWS, Azure, Hazelcast,
  SOAP, HTTP) is built per workspace through an `InvokerFactory`, from that workspace's own connection registry.
- **`services.conf` and `auth.conf`.** Service discovery and outbound auth resolve through per-workspace
  registries, so HOCON substitution sees that workspace's own environment variables. Previously a single global
  registry meant `auth.conf` files discovered through a workspace's project `additionalSources` were silently
  ignored, and outbound HTTP calls went out unauthenticated.
- **Hazelcast state.** Operation caches, state stores and stream topics are all namespaced by workspace, so
  results can't leak across boundaries.
- **The search index.** Lucene indices are per workspace, stored under `{basePath}/{orgId}/{workspaceId}/`,
  rebuilt only for the workspace whose schema changed.
- **Query history.** Every history table is stamped and filtered by org and workspace.
- **The language server.** LSP sessions resolve the workspace's live schema, and schema edits are delivered to
  the sessions bound to that workspace.
- **WebSocket streams.** Schema notifications, query status, stream results and stub updates all reconnect when
  you switch workspace, and are filtered to the workspace you're connected to.
- **Query routes, scheduled queries and persistent streams.** These maintain cross-workspace indices and now
  subscribe to schema changes from every workspace, including ones created at runtime.

Workspaces created through the API or by dropping a directory on disk are registered live — no restart needed
for schema compilation, search indexing or route registration.

### Org-scoped authorization ::feature
`@RequiresOrgPrivilege` replaces `@PreAuthorize("hasAuthority(...)")` across roughly 150 production endpoints.
It resolves the request's `{orgId}` path variable and delegates the decision to an `OrgAuthorityService`, which
has two implementations:

- The default is org-blind and checks the flat authority set, preserving behaviour for single-org and
  non-PropelAuth deployments.
- When `vyne.security.open-idp.roles.format=propelauth`, per-org roles are read from the JWT's
  `org_id_to_org_member_info` claim.

This unblocks users who belong to multiple organisations, who previously hit
*"You are a member of multiple organisations"*.

Org membership is now enforced on the workspace endpoints themselves. Previously any authenticated user could
list or create workspaces in any tenant by putting that tenant's slug in the URL. Organisations are also
auto-provisioned from JWT claims on first sign-in, so workspace creation works on a fresh deployment.

---

## Copilot
Copilot is a chat interface to your data mesh. It runs in two modes:

- **Ask** — natural-language questions answered by generating and running a TaxiQL query against the current
  workspace. *"What's Jim Patterson's account balance?"*, or *"write me a query that joins orders to
  customers"*.
- **Build** — an agent with file-editing tools that works on your Taxi projects, streaming its edits directly
  into the code editor beside the conversation.

The mode is chosen before you send the first message, and pinned for the life of the conversation — reopening
a conversation restores the mode it was started with.

Conversations are persisted with their history, can be renamed, and are listed in a sidebar. File uploads are
supported and scoped to the workspace. In Build mode, edits stream into Monaco models as they arrive, so you
watch the agent write.

### Configuring it

Copilot is a separate service. Point Orbital at it and turn on the toggle:

```yaml
vyne:
  toggles:
    copilot-enabled: true
  copilot:
    endpoint-url: http://localhost:9028
    upload-store: local
    local-store-path: ./orbital_data/copilot/uploads
```

Requests to the Copilot service carry the caller's JWT, and all calls are scoped to the current org and
workspace.

### MCP server
Orbital exposes a [Model Context Protocol](https://modelcontextprotocol.io) server, so Claude, Cursor, IDE
agents and your own tooling can ask the same questions a Copilot user can.

```yaml
vyne:
  toggles:
    mcp-server-enabled: true
```

One endpoint is mounted per workspace:

```
/api/{orgId}/{workspaceId}/mcp
```

Mounting it under the workspace path means it inherits the existing org-membership gate from the security
filter chain. No tool takes `orgId` or `workspaceId` as an argument — the URL is the only source, so an MCP
client can't reach across into another tenant's data by changing a parameter.

The server exposes a single `ask` tool:

| Field | Description |
| --- | --- |
| `question` | The question, in natural language |
| `execute` | Run the generated query and return rows, or just show the query that would run |

It answers with the natural-language `answer`, the generated `query`, a `queryHistoryId` you can use to
inspect the run in Orbital's history, the result `rows`, and any `queryError` or `diagnostics`.

The tool description and the server's MCP `instructions` are generated from the workspace's own schema, so
clients know what data they're pointed at.

Authentication accepts a PropelAuth API key presented the standard way — `Authorization: Bearer <token>` —
which is what MCP clients send. Orbital now discriminates structurally: a bearer credential shaped like a JWT
is validated as one, anything else is validated as an opaque API key. Previously an API key had to be sent
*without* the `Bearer` prefix, which MCP clients can't do.

---

## The new code editor
The old single-query editor is replaced by a multi-file editor at `/code-editor`.

- **A real file tree.** It walks the project on disk, so `taxi.conf`, READMEs, `connections.conf`, nebula
  scripts and everything else is visible — not just `.taxi` files. Content is fetched lazily when you click.
  Per-file error counts show in the tree.
- **Split panels.** Editors, query results and query history live in dockable panels you can arrange, and the
  layout persists across refreshes — keyed by workspace, so switching org or workspace doesn't restore the
  wrong location's panels.
- **File lifecycle.** Right-click to rename or delete. Both flow through the same compile-after-change pipeline
  as content edits, with atomic moves for renames.
- **Save anything.** Saving no longer runs every source in the package through the Taxi compiler and rejects
  the write on any error. Non-Taxi files are excluded from compilation entirely, and Taxi files with
  compilation errors save anyway, with the errors surfaced in the UI. The editor's job is to write what you
  typed.
- **Language support** is auto-detected from the file extension. Markdown renders. Unmapped types open as
  plain text instead of failing.

Two long-standing editor bugs are fixed. Editing or saving a file used to produce
`Symbol X is already declared` errors for every symbol in the file, caused by the same file being registered
under two different URIs — once through the language server and once through the schema editor's in-memory
edit application. And edits to files outside `src/` were being nested under `src/`, while edits already
addressed with a `src/` prefix were double-nested at `src/src/...`.

---

## Secrets management
Secrets are now first-class. Reference one anywhere in your config the same way you'd reference an environment
variable:

```hocon
connections.jdbc.warehouse {
   url:      "jdbc:postgresql://warehouse.example.com:5432/prod"
   password: ${secrets.DB_PASSWORD}
}
```

Real values never enter the merged HOCON config. The placeholder resolves to an opaque token that the consumer
resolves at the point of use — at connection time, at outbound-request time, at git-fetch time — so a secret
lives in a call frame and never in long-lived state. Rotating a secret takes effect without a restart.

### Backends

Pick one with `vyne.secrets.backend`:

| Value | Backend |
| --- | --- |
| `InMemory` (default) | In-process. Tests and demos only. |
| `File` | Local file, age-encrypted (X25519 + ChaCha20-Poly1305). Dev only — the key sits next to the data. |
| `Infisical` | [Infisical](https://infisical.com), cloud or self-hosted, via universal auth |
| `Aws` | [AWS Secrets Manager](https://aws.amazon.com/secrets-manager/) |
| `Vault` | [HashiCorp Vault](https://www.vaultproject.io/) KV v2 |
| `Gcp` | [Google Cloud Secret Manager](https://cloud.google.com/secret-manager) |

Each backend maps Orbital's scope hierarchy onto its own naming rules — paths in Vault and Infisical, flat
prefixed names in AWS, `__`-separated segments in GCP, whose names can't contain slashes.

### Scopes

| Scope | Visible to |
| --- | --- |
| `Server` | every workspace on this server |
| `Organisation` | every workspace in that organisation |
| `Workspace` | only that workspace |

Reads cascade — workspace, then organisation, then server — and the narrower scope wins on a name collision.
There is no global namespace that bypasses scoping.

### Managing them

Secrets are managed in the UI under **Secrets**, and over REST:

```
GET    /api/{orgId}/{workspaceId}/secrets
GET    /api/{orgId}/{workspaceId}/secrets/{scope}
POST   /api/{orgId}/{workspaceId}/secrets/{scope}
PUT    /api/{orgId}/{workspaceId}/secrets/{scope}/{name}
DELETE /api/{orgId}/{workspaceId}/secrets/{scope}/{name}
POST   /api/{orgId}/{workspaceId}/secrets/refresh
```

Reads need `LIST_SECRETS`, writes need `EDIT_SECRETS`. **No endpoint ever returns a secret value** — responses
carry name, scope, description, last-updated and version only. Server-scoped secrets are listable but
operator-managed: writes to that scope are rejected with a 403.

A caller at `/api/acme/prod/...` can only touch `Workspace(acme/prod)` and `Organisation(acme)` — not another
org's or another workspace's secrets.

Every backend is wrapped in a caching decorator with a 15-minute TTL (configurable via
`vyne.secrets.cache.ttl` and `.max-size`), write-through invalidation, and the `refresh` endpoint above for
forcing a reload after rotating a secret out of band.

### Git authentication
Adding a git project now captures credentials — none, username/password, token, or SSH key — as references to
secrets rather than as credentials embedded in `workspace.conf` and travelling through git. The UI rejects
plaintext entries at submit, since they would defeat the point.

Credentials resolve inside JGit's `configure(transport)` call. SSH keys load as bytes, so no key is ever
written to a temp file.

See [managing secrets](/docs/deploying/managing-secrets) for the full setup.

---

## Data validation
You can now declare validation rules against your types, and control what happens when data breaks them.
Because rules are declared against **types**, you define them once and they're enforced everywhere that type
appears, whichever service the data came from.

Three parts: **declare** rules with annotations, **activate** them with `@Valid`, **decide** what happens with
`@OnValidationFailure`.

```taxi
import com.orbitalhq.validation.NotEmpty
import com.orbitalhq.validation.Valid

@NotEmpty
type Name inherits String

@Valid
model Person {
   name : Name
}
```

A `Person` arriving as `{ "name": "" }` produces one violation — rule `com.orbitalhq.validation.NotEmpty`, at
path `name`, severity `Error` — and by default rejects the query with an HTTP 400. Remove the `@Valid` and the
same data flows through untouched. Rules do nothing until something activates them.

### The built-in rules

`@NotEmpty`, `@MinLength`, `@MaxLength`, `@Pattern`, `@Min`, `@Max`, `@MultipleOf`, `@MinItems`, `@MaxItems`,
`@UniqueItems`, `@Email`, `@Uuid`. That's feature parity with OpenAPI's schema validations. Every rule takes an
optional `message` and `severity` (`Info`, `Warning` or `Error`).

Rules can go on a type, a field, or a query parameter, and they're inherited — a rule on `Name` applies to
every type inheriting from it.

Two rule ids are produced by the engine rather than declared: `NotNull`, when a non-nullable field inside a
`@Valid` scope holds a null, and `ParseFailure`, when a value can't be parsed into its declared type.

**Note: @Valid also turns on nullability enforcement**
Type-system nullability has historically been declared but not enforced. Inside a `@Valid` scope it is. This
  is opt-in — nothing changes for schemas that don't use `@Valid` — but expect to find violations the first
  time you switch it on for an existing model.

### Deciding what happens

```taxi
@OnValidationFailure(action = ValidationAction.Warn, threshold = Severity.Error)
query GetOrder {
   find { Order }
}
```

| Action | Behaviour |
| --- | --- |
| `Reject` | Fails the request with an HTTP 400 and a structured violation report |
| `Warn` | Data passes through. Violations are still logged and reported |
| `Drop` | Skips the offending record and keeps going. Never silent — every drop is reported |

`threshold` sets the minimum severity that triggers the action. **All violations are reported regardless of
threshold** — the threshold only decides which are severe enough to act on. That pairs well with per-rule
severity: mark rules you're not yet confident about as `Warning`, watch them in your logs, promote them to
`Error` once you trust them.

Defaults are per query kind: `find` rejects at `Error`, `stream` drops at `Error`. Query arguments always
reject — dropping a record makes no sense when the record is the caller's own input.

Violation paths are precise: `cast.actors[1].actor.name`, not just "somewhere in the response".

Validation runs on query arguments, on data returned by services, and on streaming pipeline inputs.

See [validating data](/docs/data-validation/data-validation) and the
[rule reference](/docs/data-validation/validation-rules).

---

## Data sources

### Oracle
Orbital now ships an Oracle driver, with full write support: `Insert` and generated-primary-key `Upsert` use
`INSERT ... RETURNING`, `Update` runs per-row batches, and a user-supplied-PK `Upsert` uses `MERGE`. Oracle's
`MERGE` can't return rows, so the invoker echoes the input back, matching how the Postgres upsert behaves.

Table generation accounts for Oracle lacking `CREATE INDEX IF NOT EXISTS` and needing explicit lengths on
indexed `VARCHAR2` / `CLOB` columns. jOOQ's dialect is resolved by probing the live connection rather than
assuming a version, so Orbital doesn't emit `CREATE TABLE IF NOT EXISTS` (23ai and later only) against an
older database. The probe is cached per connection.

Metadata reads are scoped to the connecting user's schema. Oracle has no schema concept separate from the
user, so without that scoping Orbital crawled `SYS`, `SYSTEM` and every other user's schema.

### Database metadata is read directly, and scoped ::improvement
Table introspection — the connection UI's table list, and the schema generator behind *import from database* —
was built on SchemaCrawler, which required a full catalog crawl. On a warehouse that's slow to the point of
unusable, and it had no plugin support for Snowflake or Databricks.

It's been replaced with targeted `DatabaseMetaData` calls. What that changes for you:

- **The reactive server no longer stalls.** `listConnectionTables` and `getTableMetadata` ran their blocking
  JDBC calls on the Netty event loop, which starved the whole server until it was restarted. They now run on
  a bounded elastic scheduler.
- **Listing tables returns names only.** Column, primary-key and index metadata were an extra round-trip per
  table, on a screen that only needs names. They're opt-in now, via `includeColumns` / `includeIndexes` query
  parameters on the tables endpoint.
- **Generating Taxi for one table is independent of database size.** It reads the tables you asked for plus
  the ones their foreign keys reference, one level deep, rather than crawling the whole schema.
- **Better type coverage.** `CLOB` maps to `String` instead of failing generation outright, and Postgres
  domain types map to `Any` rather than erroring.

### Databricks and Snowflake
**Databricks** is new. Hostname, port, HTTP path and token are the user-facing parameters; `AuthMech=3` and
`UID=token` are baked into the URL template so you don't see them.

**Snowflake** existed as a twelve-line shell that had never been used in anger, with its JDBC driver commented
out of the build because the fat jar broke shading. It's been rebuilt: region is now an optional URL parameter,
and the thin driver replaces the fat one — no shading problems, no broken native build.

Neither platform supports the `ON CONFLICT` syntax the other drivers use for upserts, so both build a
`MERGE INTO` statement instead. That needs something to match existing rows on, so when a primary key column
isn't among the written fields — a `@GeneratedId` key, for instance — the write falls back to a plain
`INSERT` and logs a warning. It's a warning rather than a silent fallback because rows can duplicate under an
`@UpsertOperation` when it happens.

Driver-specific column unwrapping moved off the base JDBC invoker onto the driver itself, so Postgres owns its
`PGobject` / `PgArray` handling and any driver returning its own wrapper types can be supported cleanly.

### MongoDB
MongoDB connections can now be created and tested from the UI. The driver was already listed in the connections
editor, but test and create posted nowhere — the UI had no URL mapping for NoSQL connectors and the backend had
no endpoint. Both now exist, and MongoDB has a health-check provider like the other connectors.

### Connection failures surface properly
A database-backed saved query invoked through a routed HTTP endpoint used to return `200 OK` with an empty body
when its connection was undefined or unreachable — indistinguishable from a query that legitimately matched
nothing.

Connectors now raise typed exceptions whose HTTP status comes from the exception itself:

| Condition | Status |
| --- | --- |
| Schema references an undefined connection | 500 |
| Connection defined but upstream unreachable | 502 |

JDBC connection failures are also reported as trace events, so they show up in the query profile rather than
vanishing.

### Deletes and native SQL
`@DeleteOperation` gives you type-safe deletes through TaxiQL. A single `@Id` renders `WHERE pk IN (...)`,
composite keys use row-value `IN`, and statements are chunked at 500 rows to stay under Oracle's IN-list cap.
Values are always bound, never inlined. The operation returns a `deletedCount`.

`@SqlQuery` lets you write the SQL yourself — joins, aggregations, dialect-specific statements — for the cases
where the generated query isn't what you want. One annotation covers reads and native DML, split by taxi's
`write` keyword. Reads map joined and aggregated rows onto plain models by column name or alias, with no
`@Table` needed; writes return the affected-row count.

`:param` placeholders bind as real prepared-statement parameters, validated against the operation's parameters
before execution. Values never touch the SQL text.

Both were [announced in July](/changelog/2026-07-16-jdbc-deletes-and-native-sql), and this is the first build
they ship in.

### Also

- Overriding a service in `services.conf` can now downgrade from `https` to `http`, which it previously
  refused to do.

---

## Query engine

### Collection options: limit, sort, paginate and dedupe
You can shape the results of a query directly in TaxiQL — limiting how many rows come back, paginating through
them, sorting them, and removing duplicates:

```taxi
find { Person[]( CountryCode == "GB", orderBy: DateOfBirth desc, offset: 20, limit: 10 ) }
```

`limit`, `offset`, `orderBy` and `uniqueBy` are the standard database-style controls, and they work on **any**
query, against any source — not just databases.

Sort by more than one field, and paginate:

```taxi
find { Person[]( orderBy: [CountryCode asc, DateOfBirth desc], offset: 40, limit: 20 ) }
```

They work on projections too, including nested collections, which is handy for trimming child records per
parent:

```taxi
find { Customer[] } as {
  name : CustomerName
  // Only the 20 most recent transactions per customer
  transactions : Transaction[]( limit: 20, orderBy: TransactionDate desc )
}[]
```

Values can be literals or query parameters, so `limit: maxRows` works when `maxRows` is a query argument.

**Pushdown is decided per option, per data source.** When a source can apply an option itself, Orbital pushes
the work down to it; when it can't, Orbital does it after fetching. Either way you get the same answer, and
pushdown only happens when it can't change the result — if a source can't sort, Orbital won't push a `limit`
below the missing sort, because limiting before sorting returns the wrong rows.

| Source | Pushed down |
| --- | --- |
| SQL databases (Postgres, MySQL, MSSQL, Oracle, Redshift, Snowflake, Databricks) | `limit`, `offset`, and `orderBy` when every sort term resolves to a single column — rendered in your driver's dialect |
| MongoDB | `limit`, `offset` and `orderBy`, as `sort` / `skip` / `limit` |
| Hazelcast | `limit`, as a key-ordered page |
| REST APIs, Kafka, everything else | Nothing — Orbital applies all of it after fetching |

`uniqueBy` is never pushed down; it's always applied by Orbital. Operations annotated with `@SqlQuery` opt out
of pushdown entirely, since options can't be spliced into SQL you wrote by hand — they're applied after the
fetch instead.

On streaming queries, `limit` is supported and completes the stream once it has emitted N items. `orderBy`,
`offset` and `uniqueBy` are rejected on streams, as is cursor pagination (`after` / `before`), which parses but
is not yet implemented.

This was [announced in July](/changelog/2026-07-04-collection-options) — 0.39.1-M1 is the first build it ships in.
Full detail is in the [collection options docs](/docs/querying/collection-options).

### Operation parameters are gathered consistently
Parameters were being gathered in four separate places that disagreed with each other: the mutation path, the
graph-search path, the direct-invocation path, and again inside the invoker. The same call could bind a value
to the wrong parameter on one path, send `null` on another, and ignore a declared default entirely on a third.
An operation could also be invoked with fewer arguments than it declares, because the graph path caught any
discovery failure and quietly dropped that parameter.

There is now one shared implementation returning an explicit map of parameter to value. Callers differ in what
they do when it can't be satisfied — direct invocation tries another candidate, graph search fails the edge,
mutation returns a null — but they all gather the same way.

The operation-result cache is keyed off that map, too. Keyed off a set, as it was,
`getEligibility(orderId=A, lineId=B)` and `getEligibility(orderId=B, lineId=A)` shared a cache entry.

Nullable parameters that can be constructed from context now are. Previously `discover` short-circuited to null
the moment it saw a nullable parameter, so a nullable request body whose fields were all present in context was
sent as `null`.

### Circular type references no longer blow the stack
A model whose definition leads back to itself — `total : Total = (Total * 1.2)`, or a pair of types whose
expressions reference each other — recursed until the JVM threw `StackOverflowError`.

A closed loop now yields a null carrying the cycle path — `Order.total -> Order.total`, or `A -> B -> A` —
rather than throwing. Circular definitions are now *legible*, not valid; they remain unsatisfiable.

This matters more than it looks. `StackOverflowError` isn't recoverable the way an exception is: if it lands
while a class is initialising, that class stays unusable for the life of the JVM and every later touch throws
`NoClassDefFoundError`. We hit exactly that during a benchmark run — it broke coroutine cancellation
process-wide, with no error naming the cause.

### Collecting arrays across collection boundaries
A query like `find { GivenName[] }`, against a response whose `givenNames` sit two collection layers deep
inside the returned object, failed with *"no data sources can return GivenName[]"*. The query graph only had
scalar attribute edges, so there was no path.

There's now an edge from each provided instance to every `T[]` reachable via a path crossing at least one
collection field. Its cost sits between cheap attribute navigation and remote operation invocation, so a direct
attribute path still wins when one exists, but the planner always prefers in-memory extraction over calling a
service again.

A service returning no matching instances yields an empty `T[]` rather than failing the query.

### Query errors reach the error view
Several places in the engine caught exceptions and returned a null or an empty result without telling anyone,
so genuine failures never reached the query error view. Failures that are actually meaningful — a nullable
operation parameter that couldn't be constructed, unexpected exceptions while searching for a type — now
publish to the error stream that feeds the errors websocket and query history.

Routine search backtracking is deliberately left silent. It's normal control flow, it fires per row during
projection, and surfacing it would flood the view with alarming noise.

Separately, fast-failing queries used to tear down their error stream before the UI could connect to it, so the
same query would only intermittently show its errors. The error publisher keeps a 30-second replay buffer for
exactly this reason; eviction is now deferred past the replay window so a late-connecting websocket can still
drain it.

### Also in the engine
- A TaxiQL statement that fails to compile returns **400 Bad Request** rather than 500.
- Failed searches capture where and why they failed, with a readable display of the failure point.
- Graph search error reporting is more informative, and the error message now hints when a `find` with a
  contract would be better expressed as a `given {}`.
- Fact-bag lookups had lost their combine, dedupe and absent semantics when they moved to an `Either`-based
  API: deep searches silently dropped all secondary-bag matches whenever the primary bag had any match, merged
  collections skipped deduplication, and "absent everywhere" was reported as an empty collection rather than a
  null — which stopped the object builder falling through to discovery. All three are restored.
- Data policies are no longer applied to primitive types.
- Projections no longer attempt to construct scalar types, and fields carrying expressions are deferred to
  expression evaluation rather than being built directly.
- Object construction no longer throws when the type is nullable, matching field construction.

### Performance
This milestone adds a benchmark suite and CI guards rather than a broad optimisation pass. `query-engine-benchmarks`
carries seeded synthetic schema and data generators, 13 workloads, 7 JMH micro-benchmarks, and a guard suite
that gates merge requests on deterministic metrics — operation invocation counts, allocation budgets,
complexity-class scaling ratios and heap-slope leak checks — while tracking wall-clock time as a nightly trend
rather than a flaky gate.

One fix landed alongside it: fact-bag search caching now validates lazily on read against a generation stamp,
so adding a fact is O(1) rather than doing predicate work proportional to every cached search. Measured 86–93%
off `addFact`, with lookup cost flat.

A full review of the engine's performance characteristics is written up internally. The remaining findings are
the fix programme for subsequent releases, and each has a matching benchmark guard waiting to be switched on.

---

## UI
The navigation has been reorganised:

| Was | Now |
| --- | --- |
| Query editor | **Code editor** — the new multi-file editor |
| Query history | **Activity** — running queries and history in one paginated view |
| Authentication | **Secrets** |
| Stub Servers | **Local environments** |
| Catalog → Diagram tab | **Mesh** — promoted to a top-level item |
| — | **Copilot** |

### Endpoints
The endpoint page is now tabbed, matching the catalog layout:

- **Overview** — query plan diagram and metrics
- **Schema** — the model attribute tree for the query's return type
- **OpenAPI** — a generated OpenAPI spec, for non-streaming queries
- **Source** — the query source

A new endpoint, `GET /api/schemas/queries/{queryName}/openapi`, generates the spec for any saved query by name.
The existing `/api/q/meta/{queryName}/oas` only covered HTTP-routable queries.

Published query URLs shown in the UI now match where the endpoint is actually served — previously the raw
declared URL was shown, which didn't match the workspace-prefixed route.

### Local environments
Three unhappy paths on the Local Environments page used to fail silently or mislead:

- **Compilation errors.** A nebula script that failed to compile produced no feedback at all — the Nebula
  server killed the websocket handler. Per-stack compilation errors are now accumulated and listed against
  their stack, clearing when a corrected version compiles.
- **Disconnected empty state.** With nebula files defined but the server unreachable, the page claimed no
  environments were defined. It now shows *"Waiting to connect..."* with the names of the stacks it knows
  about from the schema.
- **Component start failures.** An invalid DDL or similar now shows its message in the tree tooltip, as a
  banner on the component detail panel, and as a red Failed state in the status bar — which previously showed
  *"Starting..."* forever.

Two connection bugs are fixed too. A dropped session (a network blip, or a non-text control frame) used to kill
the consumer coroutine permanently: the socket kept reconnecting and logging that it had connected, but nothing
consumed the new sessions, so schemas were never resubmitted. And rapid status transitions were being dropped —
the in-process HTTP component fires *Starting* and *Running* microseconds apart, and the sink was configured to
drop an update for every subscriber if any one of them was momentarily at zero demand, leaving consumers stuck
on "starting" forever.

Projects that reference `NEBULA_*` environment variables no longer flash a red config-errors banner at startup
while the local stack is still coming up. Orbital distinguishes "Nebula starting" from a real config failure
and shows a quiet *"Starting local environment…"* instead, escalating to a proper error if the connection drops.

### Query graph debugger
The old Angular `graph-vis` tool is replaced by a React Flow rewrite under `tools/graph-debugger`. Paste the
graph the engine logs while planning a query and explore it interactively to work out why a query isn't
resolving.

Large graphs used to lock the canvas, so it now starts empty: add a single type, expand outward from any node
via its off-canvas-neighbour badge, or drop the whole graph on at once for small ones. Auto-layout is
left-to-right hierarchical, matching the engine's traversal direction, and dragging a node pins it.

### Smaller UI fixes
- Picking a result in the search bar navigated twice — once to the item you chose, and again to the first item
  in the list.
- The schema diagram's markdown round-trip now serialises member keys as qualified names, so saved layouts
  resolve on re-render.
- Clicking a model node header in the schema diagram navigated to the literal path rather than the type's
  qualified name.
- Operation parameter type names link through to the catalog, matching the return type's behaviour.
- Query plans containing a Hazelcast node no longer throw.
- Loading buttons inside pop-ups are no longer obscured by incorrect styling.
- Alert action links (project errors, config errors, compilation errors) navigate to the correct workspace.
- A 404 for a type that doesn't exist no longer bounces you out to the workspace selector — only a genuine
  workspace-not-found does.

---

## Other fixes
- **HTTP header parameters were matched by position, not name.** `buildHttpHeaders` resolved each
  `@HttpHeader` parameter by its index in the supplied list, assuming that list matched the operation's
  declared parameter order. It doesn't — directly-provided parameters are appended after searched ones — so
  header values could silently be attached to the wrong header.
- **A JDBC driver returning null threw an NPE** rather than producing a null value.
- **Config loading could throw `ConcurrentModificationException`** because custom types were being registered
  on every load rather than once at class initialisation.
- **An invalid PropelAuth API key returned 500.** It now maps to 401 for a rejected token, 403 for a permitted
  token without access, and 503 when PropelAuth itself is unreachable.
- **A file watcher missed the contents of newly created directories.** `java.nio.WatchService` doesn't replay
  events from before a key is registered, so writing `orbital/config/services.conf` in one go meant the watcher
  saw only the `orbital/` directory and never the files inside it. Creating a directory now registers the watch
  and walks its existing contents.
- **Sankey lineage rows weren't stamped with the query's workspace**, so the lineage view in the query profile
  was empty for every workspace-scoped query.
- **History results for `PROVIDED` lineage nodes weren't showing.**
- **Trace events from published-query routes** were stamped with the default workspace rather than the one
  that ran the query.

---

## In case you missed it

Writing these notes turned up a gap in our own record-keeping. **0.36.0** (October 2025) and **0.38.0-M1**
(June 2026) both shipped without a release announcement, and a handful of point releases went out quietly too.
A fair amount of work reached you without us ever saying so.

None of this is new in 0.39 — it's all been in your hands for a while. It's here because you may not know
you have it.

### Custom Kotlin functions
_Available since 0.36.0_

You can extend Taxi's standard library with your own functions, written in Kotlin and loaded at runtime. Declare
the function in Taxi, implement it in Kotlin, and it's available in expressions, projections and queries like
any built-in.

This is the extension point to reach for when a transformation can't be expressed in Taxi itself — a
proprietary check-digit algorithm, a domain-specific parse, an internal encoding scheme.

It's documented at [custom functions](/docs/extending/custom-functions); it has simply never appeared in a
changelog, because 0.36.0 never got one.

It's also the mechanism the [validation framework](#data-validation) will use when user-written validation
rules land — the loader is already proven.

### Everything else

| Feature | Shipped in | Documented |
| --- | --- | --- |
| [Scheduled queries](/docs/querying/scheduled-queries) — run a query on a cron schedule | 0.36.0 | Yes |
| `in` / `not in` operators, pushed down to SQL and MongoDB | 0.36.0 | Yes |
| MongoDB native aggregations via `@MongoAggregate` | 0.36.0 | No |
| `map { }` iterates a collection rather than operating on it whole | 0.36.0 | No |
| Negate (`!`) and modulo operators | 0.36.0 | No |
| `env.conf` resolves environment variables inside annotations | 0.36.0 | Partly |
| Query-level Kafka offset override, by annotation | 0.36.0 | Yes |
| Custom error responses thrown from a `given { }` clause | 0.36.0 | No |
| Query error events persisted to the database | 0.36.0 | No |
| Roughly twenty stdlib functions — `append`, `average`, `joinToString`, `ifEmpty`, `orEmpty`, `size`, `isNullOrEmpty`, `indexOfItem`, `containsAll`, `containsAny`, `all`, `some`, `none`, `emptyInstance`, `startsWith`, `endsWith`, `matches`, `containsPattern`, `padStart`, `padEnd`, `applyFormat` | 0.36.0 | No |
| Logbook HTTP request/response capture | 0.36.0 | No |
| File-system monitors filter irrelevant files, stopping continuous polling loops | 0.36.7 | n/a |
| WebSocket transport for schema publication | 0.37.0-M4 | Partly |
| [Schema-publisher Spring Boot starter](/docs/sdks/java-and-kotlin/spring-boot-schema-publisher) | 0.37.0-M4 | Yes |
| SOAP as a loadable language via `additionalSources` | 0.38.0-M1 | Yes |
| [XSD as a source format](/docs/guides/working-with-xml) | 0.38.0-M1 | Yes |
| [JDBC batching](/docs/describing-data-sources/databases#batching-writes) — `batchSize` / `batchDuration` on database writes | 0.38.0-M1 | Yes |
| Operations usable as callable expressions | 0.38.0-M1 | No |
| [Multiple authentication tokens per service](/docs/describing-data-sources/authentication-to-services) | 0.38.0-M1 | Yes |
| Environment-specific config files (`auth.conf` overridden per environment) | 0.38.0-M1 | Partly |
| New query-plan diagram | 0.38.0-M1 | No |
| Tabbed catalog layout, and README as the first tab in the project explorer | 0.38.0-M1 | n/a |
| Endpoint for submitting parameterized query objects | 0.38.0-M1 | No |
| JSON message body accepted on the SSE streaming endpoint | 0.38.0-M1 | No |
| Git configuration changes clean up stale clones | 0.38.0-M1 | n/a |
| `schema-management` relicensed to Apache 2.0 | 0.38.0-M1 | n/a |

The undocumented rows are on our list. If one of them is something you've wanted, tell us and we'll bump it.

---

## Upgrading

This is a milestone build. Try it somewhere that isn't production first.

1. Read [Breaking changes](#breaking-changes) — particularly the license requirement and the
   `workspace-mode` toggle rename.
2. Back up your query history database. The Flyway migration takes table locks to enforce the new non-null
   columns.
3. If you use H2 as a data source, migrate off it before upgrading.
4. If you rely on the standalone Jet pipeline server, stay on 0.38.x. Streaming queries are unaffected.
5. Start Orbital and sign in. Your free license downloads automatically.

Workspaces are opt-in. Leave `workspace-mode` at `None` and Orbital behaves as it always has, with a single
workspace and legacy routes intact — you can adopt workspaces when you're ready.