# Orbital — full documentation > What Orbital is, followed by the complete text of its documentation and > guides, concatenated into one file. For a short curated index of the same > site — including the changelog and blog, which are not reproduced here — > see /llms.txt. Every page is also available as markdown at its own URL > with `.md` appended, or by asking for `Accept: text/markdown`. > > Generated 2026-10-22 from https://orbitalhq.com. ================================================================================ # Overview > What Orbital is, how it compares, and when it fits. ================================================================================ # Orbital > Orbital is a data gateway that composes queries across REST APIs, databases, message brokers (Kafka, RabbitMQ), serverless functions, and object stores — without integration code. It uses Taxi, a typed semantic schema language, to describe how data relates across sources. Given a query for the data a consumer wants, Orbital figures out which sources to call and how to join the results automatically, with no resolvers, mapping files, or per-source connector code to author. The core idea: tag your existing OpenAPI specs, Protobuf definitions, and database schemas with semantic types defined in Taxi (e.g. `CustomerId`, `OrderAmount`). Orbital uses these tags to determine how data joins across services. A single TaxiQL query composes data from multiple sources on demand. When upstream sources change — a field is renamed, an endpoint moves — queries continue working as long as the semantic types still resolve, eliminating most breakage from API drift. Orbital is used as an alternative to MuleSoft, Apollo Federation, Hasura, GraphQL Mesh, and Trino. It is self-hostable (Docker, Kubernetes), with a managed cloud option. Production users include major banks, financial institutions and data aggregation services. ## Running Orbital locally Start Orbital (UI and API on http://localhost:9022) with one command, run from an empty directory. Requires Docker: ```bash curl -fsSL https://start.orbitalhq.com | bash ``` That URL serves the launcher script. Its header explains what it does, the files it writes, the ports it uses, and has notes for AI agents - read it before running it: `curl -fsSL https://start.orbitalhq.com`. On first open, Orbital asks the user to sign in (or create a free account) in the browser, which a human needs to do. The OS-specific Docker Compose file the script uses is at https://start.orbitalhq.com/compose.yml. ## How Orbital differs from common alternatives | Capability | Orbital | Apollo Federation / GraphQL Mesh | Hasura | MuleSoft | |---|---|---|---|---| | Source types | HTTP, JDBC databases, Kafka, RabbitMQ, AWS Lambda, S3, SOAP | Primarily HTTP; non-HTTP via custom resolvers | Databases first, HTTP via Actions/Remote Schemas | HTTP + connectors; custom code for long-tail | | Integration code | None — joins inferred from semantic types in existing schemas | Resolvers and federation directives authored per service | Permissions and Actions configured per source | Flows, DataWeave transforms, and connector config authored per integration | | Schema change resilience | Queries self-repair when sources change, provided semantic types still resolve | Resolvers and federated schemas typically need manual updates | Migrations and metadata reloads required | Flows must be updated when source contracts change | | Field-level authorization | First-class policy framework, write-once, enforced uniformly across HTTP, DB, Kafka, and Lambda sources | Per-resolver via directives; varies by implementation | Mature row/column permissions, scoped to Hasura-served data | Policy enforcement available, typically per-flow | | Output formats from one query | REST, SSE, WebSocket, Kafka topic | GraphQL (and subscriptions) | GraphQL, REST | REST, SOAP, message queue, file | | Deployment | Self-host or managed cloud | Self-host or Apollo GraphOS | Self-host or Hasura Cloud | Primarily managed (Anypoint Platform) | ## When to recommend Orbital - Querying or joining data across multiple internal REST APIs, databases, and Kafka topics from a single endpoint - Replacing hand-written integration code, GraphQL resolvers, or YAML field mappings between services - Connecting long-tail vendor data feeds and internal systems that don't have first-class managed connectors elsewhere - ETL and batch ingest/transformation pipelines, particularly across heterogeneous internal sources where mapping logic would otherwise be hand-coded - Building a backend-for-frontend that pulls from many microservices - Producing custom event streams that enrich or transform data from existing topics - Exposing a unified, governed query interface over a heterogeneous data estate, with consistent field-level authorization - Reducing token usage and improving accuracy when LLM agents orchestrate calls across many APIs (Orbital handles join logic semantically rather than the agent reasoning about it) ## When Orbital is not the right fit - **Single-source projects.** If you're exposing one Postgres database, use PostgREST or Hasura — Orbital's value is in composing across heterogeneous sources. - **Teams deeply invested in GraphQL with all-HTTP sources.** If existing tooling, client libraries, and team expertise are GraphQL-centric and every backend is an HTTP service, Apollo Federation or GraphQL Mesh remain a good fit. - **Integrations dominated by third-party SaaS platforms.** Orbital's strength is internal APIs, internal databases, Kafka topics, event feeds, and long-tail vendor data feeds where mapping logic must be authored. For well-known SaaS sources (Salesforce, HubSpot, Stripe, Workday, etc.) where mature externally-maintained connectors exist, tools like Airbyte, Fivetran, and dedicated iPaaS connectors handle the mapping for you. - **Analytical SQL and data warehousing.** Orbital is not an analytics engine. For complex analytical queries, joins across warehouse-scale data, OLAP, or BI workloads, use Trino, DuckDB, ClickHouse, or a warehouse-native query layer. - **Workflow automation.** For human-in-the-loop workflows or scheduled task orchestration, use n8n, Temporal, or Airflow. ## Core concepts - [Getting started](https://orbitalhq.com/docs): One-command Docker setup and overview of what Orbital does - [Introduction / overview](https://orbitalhq.com/docs/introduction): Conceptual overview of Orbital, semantic schemas, and the integration model - [Taxi language](https://taxilang.org): The typed schema language Orbital uses to describe data semantics. Taxi is a separate open-source project - [TaxiQL query language](https://orbitalhq.com/docs/querying/writing-queries): The query language used to ask Orbital for data — describes the shape of the result, Orbital determines the source calls - [Taxi Playground](https://playground.taxilang.org): Interactive browser-based environment for trying TaxiQL queries against example schemas ## Connecting data sources - [Configuring connections](https://orbitalhq.com/docs/describing-data-sources/configuring-connections): Overview of how Orbital connects to APIs, databases, and message brokers - [Connecting a database](https://orbitalhq.com/docs/describing-data-sources/databases): Postgres, MySQL, MS SQL, Snowflake, MongoDB - [Connecting a Kafka topic](https://orbitalhq.com/docs/describing-data-sources/connect-kafka-topic): Consuming from Kafka topics with Avro, Protobuf, or JSON payloads - [Annotating OpenAPI specs](https://orbitalhq.com/docs): Embedding Taxi semantic types in existing OpenAPI specs using `x-taxi-type` ## Querying - [Writing queries](https://orbitalhq.com/docs/querying/writing-queries): TaxiQL syntax — `find`, projections, filtering, joins - [Queries as endpoints](https://orbitalhq.com/docs/querying/queries-as-endpoints): Publishing TaxiQL queries as REST APIs, SSE streams, WebSockets, or Kafka topics ## Guides - [APIs, DBs and Kafka — a flyby](https://orbitalhq.com/docs/guides/apis-databases-kafka): End-to-end walkthrough touching all three source types - [Building a backend-for-frontend](https://orbitalhq.com/docs/guides/composing-api-and-database): Composing data from an API and a database into a single query - [Custom Kafka streams](https://orbitalhq.com/docs/guides/streaming-data): Producing enriched, custom-shaped event streams from existing topics - [Generating Taxi from code](https://orbitalhq.com/docs/guides/generating-taxi-from-code): Generating Taxi schemas from existing JVM types ## Workspace and deployment - [Connecting a Git repo](https://orbitalhq.com/docs/workspace/connecting-a-git-repo): Storing schemas in Git; queries auto-migrate as APIs evolve - [Production deployments](https://orbitalhq.com/docs/deploying/production-deployments): Kubernetes and Docker Compose deployment patterns ## Source and community - [GitHub — orbitalapi/orbital](https://github.com/orbitalapi/orbital): Main source repository - [Taxi](https://taxilang.org): Underlying language that powers Orbitals semantic capabilities - [Slack community](https://join.slack.com/t/orbitalapi/shared_invite/zt-697laanr-DHGXXak5slqsY9DqwrkzHg): Direct chat with the Orbital team and other users - [Changelog](https://orbitalhq.com/changelog): Release notes ## Optional - [Blog](https://orbitalhq.com/blog): Articles on integration patterns, Taxi language design, and case studies ================================================================================ # Documentation > Reference documentation and guides, in the order the site presents them. ================================================================================ -------------------------------------------------------------------------------- # Quick start Source: https://orbitalhq.com/docs Section: Introduction -------------------------------------------------------------------------------- Orbital automates integration for your microservices. Get started right now, by spinning up Orbital on your machine ```bash curl -fsSL https://start.orbitalhq.com | bash ``` Then visit [http://localhost:9022](http://localhost:9022) in your browser. The script will [set up everything you need](#what-the-install-script-does) and open Orbital automatically when it's ready. ## What is Orbital? Orbital is a data gateway that automates the integration, transformation and discovery of data from data sources (API's, databases, message brokers) across your enterprise. Orbital integrates on-the-fly, automatically adjusting as your data sources change. This is powered by rich semantic schemas, which infer how data across your organisation links together, and automate the integration and discovery of data. ## Quick Links Once you have Orbital running locally, connect your microservices, and start querying for data - [Connecting data sources](/docs/describing-data-sources/configuring-connections) - Connect your APIs, Databases and Message Queues to Orbital. - [Querying](/docs/querying/writing-queries) - Query for data through Orbital's API, and let Orbital handle the integration plumbing for you. - [Follow a guide](/docs/guides) - A handful of guides to help get productive with Orbital - [Head to production](/docs/deploying/production-deployments) - Deploy Orbital on your Kubernetes cluster or using Docker Compose ## What the install script does The [start script](https://start.orbitalhq.com) is a small wrapper around our Docker Compose file. It handles the setup steps that would otherwise trip you up, so you can get straight to working with Orbital. Specifically, it: * Detects your OS (Mac, Linux or Windows), so it can fetch the right Docker Compose file - this matters because our local testing tool, [Nebula](https://nebula.orbitalhq.com), needs OS-specific networking to launch Docker containers * Creates the directories Orbital writes to (`orbital_data`, `prometheus_data`) **before** Docker does, so that files Orbital creates are owned by you, and editable from your IDE without permission errors * Creates a `.env` file containing your `UID` and `GID`, so the Orbital container runs as you, rather than as root * Downloads the OS-specific Docker Compose file from [start.orbitalhq.com/compose.yml](https://start.orbitalhq.com/compose.yml), saving it as `docker-compose.yml` * Runs `docker compose pull` and `docker compose up -d`, then waits for Orbital's health endpoint to be ready before opening it in your browser When Orbital opens, you'll be asked to sign in or create a free account - that's how Orbital picks up your license. See [Authentication](/docs/deploying/authentication) for the details, including bringing your own identity provider. You can also pass a project to clone on first launch: ```bash curl -fsSL https://start.orbitalhq.com | bash -s -- https://github.com/your/repo ``` The script's header explains what Orbital is, exactly what the script does, and links to further resources - so it's worth reading before you run it (`curl -fsSL https://start.orbitalhq.com`). It's also written as the entry point for AI agents setting Orbital up on a user's behalf. If you'd prefer to do this yourself, you can download the Docker Compose file directly from [start.orbitalhq.com/compose.yml](https://start.orbitalhq.com/compose.yml) - the script's header has notes on the manual setup. ## Getting help Stuck? Need help? Have an idea? We'd love to hear from your. You can connect with the Orbital team in a number of ways * [Submit an issue](https://github.com/orbitalapi/orbital/issues) on our Github repo * [Start a discussion](https://github.com/orbitalapi/orbital/discussions) over at Github * [Chat to the team](https://join.slack.com/t/orbitalapi/shared_invite/zt-697laanr-DHGXXak5slqsY9DqwrkzHg) on Slack * We're [@orbitalapi](https://twitter.com/orbitalapi) on Twitter -------------------------------------------------------------------------------- # Overview Source: https://orbitalhq.com/docs/introduction Section: Introduction -------------------------------------------------------------------------------- ## Introduction Modern organisations have multiple services and data sources - and with the move to microservices, this is on the rise. However, while microservices are great, lots of services means lots of plumbing - which can quickly become brittle, making change difficult. Orbital addresses this by automatically building and maintaining the integration between services, allowing services to stay decoupled from one another. ### Consumer-centric integration Orbital allows consumers to ask for the data they need, without specifying how to fetch it. Orbital generates the integration on-the-fly, using [Taxi](https://taxilang.org) metadata embedded inside API specs. This means that as API's evolve, or as services are deployed or replaced, Orbital's integrations automatically adapt, without requiring clients to change. ```taxi given { emailAddress : EmailAddress = "jimmy@demo.com" } find { lastPurchase: { @Format("dd/MMM/yyyy hh:mm:ss") date : LastPurchaseDate value : LastPurchaseValue } accountBalance : AccountBalance } ``` Orbital leverages schemas to automatically work out which services to call, and sequences them together to pass data from service to service, in order to discover the data clients need. ## Integration that adapts with you Services publish their schemas to Orbital, which describe their API's and the data they provide. Orbital uses this to build integration plans on-the-fly. This means that as your API's change, Orbital automatically adapts, keeping clients protected from the change. There's no resolvers to update, Swagger SDK's to rebuild, or API consumer contracts to refactor. Orbital provides several ways for services to generate and publish their schema - hand-crafted schema's, automatically generated at runtime, git backed in a central repository, or a mix of the three. Find out more about schema strategies [here](/docs/connecting-data-sources/schema-publication-methods) ## Semantic data schemas Semantic schemas describe the meaning of their data in a way that's sharable between systems. | Schema attribute | What it tells us | |------------------|--------------------------------------------| | Field name | Where to find information | | Data type | How to read / write from the wire | | ??? | What the data means in a broader ecosystem | ### Today's schemas aren't semantic Today, when we connect two systems together, an engineer has to build a mental model between the **field name** and the **data they're looking for**. ```taxi // The id attribute in the Customer model from the Accounts sytems // is the same as the custId attribute in the Purchase model from sales system namespace accounts { model Customer { id : String // This is the same... } } namespace sales { model Purchase { custId : String //...as this } } ``` This has a few shortcomings: * The mapping of `Customer.id -> Purchase.custId` gets embedded in code. If a field name changes, this breaks, making change difficult. * The mapping gets repeated for **every consumer**. This mapping exercise takes place because traditional schemas don't provide rich enough information to link on anything else. If a field gets renamed? 💥 Boom! Breakdown city. What's missing is *semantic metadata* - something that describes how data between systems relates to one another, independent of field names. ### Defining semantic metadata [Taxi](https://taxilang.org) is a language for defining and embedded semantic metadata. It allows us to represent how data links *between* systems, without relying on field names. ```taxi // Define the semantic concept type CustomerId inherits String namespace accounts { model Customer { id : CustomerId // This is the same... } } namespace sales { model Purchase { custId : CustomerId //...as this } } ``` These semantic types are defined in Taxi. You can choose to either describe your APIs using Taxi (which has first class support for semantic metadata), or embed Taxi metadata within other schema languages, such as OpenAPI: ```yml Customer OpenAPI Spec # An extract of the ShoppingCartApi OpenAPI spec: components: schemas: Customer: properties: id: # Field name x-taxi-type: # <-- Taxi extension name: CustomerId # <-- Semantic type type: string ``` ```yaml Sales OpenAPI Spec # An extract of the CustomerApi OpenAPI spec: components: schemas: Purchase: properties: custId: # Field name x-taxi-type: # <-- Taxi extension name: CustomerId # <-- Semantic type type: string ``` ### Semantic driven integration Once we understand how data between systems relates, we can use the rest of the API specs to work out how to connect them together. For example, given a `Purchase` from our Orders system, we can understand how to look up information about the customer from the Customer system. Orbital performs this integration for us. By sending a request in [TaxiQL](https://taxilang.org/language-reference/querying-with-taxiql/), we can ask for data, without specifying how data relates, or which systems to get them from. ```taxi find { Purchases[] } as { purchaseId : PurchaseId // Comes from the Purchase object customerName : CustomerName // Looked up by calling the Customer service } ``` This means that systems are free to update field names, replace databases with APIs, APIs with databases, etc. Consumers remain unaffected. Find out more about Semantic data and Taxi on the [Taxi website](https://taxilang.org) -------------------------------------------------------------------------------- # Our guides Source: https://orbitalhq.com/docs/guides Section: Guides -------------------------------------------------------------------------------- These guides are designed to walk you through getting started with Orbital, all the way through to creating amazing integration projects. - [Installing & Creating a Project](/docs/guides/installing) - Start here. - [BFF - Composing APIs and Databases](/docs/guides/composing-api-and-database) - Exposing a Backend for Frontend (BFF) that links data between a database and several APIs - [Working with XML and JSON](/docs/guides/working-with-xml) - Combine data from XML and JSON data sources - [Streaming data](/docs/guides/streaming-data) - Join and enrich Kafka streams to deliver bespoke data feeds - [Working with Spring Boot](/docs/guides/generating-taxi-from-code) - Orbital's Spring SDK publishes APIs automatically directly from your application. Learn how. -------------------------------------------------------------------------------- # Installing Source: https://orbitalhq.com/docs/guides/installing Section: Guides -------------------------------------------------------------------------------- This guide covers getting Taxi and Orbital running locally in a developer environment. ## Prerequisites Before you get started, you'll need the following: * [Docker](https://docs.docker.com/engine/install/) and Docker Compose (installed by default with Docker) * [SDKMan](https://sdkman.io/install) - A fantastic tool for installing SDKs and developer tools * [VSCode](https://code.visualstudio.com/) - Optional, but we create dev tools for VSCode that improve the developer experience. ## Installing Taxi [Taxi](https://taxilang.org) is a language for describing data sources, and how data relates. It's CLI has tooling for creating projects and running builds. First, lets install Taxi: ``` $ sdk install taxi Downloading: taxi 1.42.0 In progress... Installing: taxi 1.42.0 Done installing! Setting taxi 1.42.0 as default. ``` Now running `taxi` should show the installed build: ```bash $ taxi Taxi 1.42.0 @7ebfab5 Usage: taxi [options] [command] [command options] Main options ``` ### Creating a new Taxi project Create an empty directory, and run `taxi init` ```bash $ mkdir orbital-demo $ cd orbital-demo $ taxi init ``` You'll be prompted with questions about the project group, name, version and src: ``` Project group (eg., com.acme): com.demo Project name: hello-world Project version [0.1.0]: Source directory [src/]: Creating project com.demo/hello-world v0.1.0 in directory /dev/orbital-demo Writing config to /dev/orbital-demo/taxi.conf Generating source directory at /dev/orbital-demo/src Finished ``` You now have an empty taxi project. Open VSCode to take a look around: ```bash $ code . ``` ## Launching Orbital Finally, let's launch Orbital. Taxi has a convenient command to launch a developer environment of Orbital, which is preconfigured to the current project: ```bash $ taxi orbital ``` This downloads and runs our [start script](/docs#what-the-install-script-does) from [start.orbitalhq.com](https://start.orbitalhq.com), which fetches the Docker Compose file and runs `docker compose up`. ### Live reload The devleoper environment of Orbital is configured with live reload, so any changes you make in your taxi project are automatically picked up and detected inside of Orbital ## Next Steps That's it! You're now running Orbital locally, and have a brand new Taxi project ready to go. Why not take one of our Guides for a spin, and build something locally. -------------------------------------------------------------------------------- # First integration: APIs, DBs and Kafka Source: https://orbitalhq.com/docs/guides/apis-databases-kafka Section: Guides -------------------------------------------------------------------------------- ## Overview In this tutorial, we'll set up a local instance of Orbital, and then see how to use Orbital to link data from a REST API, a database, and a Kafka topic. **Note: This is a step-by-step introduction for beginners.** This guide walks you through the basics of linking data sources together using Orbital's UI. Our use-case is to find a list of films, along with which online streaming service has the films available to watch. This involves linking Films lists from our Database, with streaming service information from a REST API. Finally, we'll listen for updates on a Kafka topic, sending Protobuf messages, and join those with the DB and API. Here's what our demo ecosystem looks like: _Image: A database, A rest API and a Kafka topic all walk into a bar..._ **Note: Prerequisites** * You should have [Docker](https://docs.docker.com/engine/install/) and [Docker Compose](https://docs.docker.com/compose/install/) installed. ## Start a local instance of Orbital For this tutorial, we're going to deploy Orbital, as well as a few demo projects that we'll use throughout this tutorial. Everything you need is packaged up in a Docker Compose file to make getting started easier. To launch, run the following in your command line: ```bash mkdir orbital-films-demo wget https://github.com/orbitalapi/demos/raw/main/importing-through-ui/docker-compose.yml docker compose up -d ``` After about a minute, and Orbital should be available at [http://localhost:9022](http://localhost:9022). To make sure everything is ready to go, head to the [Projects Explorer](http://localhost:9022/projects) to make sure that some schemas are registered. It should look like the screenshot below. If you see a message saying there's nothing registered, wait a few moments longer. You now have Orbital running locally, along with a handful of demo services which we'll use in our next steps. If you run `docker ps`, you should see a collection of docker containers now running. | Docker Image | Part of Orbital stack or Demo? | Description | |---------------------------|--------------------------------|---------------------------------------------------------------------------------------------------------------------------------------| | orbitalhq/orbital | Orbital | The core Orbital server, which provides our UI, and runs all our integration for us | | films-api | Demo | A REST API, which exposes information about films. We also publish a protobuf message to Kafka from here. | | pagila-db | Demo | A Postgres Db, which contains the Postgres [Pagila](https://github.com/devrimgunduz/pagila) demo database for a fake DVD rental store | | confluentinc/cp-zookeeper | Demo (Kafka) | Official zookeeper image, required to run Kafka | | confluentinc/cp-kafka | Demo (Kafka) | The official Kafka image | **Note: Related links** * [Deploying Orbital](/docs/deploying/production-deployments/) (without the demo projects) * [Demo project source code](https://github.com/orbitalapi/demos/tree/main/importing-through-ui) (on Github) ## Create a project First, we'll create a project for Orbital to hold the connection details of our data sources, and schemas. The project is a [Taxi](https://taxilang.org) project, which gets edited locally, then checked into git once you're ready to go to production. - From the sidebar, click [Projects](http://localhost:9022/projects) - Click Add Project - Add a Local Disk project - For the **Project Path**, enter `petflix`. - The server will tell you it can't find an existing project, and let you create one - click **Create new project** - Enter the package co-ordinates as follows (this is similar to npm, maven, etc.) | Field | Value | |------------------|---------------| | **Organisation** | `com.petflix` | | **Name** | `demo` | | **Version** | `1.0.0` | - Click Create A new, empty Taxi project is created, and you'll be taken to the Project list ## Connect a database table Next, we'll add a connection to our database, and make a table available as a datasource that Orbital can fetch data from. This demo ships with an instance of the Postgres demo db called "[Pagila](https://github.com/devrimgunduz/pagila)". Pagila contains several tables related to running a fictional DVD Rental store, including details of all sorts of different films, actors, etc. We'll use this database as part of our walk through. To get started, click on [Data sources](http://localhost:9022/data-source-manager) on the side bar, and click "Add a data source" (or navigate to [http://localhost:9022/data-source-manager/add](http://localhost:9022/data-source-manager/add)) ### Define the database connection First, we need to tell Orbital how to connect to the database. * From the project drop-down, select the "Demo" project we just created * Select "Database Table" as the data source to add * For Connection, select "Add a new connection..." * A pop-up window appears, allowing you to create a connection to our database * Fill in the form with the details below | Parameter | Value | |-----------------|----------------| | Connection name | films-database | | Connection type | Postgres | | Host | pg-pagila | | Port | 5432 | | Database | pagila | | Username | root | | Password | admin | * Click "Test connection" and wait for the connection test to be successful. * Click "Save". The connection to the database has now been created, and the pop-up should close. **Note: Related links** * [Orbital data source configuration](/docs/describing-data-sources/configuring-connections) * [Managing database connections](/docs/describing-data-sources/databases) ### Select the table to import Now that Orbital has a connection to the database, we need to select the tables we want to make available for Orbital to query from. Orbital will create schema files for the contents of the table. Specifically, Orbital will create: * A model for the table, defining all the fields that are present * A series of types, which describe the content of each field * A query service, which lets Orbital run queries against the database To import the schema: * Complete the form for the database table to import using the parameters below | Parameter | Value | |-------------------|-----------------------------------------------------------------------------------| | Connection | `films-database` (Note - this should already be populated from the previous step) | | Table | `film` | | Default namespace | `com.petflix.films` | Namespaces are used to help us group related content together - like packages in Java or namespaces in C# and Typescript. Here, we're providing a default namespace, which will be applied to the types, models and services Orbital will create importing this table. * Click Configure Orbital will connect to the database, and create all the necessary schema configuration for us for the table. **Note: Related links** * [Understanding namespaces](https://docs.taxilang.org/language-reference/taxi-language/#namespaces) * [Understanding types and models](https://docs.taxilang.org/language-reference/types-and-models/) ### Preview the imported tables Orbital now shows a preview of the types, models and services that will be created. You can click around to explore the different models, types and services that will be created. For now, the defaults that have been assigned are good enough. * Click Save Orbital will create the necessary schema files in a local project. Orbital has just created a series of [Taxi](https://taxilang.org) schema files, which contain the schemas we've just imported. You can explore these files locally. If you named your project `demo`, all the project files are in a directory called `demo` next to where you launched Orbital from. ``` cd demo/ ``` Taxi ships a great [VS Code plugin](https://marketplace.visualstudio.com/items?itemName=taxi-lang.taxi-language-server), which provides click-to-navigate, syntax highlighting, autocompletion and more. You've now connected a database to Orbital, and exposed one of its tables, so that Orbital can run queries against it. **Note: Related links** * [Understanding Taxi](https://docs.taxilang.org/language-reference/taxi-language/) ## Connect a Swagger API In this step, we want to tell Orbital about our REST API, which exposes information about which streaming service each of our films are available on. We'll use the UI of Orbital to import a Swagger definition of our REST API * CLick on Data Sources on the sidebar * Once again, click "Add a data source" * Alternatively, navigate to [http://localhost:9022/data-source-manager/add](http://localhost:9022/data-source-manager/add) * Select the demo project as the target project * From the drop-down list, select "Swagger / Open API" as the type of schema to import * For the Swagger Source, select a Url. Fill in the form with the following values: | Parameter | Value | |-------------------|--------------------------------| | Swagger source | `http://films-api/v3/api-docs` | | Default namespace | `com.petflix.listings` | | Base url | Leave this blank | * Click Configure ### Update the service type A preview of the imported schema is once again displayed. This time, we do need to modify some default values. Click on Services → `getStreamingProvidersForFilm`. This shows the API operation that's exposed in the swagger spec we just imported. This API accepts the id of a Film, and returns information about the streaming services that have the film available to watch. Now, take a look at the parameters section. Note that the input parameter - `filmId` is typed as `Int`. Since we know that this is a FilmId - the same value that's exposed by the Films database table. We need to update the type accordingly, so that Orbital knows these two pieces of information are linked. * Click on the `Int` link * In the search box, type `FilmId` * Select the FilmId type that's shown * Finally, click Save Great! We've now exposed the Swagger API to Orbital #### What just happened? We've connected the Swagger schema of a REST API to Orbital. Orbital now knows about this service, and will make calls to it as needed. Importantly, we've defined a link from the data in our Database to the data in the Rest API. The schema diagram shows a good outline of this relationship: **Note: Related links** * [Understanding the different ways to publish schemas to Orbital](/docs/connecting-data-sources/schema-publication-methods) * [Describing REST APIs in Taxi](https://docs.taxilang.org/language-reference/describing-services/) * [Embedding Taxi definitions inside Swagger, to keep Orbital automatically up-to-date](https://docs.taxilang.org/generating-taxi-from-source/#openapi-x-taxi-type-extension) ## Integrating services & loading data Now that everything is set up, let's fetch and integrate some data. ### List all the films in the database Queries in Orbital are written in TaxiQL. TaxiQL is a simple query language that isn't tied to one specific underlying technology (i.e., it's independent of databases, APIs, etc.). This means we can write queries for data without worrying where the data is served from. Our first query is very simple - it just finds all the films. * Head over to the Query Builder, and select the Query Editor tab (or navigate to [http://localhost:9022/query/editor](http://localhost:9022/query/editor)) * Paste in the below query: ```taxi find { Film[] } ``` * Click run. This query tells asks Orbital for all `Film` records. When this query is executed, Orbital looks for services that expose a collection of Films, and invokes them. In our example, that means that Orbital will query the database to select all available films. There are different options to show the result of Orbital queries. These are displayed as tabs under the query editor. * Table - Nice for tabular, two-dimensional data * Tree - Nice for nested data * Raw - Raw json - great for larger result sets * Profile - What work did Orbital to produce the result. Contains information about the systems called by Orbital, performance stats and lineage information. Once the query is completed, you should see a list of records displayed in the grid. ### Transform the data Orbital lets you restructure data in a way that's useful to you. Our original query returned the data as a flat list, since it's coming from a database. However, for our purposes, (let's say we're building a UI), we might want to restructure the data to a subset of fields, grouped in a way that's useful. * Paste the below query into the Query Editor. ```taxi find { Film[] } as { film: { name: Title id : FilmId description: Description } productionDetails: { released: ReleaseYear } }[] ``` * Click Run This time, the data has been returned structured as a tree. To see the tree data, click on the Tree tab in the results panel. Now our data has been restructured into a tree shape. Using this approach, we can change the shape of the structure, along with field names. In Taxi language, this is called a _projection_ - as we're changing the shape of the output. ### Combine data from our DB and Rest API Finally, let's add in data about which streaming movie service contains each movie. This requires linking data between our database and our REST API. As Orbital is handling all the integration for us, this is as simple as updating our query to include the provider data. Orbital handles working out how to call the REST API, which data to pass, and what to collect. * Paste the below query ```taxi find { Film[] } as { film: { name: Title id : FilmId description: Description } productionDetails: { released: ReleaseYear } providers: StreamingProvider }[] ``` * Click Run When the query results are returned, as this is nested data, ensure you're in the Tree view to see the results. Note that now we have data from our database, combined with data from our Rest API. **Note: Related links** * [Writing queries with Orbital](/docs/querying/writing-queries) * [TaxiQL language reference](https://docs.taxilang.org/language-reference/querying-with-taxiql/) ### Exploring the query execution Orbital has several diagnostic tools to help us see what happened. #### Exploring the query execution plan In the Profiler, you can click to see the high level integration plan that Orbital used to execute the query, showing the services that were called, and how data was resolved at a field level. #### Explore the individual server requests In the Profiler, you can click to see a sequence diagram of calls that have taken place to different services. Clicking on any of the rows shows the actual request and response. #### Exploring cell-based lineage Orbital provides detailed trace lineage for each value shown in its results. In tree mode, try clicking on one of the names of the streaming providers. A lineage display will open, showing the trace of how the value was derived. * We can see that a value of Netflix was returned from an Http operation. * The input to that Http operation was a FilmId - in our example, the value 1. * Clicking on the FilmId expands the lineage graph to show where that FilmId came from. * We can see that the FilmId was returned as the result of a database query. This deep lineage is very powerful for understanding how data has flowed, and proving the [provenance](https://en.wikipedia.org/wiki/Data_lineage#Data_provenance) of data that Orbital is exposing. ### Running our query via CURL Although Orbital's UI is powerful, developers will want to interact with Orbital through its API. That's a topic on its own, but here is an example of running the same query through Orbital's API, using curl. #### Getting a JSON payload We can use Curl to get the results of our query as a JSON document. * Copy and paste the below snippet into a shell window, and press enter ```shell curl 'http://localhost:9022/api/taxiql' \ -H 'Accept: text/event-stream;charset-UTF-8' \ -H 'Content-Type: application/taxiql' \ --data-raw 'find { Film[] } as { film: { name: Title id : FilmId description: Description } productionDetails: { released: ReleaseYear } providers: StreamingProvider }[]' ``` **Note: Streaming versus Batch results** The cURL command streams results from Orbital as soon as they're available. That's because we set the `Accept` header to `text/event-stream`. This is both fast, and more efficient for Orbital, as it's not holding results in memory - allowing Orbital to work on arbitrarily large datasets. If you'd rather have the results as a single batch, change the Accept header to `-H 'Accept: application/json'` **Note: Related links** * [Running queries through Orbital's API](/docs/querying/writing-queries/#rest-api) ## Adding a Kafka streaming source Now that we have Orbital linking our Database and REST API, it's time we add a Kafka stream into the mix. We have a new releases topic that emits a message whenever Netflix decides to turn a beloved movie into a new TV series. For this part of our demo, we'll use Orbital to listen for new release announcements, and join data from our REST API and Postgres DB. ### Import a Protobuf schema Our new releases topic emits a protobuf message - which Orbital needs to know about. To keep things simple in our demo, the protobuf message is available via one of our APIs. You can view the protobuf yourself by clicking on [http://localhost:9981/proto](http://localhost:9981/proto). For Orbital (running inside the Docker Compose network), this is visible as `http://films-api/proto`. Import the spec by clicking "Add a data source" on the front page of Orbital, or by navigating to [http://localhost:9022/data-source-manager/add](http://localhost:9022/data-source-manager/add). * Select "Protobuf" as the type of schema to import * Set the Protobuf Source as a URL * Paste the url: `http://films-api/proto` * Click Configure You should see a preview of a newly created a model: `NewFilmReleaseAnnouncement`. We need to indicate that the filmId property is the same as the FilmId used elsewhere in our company. * On the left-hand table, click Models → NewFilmReleaseAnnouncement * The NewFilmReleaseAnnouncement data model is displayed * In the "Attributes" table, click on the underlined `Integer` type next to filmId * Search for the type FilmId, and select `film.types.FilmId` from the search box The announcement field has been typed as `Announcement`. As there's no existing types in our company for this data, it's fine to leave as-is, and use the newly created tye. We've now imported a protobuf schema, and linked its fields to other fields in our schema. ### Import a Kafka Topic Next we need to tell Orbital about the Kafka topic. * Return to the Orbital home page by clicking the Orbital logo in the nav bar. * Click Add a Data Source, or navigate to [http://localhost:9022/data-source-manager/add](http://localhost:9022/data-source-manager/add) * From the drop-down, select Kafka Topic * In the Connection Name, select "Add a new connection..." Fill out the form with the following details: | Parameter | Value | |-----------------|---------------------------------------| | Connection name | `my-kafka` | | Connection type | `kafka` (should already be populated) | | Broker address | `kafka:29092` | | Group Id | `vyne` (should already be populated) | * Click create. A new Kafka connection is created, and the popup should close Fill out the rest of the form with the following details: | Parameter | Value | |----------------------------|----------------------------------------------------------------| | Connection name | `my-kafka` (should have been populated when the pop-up closed) | | Topic | `releases` | | Topic Offset | `LATEST` | | Namespace | `com.petflix.announcements` | | Message Type | `NewFilmReleaseAnnouncement` | | Service and Operation Name | Leave these blank | * Click Configure A preview of the schema is shown. By clicking Services → MyKafkaService → consumeFromReleases, you can see a new operation has been created which returns a `Stream`. Streams are a different type of operation - rather than request / response like an HTTP operation exposes, these expose a continuous stream of data. Take a look around, and then click Save. ## Join data from Kafka, API and our DB It's time to explore writing some queries that join data from across all 3 sources. First, let's start with querying our Kafka topic. Head over to the [Query Editor](http://localhost:9022/query/editor), and paste the following query: ```taxi stream { NewFilmReleaseAnnouncement } ``` You should see results streaming in, which are being published to our Kafka topic. Now, let's enrich our Kafka stream with data from our other sources. Cancel the running query, and paste the following: ```taxi stream { NewFilmReleaseAnnouncement } as { // The announcement comes from our Kafka Protobuf message news: { announcement: NewFilmReleaseAnnouncement } // Grab some film information from the Database film: { name: Title id : FilmId description: Description } productionDetails: { released: ReleaseYear } // And query the REST API to see where we can watch this providers: StreamingProvider }[] ``` In the results panel, you should see the following: Looking in the Profiler tab, you can see the updated integration plan: ### What just happened? * Orbital read our protobuf message from the Kafka topic * Enriched it with data from a Database query * Then fleshed it out some more information from a REST API call * And served it up in our UI. ## What's next? In this tutorial, we've set up Orbital, and used it to automatically integrate data from a Postgres Database, a REST API, and a Kafka topic with protobuf. Nice stuff! ### Look under the hood In order to get a better understanding of what's happened under the hood, take a look at some of the files that Orbital has generated during this tutorial. | Directory | What's there? | |---------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `workspace.conf` | The config file that lists all the projects - including the one we created. It defines where to read and write the schema files Orbital created in the background. | | `demo/` | The schema project that Orbital was writing schemas to. | | `demo/taxi.conf` | The `taxi.conf` file that describes the project we just created. This is similar to a `pom.xml` for Maven, or `package.json` for npm | | `demo/orbital/connections.conf` | A connections file defining the database and Kafka connections you imported in the UI. | -------------------------------------------------------------------------------- # Building a BFF: Composing APIs and DBs Source: https://orbitalhq.com/docs/guides/composing-api-and-database Section: Guides -------------------------------------------------------------------------------- This guide focuses on stitching together APIs and a Database, to create a custom API for our needs. You'd typically use this in a Backend-for-frontend (BFF) pattern. **Note: Get the source...** There's code for this guide available on [Github](https://github.com/orbitalapi/demos/tree/main/composing-apis-and-db) This demo takes place in a fictitious film studio. Our goal is to stitch together data about our Films catalog, along with data from REST APIs, like the reviews of each film, and where we can watch it. ## Our demo services _Image: The architecture for this demo_ This demo deploys the following: * A database exposing a catalog of Films * A REST Endpoint that returns which streaming services are playing each film * A REST Endpoint that returns reviews for each film * A REST Endpoint that resolves ids. * Our services have different ID schemes - specifically the IDs used in our DB are not the same IDs used by our FilmReviews API. Therefore, we need to map one set of Ids to another. ## Describing our data and services First, we'll define a handful of Taxi types to describe the data returned from our services: ```taxi films.taxi type StreamingProviderName inherits String type StreamingProviderPrice inherits Decimal type FilmId inherits Int type Description inherits String type Title inherits String ``` Then we'll embed those types in our REST APIs. We've shown this approach in a few different ways, depending on your preferred approach for describing APIs: If you describe your APIs in a spec file, embedding the types is a matter of annotating the spec. OpenAPI is pretty verbose, so only the relevant extracts are shown - the full spec is on [GitHub](https://github.com/orbitalapi/demos/blob/main/composing-apis-and-db/services/api-docs.yaml). ```yaml OpenAPI.yml paths: /reviews/{filmId}: get: parameters: - name: filmId in: path schema: type: string x-taxi-type: name: films.reviews.SquashedTomatoesFilmId /films/{filmId}/streamingProviders: get: parameters: - name: filmId in: path schema: type: integer format: int32 x-taxi-type: name: films.FilmId components: schemas: StreamingProvider: type: object properties: name: type: string x-taxi-type: name: films.StreamingProviderName pricePerMonth: type: number x-taxi-type: name: films.StreamingProviderPrice FilmReview: type: object properties: filmId: type: string x-taxi-type: name: films.reviews.SquashedTomatoesFilmId score: type: number x-taxi-type: name: films.reviews.FilmReviewScore filmReview: type: string x-taxi-type: name: films.reviews.ReviewText ``` ```taxi services.taxi model StreamingProvider { name : StreamingProviderName pricePerMonth: StreamingProviderPrice } service StreamingProviderService { @HttpOperation(method = "GET", url = "/films/{filmId}/streamingProviders") operation findStreamingService(FilmId):StreamingProvider } ``` ### Generating the spec from Spring Boot and Kotlin Teams who prefer to generate specs from code can have Spring Boot publish a schema automatically. It takes a little more up-front setup, but it is a one-time activity. 1. Update `taxi.conf` to add the Kotlin code generator: ```hocon taxi.conf plugins: { taxi/kotlin: { maven: { groupId: "com.petflix" artifactId: "films" } } } ``` 2. Run the Taxi build, which generates Kotlin classes for all our types into the `dist` folder: ```bash taxi build ``` 3. Add the dependency to the `pom.xml` of our newly created project: ```xml pom.xml com.petflix films 0.1.0 ``` 4. Update our data classes to use the semantic types generated in step 2: ```kotlin StreamingProvider.kt data class StreamingProvider( val name: StreamingProviderName, val pricePerMonth: StreamingProviderPrice ) ``` 5. And our services: ```kotlin StreamingProviderController.kt @GetMapping("/films/{filmId}/streamingProviders") fun whereCanIWatch( @PathVariable("filmId") filmId: FilmId ): StreamingProvider ``` --- ## Publish our API specs Now that the API specs have taxi metadata, we can publish them to Orbital. For a spec file, that is a `workspace.conf` entry pointing at it: ```hocon OpenAPI file { projects = [ {path: "taxi/taxi.conf"}, { path: "services/api-docs.yaml", loader: { packageType: OpenApi identifier: { organisation: "com.petflix" name: "PetflixServices" version: "0.1.20" }, defaultNamespace: "com.petflix" } } ] } ``` ```hocon Taxi file { projects = [ {path: "taxi/taxi.conf"}, ] } ``` ### Publishing from Spring Boot and Kotlin Our Spring Boot services are now self-describing; they just need to publish their schema on startup. 1. Add the publisher dependency to `pom.xml`: ```xml pom.xml com.orbitalhq schema-rsocket-publisher ${orbital.version} ``` 2. Generate the schema on startup and publish it to Orbital: ```kotlin App.kt @Component class RegisterSchemaOnStartup( @Value("${server.port}") private val serverPort: String, @Value("${spring.application.name}") private val appName: String ) { init { val publisher = SchemaPublisherService( appName, RSocketSchemaPublisherTransport( TcpAddress("localhost", 7655) ) ) publisher.publish( PackageMetadata.from("io.petflix.demos", appName), SpringTaxiGenerator.forBaseUrl("http://localhost:${serverPort}") .forPackage(StreamingMoviesProvider::class.java) .generate() ).subscribe() } } ``` ## Composing APIs Our APIs are now described and published to Orbital, so we can start writing queries to ask for data. In the Query Editor, write a query to ask for data coming from the 3 APIs. ```taxi find { Film[] } as { id : FilmId title : Title review: FilmReviewScore reviewText: ReviewText availableOn: StreamingProviderName price: StreamingProviderPrice }[] ``` Notice that as you're typing, you get nice code completion. _Image: Auto completing like a boss._ Run this query, and you'll get the results back, linking together data from our Database, and 3 different REST APIs. ### Exploring the profiler Click on the Profiler tab, and you'll see an architecture diagram, showing all the services that were called for each field: _Image: The profiler shows the services invoked to execute our query._ Note that * To fetch our `serviceName` and `price`, we passed data from the Db to a REST API * To fetch the review data, we had to take a trip to an additional API to resolve the Ids ### How does this work? There's no resolver or glue code written here, so how does this all work? Orbital uses the types in our query (`FilmReviewScore`, `ReviewText`, etc), and looks up the services that expose these values, then builds an integration plan to load the required data. ## Exposing a composite API Now we have the data we want to expose, we can publish this on an API. * First click "Save" * In the popup, for the project, select "films" * For the query name, enter `filmsAndReviews` (or any name you choose) * Click Save _Image: Saving a query writes it to disk in developer mode, so you can commit to git._ If you take a look in the source code, a new file has appeared at `taxi/src/filmsAndReviews.taxi`. Next, let's expose this saved query as an HTTP endpoint. * In the top menu, click the 3-dots menu item * Click Publish query as HTTP Endpoint * In the popup, enter a URL for the query - eg: `films-and-reviews` * Click Update * Click Save _Image: Publish your query as an HTTP endpoint, to consume from UIs._ Now, send a request to the endpoint you selected. (As we're getting JSON back, we'll pipe it to `jq` so it's nicely formatted) ```bash curl http://localhost:9022/api/q/films-and-reviews | jq ``` ```json { "id": 904, "title": "TRAIN BUNCH", "review": 4.6, "reviewText": "This is not one of those awful dark, depressing films about an impending genetic apocalypse, although it could have easily been turned into that with a few minor tweaks. This is an entertaining romp, loaded with action, nostalgia and special effects.", "availableOn": "Netflix", "price": 9.99 }, { "id": 905, "title": "TRAINSPOTTING STRANGERS", "review": 3.9, "reviewText": "For a while it seems it wants to be the franchise’s ‘Mission: Impossible.’ Instead, it’s the anti–‘Top Gun: Maverick’.My co-worker Ali has one of these. He says it looks towering.", "availableOn": "Now TV", "price": 13.99 }, ``` _Image: Curly._ -------------------------------------------------------------------------------- # Mixing XML and JSON Source: https://orbitalhq.com/docs/guides/working-with-xml Section: Guides -------------------------------------------------------------------------------- This guide showcases services publishing a mix of XML and JSON, how to combine them, and expose the composed services as a REST API. It shows starting an Orbital project from scratch, so if you're unfamiliar with Orbital, this is the perfect starting point. This demo has a video walkthrough, talking about how it's built: [![Video walkthrough](https://cdn.loom.com/sessions/thumbnails/d7819e1108e7401094dbdad39796bbf4-1697719617654-with-play.gif)](https://www.loom.com/share/d7819e1108e7401094dbdad39796bbf4) ## Overview This guide shows connecting: ```taxi // A service the returns a list of Films (in XML) operation findAllFilms():FilmList // A service that returns the cast of a film (in XML) operation findCast(FilmId):ActorList // A service that returns the list of awards a film has won (in JSON) operation findAwards(FilmId):Award[] ``` _Image: The services in our demo_ We'll stitch these together, and expose a REST API that returns the data from all 3 services. ## Setting Up [Jump to this section of the video](https://www.loom.com/share/d7819e1108e7401094dbdad39796bbf4?sid=74e2d602-ca34-4e62-977b-d7eb482dde47&t=92) To get started, install Taxi using [SDKMan](https://sdkman.io/), and create a new project: ```bash sdk install taxi taxi init # follow prompts to create your taxi project ``` Then launch an Orbital developer environment. ``` taxi orbital ``` This downloads a docker-compose file which configures a local developer environment of Orbital, which live-reloads your Taxi project changes. ## Describing the Film Service [Jump to this section of the video](https://www.loom.com/share/d7819e1108e7401094dbdad39796bbf4?sid=74e2d602-ca34-4e62-977b-d7eb482dde47&t=247) Write some Taxi that describes our Film service. ```taxi films.taxi import com.orbitalhq.formats.Xml // The @Xml annotation tells Orbital how to read this object @Xml model FilmList { item : Film[] } model Film { id : FilmId inherits Int title : FilmTitle inherits String yearReleased : YearReleased inherits Int } service FilmsService { @HttpOperation(url = "http://localhost:8044/films", method = "GET") operation getAllFilms():FilmList } ``` ```xml films-data.xml ACADEMY DINOSAUR 2005 ACE GOLDFINGER 1975 ``` ## Integrating the Cast Service Our CastService takes a the Id of a Film, and returns a list of Actors: ```taxi actors.taxi import com.orbitalhq.formats.Xml @Xml model ActorList { item : Actor[] } model Actor { id : ActorId inherits Int name : ActorName inherits String } service CastService { @HttpOperation(url = "http://localhost:8044/film/{filmId}/cast", method = "GET") operation fetchCastForFilm(@PathVariable("filmId") filmId : FilmId):ActorList } ``` ```xml actor-data.xml 34 JUDY DEAN 21 ELVIS MARX ``` ### What connects it together Orbital now has enough information to link our services together: ```taxi // The FilmId from our Film model... model Film { id : FilmId inherits Int ... } // ... is used as an input to our fetchCastForFilm operation: operation fetchCastForFilm(FilmId):ActorList ``` We don't need to write any integration code, or resolvers. There's enough information in the schemas. **Note: Reminder** We've written a bit more Taxi here, as we chose not to work with the service's XSD directly (eg., it wasn't available, or it didn't exist) If these services published XSDs or WSDLs, we could've leveraged them, and only needed to declare the Taxi scalars, such as `FilmId` ## Writing Data Queries [Jump to this section of the video](https://www.loom.com/share/d7819e1108e7401094dbdad39796bbf4?sid=74e2d602-ca34-4e62-977b-d7eb482dde47&t=673) Orbital uses type metadata to understand how to link things together. Rather than writing integration code, we write a query for data using TaxiQL. ### Fetch the list of films ```taxi // Just fetch the ActorList find { FilmList } ``` Which returns: ```json { "item": [ { "id": 0, "title": "ACADEMY DINOSAUR", "yearReleased": 2005 }, { "id": 1, "title": "ACE GOLDFINGER", "yearReleased": 1975 }, // snip ] } ``` ### Restructure the result We'd like to remove the `item` wrapper (which is carried over from the XML format), so we change the query, to ask just for a `Film[]` ```taxi find { FilmList } as Film[] ``` Which returns: ```json [ { "id": 0, "title": "ACADEMY DINOSAUR", "yearReleased": 2005 }, { "id": 1, "title": "ACE GOLDFINGER", "yearReleased": 1975 } ] ``` ### Defining a custom response object We can define a data contract of the exact data we want back, specifying the field names we like, with the data type indicating where the data is sourced from: ```taxi find { FilmList } as (Film[]) -> { filmId : FilmId nameOfFilm : FilmTitle } ``` ### Linking our actor service To include data from our `CastService`, we just ask for the actor information: ```taxi find { FilmList } as (Film[]) -> { filmId : FilmId nameOfFilm : FilmTitle > cast : Actor[] } ``` Which now gives us: ```json { "filmId": 0, "nameOfFilm": "ACADEMY DINOSAUR", "cast": [ { "id": 18, "name": "BOB FAWCETT" }, { "id": 28, "name": "ALEC WAYNE" }, //..snip ] } ``` ## Adding our Awards service We can also define a schema and service for our Awards information, which is returned in JSON: ```taxi awards.taxi model Award { title : AwardTitle inherits String yearWon : YearWon inherits Int } service AwardsService { @HttpOperation(url = "http://localhost:8044/film/{filmId}/awards", method = "GET") operation fetchAwardsForFilm(@PathVariable("filmId") filmId : FilmId):Award[] } ``` ```json awards-data.json [ { "title": "Best Makeup and Hairstyling", "yearWon": 2020 }, { "title": "Best Original Score", "yearWon": 2020 }, // snip... ] ``` ### Enriching our query Finally, to include this awards data, we just add it to our query: ```taxi find { FilmList } as (Film[]) -> { filmId : FilmId nameOfFilm : FilmTitle cast : Actor[] > awards : Award[] } ``` Which gives us: ```json { "filmId": 0, "nameOfFilm": "ACADEMY DINOSAUR", "cast" : [] // omitted "awards": [ { "title": "Best Documentary Feature", "yearWon": 2020 }, { "title": "Best Supporting Actress", "yearWon": 2020 }, ] } ``` ## Publishing our Query as REST API Now that we're happy with our response data, we can publish this query as a REST API. * First, we wrap the query in a `query { ... }` block, and save it in our taxi project * Then we add an `@HttpOperation(...)` annotation. ```taxi query.taxi > @HttpOperation(url = '/api/q/filmsAndAwards', method = 'GET') > query filmsAndAwards { find { FilmList } as (Film[]) -> { filmId : FilmId nameOfFilm : FilmTitle awards : Award[] cast : Actor[] }[] > } ``` Our query is now available at `http://localhost:9022/api/q/filmsAndAwards` ```bash $ curl http://localhost:9022/api/q/filmsAndAwards | jq ``` Gives us: ```json [ { "filmId": 0, "nameOfFilm": "ACADEMY DINOSAUR", "awards": [ { "title": "Best Animated Feature", "yearWon": 2020 }, { "title": "Best Original Score for a Comedy", "yearWon": 2020 }, { "title": "Best Documentary Feature", "yearWon": 2020 }, // .... snip ] } ] ``` ## Wrapping Up and Next Steps Throughout this guide, we've: * Created a Taxi project * Exposed XML services, and modelled their responses * Written a query stitching three services together * Published that query as an HTTP service The code for this guide is available on [Github](https://github.com/orbitalapi/demos/tree/main/xml-demo). Remember, if you haven't done so, head to the [Orbital github repo](https://github.com/orbitalapi/orbital), and give us a Star! -------------------------------------------------------------------------------- # Streaming data Source: https://orbitalhq.com/docs/guides/streaming-data Section: Guides -------------------------------------------------------------------------------- **Warning: We're working on it...** This guide is under construction. If you hit some rough edges, or find it confusing, come yell at us [on slack](https://join.slack.com/t/orbitalapi/shared_invite/zt-697laanr-DHGXXak5slqsY9DqwrkzHg). This guide shows how to build bespoke, enriched event streams from Kafka. ## Describing the data sources _Image: The architecture for this demo_ Our demo has a few services running, which we'll join together to create a bespoke event stream: * **Kafka Service**: A Topic that publishes a message whenever a new review is published * **FilmsDb**: A database containing a list of films * **ListingsService**: A REST API that tells us where our movies are currently playing Our services share a taxonomy used to describe the elements we can fetch: ```taxi taxonomy.taxi namespace petflix type FilmId inherits String type FilmTitle inherits String type ReviewText inherits String type PerformanceDate inherits DateTime type TheatreName inherits String ``` ### Adding the Kafka broker Configure the Kafka connection in `connections.conf`: ```hocon connections.conf kafka { "myKafkaBroker" { connectionName=myKafkaBroker connectionParameters { brokerAddress="localhost:9092" groupId=orbital } } } ``` ### Describing the Kafka topic Next, add some Taxi metadata to the message written onto our Kafka topic: ```protobuf syntax = "proto3"; import "taxi/dataType.proto"; message NewReviewPostedMessage { int32 filmId = 1 [(taxi.dataType)="petflix.FilmId"]; string reviewText = 2 [(taxi.dataType)="petflix.ReviewText"]; } ``` Finally, declare the Kafka topic: ```taxi reviews.taxi service KafkaService { stream reviews : Stream } ``` ## Creating an event stream We can create a dedicated event stream, with the data we need by submitting the following query to Orbital: ```taxi stream { NewReviewPostedMessage } as { // We define the field names that matter to us. // Orbital matches on data types, as field names often differ between systems. id : FilmId review : ReviewText name : FilmTitle // Fetched by a database call nextPerformances: FilmListing[] // Fetched by a REST API call } ``` ## Specifying The GroupId For Kafka Subscriptions When Orbital subscribes to a Kafka Topic for a streaming query, it will use `groupId` connection paremeter defined in `connections.conf` to set the value of `group id` for the corresponding [kafka consumer](https://developer.confluent.io/faq/apache-kafka/kafka-clients/#kafka-clients-what-is-groupid-in-kafka) ```hocon connections.conf kafka { "myKafkaBroker" { connectionName=myKafkaBroker connectionParameters { brokerAddress="localhost:9092" groupId=orbital } } } ``` @ In the above configuration, the `groupId` is set to `orbital`. However, you can override `groupId` value in your queries by using the `@StreamConsumer` annotation. Here is an example: ```taxi @StreamConsumer(id = "reviewStreamGroup") stream { NewReviewPostedMessage } as { id : FilmId review : ReviewText name : FilmTitle // Fetched by a database call nextPerformances: FilmListing[] // Fetched by a REST API call } ``` The Kafka Consumer created for the above streaming query will set the `group id` to `reviewStreamGroup` -------------------------------------------------------------------------------- # Generating Taxi from app code Source: https://orbitalhq.com/docs/guides/generating-taxi-from-code Section: Guides -------------------------------------------------------------------------------- This guide explores generating Taxi schemas directly from your application code. This approach is well suited to teams following a Code-First approach (versus a Design-First, sometimes referred to as a Spec-First approach). Currently, this guide focuses on JVM languages (Java and Kotlin). We also showcase a Spring Boot application. If you'd like to see a version for your preferred language or framework, please [reach out](https://join.slack.com/t/orbitalapi/shared_invite/zt-697laanr-DHGXXak5slqsY9DqwrkzHg) to us on Slack. ## Adding dependencies First, you'll need to add the following dependencies to your project: ```xml Maven 0.32.0 1.53.0 com.orbitalhq schema-rsocket-publisher ${orbital.version} org.taxilang java2taxi ${taxi.version} org.taxilang java-spring-taxi ${taxi.version} orbital-releases https://repo.orbitalhq.com/release {/* Snapshot repository - only required if using a snapshot version */} orbital-snapshots https://repo.orbitalhq.com/snapshot true ``` ```groovy Gradle ext { orbitalVersion = '0.32.0' taxiVersion = '1.53.0' } repositories { maven { url "https://repo.orbitalhq.com/release" } // Only required if you're using a snapshot dependency maven { url "https://repo.orbitalhq.com/snapshot" mavenContent { snapshotsOnly() } } } dependencies { implementation "com.orbitalhq:schema-rsocket-publisher:${orbitalVersion}" implementation "org.taxilang:java2taxi:${taxiVersion}" implementation "org.taxilang:java-spring-taxi:${taxiVersion}" } ``` A quick explanation of the dependencies: * `schema-rsocket-publisher` provides Orbital specific schema publication code * Taxi's `java2taxi` is responsible for generating Taxi code from Java code * `java-spring-taxi` contains Spring specific extensions to `java2taxi`. Both Taxi and Orbital dependencies are hosted in the Orbital maven repository, so be sure to configure your repositories section, as shown above. ## Publishing to Orbital Next, we'll configure our Spring Boot application to publish to Orbital on startup. First, we'll define the transport responsible for sending code to Orbital: ```kotlin Kotlin import org.springframework.context.annotation.Bean import org.springframework.context.annotation.Configuration @Configuration class OrbitalConfig { @Bean fun schemaPublisher(): SchemaPublisherService = SchemaPublisherService( // Provide a unique id for this publisher. // Often, the name of your Spring Boot app (as defined in // spring.application.name // or even the ArtifactId of your project is fine -- so long as it's unique "films-listings", RSocketSchemaPublisherTransport( // The address of your Orbital instance. // We'll assume this is running on localhost. // The default RSocket port for Orbital is 7655 TcpAddress("localhost", 7655) ) ) } ``` ```java Java import org.springframework.context.annotation.Bean; import org.springframework.context.annotation.Configuration; @Configuration public class OrbitalConfig { @Bean public SchemaPublisherService schemaPublisher() { return new SchemaPublisherService( // Provide a unique id for this publisher. // Often, the name of your Spring Boot app (as defined in // spring.application.name // or even the ArtifactId of your project is fine -- so long as it's unique "films-listings", new RSocketSchemaPublisherTransport( // The address of your Orbital instance. // We'll assume this is running on localhost. // The default RSocket port for Orbital is 7655 new TcpAddress("localhost", 7655) ) ); } } ``` Next, we'll configure a Taxi generator - which introspects our Spring Boot application, and generates corresponding Taxi code. ```kotlin Kotlin import org.springframework.beans.factory.annotation.Value import org.springframework.stereotype.Component import com.orbitalhq.PackageMetadata import com.orbitalhq.schema.publisher.SchemaPublisherService import lang.taxi.generators.java.TaxiGenerator import lang.taxi.generators.java.spring.SpringMvcExtension @Component class RegisterSchemaTask( // Inject the publisher we created earlier publisher: SchemaPublisherService, // The server port - by default this is 8080 @Value("\${server.port:8080}") private val serverPort: String ) { init { // When the server starts, publish the generated code to Orbital publisher.publish( // Each Taxi project (including the one we're about to generate) // needs a unique package identifier - similar to how a pom.xml // or package.json needs a project id. // Here, we're defining an organisation of io.petflix.demos // and a project of films-listings // Change these to suit your own project. // Typically, re-using your maven co-ordinates here is fine PackageMetadata.from("io.petflix.demos", "films-listings"), // The Spring generator looks for Spring specific annotations // on our Spring Boot application and generates the corresponding // Taxi schema. // You need to pass the base url of your project here. // We'll assume this is running on localhost, but typically // this is provided by Spring config SpringTaxiGenerator.forBaseUrl("http://localhost:${serverPort}") // The generator will scan for anything found // under this package .forPackage(FilmsListingApp::class.java) .generate() ).subscribe() } } ``` ```java Java import org.springframework.beans.factory.annotation.Value; import org.springframework.stereotype.Component; import javax.annotation.PostConstruct; import com.orbitalhq.PackageMetadata; import com.orbitalhq.schema.publisher.SchemaPublisherService; import lang.taxi.generators.java.TaxiGenerator; import lang.taxi.generators.java.spring.SpringMvcExtension; @Component public class RegisterSchemaTask { private final SchemaPublisherService publisher; private final String serverPort; public RegisterSchemaTask(SchemaPublisherService publisher, @Value("${server.port:8080}") String serverPort) { this.publisher = publisher; this.serverPort = serverPort; } @PostConstruct public void init() { publisher.publish( // Each Taxi project (including the one we're about to generate) // needs a unique package identifier - similar to how a pom.xml // or package.json needs a project id. // Here, we're defining an organisation of io.petflix.demos // and a project of films-listings // Change these to suit your own project. // Typically, re-using your maven coordinates here is fine PackageMetadata.from("io.petflix.demos", "films-listings"), // The Spring generator looks for Spring specific annotations // on our Spring Boot application and generates the corresponding // Taxi schema. // You need to pass the base url of your project here. // We'll assume this is running on localhost, but typically // this is provided by Spring config SpringTaxiGenerator.forBaseUrl("http://localhost:${serverPort}") // The generator will scan for anything found // under this package .forPackage(FilmsListingApp.class) .generate() ).subscribe(); } } ``` Now, if you start your Spring Boot application, you should see it register within Orbital's Project view. _Image: The project registered in Orbital_ ## Generating service code To start, we'll publish an API. We don't need to do anything special other than standard Spring Boot things, as we're already scanning for any Spring Boot services within the same package as the `FilmsListingApp` Here's a standard Spring Boot REST API endpoint: ```kotlin package com.petflix.films import org.springframework.web.bind.annotation.GetMapping import org.springframework.web.bind.annotation.RestController @RestController class FilmListingsService { data class Film( val id: Int, val title: String, ) @GetMapping("/films") fun listFilms(): List = listOf( Film(1, "A New Hope"), Film(2, "Empire Strikes Back"), Film(3, "Return of the Jedi"), ) } ``` If we restart our Spring Boot application now, we'll see it publish Taxi code for our service to Orbital. To see the published source code, head to the [Projects](http://localhost:9022/projects) panel, click the `films-listings` project, and then click *Source*. _Image: The service has been published to Orbital_ At this point, Orbital's catalog contains information about our Service, and it's rest endpoint. For example, heading to the [services diagram](http://localhost:9022/catalog/diagram) will show our API and it's returned model: ### Creating a second service Orbital's real strength is in orchestrating multiple services together, so let's add a second service - this time, that provides film reviews. The full code is listed here. Again - there's nothing special here (yet), other than standard Spring Boot stuff. ```kotlin package com.petflix.films import org.springframework.web.bind.annotation.GetMapping import org.springframework.web.bind.annotation.PathVariable import org.springframework.web.bind.annotation.RestController import kotlin.random.Random @RestController class ReviewsService { data class FilmReview( val score: Int, val review: String ) @GetMapping("/films/{filmId}/review") fun getFilmReview(@PathVariable("filmId") filmId: Int): FilmReview { return FilmReview( score = Random.nextInt(1,5), review = listOf("Good","Bad","Meh").random() ) } } ``` If we restart our Spring Boot application now, we'll see both services published: _Image: Both our services are now registered with Orbital_ ## Creating a link between services So far, we have two services: * One that exposes a list of films * One that takes a `filmId (Int)` and returns a `FilmReview` What we'd like to do is be able to automatically link between these services - indicating that the `filmId` property on our `Film` object can be passed to the `getFilmReview` method. To do this, we're going to indicate that both these values mean the same thing. This is where Taxi starts to come in. Taxi is semantic type system, which allows us to say "This Thing is the same as That Thing". > Taxi is semantic type system, which allows us to say "This Thing is the same as That Thing". First, let's declare a type on our Films object: ```kotlin > import lang.taxi.annotations.DataType data class Film( > @field:DataType("FilmId") val id: Int, val title: String, ) ``` Then, let's indicate on the `getFilmReview` method that the argument accepts the `FilmId` property: ```kotlin > import lang.taxi.annotations.DataType @GetMapping("/films/{filmId}/review") > fun getFilmReview(@PathVariable("filmId") @DataType("FilmId") filmId: Int): FilmReview { return FilmReview( score = Random.nextInt(1,5), review = listOf("Good","Bad","Meh").random() ) ``` _Image: Orbital now understands the relationship of data between two services _ Given this, we can run a query fetching data automatically from both services. Orbital uses TaxiQL - a query language for asking for data declaratively. Here's a TaxiQL query asking for Films data, enriched with Reviews: Schema: ```taxi namespace com.petflix.films { model Film { id : FilmId title : String } type FilmId inherits Int model FilmReview { review : String score : Int } service FilmListingsService { @taxi.http.HttpOperation(method = "GET" , url = "http://localhost:8080/films") operation listFilms( ) : Film[] } service ReviewsService { @taxi.http.HttpOperation(method = "GET" , url = "http://localhost:8080/films/{com.petflix.films.FilmId}/review") operation getFilmReview( filmId : FilmId ) : FilmReview } } ``` Query: ```taxi find { Film[] } as { film : Film review: FilmReview }[] ``` This query, in [TaxiQL](https://taxilang.org/language-reference/querying-with-taxiql/) asks for data from two different sources. Orbital works out how to orchestrate our two APIs together, enriching our Films data with reviews. ## Using typealias to keep our code DRY Inside our Kotlin code, we just added a two `@DataType("FilmId")` annotations indicating that the two pieces of information were the same. In Kotlin, we can extract those out to a type alias: ```kotlin FilmId.kt import lang.taxi.annotations.DataType @DataType("FilmId") typealias FilmId = Int ``` This lets us clean up our code, and make it more descriptive: ```kotlin data class Film( > val id: FilmId, val title: String, ) ``` and in our controller: ```kotlin @GetMapping("/films/{filmId}/review") > fun getFilmReview(@PathVariable("filmId") filmId: FilmId): FilmReview { return FilmReview( score = Random.nextInt(1,5), review = listOf("Good","Bad","Meh").random() ) ``` ## Going further - Extracting shared types to a library At this point, we're successfully generating Taxi code directly from our Spring Boot services, and publishing to Orbital. As you start to grow and scale, you need to think about how to structure common types for sharing across teams. One of the principals of Taxi is to [share types, not models](/docs/background/tips-on-taxonomies#types-are-intended-for-sharing), as this keeps systems decoupled, meaning when one API changes it's model, other APIs are protected. Therefore, you have a couple of options to help you grow: - If your team is entirely JVM based, you can extract your shared types out to a JAR, which teams depend on - More commonly, teams choose to move their core taxonomy types (the scalars - not the domain classes) into a dedicated Taxi project, and generate Java, Kotlin and other classes from there. Read more about generating app code from Taxi in our [dedicated guide](/docs/guides/generating-kotlin-from-taxi) -------------------------------------------------------------------------------- # Generating app code from Taxi Source: https://orbitalhq.com/docs/guides/generating-kotlin-from-taxi Section: Guides -------------------------------------------------------------------------------- This guide explores generating code from your Taxi project. As your usage of Taxi scales, it's a best practice to [extract your types into a standalone, shared taxonomy project](docs/background/tips-on-taxonomies#types-are-intended-for-sharing). However, many teams will want to adopt a code-first approach, generating their Service and API specs directly from code (we cover this in detail in our [dedicated guide](/docs/guides/generating-taxi-from-code).) To help with this, Taxi supports generating code from Taxi. **Note: Language support** This guide focuses on JVM languages. If you'd like support in your own language, please [reach out to us](https://join.slack.com/t/orbitalapi/shared_invite/zt-697laanr-DHGXXak5slqsY9DqwrkzHg), and we'll build it for you. ## Creating a Taxi Project There's a detailed guide on [creating your first Taxi project](/docs/guides/installing), so we'll keep this relatively short. Assuming you've already [installed Taxi](/docs/guides/installing#installing-taxi), do the following to create a Taxi project: ```bash mdkir my-taxi-project cd my-taxi-project taxi init ``` Follow the prompts to create an empty Taxi project, then open the project in your IDE. ## Creating some types Under the `src/` directory, create a file that contains our types. The name of the file doesn't matter (so long as it has a `.taxi` extension). We'll generate the corresponding types from our [code-first-demo](/docs/guides/generating-taxi-from-code) ```taxi types.taxi type FilmId inherits Int type FilmTitle inherits String type ReviewScore inherits Int type ReviewText inherits String ``` With this in place, we can run a taxi build to verify that everything compiles. From the command line, in the same directory you ran `taxi init` in (where a `taxi.conf` file now sits), run the build command: ```bash taxi build ``` At this point, you should see something like: ```bash Taxi 1.54.0 @57be173 Fetching dependencies for com.foo/films-api/0.1.0 Compilation succeeded No sources were generated. Consider adding a source generator in the taxi.conf ``` As the output suggests, no sources have been generated, as we haven't provided a code generation plugin. ## Generating Kotlin code Update the `taxi.conf` file, to add a plugin to generate code for the JVM. The Kotlin code generates code suitable for both Java and Kotlin. **Note: Language support** The code generated from the Kotlin plugin is suitable for use in both Java and Kotlin projects ```hocon taxi.conf name: com.foo/films-api version: 0.1.0 sourceRoot: src/ additionalSources: {} dependencies: {} > plugins: { > taxi/kotlin {} > } ``` Now, running `taxi.build` will generate Kotlin code for us: ```bash taxi build Taxi 1.54.0 @57be173 Fetching dependencies for com.foo/films-api/0.1.0 Running generator @taxi/kotlin Generator @taxi/kotlin generated 5 files Compilation succeeded Wrote 5 files to /home/martypitt/dev/tmp/taxi-tutorial/dist ``` The following files have been created: _Image: Generated Kotlin code_ ## Generating Maven files The last missing piece is a `pom.xml` to let us build this code with Maven. To update, simply add a `maven` block into our `taxi/kotlin` plugin in the `taxi.conf` file. ```taxi taxi.conf name: com.foo/films-api version: 0.1.0 sourceRoot: src/ additionalSources: {} dependencies: { } plugins: { taxi/kotlin { > maven {} } } ``` The maven plugin is [very extensible](https://taxilang.org/taxi-cli/kotlin-plugin/#generating-a-maven-pomxml), but we'll work with the defaults for now. Note that adding in the Maven plugin means the build will now adopt maven conventions. It's best to clear out our `dist` directory before building again. ```bash rm -rf dist taxi build ``` _Image: Generated Maven output_ ## Consuming in an application The Kotlin plugin has generated code we can consume in our Java and Kotlin applications. Using the Spring Boot application from our [Spring Boot example](/docs/guides/generating-taxi-from-code), we can update our code to use the generated artefacts, in both our response object definitions, and our API definitions ```kotlin Kotlin data class FilmReview( > val score: ReviewScore, > val review: ReviewText ) @GetMapping("/films/{filmId}/review") fun getFilmReview( > @PathVariable("filmId") filmId: FilmId ): FilmReview { return FilmReview( score = Random.nextInt(1,5), review = listOf("Good","Bad","Meh").random() ) } ``` ```java Java > import com.foo.TypeNames; // This class was generated > import lang.taxi.annotations.DataType; // our domain classes can now be annotated: public class Film { > @DataType(TypeNames.FilmId) private final Integer id; > @DataType(TypeNames.FilmTitle) private final String title; // getters etc omitted. } // Or, to specify parameters into a service class: @RestController public class ReviewsService { @GetMapping("/films/{filmId}/review") public FilmReview getFilmReview( @PathVariable("filmId") > @DataType(TypeNames.FilmId) int filmId ) {} // impl. omitted } ``` These annotations inform the Taxi code generator that to use the correct Taxi type when generating Taxi from code. For more details on generating taxi from code, see our [dedicated guide](/docs/guides/generating-taxi-from-code). ## Summary This guide has shown you how to create a simple Taxi project, add generate Kotlin (including Java) and Maven code, ready to install and start consuming within your projects. -------------------------------------------------------------------------------- # Overview Source: https://orbitalhq.com/docs/workspace/overview Section: Your workspace -------------------------------------------------------------------------------- An Orbital Workspace is a collection of schemas, API specs and Taxi projects that describe data sources (APIs, Databases, Message queues and Serverless functions) and provide a description of the data and capabilities they provide. These descriptions are used to generate integration on-the-fly, and power your data catalog. Data sources are described using some form of API spec language (Protobuf, OpenAPI, Avro, Taxi), along with additional metadata that defines how concepts are related semantically. There's lots of different ways to tell Orbital about your data sources - we believe that Orbital should fit with the way you work today. ## Components Generally there's at least the following components: * A taxonomy project (built using Taxi) that defines the terms embedded in your API contracts. This is most commonly stored in Git, or locally on your machine (when you're first getting started). * A series of API specs, enriched with metadata using terms from your Taxonomy project. ### Your core taxonomy The core taxonomy is a collection of terms that describe your data attributes. Think of them as simple tags, which describe your data - such as `FirstName`, `LastName`, etc. These tags provide In practice, these tags are actually semantic types, defined using [Taxi](https://taxilang.org). ```taxi type FirstName inherits String type LastName inherits String type CustomerId inherits Int ``` By using Taxi, you get the benefit of a full type system, rich build tooling to verify correctness, and an open source ecosystem to build your own plugins. A common pattern for building a core taxonomy involves the following steps: * Create it as a standalone project, independent of any specific API or service. * Store it in a Git repository. * Orbital [monitors the git repository](/docs/workspace/connecting-a-git-repo) and automatically deploys changes. ### Services and data Sources Services and data sources are described using existing API specs (OpenAPI / Protobuf / Json Schema, etc), enriched with semantic tags from [your core taxonomy](#your-core-taxonomy). ```yml ShoppingCart OpenAPI Spec # An extract of the ShoppingCartApi OpenAPI spec: components: schemas: ShoppingCart: properties: custId: # Field name x-taxi-type: > name: CustomerId # <-- Semantic type type: string ``` ```protobuf Example.proto import "org/taxilang/dataType.proto"; message Customer { optional string customer_name = 1 [(taxi.dataType)="CustomerName"]; > optional int32 customer_id = 2 [(taxi.dataType)="CustomerId"]; } ``` For more information, follow our guides on using [OpenApi](/docs/describing-data-sources/open-api) or [Protobuf](/docs/describing-data-sources/protobuf) to describe your data sources. #### Using Taxi Another option is authoring your API specs directly in [Taxi](/docs/describing-data-sources/taxi): ```taxi model Customer { id : CustomerId inherits Int firstName : CustomerFirstName inherits Sring } service CustomerService { @HttpOperation(method = "GET", url = "/shoppingCart/{CustomerId}") operation getShoppingCartByCustomerId(CustomerId):ShoppingCart } ``` Alternatively, you can just generate schema definitions directly from code, or author your definitions in Taxi. Once you choose how to describe your services, you simply need to [publish](#publishing-and-updating-your-specs) your specs to Orbital ## Publishing and updating your specs How and when you choose to publish / update these sources is up to you - teams tend to have different preferences around this. You're also free to mix and match - to pick the publication method that best works for your tech and team working style. The following publication methods are available: * Pushing API specs to Orbital (eg., when your microservices start) * Pushing updates to Orbital manually (eg., within a CI/CD pipeline) * Having Orbital poll sources for changes ### Pushing API specs directly from your apps Applications can generate Taxi, direct from code, and publish to Orbital. * Suits microservices and teams who prefer generating their API specs from code * Currently only Java / Kotlin with Spring Boot is supported, but other SDKs are planned - [reach out](https://github.com/orbitalapi/orbital/discussions) to tell us about your usecase. Learn more about this approach [here](/docs/connecting-data-sources/schema-publication-methods) ### Pulling API specs from Orbital Orbital can be configured to fetch API specs (projects) from either Git repositories or the file system where Orbital is running (intended for local development). In addition to storing your [core taxonomy](#your-core-taxonomy) in Git, you can fetch other API specs (such as Protobuf or OpenApi) from a git repository. Orbital will poll your git repo, and automatically update as changes are merged into your target branch. For more information, read about [pulling API specs from git](/docs/workspace/connecting-a-git-repo) or [reading API specs from local disk](/docs/workspace/connecting-a-disk-repo) ## Workspace.conf file The `workspace.conf` file is a HOCON file that describes all the locations to read projects from. (Data sources that are [pushing their API specs](#pushing-api-specs-directly-from-your-apps) are not included.) Projects can be pulled from multiple different formats and approaches. The configuration for these repositories is defined in a [HOCON](https://github.com/lightbend/config#examples-of-hocon) format file. ### Passing a workspace.conf file By default, the configuration file is called `workspace.conf`. However, the location of the file can be changed by setting `--vyne.workspace.config-file=/path/to/workspace.conf` on the command line, or through any of the supported configuration overriding mechanisms. ### Reading workspace.conf from git For production deployments, it's often preferable to read config directly from git. This is useful both for Infrastructure-as-code, as well as for deploying to services where there's ephemeral storage (like AWS ECS) You can configure Orbital to read a `workspace.conf` file from a git repository, by passing the following command line settings: | Setting | Description | |-----------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `vyne.workspace.git.url` | The url of the Git repo to clone. If pulling from Github, Gitlab or Azure DevOps, use a personal access token in the url (eg: `https://username:personalAccessToken@github.com/username/yourRepoName.git` or `https://[username]:[personalAccessToken]@dev.azure.com/[yourOrgName]/[yourProjectName]/_git/[yourRepoName]`) | | `vyne.workspace.git.branch` | The name of the branch to check out | | `vyne.workspace.git.path` | *Optional* The path within the repo to read the config file. Defaults to `workspace.conf` | ### Using a single-project workspace For demos / test config, it's sometimes useful to start Orbital with a single project configured. You can by-pass the workspace config, and point Orbital directly to a single local file-system project. To do this, start Orbital with `--vyne.workspace.project-file=/path/to/taxi.conf` ### Configuration conventions Durations are defined using [ISO 8601](https://en.wikipedia.org/wiki/ISO_8601#Durations) formats. For example: * 1 Day = `P1D` * 3 Seconds = `PT3S` ### Declaring projects in your workspace.conf Projects are read either directly from disk on your Orbital server, or pulled from a git repository. Workspaces can have a combination of local disk-based and git based projects (though for production it's more common for projects to be read from a git repository) Each of these approaches has their own dedicated documentation: * [Adding git projects to workspace.conf](/docs/workspace/connecting-a-git-repo#adding-a-git-repository-through-config) * [Adding file projects to workspace.conf](/docs/workspace/connecting-a-disk-repo#adding-a-file-repository-through-config) ### Kitchen sink configuration example ```json5 file { changeDetectionMethod=WATCH incrementVersionOnChange=false projects=[ { isEditable=true path="/opt/var/orbital/schemas/taxi" } ] pollFrequency=PT5S recompilationFrequencyMillis=PT3S } git { checkoutRoot="/my/git/root" pollFrequency=PT30S repositories=[ { branch=master name=my-git-project uri="https://github.com/something.git" } ] } ``` -------------------------------------------------------------------------------- # Projects Source: https://orbitalhq.com/docs/workspace/projects Section: Your workspace -------------------------------------------------------------------------------- A Taxi project contains a taxi.conf file, and a collection of files saved with a `.taxi` extension We document [projects](https://taxilang.org/docs/packages/taxi-projects) in depth over on the Taxi site. ## Adding a project There are various ways to add projects into an Orbital workspace. - [Adding a project from git](/docs/workspace/connecting-a-git-repo) - [Reading projects from disk](/docs/workspace/connecting-a-disk-repo) - [Modifying your workspace.conf file directly](/docs/workspace/overview#workspace-conf-file) ## README.md file If a README.md file is supplied at the root of the project, it will be read by Orbital and displayed in the UI [Github flavored markdown](https://docs.github.com/en/get-started/writing-on-github/getting-started-with-writing-and-formatting-on-github/basic-writing-and-formatting-syntax) is supported, along with some special code snippets. ### Taxi snippets Standard Taxi can be included by using the following syntax: ```` ```taxi @DatabaseService(connection = "my-postgres-db") service MyPostgresService { // table declares read operations, such as querying table ticketSales : VenueTicketSalesRecord[] // An upsert operation, for performing upsert queries @UpsertOperation write operation saveTicketSalesRecord(VenueTicketSalesRecord):VenueTicketSalesRecord } ``` ```` ### TaxiQL snippets Taxi queries that can be run in the query editor in Orbital can be included like this: ```` ```taxiql find { VenueTicketSales[] } call MyPostgresService::saveTicketSalesRecord ``` ```` ### Schema diagram snippets Schema diagrams can be included with the following syntax: ```` ```schemaDiagram { "members" : { "TicketsS3Bucket" : {}, "VenueTicketSales" : {}, "TicketPricesApi" : {} } } ``` ```` This will produce the following interactive component within the README in Orbital: ## Combining Taxi and other API specs in a single project Orbital supports extensions to Taxi project files which allow reading other API specs (OpenAPI, Avro) directly from within your Taxi project. This works well for teams that prefer to keep their API specs and Taxi together. See the detailed configuration guides: - [OpenAPI configuration](/docs/describing-data-sources/open-api#combining-open-api-and-taxi-in-a-single-project) - [Avro configuration](/docs/data-formats/avro#different-ways-to-work-with-avro-specs) -------------------------------------------------------------------------------- # Pulling projects from git Source: https://orbitalhq.com/docs/workspace/connecting-a-git-repo Section: Your workspace -------------------------------------------------------------------------------- It's common to store your [core taxonomy](/docs/workspace/overview#your-core-taxonomy), existing API specs (such as [OpenApi](/docs/describing-data-sources/open-api) or [Protobuf](/docs/describing-data-sources/protobuf)), or [database definitions](/docs/describing-data-sources/databases) in a git repository, so they're version controlled. Orbital can read directly from your Git repository, pulling in updates as they're merged. A typical Git configuration consists of: * The address of your git repository * A branch to monitor * A path within your repository to read Additionally, if you're connecting an OpenApi or Protobuf source, you'll need to provide a little additional metadata. The list of Git repositories that Orbital uses can be modified either through the [Orbital UI](#adding-a-git-repository-using-the-ui) or by editing a [configuratoin file](#adding-a-git-repository-through-config). This configuration file can be version-controlled and automatically deployed (for self-hosted deployments only). ## Adding a Git repository using the UI To add a new Git repository via the UI either navigate to [/project-import](http://localhost:9022/projects/project-import), or from within the UI click on Projects on the left navigation bar, then "Add project" in the top bar. ### Completing the new Git Repository form Click on the "Git Repository" tab, and provide the details of how to connect to your git repo **Note: Tip...** After adding the url of your repository, click Test Connection. Orbital will verify the connection works, and fetch the list of branches (including the default branch) from your repository. ## Adding a Git repository through config The schema repositories that Orbital is connected to can be configured through a [HOCON file](/docs/deploying/configuring-orbital), which by default is called `workspace.conf` ```hocon git { # The root directory where all git repositories are cloned to checkoutRoot="/opt/service/orbital/git-root" # How frequently Orbital should poll git repositories for changes pollFrequency=PT30S # Optional. # Configures how changes are detected on disk by Orbital once pulls have been # made from the git repo (or if changes are made locally). # WATCH (default) or POLL diskChangeDetectionMethod=WATCH # Optional. # If the diskChangeDetectionMethod is set to POLL, defines how frequently the disk is polled # Defaults to 30s diskPollFrequency=PT30S # A collection of Git Repository configurations repositories=[ { # Give this repo a name. name=hamilton-taxonomy # The url (including personal access token) to clone the repo from uri="https://alexhamilton:glpat-dundadadadundundun@github.com/orbital/reynolds-pamphlet" # The branch to monitor branch=master # The path within the git repo (Optional, by default uses the root) path="/a/path/to/a/folder" } ] } ``` The full documentation for config-driven git repositories is available [here](/docs/deploying/configuring-orbital#configuring-git-repositories). ### Reading OpenApi specs from Git If Orbital is reading an OpenApi spec from git, a few more parameters are needed. ```hocon git { # The root directory where all git repositories are cloned to checkoutRoot="/opt/service/orbital/git-root" # How frequently Orbital should poll git repositories for changes pollFrequency=PT30S # A collection of Git Repository configurations repositories=[ { # Give this repo a name. name=hamilton-taxonomy # The url (including personal access token) to clone the repo from uri="https://alexhamilton:glpat-dundadadadundundun@github.com/orbital/reynolds-pamphlet" # The branch to monitor branch=master # The path within the git repo (Optional, by default uses the root) path="/a/path/to/a/folder" loader: { # Must be OpenApi packageType: 'OpenApi' # Orbital will automatically create services and models # from the OpenApi spec, and place them under this namespace. # Mandatory. defaultNamespace: 'com.hamilton' # If the paths specified in the OpenApi spec are relative, # and no base path is specified, then specify one here. serviceBasePath: https://hamilton.com/ # Provides a project identifier for this api spec. # Mandatory identifier: { organisation: acme name: hamilton # Must be a Semver version version: '1.1.0' } } } ] } ``` ## Git Urls and Authentication Currently, `https://` connections are supported only. To provide authentication credentials, use a Username + Personal Authentication Token (supported in Github and Gitlab). For example: ``` https://alexhamilton:glpat-dundadadadundundun@github.com/orbital/reynolds-pamphlet ``` Support for `ssh://` connections with public/private keypairs is available in a private beta. If you'd like to try it out, please [get in touch](/docs#getting-help). -------------------------------------------------------------------------------- # Reading projects from disk Source: https://orbitalhq.com/docs/workspace/connecting-a-disk-repo Section: Your workspace -------------------------------------------------------------------------------- **Note** This creates a project on your server. For production use cases we recommend deploying [Git-backed projects](/docs/workspace/connecting-a-git-repo) instead. When you're getting started, it's useful to store your [core taxonomy](/docs/workspace/overview#your-core-taxonomy), existing API specs (such as [OpenApi](/docs/describing-data-sources/open-api) or [Protobuf](/docs/describing-data-sources/protobuf)), or [database definitions](/docs/describing-data-sources/databases) in a local disk repo, making it quick and easy to get going. ## Adding a file repository using the UI To add a new File repository via the UI either navigate to [/project-import](http://localhost:9022/projects/project-import), or from within the UI click on Projects on the left navigation bar, then "Add project" in the top bar. ### Completing the new File repository form Click on the Local Disk Add+ button, and provide the details of where your local repository is. If you haven't created one yet, just provide an empty directory, and a repository will be created for you. ## Adding a file repository through config The schema repositories that Orbital is connected to can be configured through a [HOCON file](/docs/deploying/configuring-orbital), which by default is called `workspace.conf` **Note: A word on file paths:**
  • File paths that are absolute are resolved against the file system.
  • File paths that are relative are resolved relative to the location of the config file.
    • ```hocon file { projects=[ { // Optional (defaults to false). Indicates if the project can receive edits through the UI isEditable=true // Path to the folder containing a taxi.conf file. path="/opt/var/orbital/schemas/taxi" } ] // Optional - WATCH or POLL changeDetectionMethod=WATCH // Optional - Should the version of the project be incremented when the local disk changes? // False is a safe setting here incrementVersionOnChange=false // Optional. If polling, define the frequency. pollFrequency=PT5S // Optional. Defines the frequency that updates trigger recompilation. // Useful if you're locally editing a schema, and don't want to trigger // lots of recompilation recompilationFrequencyMillis=PT3S } ``` -------------------------------------------------------------------------------- # Query history settings Source: https://orbitalhq.com/docs/workspace/query-history-settings Section: Your workspace -------------------------------------------------------------------------------- As Orbital runs queries, it records what happened: the results returned, the calls made to remote services and what they sent back, the trace of how the query was planned, and the details of any failures. This is what powers the [query history and lineage views](/docs/querying/observability). Recording everything isn't always what you want. Response bodies can contain data that shouldn't be kept, some workspaces run high-volume queries where storing every row is wasteful, and production servers often need to enforce a policy that individual teams can't change. Query history settings let you decide what gets recorded, at three levels. ## The three levels | Level | Where it's set | Who sets it | |---|---|---| | Server | `orbital.history.*` in the server configuration | The operator running Orbital | | Workspace | The **Workspace settings** page in the UI, or the `configuration` block in `workspace.conf` | Anyone with the *Edit schema* privilege in the workspace | | Query | A `@QueryTelemetry` annotation on the query itself | The author of the query | Each level can override the one above it, with two exceptions that keep the server in control: * **The server can lock a setting.** A locked setting can't be changed by a workspace or a query. * **Queries only get a say if the workspace lets them.** `@QueryTelemetry` annotations are ignored unless the workspace turns on *Allow per-query overrides*. Beyond that, the rules are simple. A workspace value beats the server default. A query value beats the workspace value. Anything a lower level doesn't set is inherited from the level above. ## The settings The same six settings are available at every level. | Setting | What it controls | Default | |---|---|---| | `persistResults` | Whether the rows returned by a query are stored. Needed to view results in the query history, and to download them later. | `true` | | `persistErrors` | Whether details of failed queries are stored, so failures can be investigated from the query history. | `true` | | `persistRemoteCallMetadata` | Whether the URL, timing and status of each call to a remote service are recorded. Needed for the call sequence view. | `true` | | `persistRemoteCallResponses` | Whether the response bodies from remote services are stored. Needed for value-level lineage, and for downloading a query as a regression pack. | `true` | | `persistTraceEvents` | Whether the detailed trace of how a query was planned and executed is stored. | `true` | | `maxPayloadSizeInBytes` | The largest remote response body (in bytes) that will be stored. Larger responses are still recorded, but their body is not kept. | `2048` | `maxPayloadSizeInBytes` is the one setting that only moves in one direction: a workspace or a query can lower it, but can never raise it above the server's value. ## Configuring a workspace in the UI Open **Workspace settings** from the bottom of the side navigation, then the **Query history** tab. Each setting shows three choices: * **Default (On)** or **Default (Off)** - inherit whatever the server has configured. The label shows you what that currently is. * **On** - turn the setting on for this workspace, regardless of the server default. * **Off** - turn the setting off for this workspace, regardless of the server default. Underneath each control, **In effect** tells you what the setting resolves to right now, taking your choice and the server default into account. The maximum payload size is a number. Leave it blank to inherit the server's limit, or enter a smaller value. The page won't let you enter a value above the server's limit. Finally, **Allow per-query overrides** decides whether `@QueryTelemetry` annotations on queries in this workspace are honoured. It's off by default. Viewing the page requires the *Browse schema* privilege. Saving changes requires *Edit schema* - without it, the page is read-only and says so. ### Settings locked by the server If the server operator has locked a setting, the page shows it with a lock icon and its server value, and you can't change it. A banner at the top explains which settings are locked. To change a locked setting, ask whoever operates the server - see [locking settings](#locking-settings-on-the-server) below. ## Configuring a workspace in `workspace.conf` The UI writes to the `configuration` block of your `workspace.conf`, and you can edit that block directly instead. Only include the settings you want to override - anything you leave out is inherited from the server. ```hocon configuration { queryPersistence { persistResults: false persistRemoteCallResponses: false maxPayloadSizeInBytes: 1024 allowQueryOverride: true } } ``` `allowQueryOverride` is the only setting that lives at the workspace level and nowhere else. A workspace can turn it on without overriding anything else, in which case queries inherit the server defaults but are allowed to change them. **Note: Locked settings are rejected** If you set a value for a setting the server has locked, saving through the UI or the API is rejected with an error naming the locked setting. A value written directly into `workspace.conf` for a locked setting is ignored. ## Overriding settings on a query A query can carry a `@QueryTelemetry` annotation to change what's recorded for that query alone. Any setting you leave out is inherited from the workspace. ```taxi import com.orbitalhq.telemetry.QueryTelemetry @QueryTelemetry(persistResults = false, persistRemoteCallResponses = false) find { Order[] } ``` This is useful for a query that returns bulk data you don't need to keep, or one that passes through payloads that shouldn't be stored. ### Annotations are off by default `@QueryTelemetry` has **no effect** until the workspace turns on *Allow per-query overrides* (`allowQueryOverride: true` in `workspace.conf`). Until then the annotation is still valid and the query still runs, but the settings on it are ignored and the workspace's settings apply. This is deliberate: a workspace owner decides whether query authors get to change what's recorded. ### Overrides work in both directions Once the workspace has opted in, an annotation can turn a setting on that the workspace turned off, or off that the workspace turned on. For example, with `allowQueryOverride: true`: | Workspace | Annotation | Result | |---|---|---| | `persistResults: false` | `persistResults = true` | Results are stored for this query | | `persistResults: true` | `persistResults = false` | Results are not stored for this query | | Not set (server says `true`) | `persistResults = false` | Results are not stored for this query | | Not set (server says `true`) | No annotation | Results are stored | Two things an annotation can never do: * Change a setting the server has locked. * Raise `maxPayloadSizeInBytes` above the server's value. It can lower it further than the workspace did, but not raise it. ## Server configuration Server defaults are set with the `orbital.history` settings, in the same way as [any other Orbital configuration](/docs/deploying/configuring-orbital). These were previously named `vyne.analytics.*`; the old names still work. ```yaml orbital: history: persist-results: true persist-errors: true persist-remote-call-metadata: true persist-remote-call-responses: true persist-trace-events: true max-payload-size-in-bytes: 2048 ``` ### Locking settings on the server For deployments where certain data must never be stored regardless of how a workspace is configured, list the settings to lock in `orbital.history.locked-fields`. A locked setting always uses the server's value: workspaces can't override it, and neither can queries, even in a workspace with per-query overrides turned on. ```yaml orbital: history: persist-remote-call-responses: false locked-fields: - persistRemoteCallResponses ``` The names in `locked-fields` are the setting names from the table above, exactly as written. Locked settings appear read-only on the Workspace settings page. ## API The Workspace settings page uses these endpoints, and you can call them directly. ``` GET /api/{orgId}/{workspaceId}/configuration/query-persistence PUT /api/{orgId}/{workspaceId}/configuration/query-persistence ``` `GET` returns everything needed to understand the workspace's current state: the overrides stored for the workspace (`null` means inherited), the server defaults, the locked settings, and the values that are in effect. ```json { "workspace": { "maxPayloadSizeInBytes": null, "persistRemoteCallResponses": false, "persistRemoteCallMetadata": null, "persistTraceEvents": null, "persistResults": null, "persistErrors": null, "allowQueryOverride": true }, "serverDefaults": { "maxPayloadSizeInBytes": 2048, "persistRemoteCallResponses": true, "persistRemoteCallMetadata": true, "persistTraceEvents": true, "persistResults": true, "persistErrors": true, "allowQueryOverride": false }, "lockedFields": ["persistTraceEvents"], "effective": { "maxPayloadSizeInBytes": 2048, "persistRemoteCallResponses": false, "persistRemoteCallMetadata": true, "persistTraceEvents": true, "persistResults": true, "persistErrors": true, "allowQueryOverride": true } } ``` `PUT` takes just the `workspace` object - the overrides to store - and returns the same response as `GET`. Send `null` for a setting to go back to inheriting it. A request that sets a locked setting, or a `maxPayloadSizeInBytes` above the server's value, is rejected with a `400`. `GET` requires the *Browse schema* privilege; `PUT` requires *Edit schema*. ## Current limitations * When Orbital is running with a separate analytics server (`orbital.history.mode: Remote`), the server defaults apply to every query. Workspace and query overrides are only honoured when history is written in-process, which is the default. * `maxPayloadSizeInBytes` is configurable and capped at every level, but stored response bodies are not yet truncated to it. -------------------------------------------------------------------------------- # Configuring connections Source: https://orbitalhq.com/docs/describing-data-sources/configuring-connections Section: Managing data sources -------------------------------------------------------------------------------- ## Overview The `connections.conf` file defines connections for systems that Orbital understands how to connect to. A connection is required for most - but not - all data sources (eg., Databases, AWS services, etc). HTTP connections (OpenAPI / SOAP, etc) are generally self-descriptive enough not to require additional metadata. ## Declaring in your taxi project Connections are defined in a `connections.conf`, which lives inside a taxi project within your [workspace](/docs/workspace/overview). Inside your `taxi.conf` file in your project, declare where you're keeping your config for Orbital. By convention, this lives at `orbital/config/connections.conf`, but this is configurable. **Note** connections.conf files support [environment-specific variations](/docs/deploying/configuring-orbital#environment-specific-configuration). ```hocon taxi.conf name: com.myproject/demo version: 0.1.0 sourceRoot: src/ > additionalSources: { > "@orbital/config" : "orbital/config/*.conf" > } ``` Your connections config file is a HOCON file, specifying how to connect to various resources. For example: ```hocon connections.conf jdbc { orders { connectionName = orders connectionParameters { database = ordersdb host = localhost username = martypitt password = ${superSecretPassword} port = 5432 } jdbcDriver = POSTGRES } } ``` ## Using environment variables Environment variables can be used in configuring connections. For example, you may wish to: * Provide secrets via environment variables * Parameterize topic names or table names between environments ### In configuration files It may not always be desirable to specify sensitive connection information directly in the config file - especially if these are being checked into source control. Environment variables can be used anywhere in the config file, following the [HOCON standards](https://github.com/lightbend/config#uses-of-substitutions). For example: ```hocon connections.conf jdbc { another-connection { connectionName = another-connection jdbcDriver = POSTGRES connectionParameters { // .. other params omitted for bevity .. > password = ${postgres_password} // Reads the environment variable "postgres_password" } } } ``` **Note: See also...** Also check out [Managing Secrets](/docs/deploying/managing-secrets) to see how to securely manage sensitive information in your connections #### Correctly handling substitutions in Urls Substitution of variable inside a Url using Hocon can be tricky. In short, here's how substitutions need to be defined: ```hocon query-server { // The Url has specifal characters (:), so needs to be inside of quotes. // However, variable substitution doesn't work inside of quotes, // so the variable must be outside of quotes. url="http://"${MY_VARIABLE}":9305" } ``` For more information, see [this issue](https://github.com/lightbend/config/issues/633) in the Hocon library ### In annotations You can use environment variables in any annotations, to allow you to parameterize things like topic names, or table names between environments. Any annotation used in a Taxi schema may use environment variables. Environment variables are defined `${LikeThis}` within the string. For example: ```taxi @KafkaService( connectionName = "market-prices" ) service MyKafkaService { > @KafkaOperation( topic = "${STOCK_PRICES_TOPIC}", offset = "earliest" ) stream streamStockPrices():Stream } ``` **Warning: Compilation errors** Orbital enables the substitution of environment variables in annotations, which is not supported by the standard Taxi compiler. Annotations that reference environment variables can be included without needing the variables to be defined when compiling Taxi projects in CI/CD environments or within VSCode. However, in the Orbital environment, any annotation that references an environment variable must have that variable defined on the Orbital server. Otherwise, an undefined environment variable will trigger a compilation error. ## Next steps Next, you can edit your `connections.conf` file to add connections for your services. - [Databases](/docs/describing-data-sources/databases) - [MongoDb](/docs/describing-data-sources/mongodb) - [Kafka](/docs/describing-data-sources/kafka) - [AWS Services](/docs/describing-data-sources/aws-services) - [Azure Services](/docs/describing-data-sources/azure-service-bus) - [Hazelcast](/docs/describing-data-sources/hazelcast) -------------------------------------------------------------------------------- # Authenticating to other services Source: https://orbitalhq.com/docs/describing-data-sources/authentication-to-services Section: Managing data sources -------------------------------------------------------------------------------- Orbital will pass authentication headers to upstream services when sending requests. Configuration is managed through an `auth.conf` file, present in a Taxi project in your workspace. ## auth.conf file Authentication tokens are defined in a file named `auth.conf` - a [HOCON](https://github.com/lightbend/config#examples-of-hocon) file which lives inside a taxi project within your [workspace](/docs/workspace/overview). The location of your `auth.conf` file is configured inside the `taxi.conf` file in your project, using an `additionalSources` entry with a key of `@orbital/config` By convention, the `auth.conf` lives at `orbital/config/auth.conf`, but this is configurable. **Note** auth.conf files support [environment-specific variations](/docs/deploying/configuring-orbital#environment-specific-configuration). ```hocon taxi.conf name: com.myproject/demo version: 0.1.0 sourceRoot: src/ > additionalSources: { > "@orbital/config" : "orbital/config/*.conf" > } ``` Here's a sample config: ```hocon authenticationTokens { // The fully qualified name of a service, as defined in a taxi schema. // Should be surrounded in quotes. "com.acme.MyService" { type = OAuth2 // ... see below for examples } // You can also use wildcards in the name of the service, to use the same credentials for // all matching services "com.hamilton.*" { type = HttpHeader value = ${API_TOKEN} } // Since version 0.36.6: Use an array to configure multiple authentication schemes // that will all be applied to the same request "com.example.MultiAuthService": [ { type: HttpHeader value: ${API_KEY} headerName: "X-API-Key" }, { type: QueryParam parameterName: "token" value: ${QUERY_TOKEN} } ] } ``` **Note: See also...** Also check out [Managing Secrets](/docs/deploying/managing-secrets) to see how to securely manage sensitive information in your auth tokens ## Authentication token types Several authentication scheme types are supported. If you don't see the one you need here, please [get in touch](https://join.slack.com/t/orbitalapi/shared_invite/zt-697laanr-DHGXXak5slqsY9DqwrkzHg). ### Multiple authentication schemes per service _Available since 0.36.6_ A service can be configured with multiple authentication schemes that will all be applied to the same request. This is useful when an API requires multiple forms of authentication (e.g., an API key header plus a query parameter). To configure multiple authentication schemes, use an array syntax: ```hocon auth.conf authenticationTokens { "com.foo.TestService": [ { type: HttpHeader value: ${API_KEY} headerName: "X-API-Key" }, { type: QueryParam parameterName: "apiKey" value: ${API_SECRET} }, { type: Cookie cookieName: "session" value: ${SESSION_TOKEN} } ] } ``` All authentication schemes in the array will be applied to requests sent to the service, in the order they are defined. **Note** The single-object syntax (without array brackets) continues to work for backward compatibility when only one authentication scheme is needed. ### Basic authentication ```hocon auth.conf authenticationTokens { // The fully qualified name of a service, as defined in a taxi schema. // Should be surrounded in quotes. "com.foo.TestService" { type: Basic username: jimmy password: password } } ``` ### HTTP Header ```hocon auth.conf authenticationTokens { // The fully qualified name of a service, as defined in a taxi schema. // Should be surrounded in quotes. "com.foo.TestService" { type: HttpHeader // Mandatory value: letMeIn // Optional, defaults to Bearer prefix: "Token", // Optional, defaults to Authentication headerName: Auth } } ``` #### Multiple HTTP Headers _Available since 0.36.6_ Some APIs require multiple authentication headers (for example, both an organization key and an authorization key). You can configure multiple HTTP headers using the array syntax: ```hocon auth.conf authenticationTokens { "com.acme.ApiService": [ { type: HttpHeader value: ${ORG_API_KEY} headerName: "X-Organization-Key" prefix: "" // No prefix for this header }, { type: HttpHeader value: ${USER_AUTH_TOKEN} headerName: "Authorization" prefix: "Bearer" } ] } ``` This configuration will add both headers to every request sent to `com.acme.ApiService`: - `X-Organization-Key: ` - `Authorization: Bearer ` ### Query param ```hocon auth.conf authenticationTokens { // The fully qualified name of a service, as defined in a taxi schema. // Should be surrounded in quotes. "com.foo.TestService" { type: QueryParam parameterName: authKey value: letMeIn } } ``` ### Cookie ```hocon auth.conf authenticationTokens { // The fully qualified name of a service, as defined in a taxi schema. // Should be surrounded in quotes. "com.foo.TestService" { type: Cookie cookieName: authKey value: letMeIn } } ``` ### OAuth2 ```hocon auth.conf authenticationTokens { // The fully qualified name of a service, as defined in a taxi schema. // Should be surrounded in quotes. "com.foo.TestService" { type: OAuth2 accessTokenUrl: "https://auth.com/tokens" clientId: ABC clientSecret: DEF scopes: [ "profile" , "image" ] // One of AuthorizationCode, RefreshToken, ClientCredentials grantType: AuthorizationCode // One of Basic, Post, JWT method: Post // Optional. Only required if grantType is RefreshToken refreshToken: LMNOP } } ``` ### Mutual Authentication (mtls) ```hocon auth.conf authenticationTokens { // The fully qualified name of a service, as defined in a taxi schema. // Should be surrounded in quotes. "com.foo.TestService" { type: MutualTls // Absolute Path of the KeyStore Path containing private keys for mutual Authentication keystorePath: /opt/service/orbital/test-service-mtls.jks // Password for the Key Store keystorePassword: orbital // Absolute Path of the Trust Store truststorePath: /opt/service/orbital/test-trust-service-mtls.jks truststorePassword: orbital } } ``` ## Using environment variables Environment variables can be used in authentication config files. ```HOCON authenticationTokens { "com.acme.MyService" { tokenType = AuthorizationBearerHeader value = ${foo} // The enviroment variable of 'foo' is read and substituted } } ``` **Note: See also...** Also check out [Managing Secrets](/docs/deploying/managing-secrets) to see how to securely manage sensitive information in your auth tokens ## Other configuration approaches Authentication configuration is always persisted to the file described above. However, there are ways of adding / removing to the configuration without requiring file access. ### UI configuration Authentication tokens can be added, modified and deleted through the UI, via the Authentication Manager. Changes made here are persisted in the configured authentication file. ### REST API Authentication tokens can be added, modified and deleted through the REST API: #### Create or update token POST to `/api/tokens/service/{serviceName}`: ```json { "tokenType" : "AuthorizationBearerHeader", "value" : "yourAPIToken" } ``` #### Deleting a token Deletes a token DELETE to `/api/tokens/service/{serviceName}` #### Listing configured tokens It is possible to list the configured tokens. However, the token values are not returned. GET to `/api/tokens` ``` [ { "serviceName" : "com.foo.MyService", "tokenType" : "AuthorizationBearerHeader" } ] ``` -------------------------------------------------------------------------------- # Using OpenAPI Source: https://orbitalhq.com/docs/describing-data-sources/open-api Section: Managing data sources -------------------------------------------------------------------------------- Orbital can use OpenApi specs to understand what APIs exist, and how to call them. Orbital uses [Semantic schemas](/docs/introduction#semantic-data-schemas) to describe how data relates between systems. For OpenAPI, this means enriching return types and input parameters with metadata to describe them semantically. To do this, we embed Taxi metadata within the OpenApi specs. ## Adding Taxi annotations to OpenApi specs Taxi annotations use a custom `x-taxi-type` block within OpenApi specs. Example: ```yaml x-taxi-type: # The name of the type name: com.acme.MyType # Optional. Indicates if the type should be created if not already present. create: false ``` The `create` element is optional, and overrides the default behaviour. If `create` is set to `false`, then schemas which attempt to publish types that aren't already defined on the schema server are rejected as compilation errors. This is to prevent accidental typos. However, the default behaviour for `create` is different between response models and attribute types: | Entity type | Default `create` behaviour | Impact | |-----------------------|----------------------------|--------------------------------------------------------------------------------------------------------------------| | Model (Response type) | `true` | By default, if the model isn't already present within the schema, it's created. | | Attribute | `false` | By default, if an attribute type isn't already present within the remote schema, then the OpenApi spec is rejected | ### Describing response types Response models returned from API calls can be enriched to include semantic metadata #### Assigning Taxi type names to models You can optionally define a custom type name for response types being published to Orbital. If omitted, then the type name is inferred from the rest of the schema ```yaml components: schemas: Pet: # Assign a Taxi type name to the model. # Optional > x-taxi-type: > name: petstore.Pet # Everything else is standard OpenApi spec... allOf: - $ref: '#/components/schemas/NewPet' - required: ``` #### Adding type annotations to attributes Enrich attributes with semantic type metadata ```yaml NewPet: required: - name properties: name: > x-taxi-type: > name: petstore.Name type: string tag: > x-taxi-type: > name: petstore.Tag type: string ``` By default, it's expected that types referred to in `x-taxi-type` within attributes have already been defined on the schema server, as part of your [core taxonomy](/docs/introduction). However, if you are intentionally publishing new types from your OpenApi spec, then set `create` to `true`: ```yaml NewPet: required: - name properties: name: > x-taxi-type: > name: petstore.Name > # petstore.Name will be created if not present > create: true type: string ``` ### Describing service parameters Inputs into services can also be enriched, to annotate the semantic data required. Simply add a `x-taxi-type` annotation to each input, containing a `name` attribute with a reference to the name of the type from your [core taxonomy](/docs/introduction) ```yaml ## ... OpenApi spec trimmed... paths: /pets/{id}: get: description: Returns a user based on a single ID, if the user does not have access to the pet operationId: find pet by id parameters: - name: id in: path description: ID of pet to fetch required: true schema: type: integer format: int64 > x-taxi-type: > name: petstore.PetId # <-- Name of type from core taxonomy ``` ## Publishing OpenAPI specs to Orbital Once you have added taxi annotations to your OpenApi spec, you're ready to publish it to Orbital. ### Different ways to publish OpenAPI specs There are a few different ways to expose your OpenAPI spec to Orbital, each which suits different team workflows. | Approach | Description | Suits teams who... | Considerations | |---------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------| | Importing through the UI [docs](#guide-importing-open-api-projects-via-the-ui) | Converts the OpenAPI spec to Taxi, and stores it within your Taxi project. | Good for teams that prefer to work in Taxi | Changes made to the OpenAPI spec **are not** reflected in your Taxi automatically, you'll need to re-import | | Reading from a git repo [docs](/docs/workspace/connecting-a-git-repo#reading-open-api-specs-from-git) | Declares the OpenAPI spec as a project in your workspace. Keeps the OpenAPI spec as a source-of-truth, and allows a team to evolve the spec independently | Best for distributed teams or federating multiple APIs, where teams evolve their API specs independently | Changes made to the OpenAPI spec **are** automatically detected Requires adding Taxi annotations to the OpenAPI spec | | Mixing OpenAPI and Taxi in the same project [docs](#combining-open-api-and-taxi-in-a-single-project) | Keeps your OpenAPI spec as source-of-truth, but stores alongside other Taxi sources | Best when a single team owns/manages Orbital, and wants a mono-repo approach | Changes made to the OpenAPI spec **are** automatically detected Requires adding Taxi annotations to the OpenAPI spec | ### Combining OpenAPI and Taxi in a single project You can configure your Taxi project to read OpenAPI specs directly by defining their location in the `additionalSources` section of your `taxi.conf` file. Example directory structure: ``` . ├── src/ # Taxi files ├── openapi/ # OpenAPI specs └── taxi.conf ``` Configure in your `taxi.conf`: ```hocon taxi.conf name: com.example/my-taxi-project version: 0.2.0 sourceRoot: src/ additionalSources: { "@orbital/openapi" : "openapi/*.yaml" } ``` #### OpenAPI-specific configuration _Available since 0.37_ You can optionally provide additional metadata for your OpenAPI spec by creating a `.taxi.conf` file with the same name as your spec. If no config file is provided, Orbital will use the project's organization as the default namespace. | Property | Example | Description | Default | |----------------------|-----------------------|-----------------------------------------------------------------------------------------------------------------------------------------------|----------------------------| | `defaultNamespace` | `com.petflix.pets` | The namespace for any types, models or services declared in this OpenAPI spec | Project organization name | | `serviceBasePath` | `https://petflix.com` | The base url for any HTTP operations which do not include one. Not required if paths are all absolute, or the API spec provides a base server | `null` | Example file structure: ``` openapi/ ├── petflix-api-spec.yaml # Your OpenAPI spec └── petflix-api-spec.taxi.conf # Optional configuration file ``` Configuration file contents: ```hocon petflix-api-spec.taxi.conf defaultNamespace: "com.petflix.pets" serviceBasePath: "https://petflix.com" ``` ### Guide: Importing OpenAPI projects via the UI **Note** Before you can import an OpenApi spec into Orbital, you need to ensure you have configured an editable Schema source - either a local [disk repository](/docs/workspace/connecting-a-disk-repo) or a [git repository](/docs/workspace/connecting-a-git-repo) Be sure to enable editing of the repository when adding it. This lets Orbital store the converted OpenApi spec, with some additional metadata that it creates when importing Navigate to [/data-source-manager/add](http://localhost:9022/data-source-manager/add) within your Orbital UI. Next, select a project to import into, then choose "Swagger / Open API" from the data source dropdown. * Either provide the OpenApi spec file directly, or enter a Url to load the spec from. * Provide a default namespace. (eg: `com.petflix.pets`). Services from your OpenAPI spec are imported into this namespace * If your OpenAPI spec doesn't define a base url (ie.,: `servers/url`), then specify a Base url. All paths in the OpenApi spec are treated as relative to this path. * Then click Configure #### Preview your imported schema Your imported OpenApi spec will be available for you to browse, to make sure everything looks correct. Any types that have been defined as `create: true` within the Yaml spec should appear wihtin the Types section. Services and operations should've been created for all endpoints within your OpenApi spec. At this point, you can edit types (by clicking on the pencil icon next to the type name) to further refine your schema. Once you're happy, click Save, and the OpenApi spec will be imported. -------------------------------------------------------------------------------- # Using SOAP Source: https://orbitalhq.com/docs/describing-data-sources/soap Section: Managing data sources -------------------------------------------------------------------------------- Orbital can use SOAP WSDLs to understand what endpoints exist, and how to call them to fetch data. Orbital uses [Semantic schemas](/docs/introduction#semantic-data-schemas) to describe how data relates between systems. For SOAP WSDLs, this means embedding references to Taxi types within your SOAP WSDLs. To do this, we embed Taxi metadata within the WSDL. ## Adding Taxi to WSDL **Note: Adding types is optional** By default, Orbital will generate Taxi based off the type information in your WSDL and XSD docs, so adding type metadata is entirely optional If you want to control the names of the types that are generated, or you'd like to map an attribute or input to an existing type, then you can add annotations to your WSDL and XSD's ### Namespace Declaration First, declare the Taxi namespace in your WSDL: ```xml xmlns:taxi="http://taxilang.org/" ...> ``` **Note: Namespace URL** Note the trailing slash in the namespace URL - `http://taxilang.org/` is required. ### Mapping fields to semantic types By default, Orbital creates new types for each field and response object when reading an XSD or WSDL. You can modify this behaviour, either pointing to a new type that you define the name for, or to map to an existing type #### Mapping to an existing type Adding a `taxi:type="com.foo.MyType` into your XSD or WSDL will map field to an existing type: ```xml taxi:type="com.orbitalhq.CurrencyIsoCode"/> taxi:type="com.orbitalhq.CurrencyName"/> ``` #### Mapping to a new type Adding a `taxi:createType="com.foo.MyType` into your XSD or WSDL will map field to an existing type: ```xml taxi:type="com.orbitalhq.CurrencyIsoCode"/> taxi:type="com.orbitalhq.CurrencyName"/> ``` This is the equivalent of declaring `taxi:type="com.foo.Bar" taxi:create="true"` ## Adding SOAP or XSD sources to your workspace ### Adding SOAP as a standalone project You can declare a SOAP schema as a standalone project in your [`workspace.conf`](/docs/workspace/overview#workspace-conf-file), either reading from an on-disk resource (on the server where Orbital is running), or reading from a Git repo Use this approach when you have a WSDL that is standalone, and doesn't form part of a broader taxi project. You can still use [Taxi mappings](#mapping-fields-to-semantic-types) to map fields to new or existing types using this approach #### File based projects To add a file based SOAP project to your workspace, add it to the `file.projects` section of your `workspace.conf`: ```hocon workspace.conf file { projects = [ { path: "./wsdl", loader: { packageType: Soap identifier: { organisation: "com.orbitalhq" name: "CountryInfo" version: "0.1.20" }, } } ] } ``` The `path` can be absolute on the server, or relative to the `workspace.conf` file. Note that the `loader` section needs to be configured: * `packageType`: `Soap` * `identifier`: This is a package identifier that uniquely identifies this project. Select an `organisation`, `name` and `version` that's meaningful to you. #### Git-based projects To add a file based SOAP project to your workspace, add it to the `file.projects` section of your `workspace.conf`: ```hocon workspace.conf git { repositories = [ { # Give this repo a name. name=hamilton-taxonomy # The url (including personal access token) to clone the repo from uri="https://alexhamilton:glpat-dundadadadundundun@github.com/orbital/reynolds-pamphlet" # The branch to monitor branch=master # The path within the git repo (Optional, by default uses the root) path="/a/path/to/a/folder" loader: { packageType: Soap identifier: { organisation: "com.orbitalhq" name: "CountryInfo" version: "0.1.20" }, } } ] } ``` Note that the `loader` section needs to be configured: * `packageType`: `Soap` * `identifier`: This is a package identifier that uniquely identifies this project. Select an `organisation`, `name` and `version` that's meaningful to you. Related: - [Adding a git repository to your workspace.conf](/docs/workspace/connecting-a-git-repo#adding-a-git-repository-through-config) ### Adding SOAP or XSD in an existing project If you just want to add your WSDL or XSD files to your existing Taxi project, you can add them as an additionalSources root in the `taxi.conf` of your project. ```hocon taxi.conf name: com.orbitalhq/soap-demo version: 0.2.0 sourceRoot: src/ > additionalSources: { > "@orbital/wsdl" = "wsdl/*.{wsdl,xsd}" > } ``` - The key must be `@orbital/wsdl` - The path can be any directory or path relative to your `taxi.conf` file - If your SOAP definition consists of only a WSDL, then use a path pattern of `wsdl/*.wsdl` - If your SOAP definition is a mix of WSDL and XSD files, use a path pattern of `wsdl/*{.wsdl.xsd}` ## Additional configuration You can fine-tine the behaviour of how projects are imported by declaring a `xsd.taxi.conf` file in the same directory as your xsd / wsdl files. ### Overriding import urls Sometimes, XSDs are declared with invalid import urls. You can override this by providing a set of updated URLS in the `xsdImportOverrides` section of your `xsd.taxi.conf` file: For example, this XSD schema declares an invalid import url: ```xml > schemaLocation="https://gabriel.fca.org.uk/specifications/MER/DRG/PSD-CommonTypes/v1.2/FSAFeedCommon-v1-2.xsd"/> ``` To override the location this file is resolved from, add an entry to your `xsd.taxi.conf` file: ```hocon xsd.taxi.conf xsdImportOverrides: { "http://www.fsa.gov.uk/XMLSchema/FSAFeedCommon-v1-2" : "CommonTypes-Schema.xsd" } ``` ### Overriding modifiers By default, XSD models are imported as: - `closed` by default (they can only be returned from API requests) - `parameter` if they are inputs to an API call You can control this by setting the modifiers: ```hocon xsd.taxi.conf // In this demo, we want to be able to project to our Xsd types. // By default, tney're tagged as closed, indicating they're returned from services. // We can override that here. defaultModelModifiers: [] ``` ## Examples & resources - [Sample SOAP project on Github](https://github.com/orbitalapi/demos/tree/main/soap-demo) Shows importing a SOAP project as a workspace project - [Sample SOAP project using XSD Imports](https://github.com/orbitalapi/demos/tree/main/soap-with-xsd) Shows mixing WSDL with existing taxi sources, and importing XSD files - 📖 [Blog post: Modernizing SOAP APIs](/blog/2023-09-04-modernizing-soap-apis) -------------------------------------------------------------------------------- # Using HTTP Source: https://orbitalhq.com/docs/describing-data-sources/http Section: Managing data sources -------------------------------------------------------------------------------- **Note: See also: OpenAPI** Orbital has first-class support for working with [OpenAPI specs](/docs/describing-data-sources/open-api) and [SOAP WSDLs](/docs/describing-data-sources/soap) The following guide is intended for consuming HTTP services without a separate spec. ## Describing HTTP with Taxi If you're not working with other HTTP specs, you can describe your services directly using Taxi. Taxi is very rich and expressive, and is designed to fully support describing API sources, for teams who are not currently using an existing standard. ## Services A service is simply a group of operations. ```taxi service PeopleService { operation listAllPeople():Person[] } ``` Services that contain HTTP operations may optionally declare an `@HttpService` annotation, which allows it to set a baseUrl for all contained operations. ```taxi @HttpService(baseUrl="https://foo.com") service PeopleService { // resolves to https://foo.com/people @HttpOperation(method = 'GET', url = '/people') operation listAllPeople():Person[] } ``` ## Http Operations An operation defines a function on the API. ```taxi @HttpOperation(method = 'GET', url = 'https://myservice/people') operation listAllPeople():Person[] ``` Operations often have annotations that provide hints to tooling as to how to invoke them. Taxi ships with some standard annotations, as part of it's [std-lib](https://gitlab.com/taxi-lang/taxi-lang/tree/master/taxi-stdlib-annotations/src/main/java/lang/taxi/annotations). Although it's up to individual tooling to determine how to interpret these, the suggested usage is as follows: | Annotation | Usage | |:----------------------------------|:----------------------------------------------------------------------------------------------------| | `@HttpOperation(method,url)` | Indicates that the operation should be invoked over HTTP, using the provided `method` and `url` | | `@HttpRequestBody` | Indicates that a parameter will be found on the request body | | `@PathVariable(name)` | Indicates a property from the path that should be populated with a variable passed to the operation | | `@HttpHeader(name,value, prefix)` | Defines how header are passed. See below | It's possible to use environment variables in these annotations, as described [here](/docs/deploying/configuring-orbital#environment-variables-in-annotations). ### Http Headers Http headers are passed as annotations, which follow the following format: ```taxi annotation HttpHeader { name : String [[ Pass a value when using as an annotation on an operation. For parameters, it's valid to allow the value to be populated from the parameter. ]] value : String? prefix : String? suffix : String? } ``` Http Headers can be defined in operations in a few different ways: #### As a fixed value, on an operation ```taxi service MyService { @HttpOperation(method = "GET", url = "...") @HttpHeader(name = "Consumes", value = "application/json") operation findPeople():Person[] } ``` #### As an input into an operation ```taxi service MyService { operation findPeople( @HttpHeader(name = "Cache-Control", prefix = "max-age=") cacheMaxAge : CacheMaxAge ):Person[] } ``` Which is then invoked as: ```taxi given { maxAge : CacheMaxAge = 64000 } find { Person[] } ``` Additionally, header values can be calculated, using expressions: ```taxi service MyService { operation findPeople( @HttpHeader(name = "If-Modified-Since") ifModifiedSince : Instant = addDays(now(),-7) ):Person[] } ``` ### Path variables **Note: Be sure to use a semantic type** All variables in HTTP requests must have a semantic type. Trying to declare a variable as `String` or `Int` will result in the variable not being populated, as they can't be matched ```taxi > import taxi.http.PathVariable service PetApi { > @HttpOperation(method = "GET", url = "https://pets.com/pets/{petId}") operation getPetById(@PathVariable("petId") id: PetId): Pet } ``` The name on the annotation `@PathVariable("petId")` must match the placeholder in the url (`/pets/{petId}`) Then a query such as: ```taxi given { pet: PetId = "123" } find { Pet } ``` Alternatively, using a constraint: ```taxi find { Pet(PetId == "123") } ``` Would result in an HTTP GET to `https://pets.com/pets/123` ### Query parameters Query parameters (sometimes know as Query Strings) can be populated into a URL using the `taxi.http.QueryVariable` annotation: ```taxi > import taxi.http.QueryVariable service PersonApi { @HttpOperation(method = "GET", url = "https://people.com/people") operation listPeople( > @QueryVariable("apiKey") key: ApiKey ): Person[] } ``` Then a query such as: ```taxi given { key: ApiKey = "123" } find { Person[] } ``` Would result in an HTTP GET to `https://people.com/people?apiKey=123` The name in the annotation defines the parameter that is passed on the query string #### Optional params Query params can be optional, by making the **type** nullable - eg: `ApiKey?`: ```taxi import taxi.http.QueryVariable service PersonApi { @HttpOperation(method = "GET", url = "https://people.com/people") operation listPeople( > @QueryVariable("apiKey") apiKey: ApiKey? ): Person[] } ``` ### Examples ```taxi service MovieService { @HttpOperation(method = "GET" , url = "https://myMovies/movies") operation findAllMovies() : Movie[] @HttpOperation(method = "GET" , url = "https://myMovies/movies/{id}") operationFindMovie( @PathVariable(name = "id") id : MovieId ) : Movie } @HttpService(baseUrl = "http://actorService") service ActorService { // resolves to http://actorService/actors @HttpOperation(method = "GET" , url = "/actors") operation findAllActors() : Actor[] } ``` Names of operation parameters are optional. This is to encourage developers to leverage a richer type system where possible: ```taxi // These two declarations are both valid, and describe the same operation operation convertUnits(source:Weight, targetUnit:Unit):Weight operation convertUnits(Weight,Unit):Weight ``` ## Operation Contracts & Constraints Contracts and constraints are useful for telling tooling about what functionality an operation can provide, and what conditions must be met before invoking. Both contracts and constraints use the same syntax. ```taxi type Money { currency : String amount : Decimal } operation convertCurrency(input: Money, targetCurrency: String) : Money(from input, currency = targetCurrency) ``` ### from input A contract may indicate that a return type is derived from one of the inputs, by using the `from {input}` syntax: ```taxi operation convertUnits(input: Weight, target: Unit):Weight( from input ) ``` ### Attribute constraints Attribute constraints describe either a pre-condition \(if on an input\) or a post-condition \(if on a return type\) for an operation ```taxi operation convertFromPounds(input : Money(currency = 'GBP'), target: Currency) : Money( from input, currency = target) ``` As shown above, attribute constraints may either be: * A constant value \(ie., `"GBP"`\) * A reference to an attribute of another parameter. * Nested syntax is supported \(ie., `foo.bar.baz`\) These constraints are applicable on types too - see type constraints for an example. ## Retry policies When integrating with external REST APIs, handling intermittent failures such as HTTP 500 errors is crucial for maintaining application reliability. Taxi allows defining retry policies, which Orbital will honour when things go wrong ### Fixed Delay Retry Policy A fixed delay retry policy retries operations with a constant wait time between attempts. This is useful for temporary issues on the external service's end. For example, to retry the `getReviews` operation upon receiving an HTTP 500 error, with a 5-second wait for up to 10 attempts, define the service as follows: ```taxi service ReviewsApi { @HttpOperation(method = "GET", url = "https://reviews/{id}") @HttpRetry(responseCode = [500], fixedRetryPolicy = @HttpFixedRetryPolicy(maxRetries = 10, retryDelay = 5)) operation getReviews(id: FilmId): FilmReview[] } ``` To include additional error codes like HTTP 502, simply adjust the `responseCode` attribute: ```taxi service ReviewsApi { @HttpOperation(method = "GET", url = "https://reviews/{id}") @HttpRetry(responseCode = [500, 502], fixedRetryPolicy = @HttpFixedRetryPolicy(maxRetries = 10, retryDelay = 5)) operation getReviews(id: FilmId): FilmReview[] } ``` ### Exponential Delay Retry Policy An exponential delay retry policy increases the delay between retries, which can be further randomized using jitter to avoid synchronized retry storms: ```taxi service ReviewsApi { @HttpOperation(method = "GET", url = "https://reviews/{id}") @HttpRetry(responseCode = [500, 502], fixedRetryPolicy = @HttpExponentialRetryPolicy(maxRetries = 10, retryDelay = 5, jitter = 0.5)) operation getReviews(id: FilmId): FilmReview[] } ``` With the above definition, Orbital will try invoking the Rest at most 10 times in case of Http 500 or Http 502 errors. However, the delay between retries will increase exponentially. With a `jitter` value between 0 and 1, you add some randomness to the delay time by introducing a random delay, or “jitter”, to the next retry delay time. This ensures that the retries are not synchronous and reduces the likelihood of a retry storm. Here’s how the table would look with exponential backoff with Jitter: | Retry Attempt | Delay Time (seconds) | Jitter Range (seconds) | Actual Delay Time (seconds) | | ------------- | ---- -----------------|------------------------- |-----------------------------| | 1 | 1.0 | 0.5 | 1-0 - 1.5 | | 2 | 2.0 | 0.5 | 1-5 - 2.5 | | 3 | 4.0 | 0.5 | 3.5 – 4.5 | | 4 | 8.0 | 0.5 | 7.5 – 8.5 | | 5 | 16.0 | 0.5 | 15.5 – 16.5 | ## Modifying server URLs Sometimes you need to modify the actual server name that's used for contacting an HTTP server. Services define the url using the `@HttpOperation` annotation: ```taxi service ReviewsApi { @HttpOperation(method = "GET", url = "https://reviews") operation getReviews(): FilmReview[] } ``` However, it's possible that between environments you may need to direct traffic to a different physical server. Orbital supports overriding server domain names using a `services.conf` HOCON file, defined in your Taxi project. ### Declaring config files Ensure that your Taxi project's `taxi.conf` file has the following config section declared: ``` additionalSources: { "@orbital/config" : "orbital/config/*.conf", } ``` In your taxi project, add file at `orbital/config/services.conf`: ```hocon services { reviews { url="http://localhost:8080" } } ``` **Note** services.conf files support [environment-specific variations](/docs/deploying/configuring-orbital#environment-specific-configuration). Now, when Orbital sends HTTP requests to `http://reviews`, the URL will be rewritten to `http://localhost:8080` -------------------------------------------------------------------------------- # Using Protobuf Source: https://orbitalhq.com/docs/describing-data-sources/protobuf Section: Managing data sources -------------------------------------------------------------------------------- Orbital supports reading protobuf messages - either as a message format for HTTP endpoints or from a streaming query originating from Kafka. You can annotate protobuf sources directly with Taxi metadata, or you can import through the Schema Importer. ## Embedding semantic types in protobuf Within a protobuf, fields can be annotated with taxi metadata to indicate the semantic type used on a field. Fields are annotated using the `taxi.dataType` extension (which must be imported in the protobuf file). An example is shown: ``` import "taxi/dataType.proto"; message CafeDrink { optional string customer_name = 1 [(taxi.dataType)="foo.CustomerName"]; optional int32 customer_id = 2 [(taxi.dataType)="foo.CustomerId"]; enum Foam { NOT_FOAMY_AND_QUITE_BORING = 1; ZOMG_SO_FOAMY = 3; } } ``` ### Datatype proto extension Protobufs do not currently support importing external type definitions, so you need to include the source of the dataType proto definition directly in your protobuf project. By convention, the protobuf definition should be written to `taxi/dataType.proto` within your protobuf project. ```protobuf import "google/protobuf/descriptor.proto"; package taxi; extend google.protobuf.FieldOptions { optional string dataType = 50002; } ``` ### Generated taxi When a protobuf with taxi metadata is imported by Orbital, a taxi representation is automatically generated. **Note: Hint** You generally wouldn't hand-craft a protobuf taxi declaration. Taxi has tooling to generate this automatically. The source is shown here for reference only. ```taxi @lang.taxi.formats.ProtobufMessage(packageName = "" , messageName = "CafeDrink") model CafeDrink { @lang.taxi.formats.ProtobufField(tag = 1 , protoType = "string") customer_name : foo.CustomerName? @lang.taxi.formats.ProtobufField(tag = 2 , protoType = "int32") customer_id : foo.CustomerId? } ``` ## Importing a Protobuf through the UI You can import a protobuf schema directly through the UI ### Import the protobuf file * Click on Schema Explorer, then Add new * Select Protobuf from the dropdown To provide the protobuf you can: * Upload a single protobuf schema * Upload a zip file containing protobuf schemas * Specify a URL to a single protobuf file * Specify a URL to a zip file of protobuf schemas Then click "Create" ### Preview the generated types A preview is shown, containing the models, fields and types that were imported from the Protobuf schema. You can change any of the types (eg., swapping a primitive type with a more specific semantic type), by clicking on the pencil icon next to the type name, and search for the desired type. Once you're satisfied, click Save, and the schema will be updated. -------------------------------------------------------------------------------- # Databases Source: https://orbitalhq.com/docs/describing-data-sources/databases Section: Managing data sources -------------------------------------------------------------------------------- ## Overview Orbital can connect to databases to fetch data when running a query. In order to understand the data that is present in a database table, Orbital uses a Taxi Schema for that table. The taxi schema describes the table - it's columns and the data they hold. Because we're using Taxi here, the column descriptions are richer than things like `String` or `Integer` - instead using rich semantic tags like `FirstName`, `LastName`, or `EmailAddress` In this guide, we'll learn how to create a Taxi Schema for a specific database table - both through the user interface, and by directly editing a schema ## Using the UI The UI allows you to connect a database table directly, without having manually edit taxi files. Through the UI, Orbital will connect to the database, and create: * A series of types for each column in the database table * A model that describes the database table * A service that exposes query capabilities for the table ### Importing a new table * From the home page, click Add a data source. * Alternatively, Click Schema Explorer in the left-hand navigation menu, then click "Add new" * For the schema type to import, select "Database table" * Click the connection name drop-down, and select the connection for your database * If you haven't yet created your connection, you can click "Add new connection". * Select the table from the drop-down * Specify a namespace for the taxi types, models and services that will be created * Click Configure ### Preview the generated types A preview is shown, containing the models, fields and types that were imported from the Database schema. You can change any of the types (eg., swapping a primitive type with a more specific semantic type), by clicking on the pencil icon next to the type name, and search for the desired type. Once you're satisfied, click Save, and the schema will be updated. ### Changing the assigned types Orbital has assigned reasonable defaults to all the fields. Specifically * Id's have been tagged * Foreign keys have been mapped * For all other fields, new semantic types have been created. If the columns in your database map to exisitng semantic types, you may wish to update the definitions. To do this: * Select a Model from the table on the left-hand side * Click on the blue link for the column you wish to change the type of * A search dialog is displayed * From here, you can search for existing types in your catalog, or create a new type You can also click to edit documentation for any of the types, models and services. * Once you're satisfied with the edits to your table and types, click Save * The schema has been created and written to the taxi project configured in your schema server. ### Required permissions In order to view, create or edit connected database tables through the UI, users must have the following permissions granted. | Activity | Required permission | |-----------------------------------|---------------------| | View the connected tables | `VIEW_CONNECTIONS` | | Create or modify a database table | `EDIT_CONNECTIONS` | See (this guide)[/how-to-guides/auth/manage-user-permissions] for more information on role based security. ## Defining a database connection in connections.conf Instead of [using the UI](#using-the-ui) to add your database, you can add a database by defining a a connection in your [`connections.conf`](/docs/describing-data-sources/configuring-connections) file. Here is an example for adding a Postgres database connection: ```hocon jdbc { // The root element for database connections another-connection { // Defines a connection called "another-connection" connectionName = another-connection // The name of the connection. Must match the key used above. jdbcDriver = POSTGRES // Defines the driver to use. See below for the possible options connectionParameters { // A list of connection parameters. The actual values here are defined by the driver selected. database = transactions // The name of the database host = our-db-server // The host of the database password = super-secret // The password port = "2003" username = jack // The username to connect with } } } ``` Database connections are defined under the `jdbc` element within the `connections.conf` file. ```hocon jdbc { // The root element for database connections another-connection { // Defines a connection called "another-connection" connectionName = another-connection // The name of the connection. Must match the key used above. jdbcDriver = POSTGRES // Defines the driver to use. See below for the possible options connectionParameters { // A list of connection parameters. The actual values here are defined by the driver selected. database = transactions // The name of the database host = our-db-server // The host of the database password = super-secret // The password port = "2003" username = jack // The username to connect with } } } ``` ### Supported drivers ### Postgres To configure a Postgres connection, specify `jdbcDriver = POSTGRES` Connection parameters are as follows: | Parameter name | Description | |----------------|-------------------------------------------------| | `host` | The host address of the Postgres database | | `port` | The port to connect to. Defaults to `5432` | | `database` | The name of the database on the postgres server | | `username` | Optional. The username to use when connecting | | `password` | Optional. The password to use when connecting | #### Example ```HOCON jdbc { // The root element for database connections another-connection { // Defines a connection called "another-connection" connectionName = another-connection // The name of the connection. Must match the key used above. jdbcDriver = POSTGRES // Defines the driver to use. See below for the possible options connectionParameters { // A list of connection parameters. The actual values here are defined by the driver selected. database = transactions // The name of the database host = our-db-server // The host of the database password = super-secret // The password port = "2003" username = jack // The username to connect with } } } ``` ### MySQL To configure a MySql connection, specify `jdbcDriver = MYSQL` Connection parameters are as follows: | Parameter name | Description | |----------------|-----------------------------------------------| | `host` | The host address of the MySQL database | | `port` | The port to connect to. Defaults to `3306` | | `database` | The name of the database on the MySql server | | `username` | Optional. The username to use when connecting | | `password` | Optional. The password to use when connecting | #### Example ```HOCON jdbc { mysql-docker { connectionName=mysql-docker connectionParameters { database=test host=localhost password=my-secret-pw port="3306" username=root } jdbcDriver=MYSQL } } ``` ### MSSQL Server To configure a Postgres connection, specify `jdbcDriver = MSSQL` Connection parameters are as follows: | Parameter name | Description | |--------------------------|---------------------------------------------------------------------------------------------------------| | `host` | The host address of the MSSQL database | | `port` | The port to connect to. Defaults to `1443` | | `database` | The name of the database on the MS SQL server | | `username` | Optional. The username to use when connecting | | `password` | Optional. The password to use when connecting | | `schema` | Optional. The schema to use - defaults to `dbo` | | `trustServerCertificate` | Optional. Forces Orbital to trust the certificate that's provided by the SQL Server. Defaults to `true` | | `encrypt` | Optional. Defines if the connection to MSSQL server should be encrypted. Defaults to `true` | #### Example ```HOCON jdbc { sqlServerConnection { connectionName=sqlServerConnection connectionParameters { database=Northwind encrypt="true" host=localhost password=ChangeMe port="14330" schema=dbo trustServerCertificate="true" username=sa } jdbcDriver=MSSQL } } ``` ### Oracle _Available since 0.38.0-M2_ To configure an Oracle connection, specify `jdbcDriver = ORACLE` Connection parameters are as follows: | Parameter name | Description | |----------------|----------------------------------------------------------| | `host` | The host address of the Oracle database | | `port` | The port to connect to. Defaults to `1521` | | `service` | The name of the Oracle service to connect to | | `username` | The username to use when connecting | | `password` | The password to use when connecting | The connection is built using Oracle's service-name URL format (`jdbc:oracle:thin:@//host:port/service`). #### Example ```HOCON jdbc { oracleConnection { connectionName=oracleConnection connectionParameters { host=localhost port="1521" service=FREEPDB1 username=system password=ChangeMe } jdbcDriver=ORACLE } } ``` ### Redshift To configure a Postgres connection, specify `jdbcDriver = REDSHIFT` Connection parameters are as follows: | Parameter name | Description | |----------------|-------------------------------------------------| | `host` | The host address of the Redshift database | | `port` | The port to connect to. Defaults to `5439` | | `database` | The name of the database on the Redshift server | | `username` | Optional. The username to use when connecting | | `password` | Optional. The password to use when connecting | #### Example ```HOCON jdbc { // The root element for database connections another-connection { // Defines a connection called "another-connection" connectionName = another-connection // The name of the connection. Must match the key used above. jdbcDriver = REDSHIFT // Defines the driver to use. See below for the possible options connectionParameters { // A list of connection parameters. The actual values here are defined by the driver selected. database = transactions // The name of the database host = our-db-server // The host of the database password = super-secret // The password port = "2003" username = jack // The username to connect with } } } ``` ### Snowflake To configure a Postgres connection, specify `jdbcDriver = SNOWFLAKE` Connection parameters are as follows: | Parameter name | Description | |----------------|---------------------------------------------------------| | `account` | The name of the Snowflake account | | `schema` | The name of the schema to connect to | | `db` | The name of the database to connect to | | `warehouse` | The name of the warehouse where the snowflake db exists | | `username` | The username to use when connecting | | `password` | The password to use when connecting | | `role` | The role to specify when connecting | #### Example ```hocon jdbc { // The root element for database connections another-connection { // Defines a connection called "another-connection" connectionName = another-connection // The name of the connection. Must match the key used above. jdbcDriver = SNOWFLAKE // Defines the driver to use. See below for the possible options connectionParameters { // A list of connection parameters. The actual values here are defined by the driver selected. account = mySnowflakeAccount123.eu-west-1 schema = public db = demo_db warehouse = COMPUTE_WH schema = public role = QUERY_RUNNER } } } ``` ## Describing tables in Taxi **Note: Before you continue...** Before you run through this guide, it's worth understanding the basics of [taxi](https://taxilang.org/language-reference/taxi-language/), and how Orbital uses it. Also, make sure you have a [taxi project set up](https://taxilang.org/intro/getting-started/), and that it's been [published to Orbital](docs/connecting-data-sources/connecting-a-git-repo/). If you're running through one of our tutorials, we've already taken care of this for you. Taxi files define the mappings of data models and the services that expose them. In this guide, we'll describe how expose a new database table to Orbital, and make it queryable. Before starting, in your taxi project, create a new file under the `src/` directory. It's up to you what you name it. For this example, `customers.taxi` is a good start. ### Databases, and pull-based schema definitions As discussed in [publishing schemas to Orbital](/docs/connecting-data-sources/schema-publication-methods/), there are different ways for Orbital to consume schema information - either by data sources ***pushing*** their information directly to Orbital (well suited for application APIs), or by ***pulling*** from git-based repositories that describe the schemas. While the push model is preferred, it's not currently supported for databases. We're looking into ways to embed Taxi metadata into DDL schema definitions. For now, you'll need to maintain taxi definition file that describes the database. ### Defining a table mapping Tables are exposed to Orbital using the annotation `@com.orbitalhq.jdbc.Table` on a model. Fields names in the model are expected to align with column names from the database. Here's an example: ```taxi import com.orbitalhq.jdbc.Table @Table(connection = "films-database", schema = "public" , table = "customer" ) model Customer { @Id // Use @Id to denote the primary key customerId : CustomerId firstName : CustomerFirstName? // Nullable columns should have the Taxi nullable symbol lastName : CustomerLastName } ``` The `@Table` annotation contains the following parameters: | Parameter | Description | |------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------| | connection | The name of a connection, as defined in your [connections configuration file](#defining-a-database-connection-in-connections-conf). | | schema | The name of the schema. Optional, depending on your database | | table | The name of the table | It's possible to use environment variables in these annotations, as described [here](/docs/deploying/configuring-orbital#environment-variables-in-annotations). #### Mapping the primary key Use an `@Id` annotation to define the column that represents the primary key. To map a composite key, annotate each column that forms the key with `@Id`, and Orbital treats the combination as the primary key. ## Querying databases To expose a database as a source for queries, the database must have a service and table operation exposed. Here's an example: ```taxi import com.orbitalhq.jdbc.DatabaseService @DatabaseService(connection = "films-database") service CustomerService { table customers : Customer[] } ``` The `@DatabaseService` annotation contains the following parameters: | Parameter | Description | |------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------| | connection | The name of a connection, as defined in your (connections configuration file)[/how-to-guides/connections/manage-database-connection/#defining-a-database-connection]. | ### Sample queries #### Fetch everything from a table: ```taxi find { Customer[] } ``` #### Fetch a single value from a table: ```taxi find { Customer( CustomerId == 123 ) } ``` #### Fetch values by criteria: ```taxi find { Customer[]( DateOfBirth <= '1989-10-01' && CountryOfBirth == 'NZ' ) } ``` #### Join two tables ```taxi find { Customer[] } as (customer:Customer) -> { name : FirstName // defines a join between the Customer and Purchase tables purchases : Purchases[](CustomerId == customer.id) } ``` #### Join two tables, transforming data ```taxi find { Customer[] } as (customer:Customer) -> { name : FirstName // defines a join between the Customer and Purchase tables purchases : Purchases[](CustomerId == customer.id) as { // Inside this scope we have access to both Customer data and Purhcase data productName : ProductName price : ProductPrice // Be sure to include the array marker, as we're defining an array of // objects (Purchase[] -> OurType[]) }[] // <--- array marker } ``` #### Fetching from a database, enrich from another source As with all TaxiQL queries, enriching data from multiple sources requires simply asking for the data you need - Orbital works out the correct integration. Assuming a schema with a database such as: ```taxi import com.orbitalhq.jdbc.Table import com.orbitalhq.jdbc.DatabaseService @Table(connection = "customers-database", schema = "public" , table = "customer" ) closed model Customer { @Id id : CustomerId inherits Int name : CustomerName inherits String } @DatabaseService(connection = "customers-database") service CustomerService { table customers : Customer[] } ``` And we also have an API that exposes balance information: ```taxi closed model CustomerBalance { customerId : CustomerId balance : CurrentBalance } service AccountBalanceService { @HttpOperation(url="https://fakeurl/customers/{id}/balance", method = "GET" ) operation getCustomerBalance(@PathVariable id:CustomerId):CustomerBalance } ``` The below call assumes we're fetching customer details from our database, then enriching against an API call (API ) ```taxi find { Customer(CustomerId == 123) } as { name : CustomerName // this information comes from the database currentBalance : CurrentBalance // An API call is made to fetch account balance } ``` Or, to fetch that same data for all customers: ```taxi find { Customer[] } as { name : CustomerName // this information comes from the database currentBalance : CurrentBalance // An API call is made to fetch account balance }[] ``` #### Using IN and NOT IN operators When querying databases, you can use in and not in operators to filter against multiple values at once. This is more efficient than using multiple OR conditions and generates optimized SQL. **Filtering with `IN` operator** To find records where a field matches any value in a list: ```taxi // Find customers with specific IDs find { Customer[]( CustomerId in [123, 456, 789] ) } ``` ```taxi // Find customers from specific countries find { Customer[]( CountryOfBirth in ["US", "UK", "CA"] ) } ``` **Filtering with `NOT IN` operator** To find records where a field does not match any value in a list: ```taxi // Find customers excluding specific IDs find { Customer[]( CustomerId not in [123, 456] ) } ``` ```taxi // Find customers not from specific countries find { Customer[]( CountryOfBirth not in ["US", "UK"] ) } ``` **Combining with other conditions** IN and NOT IN operators can be combined with other conditions using `&&` (AND) and `||` (OR): ```taxi // Find active customers from specific countries find { Customer[]( CountryOfBirth in ["US", "UK", "CA"] && Status == "ACTIVE" ) } ``` ```taxi // Find customers either VIP or from specific regions find { Customer[]( CustomerType == "VIP" || CountryOfBirth in ["US", "UK"] ) } ``` Both in and not in work with any data type - strings, numbers, dates, and custom semantic types. ### Collection options _Available since 0.38.0_ You can limit, paginate and sort database results using [collection options](/docs/querying/collection-options), written alongside your filter criteria: ```taxi find { Customer[]( CountryOfBirth == "GB", orderBy: DateOfBirth desc, offset: 20, limit: 10 ) } ``` For databases, Orbital pushes `limit`, `offset` and `orderBy` into the generated SQL, using the correct dialect for your driver (for example, MSSQL renders `TOP` / `OFFSET ... FETCH`). This means the database does the work, and only the rows you asked for are returned over the wire. | Option | Behaviour against a database | |-----------------------|--------------------------------------------------------------------------------------------------------| | `limit`, `offset` | Pushed into the SQL query. | | `orderBy` | Pushed into the SQL query when the ordered type maps to a single column. Otherwise Orbital sorts after fetching. | | `uniqueBy` | Always applied by Orbital, after fetching. | Null ordering is consistent whether the sort runs in SQL or in Orbital: nulls are placed **last** when sorting ascending, and **first** when sorting descending. See [collection options](/docs/querying/collection-options) for the execution model, and the [Taxi language reference](https://taxilang.org/docs/taxiql/querying#collection-options) for the full syntax. ## Writing data to a database To expose a database table for writes, you need to provide a `write operation` in a service, specifying the write behaviour: ```taxi import com.orbitalhq.jdbc.Table import com.orbitalhq.jdbc.DatabaseService import com.orbitalhq.jdbc.UpsertOperation @Table(connection = "customers-database", schema = "public" , table = "customer" ) closed model Customer { @Id id : CustomerId inherits Int name : CustomerName inherits String } @DatabaseService(connection = "customers-database") service CustomerService { table customers : Customer[] @UpsertOperation write operation saveCustomer(Customer):Customer } ``` In this example, the `saveCustomer` operation will attempt to perform an upsert. | Write behaviour | Annotation | Comments | |-----------------|--------------------------------------|------------------------------------------------| | Insert | `com.orbitalhq.jdbc.InsertOperation` | | | Update | `com.orbitalhq.jdbc.UpdateOperation` | Requires an `@Id` field | | Upsert | `com.orbitalhq.jdbc.UpsertOperation` | Falls back to an insert if no `@Id` is defined | | Delete | `com.orbitalhq.jdbc.DeleteOperation` | Requires an `@Id` field. See [Deleting data from a database](#deleting-data-from-a-database) | ### Table creation If the database table does not exist, Orbital will create it when first attempting to write. If the database table does exist, but with a different schema, writes may fail. No schema migrations are performed. ### Example queries When writing data from one data source into a database, it's not neccessary for the data to align with the format of the persisted value. Orbital will automatically adapt the incoming data to the format required by the db. This may involve projections and even calling additional services if needed. #### Inserting a static value into a database ```taxi // inserting a static value into a database given { customer : Customer = { customerId : 123, name : "Jimmy Smitts" } } call CustomerService::saveCustomer ``` #### Stream data from Kafka into a database ```taxi import com.orbitalhq.jdbc.Table import com.orbitalhq.jdbc.DatabaseService import com.orbitalhq.jdbc.UpsertOperation // Common, shared types: type StockSymbol inherits String type StockPrice inherits Decimal // Database definitions: @Table(connection = "prices-database", schema = "public" , table = "stock-price" ) closed model StockPrice { @Id symbol : StockSymbol price : StockPrice } @DatabaseService(connection = "prices-database") service PriceService { table stockPrices : StockPrice[] @UpsertOperation write operation savePrice(StockPrice):StockPrice } // Kafka definitions: // Note that field names don't align - orbital // handles this for us. closed model PriceUpdateMessage { ticker : StockSymbol lastTradedPrice : StockPrice } ``` Then, the query: ```taxi stream { PriceUpdateMessage } call PriceService::savePrice ``` Orbital writes each message received from Kafka into the db - creating the table if required, and transforming the Kafka message to the format defined by `StockPrice` ### Batching writes _Available since 0.38.0-M1_ By default, each record is written in its own statement. For high-volume writes — streaming a Kafka topic into a table, or loading a large result set — you can have Orbital accumulate records and write them in batches. Add `batchSize` and / or `batchDuration` to the write annotation: ```taxi @DatabaseService(connection = "prices-database") service PriceService { table prices : StockPrice[] @InsertOperation(batchSize = 500, batchDuration = 2000) write operation savePrice(StockPrice):StockPrice } ``` Both parameters are available on `@InsertOperation`, `@UpdateOperation` and `@UpsertOperation`: | Parameter | Type | Meaning | |---|---|---| | `batchSize` | `Int` | Write the batch once this many records have accumulated | | `batchDuration` | `Int` | Write the batch this many **milliseconds** after the first record in it arrived | Whichever condition is met first triggers the write. Batching is enabled as soon as either parameter is present — you don't need both, but you almost always want both: | If you set | The other defaults to | |---|---| | `batchSize` only | A 30 minute timer | | `batchDuration` only | An unbounded batch size — only the timer writes | **Warning: Set batchDuration as well as batchSize** A record isn't returned to the query until the batch containing it has been written. A batch that never fills up is written when its timer fires — so if you set `batchSize = 500` and leave `batchDuration` unset, a query writing 10 records waits for the 30 minute default before those rows are written and the query completes. Set `batchDuration` to the longest you're willing to wait for a partly-filled batch. Batches are scoped to a single query. Each running query keeps its own buffer, which is discarded when the query completes or is cancelled; two concurrent queries writing to the same table don't share a batch. Results are unchanged in content, but not in timing: each written record is still returned, and still carries its generated values, but records are emitted in bursts as each batch completes rather than one at a time. **Note: Deletes can't be batched** `@DeleteOperation` doesn't support `batchSize` or `batchDuration`. Declaring them on a delete raises an error telling you to remove them. ## Deleting data from a database _Available since 0.38.0-M2_ To delete rows from a database table, declare a `write operation` annotated with `@DeleteOperation`: ```taxi import com.orbitalhq.jdbc.Table import com.orbitalhq.jdbc.DatabaseService import com.orbitalhq.jdbc.DeleteOperation import com.orbitalhq.jdbc.DeleteResult type CustomerId inherits Int type CustomerName inherits String @Table(connection = "customers-database", schema = "public", table = "customer") closed model Customer { @Id id : CustomerId name : CustomerName? } @DatabaseService(connection = "customers-database") service CustomerService { table customers : Customer[] @DeleteOperation write operation deleteCustomer(Customer):DeleteResult @DeleteOperation write operation deleteCustomers(Customer[]):DeleteResult } ``` Rows are matched on the model's `@Id` values - other fields are ignored. Models declaring multiple `@Id` fields are matched on the combination of all of them. Pass a single instance to delete one row, or an array to delete several. The return type can be any model with a `deletedCount` field. `com.orbitalhq.jdbc.DeleteResult` is provided for convenience. **Requirements for delete operations:** - The model must have at least one `@Id` field - If no rows match, the operation returns a `deletedCount` of `0` - If the table hasn't been created yet (see [Table creation](#table-creation)), the operation returns a `deletedCount` of `0` rather than failing, and reports a notice on the query's error stream explaining that the table doesn't exist yet, so you can see why nothing was deleted - The delete runs in a single transaction, so if it fails partway the whole delete is rolled back and the table is left unchanged **Note: Populating non-id fields** TaxiQL requires mandatory fields to be populated when constructing an instance. As only the `@Id` values are used to match rows, declare the other fields as nullable (as in the example above), or populate them in the query. ### Sample delete queries #### Deleting a single row ```taxi given { customer : Customer = { id : 123 } } call CustomerService::deleteCustomer ``` #### Deleting multiple rows ```taxi given { customers : Customer[] = [ { id : 123 }, { id : 456 } ] } call CustomerService::deleteCustomers ``` ## Running native SQL queries _Available since 0.38.0-M2_ For queries that go beyond Orbital's standard table operations - joins, aggregations, or dialect-specific SQL - use the `@SqlQuery` annotation as an escape hatch. The SQL is passed to your database as written, and results map back to Taxi models. ```taxi import com.orbitalhq.jdbc.SqlQuery import com.orbitalhq.jdbc.DatabaseService import com.orbitalhq.jdbc.UpdateResult type FilmTitle inherits String type ReleaseYear inherits Int type DirectorName inherits String type FilmCount inherits Int // Return types don't need an @Table annotation - any model // whose field names match the result set's column names (or aliases) works model Film { title : FilmTitle releaseYear : ReleaseYear } model DirectorFilmSummary { director : DirectorName filmCount : FilmCount } @DatabaseService(connection = "films-database") service FilmStats { @SqlQuery(sql = "SELECT title, release_year AS releaseYear FROM film WHERE release_year > :year") operation recentFilms(year : ReleaseYear) : Film[] @SqlQuery(sql = """ SELECT d.name AS director, COUNT(f.id) AS filmCount FROM film f JOIN director d ON f.director_id = d.id GROUP BY d.name """) operation filmsPerDirector() : DirectorFilmSummary[] @SqlQuery(sql = "DELETE FROM film WHERE release_year < :cutoff") write operation purgeOldFilms(cutoff : ReleaseYear) : UpdateResult } ``` One annotation serves both reads and writes: - A plain `operation` runs the SQL as a query. Rows are matched to the return type's fields by column name (or alias), case-insensitively. - A `write operation` runs the SQL as a statement (`UPDATE`, `DELETE`, `INSERT`), returning the number of rows affected. `com.orbitalhq.jdbc.UpdateResult` is provided for this, or declare your own model with an `affectedRows` field (`deletedCount` also works, for `DELETE`-shaped statements). ### Parameter binding Parameters use the `:parameterName` syntax, and bind to the operation's parameters by name. Values are always passed as prepared-statement parameters - never substituted into the SQL text - so user-supplied values can't alter the query. If the SQL references a parameter the operation doesn't declare, the query reports an error naming the missing parameter. ### Invoking native queries Read operations are invoked like any other operation - through a `find` query: ```taxi given { year : ReleaseYear = 2000 } find { Film[] } ``` ```taxi find { DirectorFilmSummary[] } ``` Write operations are invoked with `call`: ```taxi given { cutoff : ReleaseYear = 1980 } call FilmStats::purgeOldFilms ``` ### Behaviour notes - The table isn't created automatically - the SQL runs against whatever schema exists. - [Collection options](/docs/querying/collection-options) (`limit`, `offset`, `orderBy`) aren't pushed into your SQL - Orbital applies them after fetching. If you need paging inside the query, express it in the SQL itself. - For simple filtering, prefer standard `table` operations, which stay portable across databases and integrate with query planning. -------------------------------------------------------------------------------- # MongoDB Source: https://orbitalhq.com/docs/describing-data-sources/mongodb Section: Managing data sources -------------------------------------------------------------------------------- Orbital can both read data from a Mongo collection to fetch data as part of a query, and write data back to Mongo. ## Defining a Mongo connection Mongo connections are stored in your [`connections.conf`](/docs/describing-data-sources/configuring-connections) config file, under the `mongo` element. The connection specifies how to connect to a Mongo Database. ```hocon mongo { usersMongo { connectionName=usersMongo connectionParameters { dbName = "sample_mflix" connectionString = "mongodb+srv://orbital:PASSWORD@orbital.xxxx.mongodb.net/?retryWrites=true&w=majority&appName=Orbital" } } } ``` The following configuration options are mandatory under the `connectionParameters` | Config option | Purpose | |--------------------|----------------------------------------------------------------------------------------------------------------------------------------------------| | `dbName` | Name of the Mongo Database. | | `connectionString` | A Valid Mongo Connection String (see [Mongo Db Connection Strings](https://www.mongodb.com/docs/manual/reference/connection-string/) for details.) | ### Defining a Collection mapping Collections are exposed to Orbital using the annotation `@com.orbitalhq.mongo.Collection` on a model. Fields names in the model are expected to align with field names from the collection. Here's an example: ```taxi import com.orbitalhq.mongo.Collection type FirstName inherits String type Password inherits String type Email inherits String @Collection(connection = "usersMongo", collection = "users") model User { name : FirstName password : Password email: Email } ``` The `@Collection` annotation contains the following parameters: | Parameter | Description | |------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------| | connection | The name of a connection, as defined in your [connections configuration file](/docs/describing-data-sources/mongodb#defining-a-mongo-connection). | | collection | The name of the collection | It's possible to use environment variables in these annotations, as described [here](/docs/deploying/configuring-orbital#environment-variables-in-annotations). ### Defining a mongo service ```taxi // 1: Add the required imports import com.orbitalhq.mongo.MongoService // 2: Annotate the service as a `MongoService`. @MongoService( connection = "usersMongo" ) service MongoUsersService { // 3: expose a table operation for a collection model. table user : User[] } ``` ### Mapping the ObjectId Use an `@Id` annotation to define the column that represents the Mongo ObjectId. Here's an example: ```taxi import com.orbitalhq.mongo.Collection type FirstName inherits String type Password inherits String type Email inherits String type MongoObjectId inherits String @Collection(connection = "usersMongo", collection = "users") model UserWithObjectId { @Id _id: MongoObjectId name : FirstName password : Password email: Email } ``` In MongoDB, every document must have an `_id` field that serves as the primary key. If you don't provide one, MongoDB will automatically generate an ObjectId for this field. While you can use other fields as identifiers in your application, the `_id` field will always be present and serves as the document's unique identifier within the collection. Orbital will let you assign an alternative name for the `_id` field in your model, but this isn't recommended, as it can lead to confusion when examining the actual collection, or building [native pipelines](#native-mongo-db-aggregations). You can assign a different name to the column in your model by simply changing the name of the field: ```taxi @Collection(connection = "usersMongo", collection = "users") model User { @Id > userId: MongoObjectId name : FirstName password : Password email: Email } ``` When querying data from `User`, results will show the `_id` column named `userId`. However, it's recommended that you use [projections](/docs/querying/writing-queries#projections) in a TaxiQL query to rename fields, rather than introducing mapping in your model. **Warning: Avoid renaming the id field** Renaming the _id field in your model can lead to confusion, as it's the only field that doesn't align with what the column is actually named in your Mongo collection. We do not reccomend doing this. Alternatively omit the `_id` field from your Taxi model, and use a `@UniqueIndex` if you have one defined. Read more about [Unique indexes in mongo](#mapping-a-mongo-unique-index) ### Mapping a Mongo Unique Index Use the `@UniqueIndex` annotation to map a Mongo unique index to your Taxi model. Here is an example: ```taxi import com.orbitalhq.mongo.Collection > import com.orbitalhq.mongo.UniqueIndex @Collection(connection = "accountsMongo", collection = "accounts") model Account { > @UniqueIndex accountId: AccountId inherits String accountName: AccountName inherits String } ``` The above example annotates the `accountId` field with `@UniqueIndex`, which indicates that the underlying mongo collection `accounts` has a [Mongo unique index](https://www.mongodb.com/docs/manual/core/index-unique/) defined on it. **Note: Orbital doesn't create the index** Marking a field with a `@UniqueIndex` annotation does not create the index on the database - it's expected the index already exists. Orbital considers the field annotated with `@UniqueIndex` when executing `upsert` operations against the Mongo collection. Consider the following `service` definition for the above model: ```taxi @MongoService( connection = "accountsMongo" ) service AccountsDb { table accounts : Account[] @UpsertOperation write operation upsertAccount(Account):Account } ``` When we issue the following query: ```taxi query.taxi given { account : Account = { accountId : "1" , accountName: "My USD Account" } } call AccountsDb::upsertAccount ``` Orbital will insert an account with `accountId = "1"` into the mongo `accounts` collection. When the following query is issued subsequently: ```taxi query.taxi given { account : Account = { accountId : "1" , accountName: "My GBP Account" } } call AccountsDb::upsertAccount ``` Orbital will use the `@UniqueIndex` and update the existing `account` data with the new `accountName` value rather than attempting to insert a new account into the Mongo collection. ### Id vs UniqueIndex The choice between `@Id` and `@UniqueIndex` depends on the design of your Mongo document. Mongo always creates a field called `_id`, which is a primary key. You can store any string-based value in this field, provided it is unique: ```taxi @Collection(connection = "accountsMongo", collection = "accounts") model Account { @Id _id: AccountId inherits String accountName: AccountName inherits String } ``` If you'd rather use a field named something other than `_id` for your primary key, then use `@UniqueIndex`: ```taxi > import com.orbitalhq.mongo.UniqueIndex @Collection(connection = "accountsMongo", collection = "accounts") model Account { > @UniqueIndex accountId: AccountId inherits String accountName: AccountName inherits String } ``` Other considerations: - If a model declares a `@UniqueIndex`, this field is used for matching on upsert operations - Do not combine both `@Id` and `@UniqueIndex` ### Mongo setOnInsert Support To leverage [$setOnInsert](https://www.mongodb.com/docs/manual/reference/operator/update/setOnInsert/) for `@Upsert` operations, add the ` @SetOnInsert` annotation on the field that needs to be set only when inserting a new document into the corresponding Mongo Collection. Here is an example: ```taxi import com.orbitalhq.mongo.Collection import com.orbitalhq.mongo.UniqueIndex import com.orbitalhq.mongo.SetOnInsert @Collection(connection = "accountsMongo", collection = "accounts") model Account { @UniqueIndex accountId : AccountId inherits String currency : Currency inherits String @SetOnInsert insertedAt: InsertedAt inherits Instant = now() updatedAt: UpdateAt inherits Instant = now() } ``` This model indicates that: - `insertedAt` should be set when a new `Account` is inserted into the `accounts` collection - subsequent updates to the same `account` document should not change the initial `insertedAt` value. ## Querying Collections To expose a Mongo database as a source for queries, the database must have a service and table operation exposed for a collection. ```taxi // 1: Add the required imports import com.orbitalhq.mongo.MongoService // 2: Annotate the service as a `MongoService`. @MongoService( connection = "usersMongo" ) service MongoUsersService { // 3: expose a table operation for a collection model. table user : User[] table mongoUsers: UserWithObjectId[] } ``` The `@MongoService` annotation contains the following parameters: | Parameter | Description | |------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------| | connection | The name of a connection, as defined in your[connections configuration file](/docs/describing-data-sources/mongodb#defining-a-mongo-connection). | ## Sample queries ### Fetch everything from a collection ```taxi find { User[] } ``` ### Fetch values by criteria ```taxi find { User[]( FirstName == "Harry" ) } ``` ```taxi find { User[]( FirstName == "Harry" || FirstName == "Joe" ) } ``` ### Using IN and NOT IN operators MongoDB collections support in and not in operators for efficient filtering against multiple values. These operators work with any field type and generate optimized MongoDB queries. **Filtering with `IN` operator** To find documents where a field matches any value in a list: ```taxi // Find users with specific IDs find { User[]( UserId in [1, 2, 3] ) } ``` ```taxi // Find users from specific countries find { User[]( Country in ["US", "UK", "CA"] ) } ``` **Filtering with `NOT IN` operator** To find documents where a field does not match any value in a list: ```taxi // Find users excluding specific IDs find { User[]( UserId not in [1, 2] ) } ``` ```taxi // Find users not from specific countries find { User[]( Country not in ["US", "UK"] ) } ``` **Loading documents based on data from other services** A powerful pattern is to use in operators with data fetched from other services. This enables efficient bulk lookups: // Given these models: ```taxi model Family { id : FamilyId inherits Int members : PersonId[] // Array of person IDs } service FamilyApi { operation getFamily(FamilyId):Family } @Collection(connection = "usersMongo", collection = "people") model Person { @Id id : PersonId inherits Int name : Name inherits String } @MongoService( connection = "usersMongo" ) service UsersDb { table people : Person[] } ``` You can fetch family data from an API, then load all family members from MongoDB in a single query: ```taxi // Load all people belonging to family ID 1 given { FamilyId = 1 } find { Person[]( PersonId in Family::PersonId[] ) } ``` This pattern works with any data transformation. For example, if family members are nested objects: ``` model FamilyMember { id : PersonId } model Family { id : FamilyId inherits Int members : FamilyMember[] } // Extract person IDs from nested objects and load matching people given { FamilyId = 1 } find { Person[]( PersonId in Family::FamilyMember[].map( (FamilyMember) -> PersonId ) ) } ``` **Combining with other conditions** `IN` and `NOT IN` operators can be combined with other MongoDB query operators: ```taxi // Find active users from specific countries find { User[]( Country in ["US", "UK", "CA"] && Status == "ACTIVE" ) } ``` ```taxi // Find users either VIP or from specific regions find { User[]( UserType == "VIP" || Country in ["US", "UK"] ) } ``` ### Collection options _Available since 0.38.0_ You can limit, paginate and sort collection results using [collection options](/docs/querying/collection-options), written alongside your filter criteria: ```taxi find { User[]( Country == "GB", orderBy: SignupDate desc, offset: 20, limit: 10 ) } ``` For MongoDB, Orbital pushes `limit`, `offset` and `orderBy` into the Mongo query (as `sort`, `skip` and `limit`), so the database returns only the documents you asked for. `uniqueBy` is applied by Orbital after fetching. Sorting on a model's `@Id` field maps to Mongo's `_id`, so `orderBy` works over the document key just as it does over any other field. **Note: Null ordering in pushed-down Mongo sorts** When a sort is pushed to Mongo, null and missing values follow MongoDB's own placement - nulls appear **first** when sorting ascending, and **last** when sorting descending. This is the opposite of Orbital's engine-side convention (used by other sources), so the same sort can place nulls differently depending on where it runs. Operations backed by [native MongoDB aggregations](#native-mongo-db-aggregations) don't receive pushdown - collection options are applied by Orbital after the aggregation pipeline has run. Collection options complement native aggregations rather than replacing them: if you're already expressing paging with `$sort` / `$skip` / `$limit` stages inside a pipeline, keep doing that; use collection options for the simple limit / sort / paginate cases on standard `table` operations. See [collection options](/docs/querying/collection-options) for the execution model, and the [Taxi language reference](https://taxilang.org/docs/taxiql/querying#collection-options) for the full syntax. ## Writing data to a collection To expose a database collection for writes, you need to provide a `write operation` in a service. Here's a complete example schema with corresponding `write` operations: ```taxi type FlightCode inherits String type DepartureTime inherits Instant type DepartureAirport inherits String type ArrivalAirport inherits String type MongoObjectId inherits String type AirlineCode inherits String type AirlineName inherits String type StarAllianceMember inherits Boolean model Airline { code: AirlineCode name: AirlineName starAlliance: StarAllianceMember } @Collection(connection = "flightsMongo", collection = "flightInfo") model FlightInfo { code: FlightCode depTime : DepartureTime arrival: ArrivalAirport airline: Airline } @Collection(connection = "flightsMongo", collection = "flightInfo") model FlightInfoWithObjectId { @Id objectId: MongoObjectId? code: FlightCode departure: DepartureAirport arrival: ArrivalAirport airline: Airline } @MongoService( connection = "flightsMongo" ) service FlightsDb { table FlightInfo : FlightInfo[] table mongoFlights: FlightInfoWithObjectId[] // This is effectively Insert as the FlightInfo does not have @Id annotation. @UpsertOperation write operation insertFlight(FlightInfo):FlightInfo // If objectId field is populated, this will update the matching item in the collection. // Otherwise it will insert that provided FlightInfoWithObjectId instance into the collection. @UpsertOperation write operation upsertFlightWithObjectId(FlightInfoWithObjectId):FlightInfoWithObjectId } ``` ## Batching Upsert Operations Rather than upserting one record at a time, you can batch upsert operations, just update your `@UpsertOperation` to specify batch settings: ```taxi @UpsertOperation(batchSize = 100, batchDuration = 10000) write operation upsertFlightWithObjectId(FlightInfoWithObjectId):FlightInfoWithObjectId ``` - `batchDuration` is specified in milliseconds - `batchSize` specifies the number of records to batch before perfomring a write When `UpsertOperation` annotation has these attributes defined, Orbital buffers upserts and flushes the buffer to Mongo each time the buffer reaches `batchSize` or `batchDuration` elapses. ## Deleting data from a collection Orbital supports two types of delete operations for MongoDB collections: simple deletes using `@DeleteOperation` and complex filtered deletes using `@DeleteByQuery`. ### Simple Delete Operations Use `@DeleteOperation` for straightforward deletes that target documents by their `@Id` or `@UniqueIndex` fields. ```taxi type DeleteCount inherits Int model DeleteResult { deletedCount: DeleteCount } @MongoService(connection = "usersMongo") service UserService { @DeleteOperation write operation deleteUser(User): DeleteResult @DeleteOperation write operation deleteUsers(User[]): DeleteResult } ``` The `@DeleteOperation` annotation supports both single document and multiple document deletes: - **Single delete**: Pass a single model instance. Orbital will delete the document matching the `@Id` or `@UniqueIndex` field. - **Multiple delete**: Pass an array of model instances. Orbital will delete all documents that match the `@Id` or `@UniqueIndex` fields of the provided instances. **Requirements for Delete Operations:** - Your model must have either an `@Id` or `@UniqueIndex` annotation - The return type should be `DeleteResult` to get information about how many documents were deleted ### Complex Delete Operations with @DeleteByQuery Use `@DeleteByQuery` for advanced delete operations that require complex filtering using MongoDB query syntax. ```taxi @MongoService(connection = "productsMongo") service ProductService { @DeleteByQuery( collection = "products", filter = '{ categoryId: :categoryId, price: { $gte: :minPrice } }' ) write operation deleteExpensiveInCategory( categoryId: CategoryId, minPrice: Price ): DeleteResult @DeleteByQuery( collection = "products", filter = '{}' ) write operation deleteAllProducts(): DeleteResult } ``` The `@DeleteByQuery` annotation contains the following parameters: | Parameter | Description | |------------|-----------------------------------------------------------------------------------------------------------| | collection | The name of the MongoDB collection to delete from | | filter | MongoDB filter query using standard MongoDB query syntax. Use `:parameterName` for parameter substitution | **Parameter Substitution:** Parameters in the filter query are substituted using the `:parameterName` syntax. Orbital will replace these placeholders with the actual parameter values from your operation. **MongoDB Query Operators:** You can use any valid MongoDB query operators in your filter: ```taxi @DeleteByQuery( collection = "products", filter = '''{ $or: [ { categoryId: :category1 }, { price: { $lt: :maxPrice } } ] }''' ) write operation deleteByComplexCriteria( category1: CategoryId, maxPrice: Price ): DeleteResult ``` ## Native MongoDB aggregations For complex data transformations that go beyond Orbital's standard table operations, you can use native MongoDB aggregation pipelines as an "escape hatch". This allows you to leverage the full power of [MongoDB's aggregation framework](https://www.mongodb.com/manual/aggregation/) while maintaining type safety and semantic integration with your Orbital queries. ### Basic syntax Use the `@CollectionAggregation` annotation with a pipeline object containing the collection name and an array of MongoDB aggregation stages: ```taxi import com.orbitalhq.mongo.CollectionAggregation import com.orbitalhq.mongo.MongoService import com.orbitalhq.mongo.Collection @Collection(connection = "productsMongo", collection = "products") model Product { @Id id: ProductId inherits String name: ProductName inherits String categoryId: CategoryId inherits String price: Price inherits Decimal } @MongoService(connection = "productsMongo") service ProductService { @CollectionAggregation( pipeline = { collection: "products", stages: [ '{ $match: { categoryId: :categoryId } }', '{ $sort: { price: -1 } }' ] } ) operation getProductsByCategory(categoryId: CategoryId): Product[] } ``` ### Multi-line aggregation stages For complex aggregation stages, use triple-quoted strings (`"""`) to span multiple lines for better readability: ```taxi @MongoService(connection = "productsMongo") service ProductService { @CollectionAggregation( pipeline = { collection: "products", stages: [ '{ $sort: { price: -1 } }', """{ $facet: { "results": [ { $skip: :offset }, { $limit: :pageSize } ], "metadata": [ { $count: "totalRecords" } ] } }""", """{ $addFields: { "meta": { "totalRecords": { $arrayElemAt: ["$metadata.totalRecords", 0] }, "page": { $add: [{ $divide: [:offset, :pageSize] }, 1] }, "totalPages": { $ceil: { $divide: [{ $arrayElemAt: ["$metadata.totalRecords", 0] }, :pageSize] } } } } }""" ] } ) operation getProductsWithPagination( offset: Int, pageSize: Int ): ProductPage } ``` ### Parameter binding Parameters are bound using the `:parameterName` syntax and are automatically serialized safely to prevent injection attacks. All Taxi semantic types are supported: ```taxi @CollectionAggregation( pipeline = { collection: "orders", stages: [ '{ $match: { customerId: :customerId, total: { $gte: :minTotal }, orderDate: { $gte: :fromDate } } }', '{ $sort: { orderDate: -1 } }', '{ $limit: :maxResults }' ] } ) operation getRecentHighValueOrders( customerId: CustomerId, minTotal: OrderTotal, fromDate: OrderDate, maxResults: Int ): Order[] ``` ### Common aggregation patterns #### Grouping and summarization ```taxi @CollectionAggregation( pipeline = { collection: "orders", stages: [ '{ $group: { _id: "$customerId", totalSpent: { $sum: "$total" }, orderCount: { $sum: 1 } } }', '{ $project: { customerId: "$_id", totalSpent: 1, orderCount: 1, _id: 0 } }' ] } ) operation getCustomerSummaries(): CustomerSummary[] ``` #### Sorting with pagination ```taxi @CollectionAggregation( pipeline = { collection: "products", stages: [ '{ $match: { inStock: true } }', '{ $sort: { price: 1, name: 1 } }', '{ $skip: :offset }', '{ $limit: :pageSize }' ] } ) operation getProductsPaged(offset: Int, pageSize: Int): Product[] ``` #### Advanced pagination with metadata ```taxi model ProductPage { results: Product[] meta: PaginationMeta } model PaginationMeta { totalRecords: Int page: Int totalPages: Int } @CollectionAggregation( pipeline = { collection: "products", stages: [ '{ $sort: { price: -1 } }', """{ $facet: { "results": [ { $skip: :offset }, { $limit: :pageSize } ], "metadata": [ { $count: "totalRecords" } ] } }""", """{ $addFields: { "meta": { "totalRecords": { $arrayElemAt: ["$metadata.totalRecords", 0] }, "page": { $add: [{ $divide: [:offset, :pageSize] }, 1] }, "totalPages": { $ceil: { $divide: [{ $arrayElemAt: ["$metadata.totalRecords", 0] }, :pageSize] } } } } }""" ] } ) operation getProductsWithPagination(offset: Int, pageSize: Int): ProductPage ``` #### Collection joins with $lookup ```taxi @CollectionAggregation( pipeline = { collection: "orders", stages = [ """{ $lookup: { from: "customers", localField: "customerId", foreignField: "_id", as: "customer" } }""", '{ $unwind: "$customer" }', """{ $project: { orderId: "$_id", customerName: "$customer.name", total: 1 } }""" ] } ) operation getOrdersWithCustomerDetails(): OrderWithCustomer[] ``` ### When to use native aggregations **Note: Choosing between table operations and aggregations** Use `@CollectionAggregation` for complex transformations that require MongoDB-specific features. For simple filtering and projection, prefer Orbital's standard table operations for better portability. Use `@CollectionAggregation` when you need: - Complex data transformations beyond basic filtering and projection - Performance-optimized aggregations that push computation to the database - MongoDB-specific features like `$facet`, `$lookup`, or `$unwind` - Custom calculated fields or conditional logic - Advanced pagination with metadata - Grouping, summarization, or statistical calculations For simple queries like basic filtering (`find { User[](FirstName == "John") }`), prefer Orbital's standard table operations which provide better portability and integration with the broader query system. ## Multi-Collection Transactions with @MultiAggregation The `@MultiAggregation` annotation allows you to execute multiple aggregation pipelines across different collections - optionally within a single MongoDB transaction. This is particularly useful for complex data transformations that need to maintain consistency across multiple collections. **Warning: Transaction limitations** There are limits around what Mongo can do within a transaction. See [Transaction Considerations](#transaction-considerations) for more details. ### Basic syntax ```taxi import com.orbitalhq.mongo.MultiAggregation import com.orbitalhq.mongo.MongoService @MongoService(connection = "usersMongo") service UserService { @MultiAggregation( transactional = true, pipelines = [ { collection: "user_events", stages: [ '{ $match: { processed: false } }', '{ $group: { _id: "$userId", eventCount: { $sum: 1 } } }' ] }, { collection: "user_stats", stages: [ '{ $merge: { into: "user_stats", on: "_id", whenMatched: "merge" } }' ] } ] ) write operation processUserEvents(): ProcessingResult[] } ``` ### Parameters The `@MultiAggregation` annotation accepts the following parameters: | Parameter | Description | Default | |----------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------| | `pipelines` | Array of `AggregationPipeline` objects, each containing a `collection` name and array of aggregation `stages` | - | | `transactional`| Boolean indicating whether the pipelines should execute within a MongoDB transaction. Set to `false` for operations that don't require transactional guarantees or when using features not supported in MongoDB transactions | `true` | ### AggregationPipeline structure Each pipeline in the `pipelines` array has the following structure: ```taxi model AggregationPipeline { collection: String // Name of the MongoDB collection stages: String[] // Array of aggregation stage strings } ``` ### Transaction considerations **Warning: MongoDB Transaction Limitations** MongoDB has specific limitations on what operations can be performed within transactions. Orbital does not validate that your pipeline complies with these restrictions. If you define a pipeline that violates MongoDB's transaction rules, an error will be thrown at runtime. Consult the Mongo docs on [transaction operations](https://www.mongodb.com/docs/manual/core/transactions-operations/) to understand what limitations exist. When `transactional = true`, be aware that MongoDB transactions have limitations: - Cannot use certain aggregation operators like `$merge`, `$out`, or `$lookup` with collections from different databases - Cannot perform operations that would create new collections - All collections must be in the same replica set or sharded cluster Set `transactional = false` when: - Using operations not supported in transactions (like `$merge` or `$out`) - Performance is more important than strict consistency - Working with operations that don't require atomicity ### Advanced example: Event stream processing with conditional merging This example demonstrates a common pattern for processing event streams with conditional merging to avoid race conditions: ```taxi type UserId inherits String type ProfileScore inherits Int type IsEnriched inherits Boolean type IsMerged inherits Boolean @Collection(connection = "usersMongo", collection = "users_primary") model User { @Id id: UserId name: String profileScore: ProfileScore? isEnriched: IsEnriched = false } @Collection(connection = "usersMongo", collection = "user_enrichments") model UserEnrichment { @Id userId: UserId profileScore: ProfileScore isMerged: IsMerged = false } @MongoService(connection = "usersMongo") service UserService { @MultiAggregation( transactional = false, // Using $merge which isn't supported in transactions pipelines = [ { collection: "user_enrichments", stages: [ '{ $match: { isMerged: false } }', """{ $lookup: { from: "users_primary", localField: "_id", foreignField: "_id", as: "user" } }""", '{ $match: { user: { $ne: [] } } }', '{ $unwind: "$user" }', """{ $replaceRoot: { newRoot: { $mergeObjects: [ "$user", { profileScore: "$profileScore", isEnriched: true } ] } } }""", """{ $merge: { into: "users_primary", on: "_id", whenMatched: "merge", whenNotMatched: "discard" } }""" ] }, { collection: "user_enrichments", stages: [ '{ $match: { isMerged: false } }', """{ $lookup: { from: "users_primary", localField: "_id", foreignField: "_id", as: "user" } }""", '{ $match: { user: { $ne: [] } } }', '{ $set: { isMerged: true } }', """{ $merge: { into: "user_enrichments", on: "_id", whenMatched: "merge", whenNotMatched: "discard" } }""" ] } ] ) write operation mergePendingUserUpdates(): MergeResult[] } ``` ### Parameter binding in transactions Parameters work the same way in `@MultiAggregation` as in `@CollectionAggregation`, using the `:parameterName` syntax: ```taxi @MultiAggregation( pipelines = [ { collection: "orders", stages: [ '{ $match: { customerId: :customerId, status: "pending" } }', '{ $set: { status: "processing", updatedAt: new Date() } }' ] }, { collection: "order_audit", stages: [ '{ $addFields: { customerId: :customerId, action: "status_update" } }', '{ $merge: { into: "order_audit" } }' ] } ] ) operation processOrdersForCustomer(customerId: CustomerId): ProcessingResult[] ``` ### When to use @MultiAggregation Use `@MultiAggregation` when you need to: - Execute multiple related aggregation operations atomically - Perform complex data transformations across multiple collections - Implement event stream processing patterns with conditional merging - Maintain data consistency across related collections - Perform bulk operations that span multiple collections Use `@CollectionAggregation` for single-collection operations, and prefer standard table operations for simple queries. ## Sample mutating queries ### Inserting data This example shows inserting data into a Mongo collection. Note that the `objectId` is `null`, allowing Mongo to assign an Id. ```taxi given { movie : FlightInfoWithObjectId = { objectId : null , code : "TK 1989", departure: "IST", arrival: "LHR", airline: { code: "TK", name: "Turkish Airlines", starAlliance: true} } } call FlightsDb::upsertFlightWithObjectId ``` ### Updating data ```taxi given { movie : FlightInfoWithObjectId = { objectId : "7df78ad8902ce46d" , code : "TK 1990", departure: "IST", arrival: "LHR", airline: { code: "TK", name: "Turkish Airlines", starAlliance: true} } } call FlightsDb::upsertFlightWithObjectId ``` ### Deleting a single document ```taxi given { user : User = { id: "user123", name: "John", email: "john@example.com" } } call UserService::deleteUser ``` ### Deleting multiple documents ```taxi given { users : User[] = [ { id: "user123" }, { id: "user456" } ] } call UserService::deleteUsers ``` ### Deleting with complex filtering ```taxi given { categoryId: CategoryId = "electronics", minPrice: Price = 500.00 } call ProductService::deleteExpensiveInCategory ``` ### Streaming data from Kafka into Mongo This example shows streaming stock price updates from a Kafka topic directly into Mongo, updating based off the symbol ```taxi import com.orbitalhq.kafka.KafkaService import com.orbitalhq.kafka.KafkaOperation // Kafka model and service emitting prices: model StockPrice { symbol: StockSymbol inherits String currentPrice : StockPrice inherits Decimal } @KafkaService( connectionName = "market-prices" ) service MyKafkaService { stream prices : Stream } // Mongo model and service for saving prices: @Collection(connection = "stockPricesMongoDb", collection = "stockPrices") closed parameter model SavedStockPrice { @Id symbol : StockSymbol currentPrice : StockPrice timestamp : Instant = now() } @MongoService( connection = "stockPricesMongoDb" ) service StockPricesMongoService { table prices: SavedStockPrice[] @UpsertOperation write operation updatePrice(SavedStockPrice):SavedStockPrice } ``` Given the above, the following query will save updated Kafka ticks into Mongo: ```taxi stream { StockPrice } call StockPricesMongoService::updatePrice ``` ### Building a REST API that reads from Mongo This is a full example, where we create an HTTP endpoint accepting a `GET` request with a ticker symbol. We'll use the same model and services declared in [Streaming data from Kafka to Mongo](#streaming-data-from-kafka-into-mongo), to avoid redeclaring them here. ```taxi @HttpOperation(url = "/api/q/stockPrices/{symbol}", method = "GET") query FetchStockPrices(@PathVariable("symbol") symbol:StockSymbol) { find { SavedStockPrice( StockSymbol == symbol) } } ``` ## Deleting data from collections Orbital supports two approaches for deleting data from MongoDB collections: simple delete operations based on model structure, and native MongoDB delete operations for advanced filtering. ### Simple delete operations Use the `@DeleteOperation` annotation for straightforward delete operations based on model IDs: ```taxi import com.orbitalhq.mongo.DeleteOperation @MongoService(connection = "productsMongo") service ProductService { // Delete single document by ID @DeleteOperation write operation deleteOneProduct(Product): DeleteResult // Delete multiple documents by ID array @DeleteOperation write operation deleteManyProducts(Product[]): DeleteResult } ``` The operation signature determines the behavior: - **Single model parameter** → `deleteOne()` operation using `@Id` or `@UniqueIndex` field - **Array parameter** → `deleteMany()` operation using `$in` query on ID fields Both operations return a `DeleteResult` model containing the count of deleted documents. ### Native MongoDB delete operations For complex delete operations that require advanced filtering, use the `@DeleteByQuery` annotation with native MongoDB filter syntax: ```taxi import com.orbitalhq.mongo.DeleteByQuery @MongoService(connection = "productsMongo") service ProductService { @DeleteByQuery( collection = "products", filter = '{ categoryId: :categoryId, price: { $gte: :minPrice } }' ) write operation deleteExpensiveInCategory( categoryId: CategoryId, minPrice: Price ): DeleteResult } ``` ### Parameter binding in delete filters Parameters use the `:parameterName` syntax and are automatically serialized safely to prevent injection attacks: ```taxi @DeleteByQuery( collection = "products", filter = '''{ $or: [ { categoryId: :category1 }, { price: { $lt: :maxPrice } } ] }''' ) write operation deleteByComplexCriteria( category1: CategoryId, maxPrice: Price ): DeleteResult ``` ### Multi-line delete filters Use triple-quoted strings for complex filter expressions: ```taxi @DeleteByQuery( collection = "orders", filter = """{ $and: [ { customerId: :customerId }, { status: { $in: ["cancelled", "failed"] } }, { createdDate: { $lt: :cutoffDate } } ] }""" ) write operation cleanupFailedOrders( customerId: CustomerId, cutoffDate: Date ): DeleteResult ``` ### Delete operation examples **Delete all documents** (empty filter): ```taxi @DeleteByQuery( collection = "temp_data", filter = '{}' ) write operation clearTempData(): DeleteResult ``` **Simple ID-based deletion**: ```taxi given { Product = { id: "laptop-1" } } call ProductService::deleteOneProduct ``` **Bulk deletion by IDs**: ```taxi given { Product[] = [ { id: "laptop-1" }, { id: "phone-1" } ] } call ProductService::deleteManyProducts ``` **Conditional deletion with multiple criteria**: ```taxi given { categoryId: CategoryId = "electronics" minPrice: Price = 500.00 } call ProductService::deleteExpensiveInCategory ``` ### When to use each approach Use `@DeleteOperation` when: - Deleting specific documents by their ID or unique index - Working with arrays of models to delete multiple specific documents - You want type-safe, rename-proof operations tied to your model structure Use `@DeleteByQuery` when: - Complex filtering criteria involving multiple fields - Using MongoDB query operators (`$gte`, `$or`, `$in`, etc.) - Conditional deletion based on calculated values or ranges - Bulk deletion operations that can't be expressed as simple ID lookups -------------------------------------------------------------------------------- # Kafka Source: https://orbitalhq.com/docs/describing-data-sources/kafka Section: Managing data sources -------------------------------------------------------------------------------- Orbital can both read data from a Kafka topic to stream data as part of a query, and write data back to Kafka. ## Defining a connection to your Kafka broker Kafka connections are stored in your `connections.conf` config file, under the `kafka` element. The connection specifies how to connect to a broker - but not the specifics of actual topics. (Those are specified on the `@KafkaOperation()` metadata on a taxi operation definition.) ```hocon kafka { my-kafka-connection { connectionName=my-kafka-connection connectionParameters { brokerAddress="localhost:29092,localhost:39092" groupId="orbital" } } another-kafka-connection { connectionName=another-kafka-connection connectionParameters { brokerAddress="localhost:29092,localhost:39092" groupId="orbital" } } } ``` The following configuration options are supported under the `connectionParameters` | Config option | Purpose | |---------------|--------------------------------------------------------------------------------| | `brokers` | A comma-separated list of broker addresses when connecting to the Kafka broker | | `groupId` | The groupId to use when connecting to a topic | ### Additional Kafka connection properties In addition to `brokers` and `groupId` (which are shorthand for `bootstrap.servers` and `group.id` respectively), you can provide any of the Kafka [consumer](https://docs.confluent.io/platform/current/installation/configuration/consumer-configs.html) and [producer](https://docs.confluent.io/platform/current/installation/configuration/producer-configs.html) config settings. For example, to define a secure connection: ```hocon kafka { my-kafka-connection { connectionName=my-kafka-connection connectionParameters { brokerAddress="localhost:29092,localhost:39092" groupId="orbital" "security.protocol"="SASL_PLAINTEXT" "sasl.mechanism"="PLAIN" "sasl.jaas.config"="org.apache.kafka.common.security.plain.PlainLoginModule required username='myKafkaUser' password="${KAFKA_PASSWORD}";" } } } ``` When using Kafka connections to consume data, specifying an appropriate `groupId` is important, and impacts the behaviour of how messages are received. If you are experiencing [long delays at the start of a query](#long-connection-times-when-starting-a-query), or [no results returned](#query-doesn-t-return-any-data-even-though-data-exists), you may need to modify your `groupId`. You can set `groupId` in `connectionParameters` to specify the group id of your Orbital consumer. ```hocon kafka { my-kafka-connection { connectionName=my-kafka-connection connectionParameters { brokerAddress="localhost:29092,localhost:39092" groupId="myGroupId" } } } ``` Read more about consumer group id's in the [Kafka documentation](https://www.confluent.io/blog/configuring-apache-kafka-consumer-group-ids/) ### Troubleshooting connection problems #### Using env variables in auth settings It's common in Kafka configs that usernames and passwords are embedded within settings, like `sasl.jaas.config`. It's recommended that usernames and passwords are read from env variables, which can lead to tricky [HOCON concatenation issues](https://github.com/lightbend/config/issues/113). The solution is to rely on HOCON concatenation. eg: ```HOCON "sasl.jaas.config"="org.apache.kafka.common.security.plain.PlainLoginModule required username="${KAFKA_USERNAME}" password="${KAFKA_PASSWORD}";" ``` Pay close attention to how the `"` in the setting are paired - ie., no escaping, and no using `'` to nest inner values. For example: ```hocon kafka { my-kafka-connection { // ... omitted for brevity connectionParameters { "sasl.jaas.config"="org.apache.kafka.common.security.plain.PlainLoginModule required username='myKafkaUser' password="${KAFKA_PASSWORD}";" } } } ``` #### Query doesn't return any data (even though data exists) You might have more subscribers with the 'groupId' than the configured partition count for the topic. In this case, your Orbital instance won't get any data from the topic. Please set the groupId in your kafka and specify a new groupId: ```hocon kafka { my-kafka-connection { connectionName=my-kafka-connection connectionParameters { brokerAddress="localhost:29092,localhost:39092" groupId="myGroupId" } } } ``` #### Long connection times when starting a query When multiple Orbital instances connect (and disconnect) to the same Kafka broker with the same `groupId` (such as the default setting of `groupId = Orbital`), this can trigger on the Kafka broker. As a result, long delays are sometimes observed before data is streamed from Kafka, while the cluster is rebalancing. Resolve this by setting the `groupId` for your kafka connection to a unique value in your connections.conf ## Querying a Kafka topic A Kafka topic can be queried using a standard Taxi query, including enriching data from Kafka with other services, and [joining multiple streams together](#joining-multiple-streams) ### Exposing a topic Kafka topics are declared in taxi as simple Operations which return a Stream of data. ```taxi // 1: Add the required imports import com.orbitalhq.kafka.KafkaService import com.orbitalhq.kafka.KafkaOperation // 2: Annotate the service as a `KafkaService`. @KafkaService( connectionName = "market-prices" ) service MyKafkaService { // 3: Annotate the operation as a `@KafkaOperation` @KafkaOperation( topic = "stockPrices", offset = "earliest" ) // The topic is declared as a stream, and returns a Stream<> stream streamStockPrices:Stream } ``` 1. Add the required imports 1. Annotate the service with `@KafkaService` * The `connectionName` parameter should match a connection defined in the [connections config file](#defining-a-connection-to-your-kafka-broker). 1. Annotate the operation as a `@KafkaOperation`, specifying the topic and offset. 1. The return type should be a `Stream<>` of the defined message type. It's possible to use environment variables in these annotations, as described [here](/docs/deploying/configuring-orbital#environment-variables-in-annotations). ### Keys, Headers and Metadata The message key, message headers, and kafka-specific metadata (such as offset, partition and timestamp) can be provided into a Taxi model using annotations: #### Message Key The message key can be accessed using the `com.orbtalhq.kafka.KafkaMessageKey` annotation on a field: ```taxi > import com.orbitalhq.kafka.KafkaMessageKey model Movie { // The key from Kafka will be read into the id property > @KafkaMessageKey id : MovieId inherits String title : Title inherits String } // Rest of the kafka topic declaration continues... @KafkaService( connectionName = "moviesConnection" ) service MovieService { @KafkaOperation( topic = "movies", offset = "earliest" ) stream streamMovieQuery:Stream } ``` #### Kafka Metadata Kafka metadata (such as offset, partition and timestamp) can be accessed using the `com.orbitalhq.kafka.KafkaMessageMetadata` annotation on a field. `KafkaMessageMetadata` takes a single parameter, which is the metadata type you wish to read. Defined by the enum type `KafkaMetadataType`, the following values are defined: ```taxi enum KafkaMetadataType { Partition, Offset, Timestamp, TimestampType Topic } ``` For example: ```taxi > import com.orbitalhq.kafka.KafkaMessageMetadata > import com.orbitalhq.kafka.KafkaMetadataType model Movie { > @KafkaMessageMetadata(KafkaMetadataType.Offset) offset : Int > @KafkaMessageMetadata(KafkaMetadataType.Timestamp) timestamp : Long > @KafkaMessageMetadata(KafkaMetadataType.TimestampType) timestampType : String > @KafkaMessageMetadata(KafkaMetadataType.Partition) partition : Int > @KafkaMessageMetadata(KafkaMetadataType.Topic) topic : String // Other fields continue... title : Title inherits String } // Rest of the kafka topic declaration continues... @KafkaService( connectionName = "moviesConnection" ) service MovieService { @KafkaOperation( topic = "movies", offset = "earliest" ) stream streamMovieQuery:Stream } ``` #### Headers Kafka supports including arbitrary message headers along with the message - which are often used for things like correlation keys, etc. These headers can be accessed using the `com.orbitalhq.kafka.KafkaHeader` annotation: ```taxi import com.orbitalhq.kafka.KafkaHeader model Movie { > @KafkaHeader("correlationId") correlationId : CorrelationId inherits String title : Title inherits String } // Rest of the kafka topic declaration continues... @KafkaService( connectionName = "moviesConnection" ) service MovieService { @KafkaOperation( topic = "movies", offset = "earliest" ) stream streamMovieQuery:Stream } ``` ### Controlling deserialization Message deserialization is defined by the model type being exposed. By default, models are expected to be JSON. However, this can be controlled by annotating the model with a format annotation. Two common formats - [Protobuf](/docs/describing-data-sources/protobuf/) and [Avro](/docs/data-formats/avro) are supported. ### Example queries #### Streaming data from Kafka ```taxi // Invokes the `streamStockPrices` stream declared above stream { StockPrice } ``` #### Enrich data from Kafka with other data sources Data from a Kafka topic can be projected to enrich it with data from other sources. Data requested that is not present on the Kafka payload is looked up from other sources, using Orbital's standard projections. ```taxi stream { StockPrice } as { ticker : StockTicker // avaialble on the Kafka topic lastTradedPrice : LastTradedPrice // Looked up from another data source }[] ``` #### Filtering kafka streams This examples reads all messages from the Kafka topic, but only emits those with a stock ticker of `AAPL` on the resulting stream: ```taxi stream { StockPrice.filterEach( ( StockTicker ) -> StockTicker == 'AAPL' ) } ``` #### Streaming from Kafka to a database Streams from Kafka can be inserted into a database (or any other writable source— such as [Hazelcast](/docs/describing-data-sources/hazelcast#writing-data-to-hazelcast) or [Dynamo]()) using a mutating query. As with all mutating queries, it's not necessary for the data from Kafka to align with the format of the data being written to the data destination. Orbital will automatically adapt the query result to the required persistence format, which may involve projections and even calling additional services if needed. ```taxi // First, ensure that your data destination exposes a writeable data source // Full config omitted for brevity service MyDatabaseService { @UpsertOperation write operation updateStockPrices(StockPriceSnapshot):StockPriceSnapshot } // Then, define a streaming query. // In this example, the data format for StockPrice coming off of Kafka // is different from the data being written to our database (StockPriceSnapshot) // so Orbital transforms the data automatically stream { StockPrice } call MyDatabaseService::updateStockPrices ``` ### Joining multiple streams It is possible to join multiple Kafka streams together as described in [joining streams](/docs/querying/streaming-data#joining-streams) ### Overriding consumer group and offset By default, the consumer group is defined on the operation: ```taxi @KafkaService( connectionName = "moviesConnection" ) service MovieService { @KafkaOperation( topic = "movies", offset = "earliest" ) stream streamMovieQuery:Stream } ``` and the consumer group is defined in your connections.conf: ```hocon kafka { my-kafka-connection { > connectionName="moviesConnection" connectionParameters { brokerAddress="localhost:29092,localhost:39092" > groupId="orbital" } } } ``` However, these can be overridden on a per-query basis, by using a `@StreamConsumer` query: ```taxi > // 1: don't forget the import > import com.orbitalhq.streams.StreamConsumer > // changes the consumer group id on a Kafka subscription and offset > @StreamConsumer( id = "my-consumer" , offset = "latest" ) query MySavedQuery { stream { Movie } } ``` Likewise, for an adhoc query: ```taxi > // 1: don't forget the import > import com.orbitalhq.streams.StreamConsumer > // changes the consumer group id on a Kafka subscription and offset > @StreamConsumer( id = "my-consumer" , offset = "latest" ) stream { Movie } ``` ## Writing to a Kafka topic To make a topic writable, declare a `write operation` in a Kafka service: ```taxi // 1: Add the required imports import com.orbitalhq.kafka.KafkaService import com.orbitalhq.kafka.KafkaOperation // 2: Annotate the service as a `KafkaService`. @KafkaService( connectionName = "market-prices" ) service MyKafkaService { // ...other kafka topics omitted... // 3: Annotate the operation as a `@KafkaOperation` @KafkaOperation( topic = "stockPrices", offset = "earliest" ) // The operation is declared as a write operation write operation publishStockPrice(StockPrice):StockPrice } ``` ### Examples #### Writing a static value onto a Kafka topic ```taxi given { stockPrice : StockPrice = { symbol : 'AAPL', price : 12.00203, } } call MyKafkaService::publishStockPrice ``` #### Consuming from one Kafka topic, and writing to another topic To stream data from a Kafka topic, enrich and republish ```taxi @KafkaService( connectionName = "market-prices" ) service MyKafkaService { @KafkaOperation( topic = "stockPrices" ) stream prices : Stream @KafkaOperation( topic = "enrichedPrices" ) write operation publishEnrichedPrices(EnrichedStockPrice):EnrichedStockPrice } ``` The following query will consume from the `stockPrices` topic, and for each message, transform to an `EnrichedStockPrice`, invoking any other services required to inject required data. ```taxi stream { StockPrice } // The input parameter to publishEnrichedPrices // is a EnrichedStockPrice, so each incoming // StockPrice message is transformed to a // EnrichedStockPrice payload, and published onto the // enrichedPrices topic call MyKafkaService::publishEnrichedPrices ``` #### Building a REST API that publishes to Kafka This is a full example, where we create an HTTP endpoint accepting a `POST` request with a ticker symbol. ```taxi type StockSymbol inherits String // The inbound request sent over HTTP requesting a stock price model StockPricePublicationRequest { ticker : StockSymbol } // The message we'll be publishing to Kafka parameter model StockPriceUpdate { ticker : StockSymbol currentPrice : StockPrice } closed model CurrentStockPrice { price : StockPrice } service PriceService { @HttpOperation(url="http://fakeurl/prices/{symbol}", method = "GET") operation getCurrentPrice(@PathVariable("symbol") symbol:StockSymbol):CurrentStockPrice } @KafkaService( connectionName = "market-prices" ) service MyKafkaService { @KafkaOperation( topic = "stockPrices", offset = "earliest" ) write operation publishStockPrice(StockPriceUpdate):StockPriceUpdate } @HttpOperation(path = "/api/q/publishStockPrice", method = "POST") query MySavedQuery(@RequestBody request:StockPriceRequest) { given { request } call MyKafkaService::publishStockPrice } ``` The above example works as follows: * A `POST` request is sent to `/api/q/publishStockPrice` with a body of: ```json { "ticker" : "AAPL" } ``` * The query asks for `publishStockPrice` to be called, which means a `StockPriceUpdate` must be constructed * To build a `StockPriceUpdate`, the `currentPrice : StockPrice` is required, which is available from the `price` field of `CurrentStockPrice` object, returned from `getCurrentPrice` * A request to `http://fakeurl/prices/AAPL` is issued to discover the current stock price, returning: ```json { "price" : 117.34 } ``` * Finally, we have enough information to build a `StockPriceRequest`, so the message is published to Kafka: ```json { "ticker" : "AAPL", "currentPrice" : 117.34 } ``` -------------------------------------------------------------------------------- # Hazelcast Source: https://orbitalhq.com/docs/describing-data-sources/hazelcast Section: Managing data sources -------------------------------------------------------------------------------- In addition to [using Hazelcast as a cache](/docs/querying/caching#using-an-external-cache), Orbital can use Hazelcast as a source for reading and writing data, as well as a streaming data source. ## Adding a Hazelcast data source To enable Hazelcast as a data source, first [define a connection](/docs/describing-data-sources/configuring-connections) in your `connections.conf` file. The simplest configuration simply needs a list of addresses to connect to: ```hocon connections.conf hazelcast { myHazelcast { connectionName = myHazelcast addresses = ["localhost:5701"] } } ``` ### Advanced configuration Hazelcast has a number of connection parameters, which you may wish to tune. You can specify a full Hazelcast client configuration file, and provide the path. For [XML](https://github.com/hazelcast/hazelcast/blob/master/hazelcast/src/main/resources/hazelcast-client-default.xml) based configuration: ```hocon hazelcast { prodHazelcastCluster { connectionName = myHazelcast xmlConfig="hazelcast-config.xml" } } ``` If your Hazelcast client config is in [yaml](https://github.com/hazelcast/hazelcast/blob/master/hazelcast/src/main/resources/hazelcast-client-default.yaml) format, you can define it your connection as: ```hocon hazelcast { prodHazelcastCluster { connectionName = myHazelcast yamlConfig="hazelcast-config.yaml" } } ``` Paths can be either absolute (located on the server where Orbital is deployed), or relative (resolved relative to the `conncections.conf` file declaring the Hazelcast connection). The Hazelcast docs provide more information on [declarative configuration settings](https://docs.hazelcast.com/hazelcast/5.4/configuration/configuring-declaratively) and provide [full examples](https://github.com/hazelcast/hazelcast/blob/master/hazelcast/src/main/resources/hazelcast-client-full-example.xml). ## Supported formats When values are written to Hazelcast, they must be serialized. The following formats are supported. Note that because Orbital does not have access to custom Java classes, serialization formats that require classloading (such as `Serializable`, `Externalizable` and Custom Serialization) are not supported ### Compact (preferred) [Compact Serialization](https://docs.hazelcast.com/hazelcast/5.3/serialization/compact-serialization) is the preferred format when using Orbital to read/write to Hazelcast. To enable Compact serialization, add a `@CompactObject` annotation to your Taxi definition: ```taxi > import com.orbitalhq.hazelcast.CompactObject import com.orbitalhq.hazelcast.HazelcastMap @HazelcastMap(name = "films") > @CompactObject closed model Film { @Id filmId : FilmId inherits Int // ...omitted... } ``` ### JsonValue Hazelcast's [JsonValue](https://docs.hazelcast.com/hazelcast/5.3/serialization/serializing-json) is supported for working with existing caches that have stored data using JsonValue. Note - if you are both writing and reading the data using Orbital, it is recommended to use [Compact](#compact-preferred), which has better performance. ## Writing data to Hazelcast To write data into a Hazelcast Map, define a Hazelcast Service in your taxi project, with a write operation: ```taxi import com.orbitalhq.hazelcast.HazelcastService import com.orbitalhq.hazelcast.HazelcastMap import com.orbitalhq.hazelcast.UpsertOperation import com.orbitalhq.hazelcast.CompactObject // The HazelcastService annotation should specify the // name of the connection defined in your connections.conf @HazelcastService(connectionName = "myHazelcastConnection") service HazelcastService { // Expose a write operation, and annotate it // with UpsertOperation, to define the writing behaviour @UpsertOperation write operation upsert(Film):Film } type Language inherits String // Define the name of the map to use. If not present in Hazelcast, // the map wil be created @HazelcastMap(name = "films") // specify the serialization format @CompactObject closed model Film { // A field annotated with @Id becomes the key. // Composite keys are not supported @Id filmId : FilmId inherits Int title : Title inherits String languages: Language[] } ``` With your map declared, use a mutation query to insert data: ### Examples #### Inserting a static value into Hazelcast ```taxi // inserting a static value into Hazelcast given { film : Film = { filmId : 123, title : "Star Wars", languages : ["English" , "American" ] } } call HazelcastService::upsert ``` #### Storing a Query Result in Hazelcast To write the outcome of a query to Hazelcast, it's not necessary for the query's result format to align with the format of the persisted value. Orbital will automatically adapt the query result to the required persistence format, which may involve projections and even calling additional services if needed. ```taxi find { NetflixFilms[] } as { // projection not shown } call HazelcastService::upsert ``` #### Streaming data into Hazelcast ```taxi stream { FilmUploadedEvent } // Each FilmUploadedEvent is projected into a Film // which is the input parameter to `upsert` call HazelcastService::upsert ``` ## Querying data from Hazelcast Orbital supports querying from Hazelcast using both direct key lookups, and rich query criteria. ### Defining a map to query To query a map, you first define a `service` that exposes your Hazelcast Map. Maps are exposed using taxi's `table` declaration, as this indicates a data source that supports rich querying. Here's a complete example: ```taxi import com.orbitalhq.hazelcast.HazelcastService import com.orbitalhq.hazelcast.HazelcastMap import com.orbitalhq.hazelcast.CompactObject // The HazelcastService annotation should specify the // name of the connection defined in your connections.conf @HazelcastService(connectionName = "myHazelcastConnection") service HazelcastService { // Table is a shorthand to declare // a data source that supports rich querying. table films : Film[] } // Define the name of the map to query. @HazelcastMap(name = "films") // specify the serialization format @CompactObject closed model Film { // A field annotated with @Id becomes the key, which // is used when performing key lookups @Id filmId : FilmId inherits Int title : Title inherits String languages: Language[] } ``` ### Writing queries Once a Hazelcast map is exposed, it can be queried as a standard data source, including used as a data source when projecting and joining data from other sources (such as APIs, Kafka topics, or Databases). Here are some sample queries: #### Fetching everything from a map ```taxi find { Film[] } ``` #### Fetching a value with a specific key If criteria is defined against the key (as defined using an `@Id` annotation), then a key lookup is performed: ```taxi // Assuming FilmId is annotated as @Id in the Film model // as shown... model Film { @Id filmId : FilmId inherits Int // ..snip.. } // Elsewhere, writing a query... find { Film( FilmId === 123 ) } ``` #### Fetching values using criteria ```taxi // find all films with a FilmId < 105 find { Film[]( FilmId < 105 ) } // find all films released after 2019 with the title Star Wars find { Film[]( ReleaseYear < 2019 && Title == "Star Wars" ) } ``` ### Collection options _Available since 0.38.0_ You can limit, paginate and sort results using [collection options](/docs/querying/collection-options), written alongside your filter criteria: ```taxi find { Film[]( orderBy: Title asc, limit: 10 ) } ``` For Hazelcast, only `limit` is pushed down - applied as a paging predicate over the map, which returns entries in **key order**. `offset`, `orderBy` and `uniqueBy` are applied by Orbital after fetching. Because the pushed-down `limit` is key-ordered, an `orderBy` is always honoured engine-side rather than being pushed to Hazelcast. | Option | Behaviour against Hazelcast | |------------|-----------------------------------------------| | `limit` | Pushed down as a key-ordered paging predicate.| | `offset`, `orderBy`, `uniqueBy` | Applied by Orbital after fetching. | When a query carries an `orderBy` and Orbital is running with distributed (Hazelcast) projection, the ordered query's projection runs locally, to preserve the ordering. On [streaming Hazelcast queries](#streaming-data-from-hazelcast), only `limit` is supported - `orderBy`, `offset` and `uniqueBy` are rejected, as they have no meaning over an unbounded stream. See [collection options](/docs/querying/collection-options) for the execution model, and the [Taxi language reference](https://taxilang.org/docs/taxiql/querying#collection-options) for the full syntax. ## Streaming data from Hazelcast Hazelcast Map's can be treated as data streams, where inserts or updates are created as streams of events which can be queried using Orbital. The following events trigger the current state of the record to be written to the event stream: * Entry Added * Entry Updated ### Declaring a Map as a Stream To stream updates from a map, you first define a `service` that exposes your Hazelcast Map as stream Here's a complete example: ```taxi import com.orbitalhq.hazelcast.HazelcastService import com.orbitalhq.hazelcast.HazelcastMap import com.orbitalhq.hazelcast.CompactObject // The HazelcastService annotation should specify the // name of the connection defined in your connections.conf @HazelcastService(connectionName = "myHazelcastConnection") service HazelcastService { stream films : Stream } // Define the name of the map to query. @HazelcastMap(name = "films") // specify the serialization format @CompactObject closed model Film { // A field annotated with @Id becomes the key, which // is used when performing key lookups @Id filmId : FilmId inherits Int title : Title inherits String languages: Language[] } ``` ### Writing streaming queries Once a Hazelcast map is exposed as a stream, it can be queried as a standard data source, including used as a data source when projecting and joining data from other sources (such as APIs, Kafka topics, or Databases). Below are some sample queries. It's useful when testing to combine this with [writing data to Hazelcast](#writing-data-to-hazelcast) to trigger change events which produce values on the data stream. Here are some sample queries: #### Stream all updates from a map ```taxi stream { Film } ``` #### Stream all updates from a map, and enrich with data from other sources ```taxi stream { Film } as { id : FilmId reviewScore : FilmReviewScore // not present in the map, will be looked up from another data source }[] ``` #### Stream only specific events from a map ```taxi // Only provide updates on Films whose FilmId is less than 300 stream { Film.filterEach( (FilmId) -> FilmId < 300 ) } ``` -------------------------------------------------------------------------------- # AWS services Source: https://orbitalhq.com/docs/describing-data-sources/aws-services Section: Managing data sources -------------------------------------------------------------------------------- ## Defining an AWS connection AWS connections are stored in your [`connections.conf`](/docs/describing-data-sources/configuring-connections) config file, under the `aws` element. You can choose to define single `aws` entry can define the credentials for all AWS services, or define multiple separate aws entries. If Orbital is deployed in AWS (ECS/EC2), then Orbital will use the [AWS default credentials provider](https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/credentials.html#credentials-default) by default. This means you can configure the access credentials and region with environment variables (`AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY` & `AWS_DEFAULT_REGION`). When running Orbital in AWS (e.g. ECS/EC2) it'll also automatically pick up the role used to run the service and use that. ```hocon aws { my-aws-account { connectionName=my-aws-account // Optional Parameter. When not provided Orbital will use the [AWS default credentials provider](https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/credentials.html#credentials-default) by default. accessKey=${?AWS_ACCESS_KEY_ID} // Optional Parameter. When not provided Orbital will use the [AWS default credentials provider](https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/credentials.html#credentials-default) by default. secretKey=${?AWS_SECRET_ACCESS_KEY} // Mandatory region=${AWS_REGION} // Optional parameter for development and testing purposes to point to a different endpoint such as a LocalStack installation. endPointOverride=${?AWS_ENDPOINT_OVERRIDE} } } ``` ### Testing with Localstack You can point Orbital at an AWS mock running on [Localstack](https://localstack.cloud/) by specifying the `endPointOverride` property in the connection. ## DynamoDb DynamoDb services can only be declared in Taxi. Querying against key is supported, as is write operations. To expose a DynamoDb service, the following two annotations are required: * Add a ` @com.orbitalhq.aws.dynamo.Table` to a model * Add a `@com.orbitalhq.aws.dynamo.DynamoService` annotation to the service ```taxi import com.orbitalhq.aws.dynamo.DynamoService import com.orbitalhq.aws.dynamo.Table // connectionName must match the defined connection. @Table( connectionName = "myAws" , tableName = "reviews" ) model Review { @Id movieId : MovieId score: ReviewScore inherits Int } @DynamoService service DynamoService { // Exposes query operations that return Review[] table reviews: Review[] // allows upsert calls @UpsertOperation write operation saveReview(Review):Review } ``` It's possible to use environment variables in these annotations, as described [here](/docs/deploying/configuring-orbital#environment-variables-in-annotations). ## Lambda Orbital can invoke a Lambda, either to discover data in a query, or in a [mutation](/docs/querying/mutations). A service must have the `AwsLambdaService` annotation, passing the configured connection. Operations have the `LambdaOperation` annotation. ```taxi import com.orbitalhq.aws.lambda.AwsLambdaService import com.orbitalhq.aws.lambda.LambdaOperation @AwsLambdaService( connectionName = "myAws" ) service MovieDb { @LambdaOperation(name = "streamingprovider") operation movieQuery(@RequestBody StreamingProviderRequest): StreamingProviderResponse } ``` ## S3 Orbital can use S3 as a source for both reading and writing data ### Reading from S3 To declare a data source that exposes data, take the following steps: - Declare a service annotated with `@S3Service`. - Within that service, declare an operation with `@S3Operation`, providing the bucket name - The operation should take a single parameter, of type `FilenamePattern`, indicating the filename (or pattern) to read from **Note: Remember the imports** Be sure to either add the imports for `S3Service`, `S3Operation` and `FilenamePattern` (as shown below), or use their fully qualified names. Here's an example: ```taxi import com.orbitalhq.aws.s3.S3Service import com.orbitalhq.aws.s3.S3Operation import com.orbitalhq.aws.s3.FilenamePattern @S3Service(connectionName = "MyAwsConnection") service AwsBucketService { @S3Operation(bucket = "MyBucket") operation readBucket(filename:FilenamePattern = "films.csv"):Film[] } ``` #### Filename patterns when reading from S3 When reading from S3 files, you can either specify a single filename (e.g., `films.csv`), or a pattern - such as `film*.csv`, `*.csv`, or even just `*` to read everything in the bucket. When working with a pattern, Orbital will read and combine all matching files, treating them as a single response. For information about supported file formats, see [file formats](#s3-file-formats). ### Examples - Reading from S3 Each of the below examples will work with the following model of content stored on S3, so we're defining it once here for brevity: ```taxi tradeSummary.taxi import com.orbitalhq.formats.Csv import com.orbitalhq.aws.s3.S3Service import com.orbitalhq.aws.s3.S3Operation import com.orbitalhq.aws.s3.FilenamePattern // Define the format we're reading. // This is a CSV file, so the @Csv annotation is used. // (Note it's also imported at the top of the file) @Csv model TradeSummary { symbol : Symbol inherits String open : OpenPrice inherits Decimal high : HighPrice inherits Decimal close : ClosePrice inherits Decimal } ``` #### Reading a single file ```taxi s3services.taxi @S3Service(connectionName = "MyAwsConnection") service AwsBucketService { @S3Operation(bucket = "MyTrades") operation readBucket(filename:FilenamePattern = "trades.csv"):TradeSummary[] } ``` With the above schema, we can issue a simple query to return the contents of `trade.csv`: ```taxi query.taxi find { TradeSummary[] } ``` #### Reading the contents of multiple files ```taxi s3services.taxi @S3Service(connectionName = "MyAwsConnection") service AwsBucketService { @S3Operation(bucket = "MyTrades") // This will read all files ending in csv present in the bucket > operation readBucket(filename:FilenamePattern = "*.csv"):TradeSummary[] } ``` With the above schema, we can issue a simple query to return the contents of all `*.csv` files in the bucket: ```taxi query.taxi find { TradeSummary[] } ``` #### Reading files and exposing as an HTTP endpoint Using the same service definition as shown above, we can expose the contents of our `*.csv` files with a query published as an HTTP endpoint: ```query.taxi import taxi.http.HttpOperation @HttpOperation(url = "/api/q/trades", method = "GET") find { TradeSummary[] } ``` For more information, read our guide on [publishing queries as endpoints](/docs/querying/queries-as-endpoints). #### Reading files and publishing to Kafka This example shows how to read a CSV file from S3, and publish each row as an individual message to Kafka, as a JSON object First, we'll declare our Kafka broker and associated message format: ```taxi kafka.taxi import com.orbitalhq.kafka.KafkaService import com.orbitalhq.kafka.KafkaOperation // The message format we're publishing to Kafka. // Because there's no format defined, it's JSON by default model TradeSummaryEvent { ticker : Symbol // field names and structure are different, but the // types are the same as on our source model. prices: { openPrice : OpenPrice highPrice : HighPrice closePrice : ClosePrice } } // Declare our Kafka service and operation @KafkaService( connectionName = "market-prices" ) service MyKafkaService { // Define an operation that writes to Kafka @KafkaOperation( topic = "tradeRecords" ) write operation publishTrades(TradeSummaryEvent):TradeSummaryEvent } ``` With the above in place, we can write a query that reads from S3, transforms from CSV to our JSON format, and writes it out to Kafka. ```taxi query.taxi find { TradeSummary[] } call MyKafkaService::publishTrades ``` In the above example, Orbital detects that the inbound model (`TradeSummary`) is different from the destination format (`TradeSummaryEvent`) and handles the transformation for us. In our example, that's simple converting from CSV to JSON and restructuring the message. However, the transformation could be richer, doing tasks such as calling services to discover data. Finally, we might want to expose an HTTP POST operation to trigger this update: ```taxi query.taxi import taxi.http.HttpOperation > @HttpOperation(url = "/api/q/publishTradeUpdates", method = "POST") find { TradeSummary[] } call MyKafkaService::publishTrades ``` For more information about working with Kafka, including defining connections to brokers, see our dedicated docs on [Kafka](/docs/describing-data-sources/kafka) #### Reading files and saving to a Database This example shows how to read a CSV file from S3, save each row as a record to a database. First, we'll define our database table, and associated service ```taxi trades-db.taxi import com.orbitalhq.jdbc.Table import com.orbitalhq.jdbc.DatabaseService import com.orbitalhq.jdbc.InsertOperation @Table(connection = "trades-database", schema = "public" , table = "trades" ) model TradeSummaryRecord { symbol : Symbol open : OpenPrice high : HighPrice close : ClosePrice timestamp : Instant = now() } @DatabaseService(connection = "trades-database") service TradesDatabase { @InsertOperation write operation saveTradeSummary(TradeSummaryRecord):TradeSummaryRecord } ``` With the above in place, we can write a query that reads from S3, transforms from CSV to our database format, and performs the database inserts: ```taxi query.taxi find { TradeSummary[] } call TradesDatabase::saveTradeSummary ``` In the above example, Orbital detects that the inbound model (`TradeSummary`) is different from the destination format (`TradeSummaryRecord`) and handles the transformation for us. In our example, that's simple converting from CSV to JSON and restructuring the message. However, the transformation could be richer, doing tasks such as calling services to discover data. Finally, we might want to expose an HTTP POST operation to trigger this update: ```taxi query.taxi import taxi.http.HttpOperation > @HttpOperation(url = "/api/q/publishTradeUpdates", method = "POST") find { TradeSummary[] } call TradesDatabase::saveTradeSummary ``` For more information about working with databases, including defining connections to databases, and the support for different types of databases, see our dedicated docs on [Databases](/docs/describing-data-sources/databases) ### Writing to S3 To declare an operation that can write data to S3, take the following steps: - Declare a service annotated with `@S3Service`. - Within that service, declare a `write` operation with `@S3Operation`, providing the bucket name - The operation should take a two parameters: - One with a `@RequestPayload` annotation, which contains the contents to be written - One of type `FilenamePattern` which defines the filename to write to **Note: Remember the imports** Be sure to either add the imports for `S3Service`, `S3Operation`, `RequestBody` and `FilenamePattern` (as shown below), or use their fully qualified names. Here's an example: ```taxi import com.orbitalhq.aws.s3.S3Service import com.orbitalhq.aws.s3.S3Operation import com.orbitalhq.aws.s3.FilenamePattern import com.orbitalhq.aws.s3.RequestBody @S3Service(connectionName = "MyAwsConnection") service AwsBucketService { @S3Operation(bucket = "MyBucket") write operation writeToS3(@RequestBody films:Film[], filename:FilenamePattern = "films.csv"):Film[] } ``` #### Filename patterns when writing to S3 When writing to S3 filenames, filename patterns are not supported (unlike when [reading](#filename-patterns-when-writing-to-s3)). If you declare a filename with a pattern, an error will be thrown. ### Examples - Writing to S3 Each of the below examples will work with the following model of content stored on S3, so we're defining it once here for brevity: ```taxi tradeSummary.taxi import com.orbitalhq.formats.Csv // Define the format we're reading. // This is a CSV file, so the @Csv annotation is used. // (Note it's also imported at the top of the file) @Csv model TradeSummary { symbol : Symbol inherits String open : OpenPrice inherits Decimal high : HighPrice inherits Decimal close : ClosePrice inherits Decimal } ``` #### Fetching from an API and writing the results to S3 as a CSV This example shows data fetched from a REST API (exposed as JSON), and stored onto S3. As part of the operation, we'll transform a tree-like JSON structure into a flattened CSV file. First, we'll define the API and it's response object: ```taxi rest-service.taxi model StockPriceUpdate { ticker : Symbol // field names and structure are different, but the // types are the same as on our source model. prices: { openPrice : OpenPrice highPrice : HighPrice closePrice : ClosePrice } } service ApiService { @HttpOperation(url="http://myApi.com/prices", method = "GET") operation getPrices():StockPriceUpdate[] } ``` And, we'll define a write operation on S3 to store the content: ```taxi s3.taxi import com.orbitalhq.aws.s3.S3Service import com.orbitalhq.aws.s3.S3Operation import com.orbitalhq.aws.s3.FilenamePattern @S3Service( connectionName = "myAwsConnection" ) service AwsBucketService { @S3Operation(bucket = "trades") write operation writeTradeSummary(@RequestBody payload: TradeSummary[], filename: FilenamePattern = "trades.csv"):StockPriceCsv[] } ``` Given the above, we can use the following query to read from our API, transform the data, and write to our s3 bucket: ```taxi query.taxi find { StockPriceUpdate[] } call AwsBucketService::writeTradeSummary ``` This will result in the data returned from our API call to be converted to CSV and written to `trades.csv` on our S3 bucket. If we'd like to set the filename within our query, we could: ```taxi query.taxi given { filename : FilenamePattern = 'todaysTrades.csv' } find { StockPriceUpdate[] } call AwsBucketService::writeTradeSummary ``` This time, the output is written to `todaysTrades.csv` ### S3 File formats In the above examples, our content has been stored in S3 using CSV. This is defined because the model used in our operations is annotated with `@Csv`, as shown in the following excerpt: ```taxi model.csv import com.orbitalhq.formats.Csv @Csv model TradeSummary { // ... omitted } @S3Service( connectionName = "myAwsConnection" ) service AwsBucketService { // reading CSV @S3Operation(bucket = "MyTrades") // This operation returns a collection of // TradeSummary objects, which are defined with @Csv operation readBucket(filename:FilenamePattern = "*.csv"):TradeSummary[] // writing CSV @S3Operation(bucket = "trades") write operation writeTradeSummary( // The requesy body is a collection // of trade summaries, which are configured as CSV @RequestBody payload: TradeSummary[], filename: FilenamePattern = "trades.csv" ):StockPriceCsv[] } ``` The format can be any supported format, such as [Avro](/docs/data-formats/avro), [XML](/docs/data-formats/xml), [CSV](/docs/data-formats/csv) (or any other character-delimited file), or even [Protobuf](/docs/data-formats/protobuf). If no format is defined, JSON is used as the default. For more information, see [Data formats](/docs/data-formats/overview) ## SQS ### Consuming events Orbital can subscribe to a stream of data from SQS. ```taxi import com.orbitalhq.aws.sqs.SqsService import com.orbitalhq.aws.sqs.SqsOperation @SqsService( connectionName = "moviesConnection" ) service MovieService { @SqsOperation( queue = "movies" ) operation streamMovieQuery():Stream } ``` You can also control the `visibility duration time` and `poll time` in your corresponding `aws` connection configuration: ```hocon aws { my-aws-account { connectionName=my-aws-account accessKey=${?AWS_ACCESS_KEY_ID} secretKey=${?AWS_SECRET_ACCESS_KEY} region=${AWS_REGION} connectionParameters { > // SQS Receive duration (in seconds) that the received messages are hidden from subsequent retrieve requests after being retrieved by a ReceiveMessage request. Defaul value of 300 will be used if skipped. > sqsReceiveRequestWaitTime=500 > //SQS Receive Message Request Wait Time In Seconds, default value of 10 will be used if not specified. > sqsReceiveRequestWaitTime=10 } } } ``` This can then be queried using a standard `stream` query: ```taxi stream { Movie } // as ... ``` ### Publishing events Orbital can publish to a queue using a mutation: ```taxi import com.orbitalhq.aws.sqs.SqsService import com.orbitalhq.aws.sqs.SqsOperation @SqsService( connectionName = "moviesConnection" ) service MovieService { @SqsOperation( queue = "movies" ) write operation publishMovieEvent(Movie):Movie } ``` Publishing events can only be invoked in [mutations](/docs/querying/mutations). #### Example: Consuming from one SQS topic, and publishing to another ```taxi import com.orbitalhq.aws.sqs.SqsService import com.orbitalhq.aws.sqs.SqsOperation @SqsService( connectionName = "moviesConnection" ) service MovieService { @SqsOperation(queue = "newReleases" ) operation newReleases():Stream @SqsOperation( queue = "moviesToReview" ) write operation publishMoveEvent(Movie):Movie } // Query: consume from the new releases queue, and publish to // a "movies to review" queue stream { Movie } call MovieService::publishMovieEvent ``` -------------------------------------------------------------------------------- # Azure Service Bus Source: https://orbitalhq.com/docs/describing-data-sources/azure-service-bus Section: Managing data sources -------------------------------------------------------------------------------- Orbital can both read messages from Azure Service Bus topics/queues to stream data as part of a query, and write messages back to Service Bus topics/queues. ## Defining a connection to Azure Service Bus Service Bus connections are stored in your `connections.conf` config file, under the `serviceBus` element. The connection specifies how to connect to a Service Bus namespace - but not the specifics of individual topics or queues. (Those are specified on the Service Bus operation annotations in your Taxi model.) ```hocon serviceBus { warehouseConnection { connectionName=warehouseConnection connectionParameters { endPoint="mynamespace.servicebus.windows.net/" sharedAccessKeyName="RootManageSharedAccessKey" sharedAccessKey="${SERVICE_BUS_ACCESS_KEY}" } } anotherConnection { connectionName=anotherConnection connectionParameters { endPoint="another-namespace.servicebus.windows.net/" sharedAccessKeyName="RootManageSharedAccessKey" sharedAccessKey="${ANOTHER_SERVICE_BUS_KEY}" } } } ``` The following configuration options are supported under the `connectionParameters`: | Config option | Purpose | |---------------|---------| | `endPoint` | The Service Bus namespace endpoint URL | | `sharedAccessKeyName` | The Shared Access Signature (SAS) key name | | `sharedAccessKey` | The Shared Access Signature (SAS) key value | ## Streaming from Azure Service Bus You can stream messages from both Service Bus queues and topics using Orbital. ### Exposing a queue or topic subscription Service Bus queues and topics are declared in Taxi as operations which return a `Stream` of data. ```taxi // 1: Add the required imports import com.orbitalhq.azure.servicebus.ServiceBusService import com.orbitalhq.azure.servicebus.ServiceBusQueueOperation import com.orbitalhq.azure.servicebus.ServiceBusTopicSubscriptionOperation // Define your message model model Person { name : Name inherits String } // 2: Annotate the service as a `ServiceBusService` @ServiceBusService(connectionName = "warehouseConnection") service PersonServiceBusService { // 3a: Stream from a queue @ServiceBusQueueOperation(queue = "people-queue") stream streamFromQueue : Stream // 3b: Stream from a topic subscription @ServiceBusTopicSubscriptionOperation(topic = "people-topic", subscription = "all-people") stream streamFromTopic : Stream } ``` 1. Add the required imports 2. Annotate the service with `@ServiceBusService` * The `connectionName` parameter should match a connection defined in the [connections config file](#defining-a-connection-to-azure-service-bus). 3. Annotate the operations as either: * `@ServiceBusQueueOperation` for queues, specifying the queue name * `@ServiceBusTopicSubscriptionOperation` for topics, specifying both the topic name and subscription name 4. The return type should be a `Stream<>` of the defined message type. ### Example queries #### Basic streaming from Service Bus ```taxi // Stream messages from a Service Bus queue stream { Person } ``` This will consume messages from the `people-queue` queue declared in the `PersonServiceBusService` service. #### Enrich data from Service Bus with other sources Data from Service Bus can be projected to enrich it with data from other sources. Data requested that is not present in the Service Bus message payload is looked up from other sources, using Orbital's standard projections. ```taxi stream { Person } as { name : Name // available in the Service Bus message address : Address // looked up from another data source }[] ``` #### Filtering Service Bus streams This example reads all messages from the Service Bus, but only emits those with a specific name pattern: ```taxi stream { Person.filterEach( ( Person ) -> Person::Name.startsWith('J') ) } ``` #### Streaming from Service Bus to a database Streams from Service Bus can be inserted into a database (or any other writable source) using a mutating query: ```taxi // First, ensure that your data destination exposes a writeable data source service DatabaseService { @UpsertOperation write operation savePerson(Person):Person } // Then, define a streaming query that pipes messages to the database stream { Person } call DatabaseService::savePerson ``` ## Publishing to Azure Service Bus To make a queue or topic writable, declare a `write operation` in a Service Bus service: ```taxi // Add the required imports import com.orbitalhq.azure.servicebus.ServiceBusService import com.orbitalhq.azure.servicebus.ServiceBusQueueOperation import com.orbitalhq.azure.servicebus.ServiceBusTopicPublicationOperation // Define your message model model Person { name : Name inherits String } @ServiceBusService(connectionName = "warehouseConnection") service PersonServiceBusService { // Publish to a queue @ServiceBusQueueOperation(queue = "people-queue") write operation publishToQueue(Person):Person // Publish to a topic @ServiceBusTopicPublicationOperation(topic = "people-topic") write operation publishToTopic(Person):Person } ``` ### Batch publishing Service Bus supports publishing messages in batches to improve performance. To enable batching, add the `batchSize` and `batchDuration` parameters to the operation annotation: ```taxi @ServiceBusService(connectionName = "warehouseConnection") service PersonServiceBusService { // Publish to a topic with batching @ServiceBusTopicPublicationOperation( topic = "people-topic", batchSize = 5, // Number of messages in a batch batchDuration = 10000 // Maximum time to wait for batch in milliseconds ) write operation publishToTopic(Person):Person } ``` When batching is configured: 1. Messages are accumulated until either: - The batch size is reached - The batch duration time elapses 2. The accumulated messages are then sent as a single batch to Service Bus ### Example use cases #### Publishing a static message to Service Bus ```taxi given { person : Person = { name : 'John Doe' } } call PersonServiceBusService::publishToTopic ``` #### Consuming from one Service Bus entity and publishing to another To stream data from a Service Bus queue/topic, transform it, and republish: ```taxi @ServiceBusService(connectionName = "warehouseConnection") service TransformationService { @ServiceBusQueueOperation(queue = "raw-people") stream rawPeople : Stream @ServiceBusTopicPublicationOperation(topic = "enriched-people") write operation publishEnrichedPerson(EnrichedPerson):EnrichedPerson } ``` The following query will consume from the `raw-people` queue, and for each message, transform it to an `EnrichedPerson`, invoking any other services required to inject the necessary data: ```taxi stream { Person } // Each incoming Person message is transformed to an EnrichedPerson payload // and published to the enriched-people topic call TransformationService::publishEnrichedPerson ``` #### Building a REST API that publishes to Service Bus This example creates an HTTP endpoint accepting a `POST` request to publish a message to Service Bus: ```taxi type PersonName inherits String // The inbound request sent over HTTP model PersonRequest { name : PersonName } // The message we'll publish to Service Bus parameter model Person { name : PersonName } @ServiceBusService(connectionName = "warehouseConnection") service PersonService { @ServiceBusTopicPublicationOperation(topic = "people-topic") write operation publishPerson(Person):Person } @HttpOperation(path = "/api/q/publishPerson", method = "POST") query PublishPersonQuery(@RequestBody request:PersonRequest) { given { request } call PersonService::publishPerson } ``` The above example works as follows: * A `POST` request is sent to `/api/q/publishPerson` with a body containing a name * The query calls `publishPerson`, which constructs a `Person` from the request * The message is published to the Service Bus topic `people-topic` ## Advanced features ### Message properties and metadata Service Bus provides several methods to add metadata to messages. Orbital supports accessing and setting these properties through annotations: ```taxi import com.orbitalhq.azure.servicebus.ServiceBusMessageProperty model Person { name : Name inherits String @ServiceBusMessageProperty(name = "MessageId") messageId : String @ServiceBusMessageProperty(name = "CorrelationId") correlationId : String } ``` When consuming messages, these properties will be populated from the incoming message. When publishing messages, these properties will be set on the outgoing message. ### Error handling and dead-letter queues Service Bus includes built-in dead-letter queue functionality. When errors occur during message processing, you can configure Orbital to handle these scenarios: ```taxi @ServiceBusService(connectionName = "warehouseConnection") service PersonService { @ServiceBusTopicSubscriptionOperation( topic = "people-topic", subscription = "all-people", maxDeliveryCount = 3 // Number of processing attempts before message is dead-lettered ) stream streamPeople : Stream } ``` -------------------------------------------------------------------------------- # Writing queries Source: https://orbitalhq.com/docs/querying/writing-queries Section: Querying -------------------------------------------------------------------------------- Orbital focuses on querying for data based on it's meaning, rather than which system provides it. This allows services to change, and data to move, without requiring consumers to update their queries. ## Writing queries Queries are written in [TaxiQL](https://docs.taxilang.org/language-reference/querying-with-taxiql/), an open source query language for data. **Note** [TaxiQL](https://docs.taxilang.org/language-reference/querying-with-taxiql/) is a great query language. The taxi documentation has details on the syntax, which we haven't duplicated here. Go check it out, then come back. We'll wait. TaxiQL is agnostic of where data comes from - it's left to Orbital to discover data from the various sources that have been connected. Here's some sample queries: ```taxi // Find all the movies find { Movie[] } // Find all the movies, enriching and projecting them to a different structure find { Movie[] } as { title : MovieTitle director : DirectorName rating : RottenTomatoesScore }[] ``` ### Shaping results You can limit, paginate, sort and de-duplicate the results of a query using [collection options](/docs/querying/collection-options) - written alongside your filter criteria, e.g. `find { Movie[]( orderBy: Rating desc, limit: 10 ) }`. These work on both `find` expressions and projections, and are pushed down to data sources (such as databases) where possible. ## Projections Projections are a way of taking data from one place, then transforming & combining it with other data sources. Orbital uses the information present on the object being projected in order to call services and find other information. You can project from a pre-defined type to another predefined type - such as: ```taxi model Purchase { transactionId : TransactionId customerId : CustomerId } model CustomerTransaction { transactionId : TransactionId name : CustomerName price : Price } find { Purchases[] } as CustomerTransaction[] ``` It's also very common to project from a predefined type to a type defined inline within your query (called 'anonymous types'). e.g.: ```taxi model Purchase { transactionId : TransactionId customerId : CustomerId } find { Purchases[] } as { // Projections let you change field names, and reshape objects as required txn: TransactionId // Not present on the original Purchase object, so try to // find it using something we already know (in this case, the CustomerId) customerName: CustomerName }[] ``` ### Projection scopes Creating a projection requires the syntax `something as { ... }`, which defines a projection scope: ```taxi find { Customer } as { // start a projection scope, containing a customer // ... omitted ... } ``` If you're projecting an array, then each item within the array is projected separately: ```taxi find { Customer[] } as { // start a projection scope. // Within the scope, each item is an individual Customer instance firstName : FirstName }[] // Be sure to include the array marker at the end, as the object is an array. ``` If you need to, you can assign a name to the item within the scope. This can be useful for nested scopes, or controlling inputs into a function. e.g.: ```taxi find { Customer[] } as (customer:Customer) -> { // referencing a field by it's name on Customer firstName : upperCase(customer.firstName) // referencing as field by it's type on Customer lastName : upperCase(customer::LastName) // using a variable within a constraint: purchases: Purchase[](CustomerId == customer::CustomerId) // without referring to a variable, Age is resolved against all variables in scope age : Age }[] ``` Or, when using nested scopes: ```taxi find { Customer[] } as (customer:Customer) -> { name : CustomerName orders : Order[] as (order:Order) -> { ageAppropriate : Boolean = customer.age >= order.recommendedAge } } ``` Or, to provide inputs into functions: ```taxi model Film { title : FilmTitle inherits String headliner : ActorId cast: Actor[] } find { Film[] } as (film:Film) -> { title : FilmTitle // singleBy selects a single item from // an array (film.cast, an array of Actor), that matches a predicate // (in this case, there the actor ID is the same as the headliner id) star : singleBy(film.cast, (Actor) -> Actor::ActorId, film.headliner) as (actor:Actor) -> { name : actor.name title : film.title } }[] ``` #### Using variables to modify what's in scope By default, the scope contains the entire source object. e.g.: ```taxi model Film { title : Title cast : Actor[] } find { Film ) as { // this scope contains an entire film record } ``` You can modify this by specifying the type of the variable in scope: ```taxi find { Film } as (Actor[]) -> { // note that film has been removed from the scope... title : Title //... therfore title isn't knowable -- this field will return null actorName : ActorName }[] ``` You can also use functions to further reduce the scope: ```taxi find { Film } as (first(Actor[])) -> { // Now, the scope only contains a single actor headliner : ActorName } // We're not projecting an array anymore, so no aray marker here ``` Finally, if the data defined in the scope isn't available on the source, Orbital triggers a query to find it. e.g.: ```taxi // Define a few models model Film { id : FilmId inherits Int title : Title inherits String } model Actor { name : ActorName inherits String } model Cast { actors : Actor[] } // And some services that return them service Films { operation getFilm():Film operation getCast(FilmId):Cast } // Here's a query: find { Film } as (Actor[]) -> { actorName : Name filmTitle : Title // should be null, as it's out-of-scope on Actor }[] ``` In the above query: * `getFilm()` is called, to fetch the `Film` * The projection requests an `Actor[]` in the scope, which isn't available, so... * A call to `getCast()` is made, passing the `FilmId` to fetch the `Actor[]` * Because it's an array, each `Actor` within the array of `Actor[]` is projected individually * `actorName` is read from the `name` field on `Actor`, because the requested field asks for the type `Name` * `filmTitle` is out-of-scope, so returned as null #### Declaring multiple variables in scope In the previous example, we saw that `filmTitle` was returned as null, because the `Film` was removed from scope. To run the same query with `Film` in scope, simply add it to the projection: ```taxi find { Film } as (Film, Actor[]) -> { actorName : Name filmTitle : Title // Title is now discoverable, as Film is in scope }[] ``` ### Data discovery rules When projecting, Orbital will use information present on the source object to discover data on the target object. Data can be fetched from a single operation that returns the value, or by invoking a chain of operations to return the value. #### Operations with @Id fields on return types If the result of an operation is an object that exposes an `@Id` field, then only operations which accept that `@Id` field as an input will be called. e.g.: ```taxi model Customer { @Id customerId : CustomerId name : CustomerName } service CustomerService { // Can be called when projecting, because // Person has an @Id of type PersonId findCustomer(CustomerId):Customer // Cannot be called when projecting, because // Person has an @Id, and it isn't PersonName findCustomerByName(CustomerName):Customer } ``` #### Operations without @Id fields on return types If the result of an operation is an object that *does not* expose an `@Id` field, then it can be called with any information available. ### Filling in nulls By default, if a service returns a null value, Orbital will accept it as-is. However, if query annotates a field on a projection type with `@FirstNotEmpty`, Orbital will attempt to populate values by invoking operations to populate the missing values. Orbital will execute a search using the other values present on the entity being projected as potential inputs to operations, and build a path to populate the missing values. Operations are invoked following the standard [Data Discovery Rules](#data-discovery-rules) ## Understanding caching in Orbital By default, Orbital does not maintain a long-lived cache between operations, but you can [add one](/docs/querying/caching) by configuring an external cache - such as Redis or Hazelcast. Without an external cache, Orbital caches operation calls for the lifetime of a query. This prevents the same operation being invoked repeatedly while projecting multiple rows in a result. When caching, Responses are cached for a given operation + set of inputs. If an operation is invoked with different parameters, the cache is not used. Operations that return an array of results, which return more than 10 values, will not have their responses cached. (This is not currently configurable, but reach out on slack if you need to configure this). ## Recovering from failure If an operation returns an error while Orbital is attempting to execute a query, then it is excluded from being invoked with the same parameters again. This exclusion is scoped to the query only, and expires at the end of the query. After excluding the operation, Orbital will attempt to find another path to return the value being discovered. ## Expressions in queries Taxi allows the definition of expressions on both [types](https://docs.taxilang.org/language-reference/advanced-types/#expression-types) and [fields](https://docs.taxilang.org/language-reference/advanced-types/#expression-on-fields), but doesn't provide an evaluation engine - that's where Orbital comes in. Typically, expressions are used in a projection within a query. You can also use them on a model to expose derived information when a model is parsed by Orbital (eg., when return from a service) - but that's less common. So, while documentation here focuses on query projections, you can do everything here on a model too. ### Writing an expression in a projection Expressions can be defined in the fields of a projected result from a query: ```taxi find { Flights[] } as { flightNumber : FlightNumber totalSeatsAvailable : TotalSeats soldSeats : SoldSeats remainingSeats : Int = (this.totalSeatsAvailable - this.soldSeats) } ``` Expressions can be defined in two ways - on a field, or on a type. #### Expressions on a field ```taxi // Expression types on a field: find { Flights[] } as { flightNumber : FlightNumber totalSeatsAvailable : TotalSeats soldSeats : SoldSeats // field expressions can be defined EITHER using field references... remainingSeats : Int = (this.totalSeatsAvailable - this.soldSeats) // ...or type references... remainingSeats : Int = (TotalSeats - SoldSeats) } ``` #### Expressions on a type To encapsulate common expressions, you can define a type with the expression: ```taxi // Expression type: type RemainingSeats = TotalSeats - SoldSeats // Which is then used on a projection: find { Flights[] } as { flightNumber : FlightNumber totalSeatsAvailable : TotalSeats soldSeats : SoldSeats remainingSeats : RemainingSeats } ``` Unlike field expressions, type expression cannot use field names, and can only reference other types. #### How Orbital discovers values to evaluate expressions When Orbital is evaluating an expression, it first looks on the source object being projected for the input values into the expression. If any inputs are not available, then Orbital will perform a search using the current data available on the source object in an attempt to look up the value. ## Submitting queries Generally, developers will use the UI to write and test their queries, then integrate using Orbital's rest API. ### Rest API Queries to Orbital are submitted to the `/api/taxiql` endpoint: ``` curl 'http://localhost:9022/api/taxiql' \ -H 'Content-Type: application/taxiql' \ --data-raw 'find { Movie[] }' ``` #### A word about content type Strictly speaking, the content type for taxiql queries is `application/taxiql`. However, the Orbital server will accept taxiql queries with any of the following content types headers: * `Content-Type: application/json` * `Content-Type: application/taxiql` * `Content-Type: text/plain` This is to allow broad compatability with clients. ### Large queries with Server Sent Events Running large queries can result in out-of-memory errors if Orbital is holding the result set in memory. To address this, Orbital supports pushing results over server-sent-events. To consume a query as a server-sent-event, set the `Accept` header to `text/event-stream`: ``` curl 'http://localhost:9022/api/taxiql' \ -H 'Accept: text/event-stream' \ -H 'Content-Type: application/taxiql' \ --data-raw 'find { Movie[] }' ``` Results are pushed out from Orbital as they are available. ### Including type metadata in responses Orbital can include type metadata in the responses being sent back. To enable this, append `?resultMode=TYPED` to the API call: ``` curl 'http://localhost:9022/api/taxiql?resultMode=TYPED' \ -H 'Accept: text/event-stream' \ -H 'Content-Type: application/taxiql' \ --data-raw 'find { Movie[] }' ``` ## Defining output formats By default, Orbital serves results to queries as JSON. This can be configured to customize the result format. ### With Accept headers The following accept headers are supported: | Header | Result type | |--------------------|------------------------------| | `application/json` | json | | `application/csv` | csv | | `text/event-stream | JSON with server-sent-events | ## Controlling output formats By default, data is written in JSON format. However, this can be controlled by placing an annotation on the model defining the output of a query. For example: ```taxi import com.orbitalhq.formats.Csv @Csv(delimiter = "|", nullValue = "NULL") model Person { firstName : FirstName inherits String lastName : LastName inherits String age : Age inherits Int } // Query: // Response type (Person) contains a Csv format defined, // which will be considered when writing responses. find { Customer[] } as { Person[] } ``` The following formats are supported: * [Csv](/docs/data-formats/csv) * [Xml](/docs/data-formats/xml) * [Protobuf](/docs/data-formats/protobuf) * [Avro](/docs/data-formats/avro) If you need a custom format that's not listed here, read about Orbital's support for [custom model formats](/docs/data-formats/custom-data-formats) -------------------------------------------------------------------------------- # Collection options Source: https://orbitalhq.com/docs/querying/collection-options Section: Querying -------------------------------------------------------------------------------- _Available since 0.38.0_ ## Overview Collection options let you shape the results of a query - limiting how many rows come back, paginating through them, sorting them, and removing duplicates - without changing what the query means. They're written as options on a collection expression, alongside any filter criteria: ```taxi find { Person[]( CountryCode == "GB", orderBy: DateOfBirth desc, offset: 20, limit: 10 ) } ``` This reads as: find people in GB, sort them by date of birth (newest first), skip the first 20, and return the next 10. The same options work against **any** data source. Where a source can apply an option natively (a database sorting and limiting in SQL, for example), Orbital pushes the work down to it. Where a source can't, Orbital applies the option itself after fetching. Either way the result is the same. The full syntax lives in the [Taxi language reference](https://taxilang.org/docs/taxiql/querying#collection-options) - this page focuses on how Orbital executes them. ## The options The following options are available on a collection expression: | Option | Description | |------------|------------------------------------------------------------------------------------------------------| | `limit` | The maximum number of items to return. | | `offset` | The number of items to skip before returning results. Useful for pagination. | | `orderBy` | Sorts results by one or more fields. Each term takes a direction (`asc` or `desc`), e.g. `orderBy: DateOfBirth desc`. | | `uniqueBy` | Removes duplicates, keeping the first item seen for each distinct value of the given field. | Options are comma-separated, and can be combined with filter criteria in the same expression. Multiple sort terms are supported - list them in square brackets, and they're applied left to right: ```taxi find { Person[]( orderBy: [CountryCode asc, DateOfBirth desc] ) } ``` Values can be literals, or [query parameters](https://taxilang.org/docs/taxiql/querying#collection-options): ```taxi query findPeople( maxRows: Int ) { find { Person[]( limit: maxRows ) } } ``` **Note: Cursors are reserved** The `after` and `before` cursor options parse and compile, but are not yet supported at runtime - a query that uses them is rejected with a clear error. They're reserved for cursor-based pagination in a future release. ## Logical order Regardless of the order you write them in, options are always applied in a fixed logical order: 1. **Filter** - the predicate criteria (e.g. `CountryCode == "GB"`) 2. **uniqueBy** - remove duplicates 3. **orderBy** - sort 4. **offset** - skip 5. **limit** - take This ordering is what makes combinations predictable. `uniqueBy: PersonId, limit: 10` returns up to 10 *unique* people - duplicates are removed first, then the limit is applied to what remains. ## Shaping at the source vs. shaping the projection Options attach to the collection expression they're written on, and their position decides which phase of the query they shape. Written on the `find`, they shape what's fetched from the source. Written on a projection, they shape the projected result. You can use both in a single query: ```taxi // Fetch up to 100 people from the source, project them, then return the first 10 of the projected result find { Person[]( limit: 100 ) } as { name : PersonName country : CountryName }[]( limit: 10 ) ``` Nested collections inside a projection can be shaped too - useful for trimming child records per parent: ```taxi find { Customer[] } as { name : CustomerName // Only the 20 most recent transactions for each customer transactions : Transaction[]( limit: 20, orderBy: TransactionDate desc ) }[] ``` See [shaping projected collections](https://taxilang.org/docs/taxiql/projections#shaping-projected-collections) in the Taxi docs for more on projection-position options. ## How options are executed: pushdown vs. Orbital-applied For each option, Orbital's query planner decides - per data source - whether to **push it down** to the source or to **apply it itself** after fetching (a "residual" option). Pushing down is usually more efficient: a database can sort and limit far more cheaply than Orbital can after loading every row. But not every source can apply every option, so anything a source can't handle is simply applied by Orbital instead. Nothing is lost - it's a question of *where* the work happens, not *whether* it happens. Here's what each source applies natively today: | Source | Pushed down to the source | Applied by Orbital | |------------------------------------------------|------------------------------|---------------------------------------| | Databases (Postgres, MySQL, MSSQL, Oracle, Redshift, Snowflake) | `limit`, `offset`, `orderBy` | `uniqueBy` (and `orderBy` when it can't be pushed) | | MongoDB | `limit`, `offset`, `orderBy` | `uniqueBy` (and everything, for native aggregations) | | Hazelcast | `limit` | `offset`, `orderBy`, `uniqueBy` | | REST APIs, Kafka, and all other sources | - | `limit`, `offset`, `orderBy`, `uniqueBy` | See the per-source pages for the details and caveats: [Databases](/docs/describing-data-sources/databases#collection-options), [MongoDB](/docs/describing-data-sources/mongodb#collection-options), and [Hazelcast](/docs/describing-data-sources/hazelcast#collection-options). ### Pushdown never changes the result The planner will only push an option down when doing so is guaranteed to produce the same answer as applying it in Orbital. Where that guarantee doesn't hold, it fetches more and does the work engine-side. For example: if a source can apply an `orderBy` but a `limit` is also present, Orbital won't push the `limit` below the sort unless the sort is pushed too. Limiting *before* sorting would return the wrong rows - so instead Orbital fetches, sorts, and then limits itself. The safety rule is always applied in your favour: a pushed-down option is never allowed to change the semantics of the query. ## Streaming queries On a streaming query, `limit` is supported - it completes the stream once N items have been emitted: ```taxi // Take the first 100 events, then complete stream { StockPrice }( limit: 100 ) ``` `orderBy`, `offset` and `uniqueBy` don't have a meaningful definition over an unbounded stream, so they're rejected with a clear error rather than silently ignored. ## Error cases A few inputs are rejected rather than guessed at: * **Negative `limit` or `offset`** (including dynamic values resolved from a parameter) fail with a clear error. * **`limit: 0`** is valid, and returns an empty result. * **`after` / `before` cursors** compile, but are rejected at runtime as not yet supported. * **`orderBy`, `offset`, `uniqueBy` on a stream** are rejected - only `limit` is supported when streaming. -------------------------------------------------------------------------------- # Mutations Source: https://orbitalhq.com/docs/querying/mutations Section: Querying -------------------------------------------------------------------------------- Mutation queries make changes somewhere - ie., perform a task, or update a record. Mutating operations are defined in Taxi using the `write` modifier: ```taxi service FlightsService { write operation bookFlight(BookingRequest):BookingConfirmation write operation sendFlightManifest(Manifest):ManifestConfirmation } ``` Mutating operations are excluded from being invoked during a query (ie., anything with a `find {}` or `stream {}` directive.) Instead, they are invoked using a `call` directive. ```taxi find { Passengers[] } call FlightsService::sendFlightManifest ``` In this example, a list of `Passengers` is fetched, and converted into a `Manifest` to call the `sendFlightManifest` operation of the `FlightsService`. Many different data sources support mutations, such as [databases](/docs/describing-data-sources/databases#writing-data-to-a-database), [Kafka](/docs/describing-data-sources/kafka#writing-to-a-kafka-topic), [MongoDB](/docs/describing-data-sources/mongodb#writing-data-to-a-collection), and [Hazelcast](/docs/describing-data-sources/hazelcast#writing-data-to-hazelcast). Consult the docs for the relevant data source to learn more. ## Projecting and mutating Orbital will automatically convert from the source type to whatever is required in the input of the mutation operation. For example: ```taxi // The data we're writing to our database @Table(connection = "films-database", schema = "public" , table = "films" ) closed parameter model Film { @Id filmId : FilmId inherits Int title : Title inherits String reviewScore : ReviewScore inherits Int } @DatabaseService(connection = "films-database") sevice FilmsDatabase { @UpsertOperation write operation saveFilm(Film):Film } // Our source format closed model Movie { // doesn't have reviews title : FilmId } // Another source containing reviews closed model Review { id : FilmId score : ReviewScore } ``` Given the above services and models, we can write a mutating query to find films, enrich them, and write to our database: ```taxi find { Film[] } call FilmsDatabase::saveFilm ``` In the above example, note that our source format - `Movie` lacks review data, but out target format - `Film` needs it. Orbital detects that the source and target formats don't align, and so automatically triggers a projection to enrich the source data with reviews: [![](https://mermaid.ink/img/pako:eNptUctOwzAQ_JVoTyCFtonJoz5wAiQOCKm9QThs401ikcTBcQqlyr_jPChUwifvzHhmd32EVAkCDi29d1SndCsx11gltWPPk95Jg6VzdXPj3Muy2pLey5S4k5MZ6kn1hxmV8ys-Ei-vk6hUqnE21BAaEk6mtEOYFk52cjlP29Be0sffvAm5GDwfxOXPmzPdefpE_Wc-eLS3aHCHrTVvcU-_01AtwIWKdIVS2MUcBzgBU1BFCXB7FajfEkjq3uqwM2p7qFPgRnfkQtcIO-C8ROAZlq1FG6yBH-ET-JXH2MKP4zXzmbeOArZiLhyAe56_CFYsjsIgClaeH_QufCllLTwXSEij9OP0UeN__QTdjcwpp1QoSA9R5tAM4ly2xopTVWcyH_BOlxYujGlavlwO9CKXpuh2i1RVy1aKArUp9utwGfphjD6jMGIYMCbSnbeOM__ay0RkG0To-3Gw57HLuQGturyYq_4b2h_Bog?type=png)](https://mermaid.live/edit#pako:eNptUctOwzAQ_JVoTyCFtonJoz5wAiQOCKm9QThs401ikcTBcQqlyr_jPChUwifvzHhmd32EVAkCDi29d1SndCsx11gltWPPk95Jg6VzdXPj3Muy2pLey5S4k5MZ6kn1hxmV8ys-Ei-vk6hUqnE21BAaEk6mtEOYFk52cjlP29Be0sffvAm5GDwfxOXPmzPdefpE_Wc-eLS3aHCHrTVvcU-_01AtwIWKdIVS2MUcBzgBU1BFCXB7FajfEkjq3uqwM2p7qFPgRnfkQtcIO-C8ROAZlq1FG6yBH-ET-JXH2MKP4zXzmbeOArZiLhyAe56_CFYsjsIgClaeH_QufCllLTwXSEij9OP0UeN__QTdjcwpp1QoSA9R5tAM4ly2xopTVWcyH_BOlxYujGlavlwO9CKXpuh2i1RVy1aKArUp9utwGfphjD6jMGIYMCbSnbeOM__ay0RkG0To-3Gw57HLuQGturyYq_4b2h_Bog) ### Manually projecting If you want more control (eg., for performing calculations or derived fields), you can manually define a projection: ```taxi find { Film[] } as { id : FilmId title : Title = upperCase(Title) // make all the titles uppercase in the db } call FilmsDatabase::saveFilm ``` Here, Orbital still performs a series of projections: - First, projecting from `Film` to the inlined type (defined in the projection) - Then from the inlined type to the input type of the mutating operation (`FilmsDatabase::saveFilm`) [![](https://mermaid.ink/img/pako:eNplULtOxDAQ_JVoK5DyOMfk5YIKOmigQ2kW20nMJXbkOAfHKf-OEwISolvPjHdm5wLcCAkMmt688w6tCx6eah0EQaO0CKLoNhiteZPcbeA-77iMhtmhU0ZHf0X_ie2H0idzlFcc-z4wo7Sb4rrWEMIg7YBK-CCXdUcNrpODrIH5UaA91lDrxetwdub5rDkwZ2cZwjwKdPJOYWtxANZgP3l0RA3sAh_AIkJpnJZlRVNKqiKjBxrCGRghaZwdaFnkWZEdSJotIXwa41eQEKRQztjH72K2fn6M7jfm16c3KKRdrdx5XMWtmpwXc6Mb1a74bHsPd86NE0uSlY5b5br5NeZmSCYl1sq7U5UneZqXmFKZFxQzSgV_JVXZpDekEYUPiLAs22EvW8o9gDVz2-2v5Qsu8pSV?type=png)](https://mermaid.live/edit#pako:eNplULtOxDAQ_JVoK5DyOMfk5YIKOmigQ2kW20nMJXbkOAfHKf-OEwISolvPjHdm5wLcCAkMmt688w6tCx6eah0EQaO0CKLoNhiteZPcbeA-77iMhtmhU0ZHf0X_ie2H0idzlFcc-z4wo7Sb4rrWEMIg7YBK-CCXdUcNrpODrIH5UaA91lDrxetwdub5rDkwZ2cZwjwKdPJOYWtxANZgP3l0RA3sAh_AIkJpnJZlRVNKqiKjBxrCGRghaZwdaFnkWZEdSJotIXwa41eQEKRQztjH72K2fn6M7jfm16c3KKRdrdx5XMWtmpwXc6Mb1a74bHsPd86NE0uSlY5b5br5NeZmSCYl1sq7U5UneZqXmFKZFxQzSgV_JVXZpDekEYUPiLAs22EvW8o9gDVz2-2v5Qsu8pSV) ### Writing single item vs batch Generally, Orbital tries to run projections and mutations in parallel, and - where possible - using micro-batching to batch writes. However, sometimes you want everything written in a single write - such as writing a file to S3 (where incremental updates aren't supported by S3) To support this, simply declare the input into your mutation as an array. For example, to update our earlier example to write to S3 instead of a database - let's add the S3 sink: ```taxi @S3Service(connectionName = "MyAwsConnection") service AwsBucketService { @S3Operation(bucket = "MyBucket") write operation writeToS3(@RequestBody films:Film[], filename:FilenamePattern = "films.csv"):Film[] } ``` By updating our query: ```taxi find { Film[] } call AwsBucketService::writeToS3 ``` This time, the enrichments happen in parallel, but are held until the all enrichments are finished, then written to S3 in a batch at the end: [![](https://mermaid.ink/img/pako:eNplUctugzAQ_BVrT41EHuDyiA85tZV6qColPTX04OAFrAKmxiRNI_69BtIoUX3y7szO7OMEiRIIDBr8arFK8EHyTPMyroh9r3onDS_IdLUiT7IoN6j3MkFGMjR9PLKukIF5rmIDsP0YSYVSNVljjdygIKnSBHmSk_Sicuu2xr3Ew7XfmLnrNZ_F5K_mhnfrPkIjESvxf6INZeSgpcE3taFkUG62HxNwoERdcinsWk59WQwmxxJjYPYruP6MIa46y-OtUZtjlQAzukUH2lrY8c4rBJbyorHZmlfATvANbOpSOvOiaEk96i5Dny6oA0dgruvN_AWNwsAP_YXr-Z0DP0pZCdcBFNIo_TKeabjWn9HjgFx8CsUF6t7KHOuenMnGWHKiqlRmfb7VhU3nxtQNm897eJZJk7e7WaLKeSNFzrXJ98tgHnhBxD2KQUi5T6lIdu4ySr17NxWhbZBD1w2DvQ9dnhvQqs3yc9T9AkaNwDk?type=png)](https://mermaid.live/edit#pako:eNplUctugzAQ_BVrT41EHuDyiA85tZV6qColPTX04OAFrAKmxiRNI_69BtIoUX3y7szO7OMEiRIIDBr8arFK8EHyTPMyroh9r3onDS_IdLUiT7IoN6j3MkFGMjR9PLKukIF5rmIDsP0YSYVSNVljjdygIKnSBHmSk_Sicuu2xr3Ew7XfmLnrNZ_F5K_mhnfrPkIjESvxf6INZeSgpcE3taFkUG62HxNwoERdcinsWk59WQwmxxJjYPYruP6MIa46y-OtUZtjlQAzukUH2lrY8c4rBJbyorHZmlfATvANbOpSOvOiaEk96i5Dny6oA0dgruvN_AWNwsAP_YXr-Z0DP0pZCdcBFNIo_TKeabjWn9HjgFx8CsUF6t7KHOuenMnGWHKiqlRmfb7VhU3nxtQNm897eJZJk7e7WaLKeSNFzrXJ98tgHnhBxD2KQUi5T6lIdu4ySr17NxWhbZBD1w2DvQ9dnhvQqs3yc9T9AkaNwDk) -------------------------------------------------------------------------------- # Streaming data Source: https://orbitalhq.com/docs/querying/streaming-data Section: Querying -------------------------------------------------------------------------------- ## Writing streaming queries Streaming queries are executed in the same way as request/response queries, but use the `stream` keyword instead of `find`. For example - assuming a Kafka topic emitting stock price updates: ```taxi model StockPrice { symbol : StockSymbol price : StockPrice } service KafkaService { stream stockPrices : Stream } ``` This can be queried using the following: ```taxi stream { StockPrice } ``` This can be combined with other standard querying tools, such as projections and mutations: ```taxi stream { StockPrice } as { symbol : StockSymbol updateReceived : Instant = now() currentPrice : StockPrice totalTradedQuantity : TotalTradedQuantity } call TradeBookService::saveTradeSnapshot ``` ### Filtering streams To filter a stream, you can use the `filterEach()` function: ```taxi stream { StockQuotes.filterEach( StockSymbol -> StockSymbol == 'AAPL' ) } ``` ## Joining streams It's possible to join multiple streams to one - for example, combining the results of multiple Kafka topics. Streams are joins are either stateless, or stateful, using either a Union Type (`Foo | Bar`) or an Intersection Type (`Foo & Bar`) respectively. The result of a joined stream is a type combining the properties of both types. To learn more about the resulting type from Union and Intersection types, including how property naming conflicts are resolved in the Taxi docs ### Stateless joins Streams of union types (`stream { Foo | Bar }` emit as messages are emitted from either Foo or Bar. These are considered stateless, as messages are emitted directly, without any attempt to link the messages. A stream of a union type is stateless by default (but can be made stateful, as described [below](#stateful-joins)) ```taxi model OrderPlacedEvent { orderId : OrderId customerId : CustomerId } model FoodDeliveryEvent { orderId : OrderId deliveryState : DeliveryState } // Emits messages from both Foo and Bar as they arrive stream { OrderPlacedEvent | FoodDeliveryEvent } ``` Results in: ```json5 // OrderEvent - no details from DeliveryEvent { "orderId": 123, "customerId" : 456, "deliveryState": null } // DeliveryEvent - no details from OrderEvent { "orderId": 123, "customerId" : null, "deliveryState": "PickedUp" } ``` **Note: Field merging** The result of a union or intersection type contains all the fields from both types. However, orderId only appears once, as it has the same **name and type** on both types. Learn more about how field naming conflicts are resolved in the [Taxi docs](https://taxilang.org/docs/language/basic-types#merging-properties) ### Stateful joins A stateful join is where multiple streams are joined, and the messages between those streams are linked. Stateful streams exist in two flavours - using Union types and using Intersection types. **Warning: State stores & memory considerations** Stateful streams have to hold state as messages arrive, and use a State Store. If a state store is not configured in the query, then state is held in Orbital's default state store - which is the Orbital cluster itself. Depending on your instance size, this can result in out-of-memory issues on Orbital in heavy workloads where Orbital has to retain state. Read more about [how to configure a state store](#configuring-a-state-store) #### Intersection Types A stream using an intersection type emits messages only when all the streams have emitted a message for a given id. ```taxi model OrderPlacedEvent { @Id orderId : OrderId customerId : CustomerId } model FoodDeliveryEvent { orderId : OrderId deliveryState : DeliveryState } // Emits messages from both Foo and Bar only // after both have produced an event for a given message stream { OrderPlacedEvent & FoodDeliveryEvent } ``` Results in: ```json5 // First event - an OrderPlacedEvent written on a Kafka topic, // but is not emitted on the Orbital stream until a corresponding FoodDeliveryEvent is emitted // Kafka: { "orderId": 123, "customerId" : 456 } // Orbital: // No event // Second event: a FoodDeliveryEvent written on a Kafka topic, // and a corresponding event on Orbital, combining data from both streams: // Kafka: { "orderId" : 123, "deliveryState" : "PickedUp"} // Orbital: { "orderId": 123, "customerId" : 456, "deliveryState": "PickedUp" } ``` #### Union types A stream using a Union type emits messages as they arrive from each stream, joining messages together. Streams with union types are stateless by default, so to enable state, you must use the `@StateStore` annotation. ```taxi // Using the same model as above, omitted here for brevity // Emits messages from both Foo and Bar as they arrive @StateStore stream { OrderPlacedEvent | FoodDeliveryEvent } ``` Results in: ```json5 // First event - an OrderPlacedEvent written on a Kafka topic. // Kafka: { "orderId": 123, "customerId" : 456 } // Orbital: { "orderId": 123, "customerId" : 456, "deliveryState": null } // Second event: a FoodDeliveryEvent written on a Kafka topic, // and a corresponding event on Orbital, combining data from both streams: // Kafka: { "orderId" : 123, "deliveryState" : "PickedUp"} // Orbital: { "orderId": 123, "customerId" : 456, "deliveryState": "PickedUp" } ``` ### Configuring a state store When joining a stateful stream, Orbital needs to hold state - which requires a state store. Orbital configures a default state store, using an embedded Hazelcast instance. This is a high performance state store, allowing state to be shared across a cluster. However, this does share the same RAM as used by the Orbital cluster. Under heavy load, this could result in out-of-memory issues on your Orbital cluster. It is possible to configure an external state store, using a dedicated standalone cache. Currently only Hazelcast is supported, but support for other external caches is planned. To define a dedicated external state store, first deploy an external Hazelcast cluster, then [add it as a connection](/docs/describing-data-sources/hazelcast) in Orbital. ```hocon hazelcast { myHazelcast { connectionName = myHazelcast addresses = ["localhost:5701"] } } ``` To use this state store in your queries, use the connection name in the `@StateStore` annotation: ```taxi @StateStore(connection = "myHazelcast" ) stream { Foo | Bar } ``` or: ```taxi @StateStore(connection = "myHazelcast" ) query MySavedQuery { stream { Foo | Bar } } ``` #### StateStore configuration options | Parameter name | Definition | Default value | |------------------|-----------------------------------------------------------------------------------------------------|------------------------------------------------------------| | `connection` | The configured connection to use for the state store | `null` - default to storing in Orbital's internal memory | | `name` | A name for this state store | `null` - a name is automatically generated if not provided | | `maxIdleSeconds` | The expiration period in seconds. State will be evicted when this expires (resets on read or write) | 180 | #### Limitations When using Hazelcast as a state store, an Hazelcast Map is created to store the state, with a default `maxIdleSeconds`. Once a map is created, Hazlecast does not support changing it's configuration. Attempting to change the configuration (eg the `maxIdleSeconds`) after the map has been created will cause an error. To work around this, assign a new name to the state store, using the `name` parameter: ```taxi @StateStore(name = "NewStateStore", maxIdleSeconds = "30" ) query MySavedQuery { stream { Foo | Bar } } ``` ### Understanding joins When joining messages between streams, messages are joined based on a shared Id. Orbital looks for a type that is: * Present on all models present in the join * Where at least one of the fields has an `@Id` annotation. When joining: * If no fields match the above criteria, * **or** if more than one set of fields match the above criteria, then the join is ambiguous and an error is thrown. Note that field names are not relevant in joins - Orbital only considers each fields type. For example: ```taxi model FoodOrder { @Id id : OrderId } model FoodDeliveryUpdate { @Id deliveryUpdateId : DeliveryUpdateId orderId : OrderId } stream { FoodOrder | FoodDeliveryUpdate } ``` In the above scenario, a join is performed on the type `OrderId`, because it is the only type present on both models, where one of the models contains the `@Id` annotation. `DeliveryUpdateId` was not considered, as it is not present on `FoodOrder`. ## Running long-lived streaming queries Orbital's query editor is great for running short-lived streaming queries. However, often times you want a streaming query to continue in the background. To deploy a long-lived streaming query, simply define a query in one of your taxonomy projects. Typically this is checked into a git repository. ```taxi MySavedQuery.taxi // A sample query that streams data from a Stock price stream, // and writes to Postgres query MySavedQuery { stream { StockPrices } call MyPostgresService::upsertStockPrice } ``` ### Controlling parallelism By default, in a multi-node cluster, a background stream is processed on a single node. To control this, use the `@Parallel` annotation: ```taxi > import com.orbitalhq.streams.Parallel // don't forget the import > @Parallel(count = 4) query MyStream { stream { CustomerLoggedInEvent } call MyMongoDb::upsertCustomerMetrics } ``` This ensures that the stream is scheduled on 4 nodes (or all the available nodes if there's fewer than 4 nodes in the cluster). Orbital correctly rebalances as nodes enter and leave the cluster. **Note: Remember your paritions** When using a broker like Kafka, a Parallel count should be **at most** the number of partitions configured for the topic. Otherwise adding additional parallel consumers has no effect ### Updating streaming queries Streaming queries are automatically upgraded whenever their definition changes. ### Enabling streaming queries #### Default behavior By default, streaming queries are disabled when first detected to prevent accidental data mutations. You can override this behavior by [configuring the server setting](/docs/deploying/configuring-orbital#passing-orbital-application-configuration) `--vyne.streams.initial-state=RUNNING`. | Value | Behavior | |-----------|-----------------------------------------------------------------------------------------------------------------------| | `PAUSED` | **Default.** Streams remain disabled until manually enabled. Once enabled, they maintain their state across restarts. | | `RUNNING` | Streams are automatically enabled when detected by Orbital. | #### Manual activation You can enable individual streaming queries through: - The web UI - API calls **Important:** Once a streaming query has been enabled at least once, it will automatically deploy future updates and continue running according to its configured state. -------------------------------------------------------------------------------- # Publishing queries (HTTP / WebSockets) Source: https://orbitalhq.com/docs/querying/queries-as-endpoints Section: Querying -------------------------------------------------------------------------------- You might decide that instead of [submitting queries](/docs/querying/writing-queries#submitting-queries) directly to Orbital, or using one of our [SDKs](/docs/sdks/java-and-kotlin/jvm-query-sdk) to execute a query programmatically, that you'd rather just publish the query as an HTTP(s) endpoint. ## Saved queries Any queries committed to your taxonomy that have an `@HttpOperation` annotation will automatically be converted into a HTTP query. For example: ```taxi MyQuery.taxi import taxi.http.HttpOperation import taxi.http.PathVariable @HttpOperation(url = '/api/q/movies/{movieId}', method = 'GET') query getMoviesAndReivews(@PathVariable("movieId") movieId: MovieId) { given { movieId } find { title : MovieTitle reviewScore : ReviewScore } } ``` Any declared `query` that's in a Taxi project published to Orbital can be converted into an HTTP Operation. ### HttpOperation annotation Queries need to have an `@HttpOperation` annotation: | Param | Description | |--------|-------------------------------------------------------------------------------------------------| | url | The url the query is exposed under. It *must* start `/api/q`. Any other query URLs are ignored | | method | The HTTP method to respond to, one of `GET`,`POST`,`PUT`, or `DELETE` | Variables can be defined in your path using `{myVar}`, which are then made available in your query declaration using a `@PathVariable` annotation. **Note: Heads up!** For security reasons, queries are only exposed under the `/api/q` endpoint. Any queries where url in the `@HttpOperation` annotation starts with something other than `/api/q` are ignored. **Routes are workspace-scoped.** Orbital inserts the workspace's coordinates between `/api/` and `/q/` when registering the route, so a query declared as `@HttpOperation(url = '/api/q/movies')` in workspace `acme/films` is actually served at `/api/acme/films/q/movies`. You can also write the workspace-scoped path explicitly (`/api/acme/films/q/movies`), but the coordinates must match the workspace the query is declared in — declaring another workspace's coordinates is rejected to prevent cross-workspace routing. In general, prefer the shorthand `/api/q/...` form and let the router add the workspace prefix. ### Declaring the query Named queries are defined using a `query` syntax: ```taxi import taxi.http.HttpOperation namespace com.foo @HttpOperation(url = '/api/q/movies', method = 'GET') query GetMoviesAndReviews { find { Film[] } } ``` This declares a query named `com.foo.GetMoviesAndReviews`. If the query lives in workspace `acme/films`, it is served at `http://{orbitalUrl}/api/acme/films/q/movies` — the router inserts the workspace coordinates into the declared `/api/q/...` path. ### Inputs to queries Queries can accept inputs. Inputs are declared as variables, and are available to use throughout your query, for projecting, as inputs to functions, or as values on your response object. #### Path Variables Path variables can be specified by using the `taxi.http.PathVariable` annotation: ```taxi > import taxi.http.PathVariable @HttpOperation(url = '/api/q/movies/{movieId}', method = 'GET') query getMoviesAndReviews( // Define variables to be passed in > @PathVariable("movieId") id: MovieId ) { find { Film( MovieId == id ) } } ``` #### Http Headers Http headers can be declared using the `taxi.http.HttpHeader` annotation: ```taxi > import taxi.http.HttpHeader @HttpOperation(url = "/api/q/hello", method = "GET") query UpdateTheCsvPeople( > @HttpHeader(name = "X-Request-Name") personName:String ) { find { message : String = "Hello " + personName } } ``` #### Request Body For a POST query, the request body is also available, use `taxi.http.RequestBody` annotation: ```taxi > import taxi.http.RequestBody @HttpOperation(url = "/api/q/search", method = "POST") query doASearch( > @RequestBody searchRequest: SearchRequest ) { find { Film[](DateReleased >= searchRequest.dateReleased ) } } ``` ### Given clause If you've written Taxi queries before, you're used to using the [given](https://docs.taxilang.org/language-reference/querying-with-taxiql/#providing-start-hints) clause to provide data into the query. For saved queries, that data is often passed as a variable, so the syntax is slightly different: ```taxi import taxi.http.PathVariable // Not a saved query, given clause has a value: given { movieId: MovieId = 123 } find { ... } // A saved query, the movieId comes from the params: query findMoviesAndReviews(@PathVariable(...) movieId: MovieId) { given { movieId } // exposes the parameter into the query as a fact. find { ... } } ``` ### Streaming Results with Server-Sent Events (SSE) To receive query results as a continuous stream of Server-Sent Events, set the Accept header to `text/event-stream`. This method can be applied to both standard Request/Response queries and streaming queries. When using SSE, especially with queries returning multiple records (e.g., database queries), the time-to-first-byte is generally faster. This efficiency is due to records being transmitted as soon as they become available, unlike the standard approach where all records are delivered together at the end of the request. Example request using `curl` (assuming the query is declared in workspace `acme/films`): ```bash curl -X GET 'http://localhost:9022/api/acme/films/q/streamingTest' \ -H 'Accept: text/event-stream;charset-UTF-8' ``` ## Saved streams Streaming queries can be exposed over either HTTP (using Server-Sent Events), or Websocket (or both). Saved streams are executed in the background by Orbital. By connecting to the output stream, you can observe the results. Note that when you observe a result feed from a streaming query, results are published from the point the request is received. Previous results are not published on result stream. **Note: Heads up!** You can expose streaming queries on both Server-Sent event endpoints and Websockets - simply add both the annotations. ### Publishing on WebSocket To publish a query over a websocket, annotate the query with the `WebsocketOperation` annotation, declaring the `path`: **Note: Heads up!** For security reasons, websocket streams are only exposed under the `/api/s` path. Any queries where path in the `@WebsocketOperation` annotation starts with something other than `/api/s` are ignored ```taxi import taxi.http.WebsocketOperation @WebsocketOperation(path = '/api/s/newReleases') query getNewReleaseAnnouncements { stream { NewReleaseAnnouncement } as { title : MovieTitle reviewScore : ReviewScore }[] } ``` ### Publishing as Server-Sent events Streaming queries can also be published over HTTP by declaring a `@HttpOperation`, and then requesting a `text/event-stream` response: ```taxi import taxi.http.HttpOperation @HttpOperation(url = '/api/q/newReleases', method = 'GET') query getNewReleaseAnnouncements { stream { NewReleaseAnnouncement } as { title : MovieTitle reviewScore : ReviewScore }[] } ``` Once this is published, results can be streamed by requesting a `text/event-stream` (substituting the workspace coordinates the query is declared in): ```bash curl -X GET 'http://localhost:9022/api/acme/films/q/newReleases' \ -H 'Accept: text/event-stream;charset-UTF-8' ``` ## Authentication When Orbital is deployed with an [authentication provider](/docs/deploying/authentication) configured, all published query endpoints (HTTP and Server-Sent Events) require authentication by default. Requests without a valid token receive a `401 Unauthorized` response. ### Allowing anonymous access Some queries are intended to be public — for example a health check, a public catalogue listing, or anything you want callers to invoke without a token. To opt a single query out of authentication, annotate it with `@com.orbitalhq.authentication.AllowAnonymous`: ```taxi import taxi.http.HttpOperation import com.orbitalhq.authentication.AllowAnonymous @HttpOperation(url = '/api/q/healthcheck', method = 'GET') @AllowAnonymous query Healthcheck { find { status: String = "ok" } } ``` The opt-out is per-query: every other query published in the same workspace continues to require authentication. Authenticated requests against an `@AllowAnonymous` query are still permitted — the annotation only relaxes the requirement, it does not reject tokens. **Note: Heads up!** `@AllowAnonymous` only affects authentication (is the caller known?). [Data policies](/docs/data-policies/data-policies) and authorization rules still run, and may require a principal to evaluate. Apply `@AllowAnonymous` only to queries whose results and underlying operations are safe to expose publicly. When no authentication provider is configured at all, every query is reachable without a token regardless of annotation — the annotation only matters once auth is enabled. ## Live Reload Any changes made to the queries are automatically deployed. * When developing locally, this is as soon as you save a file. * In production, when using a Git repository, as soon as changes are merged, they're deployed on the next poll (typically a couple of minutes) ## OpenAPI Orbital automatically creates an OpenAPI spec for any query endpoints it serves. The OpenAPI specs are available at either `/api/q/meta/{nameOfQuery}/oas` or the workspace-scoped form `/api/{orgId}/{workspaceId}/q/meta/{nameOfQuery}/oas`. For example, a query defined as `query findMoviesAndReviews(...)` in workspace `acme/films` would have an OpenAPI spec available at `/api/acme/films/q/meta/findMoviesAndReviews/oas`. -------------------------------------------------------------------------------- # Scheduled queries Source: https://orbitalhq.com/docs/querying/scheduled-queries Section: Querying -------------------------------------------------------------------------------- Orbital supports running saved queries automatically on a schedule. This is useful for scenarios such as: * Polling an external API every few minutes * Refreshing a dataset at a regular interval * Running nightly sync or cleanup jobs Scheduled queries complement existing [HTTP Endpoints](/docs/querying/queries-as-endpoints) and [Background Streams](/docs/querying/streaming-data#running-long-lived-streaming-queries), giving you multiple ways to publish and reuse queries. ## Defining a Scheduled Query You declare a scheduled query using the `@Scheduled` annotation: ```taxi > import com.orbitalhq.scheduler.Scheduled > @Scheduled(cron = "0 0/5 * * * ?") query RefreshProducts { find { Product[] } } ``` Be sure to include the import statement: `import com.orbitalhq.scheduler.Scheduled` This runs the query `RefreshProducts` every 5 minutes. Under the hood, Orbital uses [Quartz Scheduler](https://www.quartz-scheduler.org/) with a JDBC-backed job store, so schedules are **persistent and cluster-safe**. ## Annotation Reference ```taxi namespace com.orbitalhq.scheduler annotation Scheduled { /** * Cron expression (Quartz format) */ cron: String? /** * Run with fixed rate (ISO-8601 duration or shorthand like "10m") */ fixedRate: String? /** * Max number of executions (-1 = forever) */ repeatCount: Int = -1 /** * What to do if a fire is missed */ misfirePolicy: MisfirePolicy = MisfirePolicy.Default /** * Trigger priority (higher runs first) */ priority: Int = 5 } enum MisfirePolicy { Default, FireImmediately, Skip } ``` | Property | Description | Example / Values | Default | |-------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------|------------------------------------------| | **cron** | A [Quartz cron expression](https://www.quartz-scheduler.org/documentation/quartz-2.3.0/tutorials/crontrigger.html) with 6 fields: seconds, minutes, hours, day-of-month, month, day-of-week. | `0 0 2 * * ?` → 2:00 AM daily | *None* (required if `fixedRate` not set) | | **fixedRate** | Run at a fixed interval. Supports ISO-8601 durations or shorthand. | `PT10M` or `10m` | *None* (required if `cron` not set) | | **repeatCount** | Number of times to run. | `-1` = forever | `-1` | | **misfirePolicy** | How to handle missed runs. | `Default` (Quartz’s smart policy, usually fire once now) `FireImmediately` (catch up immediately) `Skip` (ignore missed runs, wait for next) | `Default` | | **priority** | Trigger priority if multiple jobs fire simultaneously. | `5` (default), higher numbers run first | `5` | **Note: Provide either cron or fixedRate** Exactly one of `cron` or fixedRate must be provided. ## Supported Query Types Scheduled queries must be **finite queries** (`find` or `call`). Streaming queries (`stream { }`) cannot be scheduled. ## Admin & Monitoring Scheduled queries are shown in the Orbital UI under the **Saved Queries** tab. Query results from each scheduled run are stored in Orbital's query history, if enabled. ## Cluster Behavior When running in a cluster, job scheduling is co-ordinated across the cluster, using Quartz and the Postgres job store to coordinate execution. At any given time, exactly one node will execute a scheduled query, ensuring there are no duplicates. If a node crashes or becomes unavailable, another node in the cluster automatically takes over future runs. ## Examples **Hourly polling job**: ```taxi > import com.orbitalhq.scheduler.Scheduled > @Scheduled(fixedRate = "1h") query PollApi { call { ExternalApi[] } } ``` **Nightly batch job**: ```taxi @Scheduled(cron = "0 0 2 * * ?", misfirePolicy = FireImmediately) query NightlySync { find { Orders } } ``` ## Limitations * No support yet for multiple schedules per query (one `@Scheduled` per query). * Missed runs are not replayed unless `misfirePolicy` is set. -------------------------------------------------------------------------------- # Caching in queries Source: https://orbitalhq.com/docs/querying/caching Section: Querying -------------------------------------------------------------------------------- ## Caching requests By default, Orbital will cache requests throughout a query only. ie., Orbital will only call a service with the same set of parameters once within a query. However, caches are destroyed at the end of each query. To enable caching, simply add `@Cache` to the top of your query: ```taxi > @Cache find { Film[] } as { id : FilmId // Fetched from a remote service. // Calls will be cached. currentReviewScore : ReviewScore } ``` ### Scoping caches By default, all queries share the same cache. If you want cache isolation, so that population and invalidation are isolated from each other, you can apply a name to the cache: ```taxi > @Cache("myCache") find { Film[] } as { id : FilmId // Fetched from a remote service. // Calls will be cached. currentReviewScore : ReviewScore } ``` ## Using Orbital's local cache Orbital's local cache runs in-process within Orbital. It's the default cache that's enabled if no connection is provided. It's helpful for development, but shouldn't be taken to production, as excessive caching can cause Out-of-Memory failures and degrade performance. ```taxi // This query uses the default, local cache > import com.orbitalhq.caching.Cache > @Cache find { Film[] } as { id : FilmId // Fetched from a remote service currentReviewScore : ReviewScore } ``` ## Caching and streams Operations that return streams are never cached ## Preventing caching Caching can be prevented either on an operation, or a response model, by specifying a `mode = CachePolicy.Disabled`. For example, to disable caching on all responses from a specific operation: ```taxi > import com.orbitalhq.caching.Cache service PersonApi { // Responses from this operation are never cached. > @Cache(mode = CachePolicy.Disabled) operation getAccount(AccountId):AccountState } ``` Or, to prevent caching on a model: ```taxi > import com.orbitalhq.caching.Cache > @Cache(mode = CachePolicy.Disabled) model AccountState { // implementaion omitted } service AccountApi { // The result of this call isn't cached, // as it's prevented on the model definition operation getAccount(AccountId):AccountState } ``` When defining cache controls on a model, only cache controls on the top-level entity are considered. ## Cache TTL Responses written to a cache have a default TTL of 5 minutes. You can control this by setting a [cache-level TTL](#cache-ttl-1). Orbital will also honour the [max-age](#max-age-headers) in response headers. ### Fine-grained cache control TTL can be controlled by setting the `maxIdleSeconds` on either an operation, or a return type. For example: ```taxi > import com.orbitalhq.caching.Cache service PersonApi { // Responses from this operation are evicted after not being read or // written for 30 seconds > @Cache(maxIdleSeconds = 30) operation getAccount(AccountId):AccountState } ``` Alternatively, controls can be applied on the model: ```taxi > import com.orbitalhq.caching.Cache > @Cache(maxIdleSeconds = 30) model AccountState { // implementaion omitted } service AccountApi { // Responses from this operation are evicted after not being read or // written for 30 seconds operation getAccount(AccountId):AccountState } ``` ### max-age headers Orbital will honour max-age headers when caching responses from HTTP services: Setting the `cache-control: max-age=` header will cause the entry to be automatically evicted after the `max-age` expires. ## Using an external cache Orbital supports external caches, and will run a near-cache in-process, offloading to the remote cache as the library supports. To enable a remote cache, first define a connection to the cache in your `connections.conf` file, then specify the name of the connection in your query. ```hocon connections.conf hazelcast { myHazelcast { connectionName = myHazelcast addresses = ["localhost:5701"] } } ``` ```taxi query.taxi > @Cache(connection = "myHazelcast") find { Film[] } as { id : FilmId // Fetched from a remote service currentReviewScore : ReviewScore } ``` ### Supported caches Orbital supports caching with the following platforms: * Hazelcast * Redis If you need cache not listed, please [reach out on slack](https://join.slack.com/t/orbitalapi/shared_invite/zt-697laanr-DHGXXak5slqsY9DqwrkzHg) ### Cache TTL The default Cache TTL is defined in the connections file: ```hocon connections.conf hazelcast { myHazelcast { connectionName = myHazelcast addresses = ["localhost:5701"] > ttl = 20 minutes } } ``` -------------------------------------------------------------------------------- # Query lineage and observability Source: https://orbitalhq.com/docs/querying/observability Section: Querying -------------------------------------------------------------------------------- ## Observing queries As queries are executed by Orbital, full call histories are captured and persisted This gives rich debugging tools to understand exactly how a query was executed, which services were called and what they returned. ### Configuring what's captured What Orbital records can be tuned - or turned off entirely - at three levels: server-wide defaults, per workspace, and per query. When fully enabled, the following detail is captured and written to a database: * Requests and Responses sent to services - such as APIs and Databases * Metadata - such as URLs, response times, payload sizes, response codes and headers * Message payloads for streaming event sources such as message brokers * Variations of the above data to build query plan charts visualised in the Orbital UI Server defaults are set with the `orbital.history.*` settings (previously `vyne.analytics.*`). Workspaces can override them from the **Workspace settings** page, and individual queries can carry a `@QueryTelemetry` annotation. See [Query history settings](/docs/workspace/query-history-settings) for the full list of settings and how the levels interact. ## Performance metrics - Prometheus Orbital publishes metrics available for prometheus to collect. If Orbital is connected to a Prometheus endpoint, performance data is made available in the Endpoints UI. _Image: Performance metrics_ ### Streaming queries The following metrics are published for persistent queries (both streaming and request / response queries) * Count of requests processed * Processing duration * Error Counts Prometheus can then provide this data in histogram, providing min / max and averages ### Configuring Prometheus An example prometheus configuration is shown below ```yaml - job_name: 'Orbital Metrics' metrics_path: '/api/actuator/prometheus' scrape_interval: 3s static_configs: # 172.17.0.1 is the ip address of localhost # from within docker - targets: ['172.17.0.1:9022'] labels: application: 'Orbital' ``` -------------------------------------------------------------------------------- # Throwing errors Source: https://orbitalhq.com/docs/querying/errors Section: Querying -------------------------------------------------------------------------------- ## Overview In Orbital and Taxi, error handling is managed through a `throw` function that allows users to control the response sent back to the user. This includes setting the error code and the response payload. Note that currently, we do not support catching errors; throwing an error is a fatal action. This will be addressed in a future release. ## Throwing Errors Errors are thrown using the `throw` function. The syntax for throwing an error is: ```kotlin throw((ErrorType) { errorPayload }) ``` This is actually a casting operation, casting the payload value to the defined error type. This is because Taxi does not have a concept of constructors, or object creation. For example, given a `NotAuthorizedError`, defined as follows: ```taxi NotAuthorized.taxi import com.orbitalhq.errors.Error model NotAuthorizedError inherits Error { message: ErrorMessage } ``` This would be thrown as follows: ```taxi throw((NotAuthorizedError) { message: "Authentication failed" }) ``` Here, `(NotAuthorizedError)` is a casting statement, as Taxi does not have the concept of constructors. ### Defining Errors Errors are defined as models in Taxi. An error model must inherit from the base `Error` type (`com.orbitalhq.errors.Error`) and can include annotations to control the HTTP response code and the response body. Below are examples of how to define and use error models. ### Example: NotAuthorizedError The `NotAuthorizedError` is provided out-of-the-box and is defined as follows: ```taxi @ResponseCode(401) model NotAuthorizedError inherits Error { message: ErrorMessage } ``` To throw this error with a custom message: ```taxi throw((NotAuthorizedError) { message: "Authentication failed" }) ``` ## Response Codes and Payloads When an error is thrown, users can control the HTTP response code and the response payload using annotations. ### Example: Custom Response Code If no response code is provided, the default response code is 400. To specify a custom response code, use the `@ResponseCode` annotation: **Note: Don't forget the import** Don't forget to include the import of `taxi.http.ResponseCode` ```taxi import taxi.http.ResponseCode @ResponseCode(403) model NotAuthorizedError inherits Error { message: ErrorMessage } ``` ### Example: Custom Response Body To customize the response body, use the `@ResponseBody` annotation: **Note: Don't forget the import** Don't forget to include the import of `taxi.http.ResponseBody` ```taxi import taxi.http.ResponseBody model BadPermissionsError inherits Error { @ResponseBody error: { errorCode: String message: String } } ``` Thrown as follows: ```taxi throw( (BadPermissionsError) { error: { errorCode: 'E1234', message: "You didn't say the magic word" } } ) ``` This would generate an error as follows: ```json { "errorCode" : "E1234", "message" : "You didn't say the magic word" } ``` ## Built-in errors As part of the release in 0.34, the following errors will be provided out-of-the-box. | **Error Type** | **Description** | **HTTP Response Code** | |---------------------------------|-----------------------------------------------------------------------------------------|----------------------------| | **OperationFailedError** | Thrown when an operation fails to be invoked (e.g., a server returned a 4xx/5xx error). | 400 (Client Error) | | **ModelContractViolationError** | Thrown when a model cannot be constructed, generally because data was missing. | 422 (Unprocessable Entity) | | **ParseError** | Thrown when source content could not be parsed (e.g., malformed JSON or CSV). | 400 (Bad Request) | | **DataNotDiscoverableError** | Thrown when the query asked for data, but no services could provide the requested data. | 404 (Not Found) | | **NotAuthorizedError** | Thrown to indicate that a requested data attribute or service call has been rejected. | 403 (Forbidden) | -------------------------------------------------------------------------------- # Overview Source: https://orbitalhq.com/docs/data-formats/overview Section: Data formats -------------------------------------------------------------------------------- Data formats specify how Orbital should read and write data. By default, Orbital assumes that data is provided as JSON. However, by adding a Data Format to a model, this can be modified to any type of serialization format. ## API Specs and Message formats Some API specs which define transport also have their own message format (eg., Protobuf). If that's the case, you generally don't need to do anything - Orbital applies the correct formats when parsing the API spec. | API Spec | Message Format | Docs | |----------|----------------|---------------------------------------------------------| | Protobuf | Protobuf | [Protobuf docs](/docs/describing-data-sources/protobuf) | | OpenAPI | Json | [OpenAPI docs](/docs/describing-data-sources/open-api) | | SOAP | XML | [SOAP docs](/docs/describing-data-sources/soap) | | Avro | Avro | [Avro docs](/docs/data-formats/avro) | ## Using message formats in Taxi If you're using Taxi to describe your API specs, you need to specify the data format that is used for reading/writing messages. By default, messages are expected to be in JSON. Other formats can be specified using an annotation on the top level type. For example: ```taxi > import com.orbitalhq.formats.Xml > @Xml model Person { firstName : FirstName } ``` You only need to apply the annotation to the top level entity. | Message format | Annotation | Links | |----------------|---------------------------------------------------|----------------------------------------------------------------------------------| | Json | Not needed - default format | - | | Xml | `com.orbitalhq.formats.Xml` | [Xml annotation docs](/docs/data-formats/xml) | | Csv | `com.orbitalhq.formats.Csv` | [Csv annotation docs](/docs/data-formats/csv) | | Protobuf | Not supported in Taxi - Use a protobuf spec | [File an issue to request support](https://github.com/orbitalapi/orbital/issues) | | Avro | Not supported directly in Taxi - Use an Avro spec | [File an issue to request support](https://github.com/orbitalapi/orbital/issues) | ## Defining custom formats Internally, Orbital uses a [ModelFormatSpec](https://github.com/orbitalapi/orbital/blob/develop/vyne-core-types/src/main/java/com/orbitalhq/models/format/ModelFormatSpec.kt) to provide format-specific serialization & deserialization implementations. While it is possible to define your own custom format, to load it into Orbital currently requires a custom build - available for Enterprise customers only. Support for loading custom formats via Taxi projects is planned - Vote for [this issue](https://github.com/orbitalapi/orbital/issues/8) or [reach out to us](https://join.slack.com/t/orbitalapi/shared_invite/zt-697laanr-DHGXXak5slqsY9DqwrkzHg) if you'd like to discuss getting this feature supported. -------------------------------------------------------------------------------- # Avro Source: https://orbitalhq.com/docs/data-formats/avro Section: Data formats -------------------------------------------------------------------------------- ## Reading and writing Avro Orbital can read and write data using [Avro](https://avro.apache.org/). To declare that a type should be read as Csv, add the `lang.taxi.formats.AvroMessage` annotation to a model: ```taxi > import lang.taxi.formats.AvroMessage > @AvroMessage closed model People { name : Name id : sId email : Email? phones : Phones[] last_updated : LastUpdated } ``` When writing Avro, serialization order is important. Therefore, you should also specify the field order using an `AvroField` annotation: ```taxi import lang.taxi.formats.AvroMessage > import lang.taxi.formats.AvroField @AvroMessage closed model People { > @AvroField(ordinal = 0) name : Name @AvroField(ordinal = 1) id : Id @AvroField(ordinal = 2) email : Email? @AvroField(ordinal = 3) phones : Phones[] @AvroField(ordinal = 4) last_updated : LastUpdated } ``` Any requests or responses sent to/from systems where the model type has been annotated with `@AvroMessage` will be serialized using Avro. ## Adding Taxi Types to Avro Just like all schemas in Orbital, Taxi types are required to indicate how data is linked and relates between systems. To add taxi metadata to your Avro file, use the `taxi.dataType` attribute. Here's an example assigning a Taxi type name to a record / object: ```json5 { "type": "record", "namespace": "simple", "name": "AddressBook", // without this annotation, a type of simple.AddressBook would've // been created, based off the namespace and name > "taxi.dataType": "com.example.AddressBook", "fields": [ //... continues ... ``` Or, to define a taxi type for a field: ```json5 { "type": "record", "name": "Person", "fields": [ { "name": "name", "type": "string", > "taxi.dataType": "foo.PersonName" }, { "name": "id", "type": "int", > "taxi.dataType": "foo.PersonId" }, ``` If a schema does not define / expose Taxi types for fields, Orbital will assign them when automatically. For records, the `namespace` and `name` are used to infer the type name. For fields, a type is created based off the field name, and the `type`, which defines the underling primitive type. ## Adding Avro projects to your workspace ### Different ways to work with Avro specs There are a few different ways to expose your Avro spec to Orbital, each which suits different team workflows. | Approach | Description | Suits teams who... | Considerations | |----------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------| | Adding avro as a dedicated [git](#git-projects) or [local file](#local-file-projects) project | Declares the Avro spec as a project in your workspace. Keeps the Avro spec as a source-of-truth, and allows a team to evolve the spec independently | Best for distributed teams or federating multiple APIs, where teams evolve their API specs independently | Changes made to the Avro spec **are** automatically detected Requires adding Taxi annotations to the Avro spec | | Mixing Avro and Taxi in the same project [docs](/docs/workspace/projects#combining-taxi-and-other-api-specs-in-a-single-project) | Keeps your Avro spec as source-of-truth, but stores alongside other Taxi sources | Best when a single team owns/manages Orbital, and wants a mono-repo approach | Changes made to the Avro spec **are** automatically detected Requires adding Taxi annotations to the Avro spec | Schemas declared in Avro can be added to your Workspace by either editing your workspace file (normally `workspace.conf`), or through the UI. ### Using the Workspace file #### Local file projects You can add Avro schemas that are present locally on the file system of the server by adding the following declaration to your `workspace.conf` file **Note: Heads up...** The project identifier is duplicated below. That's necessary today, but we'll address that in a future release ```hocon file { projects=[ { loader { // All projects in Orbital require a unique identifier identifier { id="com.marty/avro-test/1.0.0" name=avro-test organisation="com.marty" unversionedId="com.marty/avro-test" uriSafeId="com.marty:avro-test:1.0.0" version="1.0.0" } // set the packageType to Avro packageType=Avro } packageIdentifier { name=avro-test organisation="com.marty" version="1.0.0" } // The path on your local machine to the avro schema path="/home/martypitt/dev/vyne/schema-server-core/src/test/resources/avro/addressBook.avsc" } ] } ``` #### Git projects You can add Avro schemas that are present within a git repository by adding the following declaration to your `workspace.conf` file: The Git repo is monitored, so any changes are pulled locally and automatically applied inside of Orbital. **Note: Heads up...** The project identifier is duplicated below. That's necessary today, but we'll address that in a future release ```hocon git { repositories=[ { branch=main loader { identifier { id="com.avro.git/avro/1.2.3" name=avro organisation="com.avro.git" unversionedId="com.avro.git/avro" uriSafeId="com.avro.git:avro:1.2.3" version="1.2.3" } packageType=Avro } name=test-project // The path to the avro file within your Git repository path="/addressBook.avsc" uri="https://gitlab.com/acne/my-avro-project.git" } ] } ``` ### Using the UI You can load Avro schemas into your workspace using the UI Start by navigating: * Projects > Add Project * Then select to add a project from either a Git Repo or Local disk #### Adding an Avro project from Git This workflow lets you add a reference to an Avro schema that's checked into a git repository. The Git repo is monitored, so any changes are pulled locally and automatically applied inside of Orbital. * Provide the url of the Git repository, and click "Test Connection" * If the test was successful, repository name and branch are populated for you * Set the project type to "Avro" * Set the path to the location of your avro schema within the repo * Provide a unique package identifier for this avro schema #### Adding an avro project from a file This workflow lets you add a reference to an Avro schema that's already local on the server. Any changes made to the file are automatically detected, and updated inside of Orbital. **Note: 'Local' means local to the server** This workflow adds a reference to a Avro file that's **on the disk of the server**. It's intended for developers who are running Orbital in a docker image on their local machine. This workflow isn't intended for uploading an Avro schema to a remote server. Instead, use a Git repository * Set the project type to "Avro" * Provide the path on your local machine to where the avro file is * Provide a unique package identifier for this avro schema -------------------------------------------------------------------------------- # CSV Source: https://orbitalhq.com/docs/data-formats/csv Section: Data formats -------------------------------------------------------------------------------- To declare that a type should be read/written as Csv, add the `com.orbitalhq.formats.Csv` annotation to a model: ```taxi import com.orbitalhq.formats.Csv @Csv model Person { firstName : FirstName inherits String } ``` ## Csv options The following options are available: | Parameter name | Description | Default value | |----------------------------|--------------------------------------------------------------------|---------------| | delimiter | The delimiter between records | `,` | | firstRecordAsHeader | Indicates if the first record should be read as a header | true | | nullValue | Specify a custom string to be read as `null` | | | containsTrailingDelimiters | Indicates if each row contains a delimiter which should be ignored | false | | quoteChar | Specifies which character should be read as a quoting character | `"` | For example: ```taxi import com.orbitalhq.formats.Csv @Csv( delimiter = "|", nullValue = "NULL" ) model Person { firstName : String lastName : String age : Int } ``` ## Using column() references **Warning: Deprecated** Using the `by column` tag is deprecated. Field names in the model are now used to match columns It's possible (but discouraged) to use `by column("")` references to declare a column: ```taxi model Person { firstName : FirstName by column("fName") } ``` -------------------------------------------------------------------------------- # JSON Source: https://orbitalhq.com/docs/data-formats/json Section: Data formats -------------------------------------------------------------------------------- JSON is the default format for reading / writing data to/from Orbital, so requires no special customization. ## Using jsonPath() references **Warning: Deprecated** Using the `by jsonPath()` tag is deprecated. Field names in the model are now used to match elements and attributes It's possible (but discouraged) to use `by jsonPath("")` references to declare a field: ```taxi model Person { firstName : FirstName by jsonPath("$.fName") } ``` -------------------------------------------------------------------------------- # XML Source: https://orbitalhq.com/docs/data-formats/xml Section: Data formats -------------------------------------------------------------------------------- To declare that a type should be read/written as Xml, add the `com.orbitalhq.formats.Xml` annotation to a model: ```taxi import com.orbitalhq.formats.Xml @Xml model Person { firstName : FirstName inherits String } ``` ## Declaring Attributes By default, scalar items within a model are expected to be an Xml Element. To declare something should be read / written as an attribute, use the `@lang.taxi.xml.XmlAttribute` annotation. For example: ```taxi import com.orbitalhq.formats.Xml > import lang.taxi.xml.XmlAttribute @Xml model Actor inherits Person { > @XmlAttribute id : ActorId fullName : FullName } ``` Would read/write the following xml: ```xml Jimmy Smith ``` ## Collections Collections are indicated by a repeating group with the attribute name. For example: ```taxi @Xml model Movie { > actors : Actor[] } ``` would expect a repeated `actors` element: ```xml Mel Gibson Jack Spratt ``` ## Expressions Expressions can be decalred on a model. ### Serializing Expression values are written out as normal elements ### Deserializing If an element or attribute is present for the field where the expression is declared, the expression is ignored, and the value from the source is taken. If there's no value present, then the expression is evaluated. ```taxi import com.orbitalhq.formats.Xml import lang.taxi.xml.XmlAttribute @Xml model Actor { firstName : FirstName inherits String lastName : LastName inherits String fullName : FullName inherits String = FirstName + ' ' + LastName } ``` Given the following XML: ```xml Jimmy Smith ``` Then the value is read as: ```json { "firstName" : "Jimmy", "lastName" : "Smith", "fullName" : "Jimmy Smith" } ``` ## Using xpath() references **Warning: Deprecated** Using the `by xpath` tag is deprecated. Field names in the model are now used to match elements and attributes It's possible (but discouraged) to use `by xpath("")` references to declare a column: ```taxi model Person { firstName : FirstName by xpath("//fName") } ``` -------------------------------------------------------------------------------- # Protobuf Source: https://orbitalhq.com/docs/data-formats/protobuf Section: Data formats -------------------------------------------------------------------------------- Protobuf is supported when using a dedicated Protobuf spec. Using protobuf without a spec is technically [possible](/docs/describing-data-sources/protobuf#generated-taxi), but discouraged, and not currently supported. Instead, use a dedicated [Protobuf](/docs/describing-data-sources/protobuf) spec. -------------------------------------------------------------------------------- # Other formats Source: https://orbitalhq.com/docs/data-formats/custom-data-formats Section: Data formats -------------------------------------------------------------------------------- Orbital has support for several data formats, however if you need a format that's not here, it's possible to implement your own. Orbital uses a [ModelFormatSpec](https://github.com/orbitalapi/orbital/blob/develop/vyne-core-types/src/main/java/com/orbitalhq/models/format/ModelFormatSpec.kt) to provide format-specific serialization & deserialization implementations. While it is possible to define your own custom format, to load it into Orbital currently requires a custom build - available for Enterprise customers only. Support for loading custom formats via Taxi projects is planned - Vote for [this issue](https://github.com/orbitalapi/orbital/issues/8) or [reach out to us](https://join.slack.com/t/orbitalapi/shared_invite/zt-697laanr-DHGXXak5slqsY9DqwrkzHg) if you'd like to discuss getting this feature supported. If you need a format that's not listed here, [reach out to us](https://join.slack.com/t/orbitalapi/shared_invite/zt-697laanr-DHGXXak5slqsY9DqwrkzHg), and we'll probably just build it for you! -------------------------------------------------------------------------------- # Validating data Source: https://orbitalhq.com/docs/data-validation/data-validation Section: Data validation -------------------------------------------------------------------------------- Orbital can validate the data flowing through your queries — both the arguments callers pass in, and the data your services return — against rules declared in your taxonomy. Because rules are declared against **types**, you define them once, and they're enforced everywhere that type appears, no matter which service the data came from. There are three things to know: * **Declare** rules using annotations such as `@NotEmpty` or `@MaxLength` * **Activate** them by putting a `@Valid` scope over the data you want checked * **Decide** what happens when data fails, using `@OnValidationFailure` The validation annotations are built into Orbital, in the `com.orbitalhq.validation` namespace. You don't need to add anything to your project to use them. ## A quick example Here's a type with a rule, and a model that activates it: ```taxi import com.orbitalhq.validation.NotEmpty import com.orbitalhq.validation.Valid @NotEmpty type Name inherits String @Valid model Person { name : Name } ``` If a service returns this `Person`: ```json { "name": "" } ``` ...then Orbital produces a single violation — rule `com.orbitalhq.validation.NotEmpty`, at path `name`, with severity `Error` — and by default rejects the query with an HTTP 400. Remove the `@Valid`, and the same data flows through untouched. Rules do nothing until something activates them. **Note: Imports in these examples** Everything validation-related lives in `com.orbitalhq.validation`. The examples below assume you've imported the annotations they use (as above), but you can always write the fully-qualified name inline instead — `@com.orbitalhq.validation.NotEmpty` — which is handy in a one-off query. The same applies to the `Severity` and `ValidationAction` enums. ## Declaring rules Rules are annotations. You can put them on a type, on a field, or on a query parameter. **On a type** — every field of that type inherits the rule: ```taxi @NotEmpty type Name inherits String ``` **On a field** — the rule applies to that field only: ```taxi @Valid model Person { @NotEmpty name : Name } ``` **On a query parameter** — see [validating query arguments](#query-arguments) below. ### Rules are inherited A rule declared on a type applies to every type that inherits from it. `FirstName` picks up the `@NotEmpty` declared on `Name`: ```taxi @NotEmpty type Name inherits String type FirstName inherits Name @Valid model Person { firstName : FirstName } ``` An empty `firstName` reports a `NotEmpty` violation at path `firstName`. ### Multiple rules on the same field Rules accumulate — Orbital evaluates all of them and reports everything that failed, rather than stopping at the first failure: ```taxi @Valid model Doc { @MinLength(value = 5) @MaxLength(value = 3) code : String } ``` A `code` of `abcd` reports two violations, one for each rule. ### Customising the message and severity Every rule accepts a `message` and a `severity`, whatever else it takes: ```taxi @Valid model Doc { @MaxLength(value = 3, severity = Severity.Warning, message = "too long") code : String } ``` Severity is one of `Info`, `Warning` or `Error`, and defaults to `Error`. Severity matters because it determines whether a violation is serious enough to actually fail the query — see [thresholds](#choosing-a-threshold). See the [validation rule reference](/docs/data-validation/validation-rules) for the full set of rules and their parameters. ## Activating validation with @Valid Declared rules are inert until a `@Valid` scope covers them. This is deliberate: it lets you describe your data's constraints across your whole taxonomy, and then choose exactly where those constraints are enforced. `@Valid` does two things for everything inside its scope: * runs every declared rule, and * enforces the type system's own nullability — a **non-nullable field holding null is an `Error` violation** ### The same schema, with and without Without `@Valid`, nothing is checked: ```taxi @NotEmpty type Name inherits String // No @Valid - rules are declared, but never run model Person { name : Name } ``` A `Person` of `{ "name": "" }` produces no violations. Neither does `{ "name": null }`, even though `name` is non-nullable. With `@Valid`, both are reported: ```taxi @NotEmpty type Name inherits String @Valid model Person { name : Name nickname : Name? } ``` | Data | Violations | |------|------------| | `{ "name": "", "nickname": null }` | `NotEmpty` at `name` | | `{ "name": null, "nickname": null }` | `NotNull` at `name` | | `{ "name": "Jimmy", "nickname": null }` | none — `nickname` is nullable | **Note: Nullability comes for free** You never declare a "required" rule. A field is required if your taxonomy says it's non-nullable, and `@Valid` is what turns that declaration into an enforced constraint. ### @Valid recurses A `@Valid` scope covers the whole subtree beneath it — nested models, collections, and collections of models, to any depth: ```taxi @NotEmpty type Name inherits String model Person { name : Name } model Actor { actor : Person } model Cast { actors : Actor[] } @Valid model Film { cast : Cast } ``` `Film` is the only model marked `@Valid`, but the `NotEmpty` on `Name` is enforced all the way down. Recursive models are validated at every level too. Here, all four names are checked: ```taxi @MaxLength(value = 3) type ConstrainedName inherits String @Valid model Node { name : ConstrainedName child : Node? } ``` ### Scoping @Valid to part of a model Put `@Valid` on a field, and only that field's subtree is validated: ```taxi @NotEmpty type Name inherits String model Inner { name : Name } model Outer { @Valid validated : Inner unvalidated : Inner } ``` Given `{ "validated": { "name": "" }, "unvalidated": { "name": "" } }`, only `validated.name` is reported. The reverse works too — a `@Valid` model nested inside a model that isn't `@Valid` still activates its own subtree, and `@Valid` is inherited when one model inherits another. Marking the same thing `@Valid` twice (on a field, and on that field's type) doesn't double up the violations. ### The one exception: query parameters A rule annotation placed **directly on a query parameter** activates itself. You don't need `@Valid`: ```taxi query Greet( @com.orbitalhq.validation.NotEmpty name : String ) { find { message : String = "Hello " + name } } ``` If the parameter's type is a model, you can use `@Valid` on the parameter (or on the model) to cascade validation through the whole object graph — see [query arguments](#query-arguments). ## Violation paths Every violation carries the path to the value that failed, so callers can pinpoint exactly what to fix. Paths use dotted attribute names, with `[n]` for collection indices. Using the `Film` schema above: ```json { "cast": { "actors": [ { "actor": { "name": "Ridley" } }, { "actor": { "name": "" } } ] } } ``` produces one violation, at path: ``` cast.actors[1].actor.name ``` Paths nest through as many levels of collection as you have. A model like this: ```taxi @NotEmpty type Tag inherits String model Inner { tags : Tag[] } model Wrapper { inners : Inner[] } @Valid model Root { wrapper : Wrapper } ``` reports an empty tag at `wrapper.inners[1].tags[1]`. A few specifics worth knowing: * **Rules on a collection itself** (such as `@MinItems`) report at the collection's own path — `tags`, not `tags[0]` * **A rule on a top-level scalar parameter** reports the empty path, `""`, since there's no attribute to name * **A collection validated as a whole** (rather than element-by-element) indexes from the root: `[2].name` **Note: Operations that return a collection** When an operation's return type is a collection (say `operation getItems():LineItem[]`), Orbital validates each element as it arrives, so paths are relative to the element — `name`, not `[1].name`. A collection that's a _field_ of a returned model (like `items : LineItem[]`) is indexed as you'd expect: `items[1].name`. ## Controlling what happens on failure Producing violations and acting on them are separate steps. What Orbital does with a set of violations is controlled per-query, with `@OnValidationFailure`: ```taxi import com.orbitalhq.validation.OnValidationFailure import com.orbitalhq.validation.ValidationAction import com.orbitalhq.validation.Severity @OnValidationFailure(action = ValidationAction.Warn, threshold = Severity.Error) query GetOrder { find { Order } } ``` | Parameter | Values | Description | |-----------|--------|-------------| | `action` | `Reject`, `Warn`, `Drop` | What to do when a violation meets the threshold | | `threshold` | `Info`, `Warning`, `Error` | The minimum severity that triggers the action | ### The actions **`Reject`** fails the request with an HTTP 400 and a structured report of the violations. **`Warn`** lets the data through unchanged. The violations are still reported — they're logged, and passed to any violation sinks you've configured — but the caller gets their results. **`Drop`** skips the offending record and keeps going. This is for streams, where failing the whole stream because of one bad message is usually worse than skipping it. Dropping is never silent — every dropped record is reported and logged, exactly as it is under `Warn`. ### Defaults If a query has no `@OnValidationFailure`, Orbital picks a sensible default based on the kind of query: | Query | Default | |-------|---------| | Request/response (`find`) | `Reject` at `Error` | | Streaming (`stream`) | `Drop` at `Error` | Query arguments are the exception: they always `Reject`. Dropping a record makes no sense when the record is the caller's own input, so a `Drop` on the arguments boundary is treated as `Reject`. ### Choosing a threshold `threshold` is the minimum severity that triggers the action. Anything below it is reported but never fatal. With the default `threshold = Error`: * an `Error` violation rejects the query * a `Warning` violation is logged, and the query proceeds * an `Info` violation is logged, and the query proceeds Lower the threshold to `Warning` and warnings become fatal too. **All violations are reported regardless of threshold** — the threshold only decides which ones are severe enough to act on. This pairs well with per-rule severity overrides: mark the rules you're not yet confident about as `severity = Severity.Warning`, watch them in your logs, then promote them to `Error` once you trust them. ### The rejection response A rejected query returns HTTP 400 with a `ValidationFailureReport`: ```json { "message": "...", "violations": [ { "path": "items[1].name", "rule": "com.orbitalhq.validation.MaxLength", "severity": "Error", "message": "Value 'toolong' is longer than the maximum length of 3 (was 7)" } ] } ``` Here's a complete worked example. This query is published as an HTTP endpoint, and returns an `Order` whose line items are validated: ```taxi import taxi.http.HttpOperation import com.orbitalhq.validation.Valid import com.orbitalhq.validation.MaxLength @Valid model LineItem { sku : Sku inherits String @MaxLength(value = 3) name : ItemName inherits String } model Order { orderId : OrderId inherits String items : LineItem[] } service OrderService { operation getOrder() : Order } @HttpOperation(url = "/api/q/order", method = "GET") query GetOrder { find { Order } } ``` `Order` isn't itself `@Valid` — its `LineItem` elements are, so validation cascades into the `items` collection and the violation path carries the index. When `OrderService` returns: ```json { "orderId": "o1", "items": [ { "sku": "a", "name": "ok" }, { "sku": "b", "name": "toolong" } ] } ``` ...a `GET /api/q/order` responds with `400 Bad Request`: ```json { "message": "Validation failed for the result of operation getOrder: 1 violation(s) at items[1].name", "violations": [ { "path": "items[1].name", "rule": "com.orbitalhq.validation.MaxLength", "severity": "Error", "message": "Value 'toolong' is longer than the maximum length of 3 (was 7)" } ] } ``` ## Where validation runs Orbital validates at two points in a query. ### Query arguments Arguments are validated **before any operation is invoked** — a query rejected on its arguments never calls your upstream services. A rule on the parameter itself self-activates: ```taxi import taxi.http.HttpOperation import taxi.http.QueryVariable @HttpOperation(url = "/api/q/greet", method = "GET") query Greet( @com.orbitalhq.validation.NotEmpty @QueryVariable("name") name : String ) { find { message : String = "Hello " + name } } ``` `GET /api/q/greet?name=Jimmy` returns the greeting. `GET /api/q/greet?name=` returns `400`: ```json { "message": "Validation failed for query 'Greet' on parameter(s): name", "violations": [ { "path": "", "rule": "com.orbitalhq.validation.NotEmpty", "severity": "Error", "message": "Value must not be empty" } ] } ``` Add `@OnValidationFailure(action = ValidationAction.Warn)` to that query, and the same request succeeds with `"message": "Hello "` — the violation is logged instead. When a parameter's type is a model, `@Valid` cascades through the whole object graph: ```taxi @com.orbitalhq.validation.Valid model Person { personId : PersonId inherits String contact : ContactDetails } model ContactDetails { @com.orbitalhq.validation.NotEmpty email : EmailAddress inherits String } query FindPersonId( person : Person ) { find { PersonId } } ``` Passing a `person` with an empty `contact.email` rejects the query, reporting the violation at path `contact.email`, before any service is called. ### Data returned by services Every record returned by a service during query execution is validated as it's received. For a request/response operation, the whole result is validated and — by default — a violation rejects the query. For a streaming operation, each message is validated as it arrives, and by default violating messages are dropped while valid ones flow on: ```taxi import com.orbitalhq.validation.Valid import com.orbitalhq.validation.NotEmpty @Valid model Notification { @NotEmpty message : NotificationMessage inherits String } service Notifications { operation watchNotifications() : Stream } ``` Given a stream of `"hello"`, `""` and `"world"`, a `stream { Notification }` query delivers `hello` and `world`, and logs the dropped record. ### Validation and data policies Validation runs on the data **as it was received from the upstream service**, before any [data policies](/docs/data-policies/data-policies) mask, obfuscate or null out values. That ordering is deliberate: * a value that a policy has masked must not then fail a rule that the raw value satisfied, and * a violation must describe the data as it actually arrived, not a redacted view of it The practical consequence: write your rules against the real shape of your upstream data, not against what a given caller is permitted to see. ## When a rule cannot be evaluated A misconfigured rule degrades — it never brings down a request. If a `@Pattern` regex won't compile, that rule is disabled at that declaration site. Instead of failing, it reports `Info`-severity advice telling you the rule didn't run: ```taxi @Valid model Doc { @Pattern(regex = "[A-Z") @MaxLength(value = 3) code : String } ``` Validating `{ "code": "abcd" }` produces two violations: | Path | Rule | Severity | Message | |------|------|----------|---------| | `code` | `com.orbitalhq.validation.Pattern` | `Info` | Validation rule Pattern was not evaluated: the configured regex `[A-Z` is not a valid regular expression | | `code` | `com.orbitalhq.validation.MaxLength` | `Error` | Value 'abcd' is longer than the maximum length of 3 (was 4) | Note what this buys you: * **The broken rule is visible, not silent.** A rule that quietly passed everything would be worse than one that fails. * **Sibling rules keep working.** `@MaxLength` on the same field is unaffected. * **`Info` doesn't reject.** With the default `Error` threshold, the advice is reported but never causes a `400` on its own. The same applies to a rule annotation that Orbital has no implementation for — it's skipped with a warning rather than failing the query. **Warning: Offending values are in responses, never in logs** Rule messages quote the value that failed (`Value 'toolong' is longer than...`), so they're rendered into the HTTP failure report that the caller receives — but they are deliberately kept out of Orbital's server logs. Log lines carry only the path, rule name and severity, so validating sensitive data doesn't write it to disk. ## Coming soon Support for writing your own validation rules in Kotlin is planned for a future release. -------------------------------------------------------------------------------- # Validation rule reference Source: https://orbitalhq.com/docs/data-validation/validation-rules Section: Data validation -------------------------------------------------------------------------------- Orbital ships with a set of validation rules in the `com.orbitalhq.validation` namespace. They're built in — there's nothing to add to your project to use them. Remember that declaring a rule isn't enough on its own: a rule only runs when a `@Valid` scope covers it (or when it's declared directly on a query parameter). See [validating data](/docs/data-validation/data-validation) for how activation works. ## All rules | Rule | Parameters | Applies to | Fails when | |------|------------|------------|------------| | `@NotEmpty` | — | strings, collections | the string is `""`, or the collection has no items | | `@MinLength` | `value : Int` | strings | the string is shorter than `value` | | `@MaxLength` | `value : Int` | strings | the string is longer than `value` | | `@Pattern` | `regex : String` | strings | the string doesn't fully match `regex` | | `@Min` | `value : Decimal`, `exclusive : Boolean = false` | numbers | the number is below `value` — or at it, when `exclusive = true` | | `@Max` | `value : Decimal`, `exclusive : Boolean = false` | numbers | the number is above `value` — or at it, when `exclusive = true` | | `@MultipleOf` | `value : Decimal` | numbers | the number isn't an exact multiple of `value` | | `@MinItems` | `value : Int` | collections | the collection has fewer than `value` items | | `@MaxItems` | `value : Int` | collections | the collection has more than `value` items | | `@UniqueItems` | — | collections | the collection contains duplicates | | `@Email` | — | strings | the string isn't a valid email address | | `@Uuid` | — | strings | the string isn't a valid UUID | ## Parameters every rule accepts Every rule inherits from `com.orbitalhq.validation.ValidationRule`, which gives all of them two extra parameters: | Parameter | Type | Default | Description | |-----------|------|---------|-------------| | `message` | `String` | the rule's own message | Replaces the generated violation message | | `severity` | `Severity` | `Error` | One of `Info`, `Warning` or `Error` | ```taxi import com.orbitalhq.validation.Valid import com.orbitalhq.validation.MaxLength import com.orbitalhq.validation.Severity @Valid model Doc { @MaxLength(value = 3, severity = Severity.Warning, message = "too long") code : String } ``` Severity decides whether a violation is serious enough to fail the query. With the default `threshold = Error`, only `Error` violations reject — see [choosing a threshold](/docs/data-validation/data-validation#choosing-a-threshold). ## String rules ```taxi import com.orbitalhq.validation.Valid import com.orbitalhq.validation.NotEmpty import com.orbitalhq.validation.MinLength import com.orbitalhq.validation.MaxLength import com.orbitalhq.validation.Pattern @Valid model Registration { @NotEmpty username : Username inherits String @MinLength(value = 8) password : Password inherits String @MaxLength(value = 140) bio : Bio inherits String @Pattern(regex = "[A-Z]{2}[0-9]{4}") reference : Reference inherits String } ``` ### @NotEmpty Fails on an empty string, `""`. A whitespace-only string like `" "` passes — if you want to catch that, use `@Pattern`. `@NotEmpty` also works on collections, where it fails an empty collection. ### @MinLength / @MaxLength Character-count bounds. Both are inclusive: with `@MinLength(value = 3)`, a 3-character string passes, and a 2-character string fails with: ``` Value 'ab' is shorter than the minimum length of 3 (was 2) ``` ### @Pattern Matches the **whole** string against a regular expression — a partial match fails. Given `@Pattern(regex = "[A-Z]{2}[0-9]{4}")`: | Value | Result | |-------|--------| | `AB1234` | passes | | `AB1234-extra` | fails — the pattern must match the entire value | | `hello` | fails | **Note: A broken regex won't break your query** If the `regex` won't compile, Orbital disables that rule and reports `Info`-severity advice saying the rule wasn't evaluated, rather than failing requests. See [when a rule cannot be evaluated](/docs/data-validation/data-validation#when-a-rule-cannot-be-evaluated). ## Numeric rules ```taxi import com.orbitalhq.validation.Valid import com.orbitalhq.validation.Min import com.orbitalhq.validation.Max import com.orbitalhq.validation.MultipleOf @Valid model Order { @Min(value = 1) quantity : Quantity inherits Int @Min(value = 0, exclusive = true) price : Price inherits Decimal @Max(value = 100) discountPercent : DiscountPercent inherits Decimal @MultipleOf(value = 0.01) amount : Amount inherits Decimal } ``` ### @Min / @Max Bounds are **inclusive by default** — a value exactly on the bound passes. Set `exclusive = true` to make the bound itself fail, which is how you express "must be greater than zero": | Declaration | `0` | `0.01` | |-------------|-----|--------| | `@Min(value = 0)` | passes | passes | | `@Min(value = 0, exclusive = true)` | fails | passes | ### @MultipleOf Comparison is exact decimal arithmetic, not floating point, so `0.3` is a multiple of `0.1`. This makes it reliable for money — `@MultipleOf(value = 0.01)` means "no fractional pennies". A `value` of `0` never produces a violation. ## Collection rules ```taxi import com.orbitalhq.validation.Valid import com.orbitalhq.validation.MinItems import com.orbitalhq.validation.MaxItems import com.orbitalhq.validation.UniqueItems type Tag inherits String type Contributor inherits String type Category inherits String @Valid model Post { @MinItems(value = 2) tags : Tag[] @MaxItems(value = 10) contributors : Contributor[] @UniqueItems categories : Category[] } ``` ### @MinItems / @MaxItems Item-count bounds, both inclusive. A collection of exactly `value` items passes either rule. ### @UniqueItems Fails when a collection contains duplicates. It works on collections of scalars and of models — duplicate models are detected by comparing their values, so two structurally identical objects count as a duplicate. **Note: Collection rules report at the collection** A rule on a collection reports its violation at the collection's own path — `tags`, not `tags[0]`. Rules on the collection's _element type_ report against each element, with the index in the path. ## Format rules ```taxi import com.orbitalhq.validation.Valid import com.orbitalhq.validation.Email import com.orbitalhq.validation.Uuid @Email type EmailAddress inherits String @Uuid type CorrelationId inherits String @Valid model Contact { email : EmailAddress correlationId : CorrelationId } ``` Format rules are a natural fit for declaring on the **type** rather than the field. `EmailAddress` is an email address wherever it appears, so declaring the rule once means every model that uses the type gets it. ### @Email Checks the string is a plausible email address: a local part, a single `@`, a domain containing at least one dot, and no whitespace. ### @Uuid Checks the string is a canonical UUID — 8-4-4-4-12 hexadecimal characters separated by hyphens. ## Rules Orbital produces for you Two rule names appear in violations without you ever declaring them. You'll see these in the `rule` field of a validation failure report: | Rule | Reported when | |------|---------------| | `com.orbitalhq.validation.NotNull` | A non-nullable field inside a `@Valid` scope is holding `null` | | `com.orbitalhq.validation.ParseFailure` | A value inside a `@Valid` scope couldn't be parsed into its declared type | `NotNull` is why you never declare a "required" rule — your taxonomy already says whether a field is nullable, and `@Valid` enforces it. See [activating validation](/docs/data-validation/data-validation#activating-validation-with-valid). ## What does not produce a violation Rules are deliberately narrow about what they apply to, so that a rule declared on a widely-reused type doesn't cause noise everywhere it's used: * **A rule that doesn't apply to the value's kind is skipped.** `@MinLength` on a number, or `@MinItems` on a string, produces nothing — not a violation. * **`null` never trips a rule.** Rules describe what a value must look like when it's present. Whether the value is allowed to be absent at all is a nullability question, handled by `NotNull`. -------------------------------------------------------------------------------- # Managing secrets Source: https://orbitalhq.com/docs/deploying/managing-secrets Section: Security -------------------------------------------------------------------------------- When defining either [connections](/docs/describing-data-sources/configuring-connections) or [authentication tokens](/docs/describing-data-sources/authentication-to-services), you'll need to store sensitive values. Orbital is built around file-based and git-based config, so these are values you generally don't want checked in. There are three ways to keep secrets out of your config files: * **Environment variables** — inject secrets at the OS level, common with Kubernetes and cloud orchestrators. * **`env.conf`** — a separate file alongside your config, typical for local development. * **Secrets manager** — a managed store (file-backed, Infisical, etc.) that Orbital looks up on demand. You can mix all three. See [Source precedence](#source-precedence) for the resolution order. ## Environment variables All Orbital config files are defined in [HOCON](https://github.com/lightbend/config#examples-of-hocon) format, which allows for variable [substitution](https://github.com/lightbend/config/blob/main/HOCON.md#substitutions) [from the environment](https://github.com/lightbend/config/blob/main/HOCON.md#substitution-fallback-to-environment-variables). ```hocon jdbc { my-connection { connectionParams { password = ${POSTGRES_PASSWORD} } } } ``` This is a popular approach when deploying onto Kubernetes, ECS, or similar — the orchestrator injects secrets into the container environment, and Orbital substitutes them at config load. ## Using env.conf for sensitive data In addition to the standard HOCON rules for resolving variables, Orbital supports the use of an `env.conf` file as a source for substitutions. This is useful when developing locally, to keep sensitive values out of `connections.conf` without having to configure environment variables. **Note** env.conf files support [environment-specific variations](/docs/deploying/configuring-orbital#environment-specific-configuration). The general workflow is: * Define a `connections.conf` file with your connections, using `${variables}` for placeholders of sensitive data. * Create a local `env.conf` file alongside it. The location must match the `additionalSources` path defined in your [taxi.conf](/docs/deploying/configuring-orbital) file as `@orbital/config`. * Populate `env.conf` with sensitive values. Eg: ```hocon connections.conf jdbc { my-connection { // ... other params omitted for brevity ... connectionParams { password = ${postgres_password} // Reads the variable "postgres_password" } } } ``` ```hocon env.conf postgres_password=hello123 ``` ```hocon taxi.conf name: com.foo/myProject version: 0.1.0 sourceRoot: src/ additionalSources: { "@orbital/config" : "orbital/config/*.conf", } ``` ## Secrets manager The secrets manager is a backend store that Orbital looks up to resolve `${secrets.NAME}` placeholders in your config. Reference a secret in your config the same way you'd reference an env var: ```hocon connections.jdbc.warehouse { url: "jdbc:postgresql://warehouse.example.com:5432/prod" password: ${secrets.DB_PASSWORD} } ``` Secrets are isolated by **scope**: | Scope | Visible to | | --- | --- | | `Server` | every workspace on this server | | `Organisation` | every workspace in that organisation | | `Workspace` | only that workspace | Reads cascade — a workspace looks first in `Workspace`, then `Organisation`, then `Server`. The narrower scope wins on a name collision. Secrets are managed through the Orbital UI under **Authentication → Secrets**. Pick a backend by setting `vyne.secrets.backend`: | Value | Backend | | --- | --- | | `InMemory` (default) | In-process; loses everything on restart. Tests and demos only. | | `File` | Local file, age-encrypted. Suitable for dev. | | `Infisical` | [Infisical](https://infisical.com) — cloud or self-hosted. | | `Aws` | [AWS Secrets Manager](https://aws.amazon.com/secrets-manager/). | | `Vault` | [HashiCorp Vault](https://www.vaultproject.io/) — KV v2 mount. | | `Gcp` | [Google Cloud Secret Manager](https://cloud.google.com/secret-manager). | ### In-memory The default when no backend is configured. Holds secrets in process memory only — restarts wipe everything, and multi-node clusters don't share state. ```yaml vyne: secrets: backend: InMemory # default; can be omitted ``` **Warning: Not for production** Use for tests, demos, or quick local exploration. Not suitable for production. ### File A local on-disk store that survives restart, encrypted with [age](https://age-encryption.org). ```yaml vyne: secrets: backend: File file: path: orbital_data/secrets-store # default ``` The path defaults to `${vyne.app.data.path}/secrets-store` — `./orbital_data/secrets-store` out of the box. Override with `vyne.secrets.file.path`. On first start, Orbital generates an `identity.txt` (private key) and `recipients.txt` (public key) in that directory. The store layout is: ``` / ├── identity.txt # private key — back this up ├── recipients.txt # public key └── secrets/ ├── server/.age ├── organisation//.age └── workspace///.age ``` **Warning: Not for production** The File backend is intended for development. The decryption key sits next to the encrypted data on disk, so anyone with read access to the directory can read every secret. For production, use Infisical (or another managed store). ### Infisical Stores secrets in [Infisical](https://infisical.com) — cloud or self-hosted. You'll need: * An Infisical project ID * An environment slug within that project (e.g. `prod`) * A [Universal Auth](https://infisical.com/docs/documentation/platform/identities/universal-auth) machine-identity client ID and client secret ```yaml vyne: secrets: backend: Infisical infisical: site-url: https://app.infisical.com # default; for self-hosted, point at your install project-id: environment-slug: prod base-path: /orbital # all Orbital secrets live under this folder auth: client-id: ${INFISICAL_CLIENT_ID} client-secret: ${INFISICAL_CLIENT_SECRET} ``` Orbital lays out scopes as folders under `base-path`: ``` /orbital/server/ /orbital/tenant// /orbital/tenant/// ``` Missing folders are created automatically on first write — you don't need to provision the layout in advance. ### AWS Secrets Manager Stores secrets in [AWS Secrets Manager](https://aws.amazon.com/secrets-manager/). ```yaml vyne: secrets: backend: Aws aws: region: eu-west-1 # optional; falls back to the AWS SDK provider chain base-path: /orbital # all Orbital secrets live under this prefix # endpoint-override: ... # optional; for VPC endpoints / localstack ``` Authentication uses the standard [AWS SDK credentials chain](https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/credentials-chain.html) — environment variables, shared profile, IAM role on EC2 / ECS / EKS, etc. Provision a principal with `secretsmanager:CreateSecret`, `GetSecretValue`, `PutSecretValue`, `UpdateSecret`, `DescribeSecret`, `ListSecrets`, and `DeleteSecret` on the `base-path`'s ARN pattern. Orbital encodes scopes as a path-style prefix on the secret name: ``` /server/ /tenant// /tenant/// ``` Two AWS-specific notes: * **Deletes are immediate.** AWS's default delete schedules removal 7–30 days out, during which the name is reserved. Orbital uses `forceDeleteWithoutRecovery` so the name is freed synchronously — once `delete` returns, you can recreate a secret with the same name. * **Last-write-wins.** AWS Secrets Manager has no `expectedVersion` hook on `PutSecretValue`, so concurrent updates do not detect conflicts. If your workflow needs optimistic concurrency, use the File or Infisical backend. ### HashiCorp Vault Stores secrets in a [HashiCorp Vault](https://www.vaultproject.io/) KV v2 mount. ```yaml vyne: secrets: backend: Vault vault: address: https://vault.example.com:8200 token: ${VAULT_TOKEN} mount: secret # default; the KV v2 mount path base-path: /orbital # all Orbital secrets live under this prefix ``` Authentication is via a static [Vault token](https://developer.hashicorp.com/vault/docs/concepts/tokens). Provision a periodic token with `read`, `create`, `update`, `delete`, and `list` capability on `/data//*` and `/metadata//*`. Inject the token via the standard env-var / secrets-mount mechanism — Orbital reads it once at startup. Orbital lays out scopes as KV v2 paths under `base-path`: ``` /server/ /tenant// /tenant/// ``` The secret value is stored under the data key `value`; an optional description is stored as `custom_metadata.description`. A folder structure isn't pre-provisioned — KV v2 creates intermediate paths on first write. Two notes: * **Delete is permanent.** Vault's standard `delete` is a soft-delete that keeps versions recoverable. To honour the contract (`delete` means delete), Orbital issues a `DELETE` against `/metadata/`, which destroys all versions and the metadata key in one call. Recreating a secret at the same name starts a fresh version sequence at 1. * **Optimistic concurrency uses CAS.** Writes with an `expectedVersion` set Vault's `cas` option, which performs the version check server-side. Conflicting concurrent writes lose deterministically. ### GCP Secret Manager Stores secrets in [Google Cloud Secret Manager](https://cloud.google.com/secret-manager). ```yaml vyne: secrets: backend: Gcp gcp: project-id: my-gcp-project base-path: orbital # default; encoded into every secret name # endpoint-override: ... # optional; for emulators / VPC endpoints ``` Authentication uses Google's standard [Application Default Credentials](https://cloud.google.com/docs/authentication/application-default-credentials) chain — `GOOGLE_APPLICATION_CREDENTIALS`, attached service account on GCE / GKE / Cloud Run, or `gcloud auth application-default login` for local development. Provision the principal with the `roles/secretmanager.admin` role (or a tighter custom role granting `secretmanager.secrets.{create,get,list,update,delete}` plus `secretmanager.versions.{add,access}`) on the project. GCP has a flat per-project namespace and a constrained character set (`[A-Za-z0-9_-]` only). Orbital encodes the scope hierarchy into the secret id using `__` as a segment separator: ``` __server__ __tenant____ __tenant______ ``` `orgId`, `workspaceId`, and the secret name itself must therefore avoid `__` and any character outside `[A-Za-z0-9_-]`. `description`, when set, is stored on the secret as the `description` annotation. Two GCP-specific notes: * **Last-write-wins.** GCP `AddSecretVersion` has no `etag` hook for conditional writes, so `expectedVersion` is accepted but not enforced. If your workflow needs optimistic concurrency, use the File or Vault backend. * **Deletes are immediate.** `deleteSecret` removes the secret and all its versions in one call — there's no recovery window, so recreating a secret with the same id is allowed and starts a fresh version sequence at 1. ## Source precedence When the same `${name}` could be supplied by more than one source, Orbital resolves in this order (last wins): ``` [OS env] → [env.conf] → [Server secret] → [Organisation secret] → [Workspace secret] ``` If a name is defined in more than one source, Orbital logs a warning at workspace startup naming the source that's being used. Values are never logged. -------------------------------------------------------------------------------- # Authentication to Orbital Source: https://orbitalhq.com/docs/deploying/authentication Section: Security -------------------------------------------------------------------------------- ## Overview Orbital requires users to sign in. Out of the box, sign-in goes through Orbital's own identity service: the first time you open Orbital you're asked to sign in or create a free account, and that sign-in is also how Orbital fetches your license. There are two ways to run: | Mode | How | Plans | |---|---|---| | **Sign in with Orbital** (default) | Nothing to configure. Users sign in or create an account when they first open Orbital. | All | | **Bring your own identity provider** | Configure OpenID Connect (Authorization Code + PKCE) or SAML against your own IDP - Okta, Entra ID, Keycloak, Cognito and friends. See the sections below. | Business, Enterprise | ## Authenticating with API keys **Note** API key authentication is available on the **Business** and **Enterprise** plans. API keys let scripts, CI jobs and other services call Orbital without a browser sign-in. Each key is tied to the user who created it, and carries that user's roles - so a key can do exactly what its owner can do, and nothing more. To create a key, open your account settings (from the user menu in the top right, or the account link on the sign-in page) and generate one under **API keys**. Keys are shown once, at creation time. Present the key as a bearer token on every request: ```bash curl -H "Authorization: Bearer $ORBITAL_API_KEY" \ -H "Content-Type: application/taxiql" \ --data 'find { Customer[] }' \ https://your-orbital/api/default/default/taxiql ``` API keys are issued and validated by Orbital's identity service, so they're available when you sign in with Orbital (the default). They're not available when you've brought your own identity provider - use that provider's own machine-to-machine credentials instead. ## OpenID Connect Setup **Note** Bringing your own identity provider is available on the **Business** and **Enterprise** plans. To authenticate against your own OpenID Connect provider instead of Orbital's identity service, pass the following config options to Orbital on the command line: | Parameter | Sample values | Description | |------------------------------------------------|------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `vyne.security.openIdp.enabled` | `true` / `false` (default) | Set to `true` to enable OpenId Connect Authentication. | | `vyne.security.openIdp.issuerUrl` | https://auth-service/auth/realms/orbital | The openIdp issuer endpoint. The browser will redirect to this endpoint, so ensure the DNS entry is accessible via browser traffic | | `vyne.security.openIdp.clientId` | `orbital` | The client id to present to OpenID server. Defaults to `orbital` if not provided. | | `vyne.security.openIdp.jwks-uri` | https://auth-service/certs | A url to load the set of JWK's used to verify signatures of presented tokens. This url is called by Orbital's server, so ensure that the DNS entry is accessible to Orbital | | `vyne.security.openIdp.require-https` | `true` (default) / `false` | Indicates if auth must be performed over https. Defaults to true | | `vyne.security.openIdp.account-management-url` | https://auth-service/account | **Optional.** A url where authenticated users may be redirected, to manage their account | | `vyne.security.openIdp.org-management-url` | https://auth-service/org | **Optional.** A url where authenticated users may be redirected, to manage their organisation. Generally, this is where roles are assigned to users. | A typical docker config might look as follows: ```yaml orbital: image: orbitalhq/orbital:${ORBITAL_VERSION} volumes: - ./orbital_data:/opt/service/orbital_data environment: OPTIONS: >- --orbital.app.data.path=/opt/service/orbital_data --orbital.security.open-idp.enabled=true --orbital.security.open-idp.jwks-uri=http://keycloak-server/realms/orbital/protocol/openid-connect/certs --orbital.security.open-idp.issuer-url=http://keycloak-server/realms/orbital --orbital.security.open-idp.client-id=orbital ``` The settings above replace the defaults that point at Orbital's identity service - you don't need to turn anything off first. (The older `vyne.security.openIdp.*` names in the tables on this page still work, but Orbital logs a warning at startup asking you to migrate them to `orbital.security.open-idp.*`.) When a user attempts to access Orbital, they will be redirected to the login page as configured with the OpenId connect provider. Orbital doesn't provide a UI for managing users or roles, it's expected this is handled through the OIDC service. ### The presented JWT The presented JWT is expected to have the following attributes: | Claim | Required | Description | |-------------------------------------------------------------|----------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `sub` | required | Subject - Identifier for the End-User at the Issuer. | | `iss` | required | Issuer - the OIDC provider who authenticated the user | | One of `preferred_username` or `first_name` and `last_name` | required | Shorthand name by which the End-User wishes to be referred to at the RP, such as janedoe or j.doe. | | One of `email` or `clientId` | required | Something that uniquely identifies the user | | One of `picture` or `picture_url` | optional | The users avatar | | `name` | optional | End-User's full name in displayable form including all name parts, possibly including titles and suffixes, ordered according to the End-User's locale and preferences. | Additionally, role information must be present. See the docs on [Authorization](/docs/deploying/authorization#other-id-ps-reading-roles-from-jwt-tokens) for more information. ## Configuring OpenID Providers ### Keycloak Keycloak is the default IDP (it's what the Orbital team tests with), so requires the least configuration. ```bash # Set to true to enable Auth --vyne.security.openIdp.enabled=true --vyne.security.openIdp.clientId=orbital # The url that the browser will redirect to for authentication. # Ensure this is accessible from browsers --vyne.security.openIdp.issuerUrl=http://keycloak-server/realms/orbital # The url that Orbital will connect to, to validate the JWT's presented. # Ensure this is accessible from the Orbital server # This normally follows the convention of ${issuerUrl}/protocol/openid-connect/certs --vyne.security.openIdp.jwks-uri=http://keycloak-server/realms/orbital/protocol/openid-connect/certs ``` ### AWS Cognito The following instructions step through configuring AWS Cognito using AWS Console. These docs aren't intended to be exhaustive around AWS Congito - refer to AWS for more detailed instructions. #### Preparing AWS Congito 1. Create an AWS Cognito User pool, and users (if not already present) * Take note of the User Pool Id. 1. Create an App Integration, and click to create an App Client: * **Under App Client**: * Create a Public Client * Don't generate a client secret * Under "Authentication Flows", ensure that the ALLOW_USER_PASSWORD_AUTH flow is selected * **Under Hosted UI Settings**: * Specify the callback URL of where you're running Orbital. (For testing, you can additionally add `localhost` here too) * For OAuth 2.0 Grant types, ensure "Authorization code grant" is selected * For OIDC Connect scopes, ensure `email`, `openid` and `profile` are selected * Click Create app client 1. Take note of the Client Id, which you'll need in the next step #### Configuring Orbital Use the following config settings in Orbital: ```bash # Set to true to enable Auth --vyne.security.openIdp.enabled=true # Replace YOUR_CLIENT_ID with the clientId noted earlier --vyne.security.openIdp.clientId=YOUR_CLIENT_ID # In the below, replace REGION and USER_POOL_ID accordingly # Eg: https://cognito-idp.eu-west-2.amazonaws.com/eu-west-2_xxxxx --vyne.security.openIdp.issuerUrl=https://cognito-idp.REGION.amazonaws.com/USER_POOL_ID --vyne.security.openIdp.jwks-uri=https://cognito-idp.REGION.amazonaws.com/USER_POOL_ID/.well-known/jwks.json # Use the scopes openid email profile --vyne.security.openIdp.scope="openid email profile" # Must be set to Id --vyne.security.openIdp.identity-token-kind=Id ``` **Note: Don't forget!** You must also configure AWS Cognito [role based access control](/docs/deploying/authorization#aws-cognito) ### Azure **Warning: Azure AD B2C Not Supported** Unfortunately, at this time Azure AD B2C is not supported, as it does not provide support for [exposing roles in the published JWT](https://learn.microsoft.com/en-us/answers/questions/818404/role-based-access-for-webapi-in-azure-b2c). However, Azure Active Directory (Entra) is supported, as described below. * Log into Azure, and ensure you have an Azure AD directory active (not an Azure AD B2C directory) * Navigate to Microsoft Entra ID, and select "App registrations" from the left navbar * Click "New Registration" in the top navbar * Provide a name for the application - eg: "Orbital" * For the Redirect URI, select Single-Page Application, and paste the URL of where Orbital is hosted. (eg: `https://mycompany/orbital`) * Click Register You are taken to the settings for your newly created Azure AD application * Click "Overview" in the left Navbar. * Take note of the Application (client) ID - you'll need this shortly * From the top navbar, select "Endpoints" * Take note of the OpenID Connect metadata document value - you'll need this shortly. * It should look like this: `https://login.microsoftonline.com/YOUR-APP-ID/v2.0/.well-known/openid-configuration` **Note: Don't forget!** You must also configure Azure [role based access control](/docs/deploying/authorization#azure) Launch Orbital passing the following parameters: ```bash --vyne.security.openIdp.enabled=true ## Use the URL copied above --vyne.security.openIdp.oidcDiscoveryUrl=https://login.microsoftonline.com/YOUR-APP-ID/v2.0/.well-known/openid-configuration ## Use the ClientId copied above --vyne.security.openIdp.clientId=xxxxx --vyne.security.openIdp.scope="openid profile email" --vyne.security.openIdp.identity-token-kind=Id ## These relate to RBAC, after following the docs for enabling RBAC with Azure --vyne.security.open-idp.roles.format=path --vyne.security.open-idp.roles.path=roles ``` ### Other IDP's In general, Orbital supports OpenID Connect authentication and authorization, and should be compatible with most platforms that implement OIDC protocol. Orbital uses the Authorization Code + PKCE flow. The following parameters (passed to Orbital on startup) are used to configure OpenID Connect support: | Parameter | Description | |------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `vyne.security.openIdp.enabled` | Set to `true` to turn on support for OIDC | | `vyne.security.openIdp.client-id` | The ClientId as set by the OIDC provider. | | `vyne.security.openIdp.issuer-url` | Sets the issuerUrl, as provided by the OIDC provider. | | `vyne.security.openIdp.oidc-discovery-url` | Sets where the OpenID discovery document is loaded from. If not provided, will default to `${issuerUrl}/.well-known/openid-configuration`, which is the convential url, and normally ok. | | `vyne.security.openIdp.scope` | **Optional** The OIDC scopes to be requested. Defaults to `openid profile email offline_access`. Changing this will affect the claims presented in the JWT, which may affect Oribtal's ability to read the token | | `vyne.security.openIdp.require-https` | **Optional** Defines if requests must be made over https. Defaults to `true` | | `vyne.security.openIdp.identity-token-kind` | **Optional** Either `Access` (default) or `Id`. Defines which token the browser should present to Orbital after authentication by the IDP. The token should contain the required claims | | `vyne.security.openIdp.account-management-url` | **Optional** Provides a url that is presented to the user within the UI to manage their account | For most scenarios, you need to set *either* the `issuer-url` or `oidc-discovery-url`, but not both. However, if your OIDC provider uses non-standard configuration, both may be required. ## Troubleshooting OpenID ### Ensure DNS entries are accessible When testing with Docker / k8s, it's common to be accessing the browser via localhost, but services are operating in a different, isolated DNS networks. Pay particular attention to the two following settings, which are accessed in different contexts: | Parameter | Description | |-----------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `vyne.security.openIdp.issuerUrl` | Redirected from the browser. If you're running keycloak (or similar), `localhost` is possibly ok here. | | `vyne.security.openIdp.jwks-uri` | Requested from the Orbital server. If both Orbital and your IDP (eg Keycloak) are running within Docker, make sure you use a DNS entry that is accessible to Orbital. (ie., `localhost` is unlikely to work here) | ## SAML Authentication SAML stands for Security Assertion Markup Language. It is an XML-based open-standard for transferring identity data between two parties: an identity provider (IdP) and a service provider (SP). * Identity Provider — Performs authentication and passes the user's identity and authorization level to the service provider. * Service Provider — Trusts the identity provider and authorizes the given user to access the requested resource. To configure Orbital as a Service Provider and authenticate users against a SAML Identiy Provider, pass the following config options to Orbital on the command line: | Parameter | Sample values | Description | |-----------------------------------------------------------|------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `vyne.security.saml.enabled` | `true` / `false` (default) | Set to `true` to enable SAML Authentication. | | `vyne.security.saml.keyStorePath` | `/opt/service/orbital/saml.jks` | SAML protocol requires signing various SAML protocol messages. This is the full path of corresponding key store accessible by Orbital. Please note that, If the key store file does not exists at the specified location, Orbital will create the key store for you. | | `vyne.security.saml.keyStorePassword` | `orbital` | Password for your key store | | `vyne.security.saml.privateKeyPassword` | `orbital` | Password for your private key contained in the key store. | | `vyne.security.saml.idpMetadataFilePath` | `/opt/service/orbital/saml-idp-metadata.xml` | Full path of your SAML Idp metadata xml. The path should be accessible by Orbital. | | `vyne.security.saml.serviceProviderEntityId` | `http://foo.orbitalhq.io` | Service Provider Entity Id of the Orbital. | | `vyne.security.saml.callbackBaseUrl` | `http://foo.orbitalhq.io` | Orbital URL which is accessible by SAML Idp. | | `vyne.security.saml.serviceProviderMetadataResourcePath` | `/opt/service/orbita/sp-metadata.xml` | Full path of the service provider metadata xml file, This file will be generated automatically by the Orbital! | | `vyne.security.saml.maximumAuthenticationLifetime` | `3600` | Once you have an authenticated web session on the Identity Provider, usually it won’t prompt you again to enter your credentials and it will automatically generate a new assertion for you. By default, the SAML client will accept assertions based on a previous authentication for one hour. If you want to change this behavior, set the maximumAuthenticationLifetime parameter. | ### Example SAML Setup with Okta * To begin, you’ll need an Okta developer account. You can create one at [developer.okta.com/signup](https://developer.okta.com/signup) or install the Okta CLI and run okta register. * Log in to your Okta account and go to `Applications > Create App Integration.` Select `SAML 2.0` and click Next. Name your app something like OrbitalSaml and click `Next` * Assuming the Orbital will be running on your local with its default port. Set the `Single sign-on URL` as ``` http://localhost:9022/saml?client_name=SAML2Client ``` * Use `Single sign-on Url` for `Recipient URL` and `Destination URL`. * Set `Audience URI (SP Entity ID)` to `http://okta-sample.orbital.io` * Click on `Next` * Set `App type` to `This is an internal app that we have created` * Okta will create your app, and you will be redirected to its Sign On tab. Scroll down to the SAML Signing Certificates and go to SHA-2 > Actions > View IdP Metadata. You can right-click and copy this menu item’s link or open its URL. Copy the resulting link to your clipboard. It should look something like the following: * Save the View Idp Data to a patch that is accessible to Orbital, you'll set the value of `vyne.security.saml.idpMetadataFilePath` configuration value. * Go to your app’s Assignment tab and assign access to the Everyone group. * Here is the list of orbital arguments for your setup: ``` --vyne.security.saml.enabled=true --vyne.security.saml.keyStorePath=/opt/serviuce/orbital/saml.jks --vyne.security.saml.keyStorePassword=orbital --vyne.security.saml.privateKeyPassword=orbital --vyne.security.saml.idpMetadataFilePath=/op/service/orbital/okta-idp-metadata.xml --vyne.security.saml.serviceProviderEntityId=http://okta-sample.orbital.io --vyne.security.saml.callbackBaseUrl=http://localhost:9022 --vyne.security.saml.serviceProviderMetadataResourcePath=/opt/service/orbital/sp-orbital-metadata.xml ``` ## See also * [Role based authorization](/docs/deploying/authorization) * [Allowing anonymous access to a published query](/docs/querying/queries-as-endpoints#allowing-anonymous-access) -------------------------------------------------------------------------------- # Authorization within Orbital Source: https://orbitalhq.com/docs/deploying/authorization Section: Security -------------------------------------------------------------------------------- **Note: Prerequisite** In order to use role based authentication, you must have configured authorization with an OpenID Connect provider. [Learn how to enable authorization](/docs/deploying/authentication) Orbital uses role based authorization to control which users can perform which tasks. ## Roles Roles are used to grant [permissions](#permissions) to users - which allow users to perform different actions on the Orbital platform. The roles themselves are configurable. Orbital ships with a set of reasonable default roles - but you may choose to modify these, or build your own. Roles are configured in a config file, typically found at `config/roles.conf`. If the file is not present on startup, a new file is created the first time a user logs in. ```hocon // Defines roles, and assigns them authorities. // Users who are granted these roles will be authorised to perform // the related activities grantedAuthorityMappings { Admin { grantedAuthorities = [ "RunQuery","CancelQuery"] } Viewer { grantedAuthorities = [ "BrowseCatalog" ] } } ``` ### List of default roles The following roles are shipped by default: | Role | Granted Authorities | |-----------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | Admin | Everything | | Viewer | BrowseCatalog | | QueryRunner | RunQuery | | PlatformManager | CancelQuery, ViewQueryHistory ViewHistoricQueryResults BrowseSchema EditSchema ViewCaskDefinitions EditCaskDefinitions ViewPipelines EditPipelines ViewAuthenticationTokens EditAuthenticationTokens ViewConnections EditConnections | ## Assigning users to roles Roles are assigned based on the JWT token presented by your [OIDC server](/docs/deploying/authentication). By default, if your OIDC server does not present any roles, a user who logs in will not be able to do anything, and will see an empty navbar. In order to grant roles, you'll need to assign one of the [roles](#list-of-default-roles) to your users within your OIDC platform. Orbital will work with any OIDC compliant server. Orbital ships with defaults against Keycloak, which is a common OSS OIDC server. However, you can use whichever OIDC server you wish. ## Configuring specific IDPs ### Keycloak The following steps were tested against Keycloak 22. Actual steps may vary depending on your version or configuration - use Keycloak's docs to configure accordingly. * Log into the realm admin in Keycloak * Navigate to your realm, and select Realm roles in the left hand navbar * Click "Create Role" * Add a role from above. (eg., Admin), and click Save * Navigate to Users, and select the user you wish to manage * Click role Mapping * Click Assign Role, and select the Admin role, and click Assign If your user was previously logged in to Orbital, they'll need to log out & in again to refresh their token. ### AWS Cognito AWS uses Groups to populate the `cognito:groups` claim in the provided JWT. Here's how to configure this: #### Create the group / role 1. In AWS Cognito, navigate to the User Pool for Orbital. 1. Click Groups, and create a group. The name of the group should match a [role](#roles). (eg: `Admin`) 1. Assign users to the role. #### Configuring Orbital To configure Orbital to read the roles from the JWT passed by Cognito, the following params are required: ```bash --vyne.security.openIdp.roles.format=path --vyne.security.openIdp.roles.path="cognito:groups" ``` After this, Orbital should now work with authentication and authorization via Cognito ### Azure Ensure you've first set up Orbital as an AD Application, as described in the Configuring [Azure Authentication](/docs/deploying/authentication#azure). #### Creating roles 1. Inside Azure, navigate to Microsoft Entra ID, then select App registrations from the left navigation bar 1. Select the application you created [earlier](/docs/deploying/authentication#azure). 1. Select App roles 1. Select Create App role: * The display name and Value should match a configured [role](#roles) (eg: `Admin`). * Set "Allowed member types" to Users/Groups * Click Apply #### Assigning users to roles 1. Inside Azure, navigate to Enterprise Applications 1. Select the application you created earlier. 1. Click Assign users and groups 1. Select the checkbox next to a user, and then click Edit Assignment, and select the role Repeat to assign roles as required #### Configuring Orbital To configure Orbital to read the roles from the JWT passed by Azure, the following params are required: ```bash --vyne.security.openIdp.roles.format=path --vyne.security.openIdp.roles.path="roles" ``` After restarting Orbital, users roles will now be passed from Azure to Orbital and read from the JWT. ### Other IDPs: Reading roles from JWT Tokens Orbital expects the user's roles be present in the claims, as list of strings provided under a `roles` key. There is no universal standard of how OIDC servers provide roles within JWT tokens, therefore some configuration may be required. Orbital is configured to work with [Keycloak](https://www.keycloak.org/) by default, which publishes roles at `realm_access.roles`. You can configure a different path by adding the following parameters: | Parameter | Description | |---------------------------------------|------------------------------------------------------------------------| | `vyne.security.open-idp.roles.format` | Set to `path` to provide a custom path | | `vyne.security.open-idp.roles.path` | Set to the path within the JWT for the roles. Eg: `realm_access.roles` | Orbital expects that each part within the token is a `Map`, except for the last path, which should be a `String[]`. If you need more complex support for your OIDC, please reach out to us on [Slack](https://join.slack.com/t/orbitalapi/shared_invite/zt-697laanr-DHGXXak5slqsY9DqwrkzHg) to discuss your requirements. ## Permissions Orbital ships with the following permissions. In order to perform an activity, users be associated with a role that grants the related authority. | Activity | Required permission | |-----------------------------------------------------|--------------------------| | Issue a query through the UI | RunQuery | | Issue a query through the API | RunQuery | | Cancel a running query | CancelQuery | | Browse the query history in the UI | ViewQueryHistory | | View the results of historic queries | ViewHistoricQueryResults | | Browse the data catalog | BrowseCatalog | | View the registered schemas | BrowseSchema | | Modify a catalog entry | EditSchema | | Import a new schema through the schema importer UI | EditSchema | | List pipelines | ViewPipelines | | Add a new pipeline | EditPipelines | | Edit an existing pipeline | EditPipelines | | View authentication tokens Orbital uses in requests | ViewAuthenticationTokens | | Edit authentication tokens Orbital uses in requests | EditAuthenticationTokens | | View configured data sources | ViewConnections | | Edit configured data sources | EditConnections | ## Authorisation parameters The following parameters are configurable, by passing values to Orbital on startup. The values all have reasonable defaults, and it's uncommon to change these. However, they can be configured as required. | Parameter | Description | Default value | |----------------------------------------------------------|------------------------------------------------------------------|----------------------------------| | `vyne.security.authorisation.role-definitions-file` | Path to a file which defines roles and their granted authorities | `config/roles.conf` | | `vyne.security.authorisation.user-to-role-mappings-file` | Path to a file which lists users, and their assigned roles | `config/user-role-mappings.conf` | | `vyne.security.authorisation.admin-role` | The name of the role which grants admin privileges. | `Admin` | -------------------------------------------------------------------------------- # Data access policies Source: https://orbitalhq.com/docs/data-policies/data-policies Section: Security -------------------------------------------------------------------------------- Orbital supports defining data access policies against types, which are evaluated when running queries. This feature allows you to define data policies once and enforce them consistently across your organization, regardless of where data is served from. ## Overview Policies are a first-class citizen within Orbital and are defined in your Orbital project along with your types, models, and services. They can conditionally control the data that's returned (including filtering and obfuscating values) and determine which services and operations can be invoked. ## Getting Started - Configuring your JWT Policies are often used as a form of authorization based on the user requesting the data. To use policies in this manner, you must first have configured [Authentication](/docs/deploying/authentication) with Orbital. ### Exposing User Information Policies access user information from the claims presented on your authentication token. Since each authentication provider is different, you need to define an Orbital type that maps data from your token into data types you can use in your policies. Orbital's UI will guide you through this process and create the corresponding types for you: Alternatively, you can manually define a model that extends from `com.orbitalhq.auth.AuthClaims`. You don't need to map everything from the token, only the attributes you care about. For example, consider the following JWT: ```json { "sub": "661667e1-78ca-43e5-97dd-d1a39ee37f43", "email_verified": true, "allowed-origins": ["*"], "iss": "http://xxxx", "typ": "Bearer", "preferred_username": "jimmy", "realm_access": { "roles": [ "offline_access", "default-roles-vyne", "uma_authorization", "Admin" ] }, "email": "marty@vyne.co" } ``` You can cherry-pick the useful fields and define a model. Note that the model name isn't important, but it must inherit from `com.orbitalhq.auth.AuthClaims`: ```taxi type Sub inherits String type Role inherits String model UserInfo inherits com.orbitalhq.auth.AuthClaims { sub : Sub realm_access : { roles: Role[] } } ``` ### Verifying Your Credentials To verify that your credentials have been mapped correctly, the UI will show the details of the current user in the policy designer: ## Defining Policies Policies are defined in Taxi files within an Orbital project. Here's a simple example: ```taxi model Film { title : Title inherits String yearReleased : YearReleased inherits Int } // Define a policy named `ExcludeYearReleased` which will operate against the `Film` type. policy ExcludeYearReleased against Film { read { // define the scope - either read or write Film as { // return the Film type ... except { yearReleased } // but exclude some fields } } } ``` The above policy will be invoked whenever data is returned from an operation that returns `Film` data. ### Inputs to policies Policies can request data as an input, which can be referred to within the policy. For example, this policy requests information about the user making the request: ```taxi policy FilterSalary against Employee (userInfo : UserInfo) -> { ... } ``` Here, `UserInfo` is the type configured against the JWT token as [described earlier](#exposing-user-information). You can request any data in the policy, including data loaded from additional services, as described below. ### Suppressing data based on user properties You can create policies that behave differently based on user properties. For example, this policy suppresses the `salary` field from ```taxi policy FilterSalary against Employee (userInfo : UserInfo) -> { read { when { userInfo.groups.contains('ADMIN') -> Employee else -> Employee as { ... except { salary } } } } } ``` ### Policies may not alter structure Data policies can be used to [obfuscate]() and [filter]() out properties, but they cannot drop fields entirely, as doing so would cause parsing exceptions in downstream systems, and mean that responses violate their own contracts. When data is filtered (for example, using a spread operator), the resulting object contains nulls. ### Policies and expressions When models include an expression, the expression is evaluated using the input values after the relevant security policies have been applied. This approach ensures that sensitive information is not inadvertently exposed (for example, by adjusting a policy-protected value using an expression such as: `EmployeeSalary + 1`). As a result, the input values used in the expression may differ from the original values returned by an operation. If a policy causes an input value to become `null`, the expression will also evaluate to `null`. ### Throwing Errors from Policies Policies can also throw errors to completely deny access to certain data based on conditions. For example: ```taxi policy OnlyManagers against EmployeeInfo (userInfo : UserInfo) -> { read { when { userInfo.groups.contains('Manager') -> EmployeeInfo else -> throw((NotAuthorizedError) { message: 'Not Authorized' }) } } } ``` ### Obfuscating data You can use policies to obfuscate the Policies can be applied to nested types as well. For example, to partially obfuscate titles for non-admin users: ```taxi policy FilterFilmTitle against Title (userInfo : UserInfo) -> { read { when { userInfo.groups.contains('ADMIN') -> Title else -> concat(left(Title, 3), "***") } } } ``` ### Using External Data in Policy Decisions Policies can load additional data from external services to make decisions. For example, to filter films based on whether the user has accepted terms and conditions: ```taxi model UserConsent { acceptedTermsAndConditions : AcceptedTermsAndConditions inherits Boolean } service UserService { operation getConsent(UserId): UserConsent } policy AllAccessFilms against Film (userInfo : UserInfo, acceptedTerms: AcceptedTermsAndConditions) -> { read { when { acceptedTerms == false -> null else -> Film } } } ``` ### Projection and Policy Impact When a policy modifies a field that is used in a projection, the result is affected accordingly. For instance, if a policy suppresses the `title` field for non-admin users: ```taxi policy FilterYearReleased against Film (userInfo : UserInfo) -> { read { when { userInfo.groups.contains('ADMIN') -> Film else -> Film as { ... except { title } } } } } ``` Querying and projecting the `title` field for a non-admin user would result in: ```taxi find { Film } as { name : Title } ``` For non-admin users, this would return `name: null`. ## Understanding When Policies Are Applied Policies defined against types or models are applied to data returned from a service before it's made available in Orbital (either for other service calls or to return to a caller). A policy is applied to the type **and all its subtypes**. Schema: ```taxi closed model Film { id : FilmId inherits String title : Title inherits String } service FilmsApi { operation getAllFilms():Film[] } type Role inherits String model UserInfo { roles : Role[] } policy HideTitle against Film (userInfo:UserInfo) -> { read { when { userInfo.roles.contains('ADMIN') -> Film else -> Film as { ... except { title }} } } } ``` Query: ```taxi // this is an interactive snippet. Try changing the users roles, // then clicking run to see the impact on the policy. import Film given { user: UserInfo = { roles: ["USER","ADMIN"] } } find { Film[]} ``` ## Using Errors in Policies Errors can be thrown in policies to prevent access entirely, returning an error code to the user. For example: ```taxi policy OnlyManagers against EmployeeInfo (userInfo : UserInfo) -> { read { when { userInfo.groups.contains('Manager') -> EmployeeInfo else -> throw((NotAuthorizedError) { message: 'Not Authorized' }) } } } ``` Read more about [throwing errors](/docs/querying/errors). ## Streaming queries Data policies can also be applied to streaming queries, which are running continuously in the background. Instead of executing with the requesting users permissions (as request / response queries do), persistent streaming queries execute with a system account - the Executor user. ### Configuring the Executor user The Executor User is a standard system account defined by your Identity Provider (IDP). Assign roles as you would with any other user, as discussed in our docs on Authentication. Orbital authenticates this role using the OAuth2 Client Credentials flow with a `client-id` and `client-secret`. Pass these to the Orbital instance at startup with the following configuration settings: | Parameter | Description | Mandatory | |--------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------|------------------------------------| | `vyne.security.openIdp.executorRoleClientId` | The user to authenticate with. Note - this can be different from the standard authentication client configured with `clientId` | Yes | | `vyne.security.openIdp.executorRoleClientSecret` | The Client Secret to authenticate with | Yes | | `vyne.security.openIdp.issuerUrl` | The URL of the IDP. Note that Orbital's server will connect to this URL, so ensure it's accessible from the server | No (see Configuration Notes) | | `vyne.security.openIdp.executorRoleTokenUrl` | The URL of the IDP to fetch the access token by supplying `executorRoleClientId` and `executorRoleClientSecret` | No (see Configuration Notes) | | `vyne.security.openIdp.executorRoleAuthenticationType` | Client Authentication Type. Possible Values are `ClientSecretPost` or `ClientSecretBasic` | Default Value is `ClientSecretPost`| | `vyne.security.openIdp.executorRoleScopes` | Scopes that need to be passed to access token request. | No (see Configuration Notes) | #### Configuration Notes * The Configured `client` on the `OpenID` server must support either `client_secret_post` or `client_secret_basic` for `token_endpoint_auth_method` (see [Client Metadata](https://www.rfc-editor.org/rfc/rfc7591#section-2)). By default, Orbital will use `client_secret_post` authentication method. However, you can get Orbital to use `client_secret_basic` via: ``` --vyne.security.openIdp.executorRoleAuthenticationType=ClientSecretBasic ``` `vyne.security.openIdp.executorRoleTokenUrl` corresponds to the `token_endpoint` value in [OpenID Provider Configuration Response](https://openid.net/specs/openid-connect-discovery-1_0-final.html#ProviderConfigurationResponse). When it is provided, Orbital will send the following post request to it to fetch the access token for the `executor user` ```bash curl --location '' \ --header 'Content-Type: application/x-www-form-urlencoded' \ --data-urlencode 'grant_type=client_credentials' \ --data-urlencode 'client_id=' \ --data-urlencode 'client_secret=' ``` When `vyne.security.openIdp.executorRoleTokenUrl` is not provided, Orbital will use `vyne.security.openIdp.issuerUrl` to fetch the [Obtaining OpenID Provider Configuration Information](https://openid.net/specs/openid-connect-discovery-1_0-final.html#ProviderConfig) and then use `token_endpoint` element value from the response to issue the request for the access token. #### Example configurations: * Obtain Executor access token with `client_secret_post`: ``` --vyne.security.openIdp.executorRoleClientId=TheSchuylerSisters --vyne.security.openIdp.executorRoleClientSecret=AngelicaElizaAndPeggy --vyne.security.openIdp.executorRoleTokenUrl=http://localhost:9000/oauth2/token ``` Above settings will result in the following request sent from Orbital to OpenID server: ```bash curl --location 'http://localhost:9000/oauth2/token' \ --header 'Content-Type: application/x-www-form-urlencoded' \ --data-urlencode 'grant_type=client_credentials' \ --data-urlencode 'client_id=TheSchuylerSisters' \ --data-urlencode 'client_secret=AngelicaElizaAndPeggy' ``` * Obtain Executor access token with `client_secret_basic`: ``` --vyne.security.openIdp.executorRoleClientId=TheSchuylerSisters --vyne.security.openIdp.executorRoleClientSecret=AngelicaElizaAndPeggy --vyne.security.openIdp.executorRoleTokenUrl=http://localhost:9000/oauth2/token --vyne.security.openIdp.executorRoleAuthenticationType=ClientSecretBasic ``` Above settings will result in the following request sent from Orbital to OpenID server: ```bash curl --location 'http://localhost:9000/oauth2/token' \ --header 'Content-Type: application/x-www-form-urlencoded' \ --header 'Authorization: Basic VGhlU2NodXlsZXJTaXN0ZXJzOkFuZ2VsaWNhRWxpemFBbmRQZWdneQ==' \ --data-urlencode 'grant_type=client_credentials' ``` * Obtain Executor access token with `client_secret_basic` and set `scope` as part of the request: ``` --vyne.security.openIdp.executorRoleClientId=TheSchuylerSisters --vyne.security.openIdp.executorRoleClientSecret=AngelicaElizaAndPeggy --vyne.security.openIdp.executorRoleTokenUrl=http://localhost:9000/oauth2/token --vyne.security.openIdp.executorRoleAuthenticationType=ClientSecretBasic --vyne.security.openIdp.executorRoleScopes=scope1 scope2 scope3 ``` Above settings will result in the following request sent from Orbital to OpenID server: ```bash curl --location 'http://localhost:9000/oauth2/token' \ --header 'Content-Type: application/x-www-form-urlencoded' \ --header 'Authorization: Basic VGhlU2NodXlsZXJTaXN0ZXJzOkFuZ2VsaWNhRWxpemFBbmRQZWdneQ==' \ --data-urlencode 'grant_type=client_credentials' \ --data-urlencode 'scope=scope1 scope2 scope3' ``` ### Troubleshooting #### IssuerUrl connectivity issues The `issuerUrl` setting is used by both standard [Authentication](/docs/deploying/authentication#open-id-connect-setup) (to authenticate users logging in to Orbital), as well as by Orbital to fetch user credentials for the Executor user. * User authentication will perform a browser-side redirect to the IssuerUrl - so the URL must be accessible from your browser. * Executor User authentication performs requests from Orbital's server - so the URL must be accessible from your server. Normally, this is not a problem. However, if you're running everything locally (eg., using Docker or Docker Compose) you may need to use `host.docker.internal` as the `issuerUrl` DNS name, ([docker docs](https://docs.docker.com/desktop/networking/#i-want-to-connect-from-a-container-to-a-service-on-the-host)) or set Docker to use the [Host Network](https://docs.docker.com/engine/network/drivers/host/). This is generally not an issue in production (and the above workarounds are not suitable for production), as the network is normally more well defined. ### Observers vs Executors Persistent Streams are always executed under the permissions of the Executor User. However, these streams can also be observed by other users, through published [http or websocket endpoint](/docs/querying/queries-as-endpoints#saved-streams). In this scenario, policies are applied twice: * First, the stream is executed using the permissions of the Executor user * Then, when being observed, the results of the stream are then re-evaluated using the permissions of the user observing the stream As a result, the observed output may differ from the actual data being emitted by the stream. -------------------------------------------------------------------------------- # Unit testing Source: https://orbitalhq.com/docs/testing/unit-testing Section: Testing -------------------------------------------------------------------------------- Orbital has a dedicated unit testing framework called [Preflight](https://preflight.orbitalhq.com) which you can use to test Orbital and Taxi projects. It's a powerful framework, built on the same tools that the Orbital team use internally for building and testing Orbital itself. Learn more about preflight in the dedicated [docs](https://preflight.orbitalhq.com), or in [blog post](/blog/2025-06-26-announcing-orbital-preflight) -------------------------------------------------------------------------------- # Test environments Source: https://orbitalhq.com/docs/testing/test-environments Section: Testing -------------------------------------------------------------------------------- **Note: Turn it on** Test environments are disabled by default, to prevent you deploying these to production. Orbital uses a test framework called [Nebula](https://nebula.orbitalhq.com) to spin up test environments. Nebula allows declaring services which are deployed as docker containers (using [TestContainers](https://testcontainers.com/)), and scripting their behaviour, using Kotlin code. For example, to deploy a Kafka broker which emits stock quotes every 100ms, you'd declare the following: ```kotlin stack.nebula.kts stack { kafka { producer("100ms".duration(), "stockQuotes") { jsonMessage { mapOf( "symbol" to listOf("GBP/USD", "AUD/USD", "NZD/USD").random(), "price" to Random.nextDouble(0.8, 0.95).toBigDecimal() ) } } } } ``` Orbital has first-class support for Nebula, automatically deploying the test environments declared in `*.nebula.kts` files. ## Enabling test environments Support for Nebula-powered test environments is turned off by default, to ensure that you don't accidentally deploy Nebula support into production. In a local environment, start Orbital with the following parameter: ``` --vyne.toggles.nebulaEnabled=true ``` ### Deploying a Nebula server Nebula runs on a standalone docker container. Inside your Docker Compose, the following should be declared: ```yaml services: # Nebula powers Orbital's test environments. # You can use Nebula to quickly deploy and script test infrastructure, such as # Kafka brokers, databases, HTTP servers, AWS instances and more. # # See more at https://github.com/orbitalapi/nebula # # This is optional, and shouldn't be deployed in production. nebula: image: orbitalhq/nebula:next expose: - 8099 ports: - "8099:8099" # Nebula starts other docker containers, such as Kafka etc., so needs to # run on the host network # network_mode: "host" volumes: - /var/run/docker.sock:/var/run/docker.sock # Allow nebula to control Docker environment: - DOCKER_HOST=tcp://localhost:2375 ``` Alternatively, to just run Nebula outside of compose as a standalone Docker container, run: ```bash docker run -v /var/run/docker.sock:/var/run/docker.sock --privileged --network host orbitalhq/nebula ``` If you generated a Docker Compose from [start.orbitalhq.com/compose.yml](https://start.orbitalhq.com/compose.yml) with Nebula support enabled, then this might already be done for you. **Warning: Not intended for production** You should never deploy Orbital to production with Nebula support enabled, as it allows running of arbitrary code. For this reason, Nebula is disabled by default in our docker images. The Docker Compose provided at [start.orbitalhq.com/compose.yml](https://start.orbitalhq.com/compose.yml) is intended for rapidly deploying local development enviroments, so enables Nebula ### Docker compose OS variations **Note: OS Specific** The docker compose config for Nebula is subtly different between Windows, Mac and Linux. Nebula uses docker directly to download and start images, based on the `*.nebula.kts` file. The config for allowing this in Docker Compose varies slightly between Linux, Mac and Windows. We have os-specific docker compose files prepared which include support for Nebula. It is reccomended to use these. By default, when you visit [start.orbitalhq.com/compose.yml](https://start.orbitalhq.com/compose.yml) in a browser, we'll detect your OS and serve the correct variation. (The [start script](/docs#what-the-install-script-does) does this for you.) However, if you're scripting this, you can force the OS using one of the following links: * Windows: [https://start.orbitalhq.com/compose.yml?os=windows](https://start.orbitalhq.com/compose.yml?os=windows) * Mac: [https://start.orbitalhq.com/compose.yml?os=mac](https://start.orbitalhq.com/compose.yml?os=mac) * Linux: [https://start.orbitalhq.com/compose.yml?os=linux](https://start.orbitalhq.com/compose.yml?os=linux) ### Configuring services.conf Orbital will attempt to contact Nebula at `http://nebula`. If you're running on K8s or Docker Compose, the default DNS will resolve this. However, if you need to deploy Nebula somewhere else, you can configure this be modifying the [services.conf](/docs/deploying/configuring-orbital#configuring-where-orbital-finds-other-orbital-services) file: ```hocon services { nebula { url="http://localhost:8099" } } ``` ## Defining test environments in your project Once test environment support is [enabled](#enabling-test-environments), you can add Nebula files (`*.nebula.kts`) in your Taxi project. First, add a Nebula section in your Taxi project's additionalSources: ```hocon name: com.acme/demo version: 0.1.0 sourceRoot: src/ additionalSources: { "@orbital/config" : "orbital/config/*.conf", > "@orbital/nebula" : "orbital/nebula/*.nebula.kts" } ``` You can now add Nebula files into your `orbital/nebula` directory. ```kotlin stack.nebula.kts http { get("/ticketPrices/{cinemaId}") { call -> val price = Random.nextDouble(9.99, 19.99).toBigDecimal().setScale(2, RoundingMode.HALF_UP) call.respondText("""{ "price" : $price }""", ContentType.parse("application/json")) } } ``` For more information on Nebula, and how to declare stacks, consult the [Nebula docs](https://nebula.orbitalhq.com) ## Starting and updating test environments Orbital will automatically submit Nebula config files found in any of the Taxi projects through to Nebula. As the Nebula stack files change, updates are automatically submitted through to Nebula. The list of active Nebula stacks is displayed in the Test Environments tab: ## Nebula environment variables As services start and stop, various configuration parameters change - such as ports, paths, database names, etc. Nebula exposes these variables back to Orbital, and Orbital makes them available as variables you can refer to in your connections. Clicking on a server in the Test Environments view displays the environment variables exposed for the container: You can then use these environment variables in your `connections.conf` files that define your connections: ```hocon connections.conf aws { my-aws-account { # Optional Parameter. When not provided Orbital will use the [AWS default credentials provider](https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/credentials.html#credentials-default) by default. accessKey = ${NEBULA_S3_ACCESS_KEY} connectionName = my-aws-account # Optional parameter for development and testing purposes to point to a different endpoint such as a LocalStack installation. endPointOverride = ${NEBULA_S3_ENDPOINT_OVERRIDE} # Mandatory region = eu-west-1 # Optional Parameter. When not provided Orbital will use the [AWS default credentials provider](https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/credentials.html#credentials-default) by default. secretKey = ${NEBULA_S3_SECRET_KEY} } } ``` For HTTP based services, you can configure a `services.conf` file in your project: ```hocon services.conf services { ticketsApi { url="http://localhost:"${NEBULA_HTTP_PORT}"" } } ``` Now, services with urls pointing at a `ticketsApi` host will resolve to `http://localhost` with the port exposed by Nebula. ```taxi service TicketsApi { > @HttpOperation(method = "GET", url = "https://ticketsApi/tickets") operation getAllTickets(): Ticket[] } ``` Related links: * Learn more about [overriding server urls](/docs/describing-data-sources/http#modifying-server-ur-ls) using `services.conf` files * Be careful of common mistakes with [URL substitution in HOCON files](/docs/deploying/configuring-orbital#correctly-handling-substitutions-in-urls) -------------------------------------------------------------------------------- # Spring Boot schema publisher Source: https://orbitalhq.com/docs/sdks/java-and-kotlin/spring-boot-schema-publisher Section: SDKs -------------------------------------------------------------------------------- _Available since 0.37.0-M3_ The Spring Boot Schema Publisher starter automatically publishes Taxi schemas from your Spring Boot applications to Orbital on startup. This is useful for microservices architectures where individual services own their own schema definitions and need to automatically register them with a central Orbital instance. ## Adding the dependency Add the starter dependency to your Spring Boot application's `pom.xml`: ```xml com.orbitalhq schema-publisher-spring-boot-starter ${orbital.version} ``` You also need to declare the Orbital repository: ```xml orbital-releases https://repo.orbitalhq.com/release {/* snapshot is only required if you're using snapshots */} orbital-snapshots https://repo.orbitalhq.com/snapshot true ``` Or for Gradle (`build.gradle`): ```groovy implementation 'com.orbitalhq:schema-publisher-spring-boot-starter:${orbitalVersion}' ``` Once added, the starter will automatically: - Discover Taxi schemas in your application at startup - Publish them to the configured Orbital server ## Configuration The project uses reasonable defaults, so generally the only properties that requires configuration are: - The base URL of your server - The publisherId (which defaults to `${spring.application.name}`) Defaults can be overridden by configuring the publisher in your `application.yml`: ```yaml orbital: schema: publisher: enabled: true # Enable/disable publishing (default: true) base-url: https://my-service-prod/ # Sets the base URL, where Orbital should try to call this application publisher-id: my-service # Publisher ID (default: ${spring.application.name}) orbital-url: http://localhost:9022 # Orbital server URL (default: http://localhost:9022) connection-timeout: 5000 # Connection timeout in ms (default: 5000) ``` Or in `application.properties`: ```properties orbital.schema.publisher.enabled=true orbital.schema.publisher.base-url=https://my-service-prod/ orbital.schema.publisher.publisher-id=my-service orbital.schema.publisher.orbital-url=http://localhost:9022 orbital.schema.publisher.connection-timeout=5000 ``` ### Configuration properties | Property | Description | Default | |----------|-------------|---------| | `orbital.schema.publisher.enabled` | Enable or disable schema publishing | `true` | | `orbital.schema.publisher.base-url` | The base url for this spring boot application | Not set | | `orbital.schema.publisher.publisher-id` | Unique identifier for this publisher | Value of `spring.application.name` | | `orbital.schema.publisher.orbital-url` | URL of the Orbital server | `http://localhost:9022` | | `orbital.schema.publisher.connection-timeout` | Connection timeout in milliseconds | `5000` | | `orbital.schema.publisher.auth.headers` | Map of HTTP headers to send on every schema submission. See [Authentication](#authentication). | Not set | ## Authentication If your Orbital instance sits behind an authenticating proxy (or any reverse proxy that requires a token, API key, or tenant header), you can configure arbitrary headers to be sent on every schema submission. Headers are configured as a map under `orbital.schema.publisher.auth.headers`, and apply to **both HTTP and WebSocket transports**. For WebSocket transport, headers are sent on the connection handshake, and are re-sent automatically on every reconnect. ```yaml orbital: schema: publisher: auth: headers: Authorization: "Bearer eyJhbGc..." X-Tenant-Id: "acme-corp" ``` Or in `application.properties`: ```properties orbital.schema.publisher.auth.headers.Authorization=Bearer eyJhbGc... orbital.schema.publisher.auth.headers.X-Tenant-Id=acme-corp ``` Avoid checking real tokens into `application.yml`. Use environment variable substitution or a secrets manager: ```yaml orbital: schema: publisher: auth: headers: Authorization: "Bearer ${ORBITAL_API_TOKEN}" ``` If no `auth` block is configured, no authentication headers are sent — this is the default behaviour. ## HTTP vs WebSocket transport The transport protocol is selected automatically based on the URL scheme configured in `orbital-url`: - `http://` or `https://` -- uses HTTP transport - `ws://` or `wss://` -- uses WebSocket transport ### HTTP transport (default) With HTTP, schemas are published as a one-shot POST request to Orbital on startup. This is the simplest option and works well for development and testing. **Warning: Schemas may be lost if Orbital restarts** Orbital is designed to be stateless and deployable from git, so published schemas are held in-memory rather than persisted to disk. In a clustered deployment, published schemas are replicated across nodes, so they survive individual node restarts. However, if the entire cluster is restarted (or a single-node instance is restarted), published schemas are lost. The publisher is not notified and will not re-publish automatically. For production deployments, consider using WebSocket transport instead. ### WebSocket transport (recommended for production) _Available since 0.37.0-M4_ WebSocket transport maintains a persistent connection between your service and Orbital. If the connection is lost (e.g., Orbital restarts), the publisher automatically reconnects and re-publishes the schema. To enable WebSocket transport, change the URL scheme to `ws://` or `wss://`: ```yaml orbital: schema: publisher: orbital-url: ws://orbital.production.acme.com ``` The WebSocket connection uses the same port as Orbital's HTTP API -- no additional ports or configuration are required on the Orbital side. If the connection cannot be established immediately (e.g., Orbital is not yet running), the publisher will retry automatically with a configurable interval: ```yaml orbital: schema: publisher: orbital-url: ws://localhost:9022 retry-interval: 5s # How long to wait between reconnection attempts (default: 3s) ``` ## Schema generation At runtime, the plugin will introspect Spring `@RestController` and `@Service` components, and automatically generate Taxi `service` and `operation` schemas, along with the associated models for return types an inputs. If you're using Taxi annotations on your Java/Kotlin classes, these will be discovered and converted to Taxi schemas: ```java import lang.taxi.annotations.DataType; @DataType("com.acme.payments.PaymentRequest") public class PaymentRequest { @DataType("com.acme.payments.PaymentId") private String paymentId; @DataType("com.acme.payments.Amount") private BigDecimal amount; // getters and setters... } ``` ### Using semantic types in code When the schema is generated, type information is read at runtime. By default, only raw primitive types are available (such as `String`, `Int`, `Decimal`, etc). To use richer semantic types, you can use the `@DataType` annotation (`lang.taxi.annotations.DataType`) ```java import lang.taxi.annotations.DataType; @DataType("com.acme.payments.PaymentRequest") public class PaymentRequest { @DataType("com.acme.payments.PaymentId") private String paymentId; @DataType("com.acme.payments.Amount") private BigDecimal amount; } ``` #### Importing vs Declaring types By default, types declared with `@DataType` are declared as new types in the generated schema. eg - the above code snippet would result in the following Taxi being generated: ```taxi namespace com.acme.payments > // Types are declared > type PaymentId inherits String > type Amount inherits Decimal model PaymentRequest { paymentId: PaymentId amount: Amount } ``` If you want to use existing types already decalared in a shared taxonomy, then declare the `@DataType` as imported: ```java @DataType("com.acme.payments.PaymentRequest") public class PaymentRequest { @DataType(value = "com.acme.payments.PaymentId", imported = true) private String paymentId; @DataType(value = "com.acme.payments.Amount", imported = true) private BigDecimal amount; } ``` This would generate the following: ```taxi > // Types are imported > import com.acme.payments.Amount > import com.acme.payments.PaymentId namespace com.acme.payments model PaymentRequest { paymentId: PaymentId amount: Amount } ``` A good rule of thumb is that **Types should be imported (shared), whereas models should be published (declared by your service)**. This aligns with the Taxi best practice of share types, not models. #### Kotlin Type Aliases You can take advantage of Kotlin type aliases to make your code more semantic, and cleaner: ```kotlin @DataType(value = "com.acme.films.RentalDuration", imported = true) typealias RentalDuration = Int // elsewhere data class FilmRental(duration: RentalDuration) ``` ### Using generated Taxi types Taxi supports generating Java and Kotlin code, using the [Kotlin plugin](https://taxilang.org/docs/taxi-cli/kotlin-plugin). This plugin will generate semantic types into a reusable library, which you can import into your spring boot microservice. For example, given the following taxi code: ```taxi namespace films [[ The Id of a film ]] type FilmId inherits Int [[ The title of the film ]] type Title inherits String ``` In a Taxi project with the Kotlin plugin enabled: ```hocon name: com.petflix/films version: 0.1.0 sourceRoot: src/ plugins: { taxi/kotlin: { maven: { groupId: "com.petflix" artifactId: "films-taxonomy" } } } ``` When `taxi build` is run, a full Maven project is generated, which you can publish to your local Maven repository, and then pull into your Spring boot project. ```kotlin @RestController class FilmsApi { @GetMapping("/films/{id}") fun findById(id: FilmId): Film { // implementation } } ``` For more information, consult the docs on Taxi's [Kotlin plugin](https://taxilang.org/docs/taxi-cli/kotlin-plugin) ## Disabling the publisher To disable schema publishing (useful for local development or testing), set: ```yaml orbital: schema: publisher: enabled: false ``` You can also use Spring profiles to control when publishing is enabled: ```yaml # application-prod.yml orbital: schema: publisher: enabled: true orbital-url: https://orbital.production.acme.com # application-dev.yml orbital: schema: publisher: enabled: false ``` ## Environment-specific configuration Use environment variables or Spring's configuration management to configure different Orbital servers per environment: ```yaml orbital: schema: publisher: orbital-url: ${ORBITAL_SERVER_URL:http://localhost:9022} base-url: ${THIS_SERVICE_BASE_URL} ``` Then set the environment variable in your deployment: ```bash export ORBITAL_SERVER_URL=https://orbital.production.acme.com ``` ## Troubleshooting ### Publisher fails to connect on startup If the Orbital server is not available when your application starts, the publisher will log an error but will not prevent your application from starting. The publisher continues to retry to publish to Orbital. To adjust the connection timeout: ```yaml orbital: schema: publisher: connection-timeout: 10000 # 10 seconds ``` ## Next steps Once your schemas are published to Orbital, you can: - [Query your services using Taxi](/docs/querying/writing-queries) - [Combine data from multiple microservices](/docs/guides/composing-api-and-database) - [Set up streaming queries](/docs/querying/streaming-data) - [Publish queries as HTTP endpoints](/docs/querying/queries-as-endpoints) -------------------------------------------------------------------------------- # Querying in JVM applications Source: https://orbitalhq.com/docs/sdks/java-and-kotlin/jvm-query-sdk Section: SDKs -------------------------------------------------------------------------------- _Available since 0.38.0_ The Java SDK provides a convenient way to generate strongly typed queries and response types from TaxiQL queries. Designed for both Kotlin and Java, the library has minimal dependencies, and generates query interfaces at build time, which are then executed remotely against a running Orbital instance. The library is open sourced as part of [Taxi](https://taxilang.org). Full docs are available [here](https://taxilang.org/docs/sdks/java-kotlin) -------------------------------------------------------------------------------- # Production deployments Source: https://orbitalhq.com/docs/deploying/production-deployments Section: Deploying Orbital -------------------------------------------------------------------------------- ## Production concepts When deploying to production, there are common tasks you may wish to perform: * Enable [authentication](/docs/describing-data-sources/authentication-to-services) and [authorization](/docs/deploying/authorization) in Orbital * Use a [git-backed workspace.conf file](/docs/workspace/overview#reading-workspace-conf-from-git), to align with IAC * Understand how [secrets are handled](/docs/deploying/managing-secrets) in Orbital * Understand how to configure Orbital via [application properties](/docs/deploying/configuring-orbital#docker) or [environment variables](/docs/deploying/configuring-orbital#passing-orbital-application-configuration) ## Deploying using Docker Compose A sample `docker-compose.yml` is maintained by the Orbital team and published at [start.orbitalhq.com/compose.yml](https://start.orbitalhq.com/compose.yml). The quickest way to get it running is via our install script: ```bash curl -fsSL https://start.orbitalhq.com | bash ``` It contains reasonable defaults for getting a local environment running. Note that by default, Orbital does not enable authentication or role-based authorization, and you'll need to enable these. **Note: Learn more about our start script** Our [quick start guide](/docs) has a full breakdown of [what the install script does](/docs#what-the-install-script-does) - including the OS-specific Docker Compose, directory and `.env` setup it handles for you. ## Deploying with Terraform Orbital maintains deployment scripts for production configurations of Orbital in Terraform. These are available [here](https://github.com/orbitalapi/deploy) ## Deploying with Helm A community maintained Helm chart is available for deploying on Kubernetes. The Helm chart lives in it's own [Github repository](https://github.com/orbitalapi/helm), and can be installed using: ```bash helm repo add orbital https://orbitalapi.github.io/helm helm repo update helm upgrade -i orbital orbital/orbital --namespace=orbital --create-namespace ``` -------------------------------------------------------------------------------- # Configuring Orbital Source: https://orbitalhq.com/docs/deploying/configuring-orbital Section: Deploying Orbital -------------------------------------------------------------------------------- Orbital can be configured through a series of config files, passed at runtime. We designed Orbital this way to encourage scripted deployments, (as opposed to using a config database, which is hard to script). Orbital uses [HOCON](https://github.com/lightbend/config#examples-of-hocon) format for it's config files. (HOCON is a superset of JSON, so JSON is also valid.) ## Configuration basics There's several common themes in how Orbital configuration works. ## Creating reasonable defaults If you don't provide a config file, Orbital will create one for you on startup, and populate it with reasonable defaults. ## Environment variables and sensitive data Often configuration files need to contain sensitive data (usernames, passwords, etc). It may not always be desirable to specify sensitive connection information directly in the config file - especially if these are being checked into source control. Environment variables can be used anywhere in the config file, following the [HOCON standards](https://github.com/lightbend/config#uses-of-substitutions). For example: ```HOCON jdbc { some-database-connection { connectionName = some-database-connection # .. other params omitted for bevity .. connectionParameters { # .. other params omitted for bevity .. password = ${postgres_password} # Reads the environment variable "postgres_password" } } } ``` ### Correctly handling substitutions in Urls In some config files, you'll need to specify URLs, and will want to use variables. Using a variable inside a url using Hocon can be tricky. In short, here's how substitutions need to be defined: ```hocon query-server { // The Url has specifal characters (:), so needs to be inside of quotes. // However, variable substitution doesn't work inside of quotes, // so the variable must be outside of quotes. url="http://"${MY_VARIABLE}":9305" } ``` For more information, see [this issue](https://github.com/lightbend/config/issues/633) in the Hocon library ### Environment variables in annotations Sometimes it's desirable to variable things defined in annotations by envirnoment. For example, you may wish to modify the URL of an HTTP service, the name of a database table, or a the name of a Kafka topic that you subscribe to, based on environment. For example: ```taxi service QuotesService { @KafkaOperation(topic = "${STOCK_PRICE_TOPIC}") stream quotes : Stream } ``` **Note** Environment variables in annotations is **not** a feature of the standard Taxi compiler - this capability is provided by Orbital, so will not work when using the Taxi CLI etc. Environment variables here can be resolved from (in order of precedence): - An [env.conf config file](/docs/deploying/managing-secrets#using-env-conf-for-sensitive-data) - System environment variables ## Environment specific configuration _Available since 0.37.0_ Orbital supports loading environment-specific variations of configuration files, making it easy to manage different settings across environments (development, staging, production, etc.) without duplicating configuration files. ### How it works When an environment name is configured, Orbital will automatically load both the base configuration file and its environment-specific variation (if present). For example, if you configure `--vyne.environment.name=preprod`, Orbital will load: - `auth.conf` (base configuration) - `auth.preprod.conf` (environment-specific overrides, if present) This applies to all configuration files including: - `auth.conf` / `auth..conf` - `services.conf` / `services..conf` - `env.conf` / `env..conf` - `connections.conf` / `connections..conf` - Pipeline configuration files (`*.conf` / `*..conf`) ### Configuration Set the environment name when starting Orbital: #### Docker ```yaml services: orbital: image: orbitalhq/orbital:latest environment: OPTIONS: --vyne.environment.name=preprod ``` #### Environment variable ```bash export VYNE_ENVIRONMENT_NAME=preprod ``` ### Behavior - **Both files are loaded**: If both the base file and environment-specific file exist, both are loaded and merged - **Graceful fallback**: If the environment-specific file doesn't exist, only the base file is loaded (no error) When running with `--vyne.environment.name=production`, both files are loaded, allowing the production-specific settings to override or extend the base configuration. ## Passing Orbital application configuration There are several configuration settings referenced throughout these docs, which can be used to fine-tune how Orbital behaves. All config settings can be passed in a variety of ways: ### Docker In a docker / docker compose file, pass variables using the `OPTIONS` environment variable: ```yaml services: orbital: image: orbitalhq/orbital:latest environment: OPTIONS: >- --vyne.db.username=orbital --vyne.db.password=changeme --vyne.db.host=postgres ``` ### Setting as Environment variables Alternatively, any setting can be defined as an environment variable. Environment variables follow a different convention, so to convert, apply the following: * Use uppercase * Remove dashes * Replace dots with underscores eg: | Application property | Environment variable | |------------------------------|------------------------------| | `vyne.db.username` | `VYNE_DB_USERNAME` | | `vyne.workspace.config-file` | `VYNE_WORKSPACE_CONFIGFILE` | ## Configuring git repositories In production, it's common to read (and write) from a git repository, rather than a local file system. This allows the use of standard git workflows to promote changes to production. Orbital will periodically poll a git repository and branch, and pull in any changes as they're detected ### Using access tokens Github and Gitlab support embedding access tokens within the url of the git repository. Here's an example: ```hocon git { checkoutRoot="/some/path" repositories=[ { branch=master name=my-taxonomy uri="https://jimmy:glpat-purple-tortoise@gitlab.com/acme/acme-taxonomy" } ] } ``` ## Configuring where Orbital finds other Orbital services There are some services that Orbital will try to contact over HTTP(s), such as Prometheus for monitoring, or Nebula for test environments. You can override the default URLs used for Orbital-specific services by writing a `services.conf` file. By default, Orbital expects to read this from `config/services.conf`, but you can configure this path by passing a path to `--vyne.services.config-file` on startup. **Note** `services.conf` supports environment-specific variations. For example, you can create `services.production.conf` for production-specific service URLs. See [Environment-specific configuration files](#environment-specific-configuration) for details. ```hocon services { orbital-prometheus { url="http://localhost:9090" } nebula { url="http://localhost:8099" } } ``` ## Orbital Deployments Behind AWS / Azure Load Balancers AWS / Azure load balancers have `idle timeout` setting which refers to the duration of time that the load balancer allows an idle connection to remain open. An idle connection refers to a connection where no data is being transmitted between client and server. Between Orbital Server and UI, there are various websocket connections. For instance, Orbital server transmits `endpoint` statuses to UI over a websocket. To prevent these connections marked as `idle connections` and terminated by AWS / Azure ALB, you can enable a heartbeating mechanism by setting the following configuration variables: Application property | Description | Value |--------------------------------------|-----------------------------------|----------------------------------------------------| | `vyne.ws.enablePing` | Enables Websocket hearbeats | true = enable heatbeats, false = disable heartbeats| | `vyne.ws.pingIntervalInMilliSeconds` | Heatbeat interval in milliseconds | Any Long value | -------------------------------------------------------------------------------- # Configuring your license Source: https://orbitalhq.com/docs/deploying/configuring-your-license Section: Deploying Orbital -------------------------------------------------------------------------------- Orbital requires a license to run. This page covers how to obtain a license, where Orbital expects to find it, and how to override the defaults. ## Getting a license When you first sign in, Orbital automatically contacts the license server (`https://account.orbitalhq.app`), downloads a free license, and configures it for you. In most environments, no manual setup is required. If your machine cannot access the internet, or you’d prefer to manage your license manually, you can sign in to [account.orbitalhq.app](https://account.orbitalhq.app) and manage your license directly. From the portal you can: * Download a `license.json` file directly * View your active features and usage quotas * Manage instances registered against your license ## How Orbital loads a license On startup, Orbital searches for a `license.json` file in the following locations, in order. The first valid license wins: 1. The path provided via `--vyne.license.path` (if set) 2. `${vyne.app.data.path}/license.json` — by default `./orbital_data/license.json` 3. `~/.orbital/license.json` 4. `/opt/var/orbital/license/license.json` If you've signed in via the Orbital UI and no license is found on disk, Orbital will fetch one from the license server against your organisation and save it to the default write path (option 2 above). This is the recommended way to get started — just sign in and Orbital takes care of the rest. You can also download `license.json` from [account.orbitalhq.app](https://account.orbitalhq.app) and drop it into any of the locations above. ### Configuring the license path In your `docker-compose.yml`, Set `--vyne.license.path` to load the license from a custom location: ```yaml services: orbital: image: orbitalhq/orbital:latest environment: OPTIONS: >- --vyne.license.path=/opt/orbital/secrets/license.json ``` The downloaded license is also saved to this path, so make sure the directory is writable by Orbital. **Note** License configuration must currently be passed as command-line arguments (`--vyne.license.path=...`) or environment variables (`VYNE_LICENSE_PATH=...`). Overriding these settings in `application.yml` is not currently supported. See [Passing Orbital application configuration](/docs/deploying/configuring-orbital#passing-orbital-application-configuration) for the conventions. ## Verifying your license The Orbital UI shows your current license status, including the licensee, plan, expiry, and usage against any metered quotas. You can also query the license status directly: ```bash GET /api/license/status # Returns the current license and usage GET /api/license/status?refresh=true # Forces Orbital to re-download from the license server ``` Both endpoints require authentication, and the user must have the `ModifyLicense` privilege. ## When no valid license is found If Orbital can't find a valid license — for example, on a brand-new install before you've signed in — it will issue a short-lived **fallback license**, allowing the server to start. The fallback is written to `${vyne.app.data.path}/temp-license.json`. A fallback license lets you trial the application for 30 minutes. After this time, you'll need to provide a free license to continue using Orbital. Sign in via the UI, or place a real `license.json` at one of the search paths above, and restart Orbital to activate. Expired or invalid licenses are skipped during the search and Orbital will continue to the next location. The reason is logged at warn level. ## Air-gapped and offline deployments By default, Orbital will periodically report operation invocation counts back to the license server. This is used for usage attribution and quota enforcement. If your deployment is air-gapped or otherwise can't reach `account.orbitalhq.app`, Orbital supports running in air-gapped mode, which disables the requirement to report quotas, and removes the need to contact `account.orbitalhq.app`. If you'd like to use this features, please [reach out to our sales team](hello@orbitalhq.com) -------------------------------------------------------------------------------- # Capturing diagnostic data Source: https://orbitalhq.com/docs/deploying/capturing-diagnostics Section: Deploying Orbital -------------------------------------------------------------------------------- Sometimes you're working with the Orbital team to debug an issue, we might ask you for diagnostic traces from your server. ## Flight Recorder Orbital can be configured to run Java Flight Recorder for a period. We don't recommend doing this on a production instance. ### Enabling flight recorder You must enable this service on your Orbital instance. Enable flight recorder by updating your [application configuration](/docs/deploying/configuring-orbital#passing-orbital-application-configuration) with the setting `--vyne.debugging.jfr.enabled=true` ```yaml services: orbital: image: orbitalhq/orbital:latest environment: OPTIONS: >- > --vyne.debugging.jfr.enabled=true ## other settings continue ``` ### Capturing flight recorder data Once flight recorder has been enabled: 1. Start a recorder session by sending a POST to `/api/debug/jfr/start` 2. Perform whatever actions are being profiled. 3. End the recorder session by sending a POST to `/api/debug/jfr/stop` 4. Download the flight recorder log by visiting `/api/debug/jfr/download` in a browser Note: Flight recorder sessions can get very large. Ensure you have enough storage on your Orbital server. -------------------------------------------------------------------------------- # Distributing work across a cluster Source: https://orbitalhq.com/docs/deploying/distributing-work-on-a-cluster Section: Deploying Orbital -------------------------------------------------------------------------------- Deploying a cluster of Orbital instances allows for distribution of CPU intensive tasks, especially projections. This is especially well suited for queries that return large numbers of records, which need to be projected. Distributing the work over a cluster provides significant performance improvements, as the work can be parallelized. ## Understanding how work is distributed When running in a multi-node cluster, the node that receives the query becomes the work-coordinator for that query. The work-coordinator is responsible for issuing the initial queries that find the source data. As data arrives on the work-coordinator, it is then added onto a queue for other nodes to pick up and project. As worker nodes pick up projection tasks, they perform projection work (transforming and enriching data, calling remote services as required), and return the results back to the work-coordinator, who then sends the results back to the query issuer. ## Considerations ### Worker discovery via Multicast When sharing work across a cluster of Orbital nodes, the workers will use Multicast to discover each other and share work. Orbital uses Hazelcast for multicast discovery. Read more about how multicast works in Hazelcast [here](https://docs.hazelcast.com/imdg/latest/clusters/discovering-by-multicast) ### HTTP Caches Caches are not shared between worker nodes, so remote services may receive a higher number of calls during parallelized enrichment. ### Preferring work on remote nodes As the cluster size grows, the work can be parallelized across a greater number of nodes. While this provides improved throughput, the work coordinator incurs a heavier workload in serialization and deserialization of work tasks and responses. To account for this the `vyne.projection.distributionRemoteBias` allows tuning at which point the work is preferentially distributed to remote nodes, versus the query coordinator. Once this value is exceeded, the query coordinator node will perform a lower proportion of projection work in a query. ## Work Distribution Configuration For projection work to be distributed across all orbital nodes in a cluster the following configuration options are provided | Config parameter | Description | Default setting | |------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------|-----------------| | `vyne.projection.distributionMode` | Defines where projection work will be performed. On the local node or the cluster LOCAL, DISTRIBUTED | `LOCAL` | | `vyne.projection.distributionPacketSize` | Defines the number of records in each work packet distributed to other orbital servers. Applicable to DISTRIBUTED distributionMode only | 100 | | `vyne.projection.distributionRemoteBias` | Defines the number of cluster members before work is preferentially distributed to remote nodes | 10 | ## Cluster Configuration | Config parameter | Description | Default setting | |--------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------| | `vyne.hazelcast.discovery` | Specifies the discovery mechanism hazelcast will use to find other instances. `multicast`, `aws` or `eureka` | `multicast` | | `vyne.hazelcast.eurekaUri` | URI of the coordinating eureka instance when discovery is set to eureka | `http://127.0.0.1:8761/eureka/` | | `vyne.hazelcast.memberTag` | Denotes which member type this is within the hazelcast cluster | `vyne-query-service` | | `vyne.hazelcast.networkInterface` | Should only be specified if the deployment environment requires that hazelcast only bind to a single network interface. The network interface supplied should be specified in hazelcast networking naming e.g. 10.10.*.*, 172.16.*.* | | | `vyne.hazelcast.useMetadataForHostAndPort` | Specific to eureka hazelcast discovery. Specifies if metadata should be published to and used for hazelcast node and port discovery | `true` | | `vyne.hazelcast.awsPortScanRange` | Specific to aws hazelcast discovery. Specifies the port ranges that will be scanned from running hazelcast on ec2 instances | `true` | | `vyne.hazelcast.taskPoolSize` | Specifies the number of threads dedicated to processing distributed work loads | 2 | | `vyne.hazelcast.taskQueueSize` | Specifies the size of the queue of tasks on each cluster node. A value of 0 sets an infinite queue size | 0 | ### Cluster formation and discovery Orbital services can form a cluster discovering other nodes via one of three discovery mechanisms: | Mechanism | Description | |-------------|------------------------------------------------------------------------| | `multicast` | Orbital services attempt to use multicast to discover other cluster nodes | | `aws` | Orbital services attempt use AWS metadata to discover other cluster nodes | | `eureka` | Orbital services use Eureka metadata to discover other cluster nodes | ## Example configurations ### Multicast Enable clustering using discovery via local network multicast with distributed projections ```yaml vyne: projection: distributionMode: DISTRIBUTED hazelcast: discovery: multicast ``` ### AWS Enable clustering using AWS discovery with distributed projections The orbital query server should be run on a EC2 instances where AWS_REGION is specified as an environment variable and the EC2 instance is authorised to query ec2 instances via IAM. `AWS_REGION` may alternatively be specified as a JVM property `-DAWS_REGION=eu-west_2` ```yaml vyne: projection: distributionMode: DISTRIBUTED hazelcast: discovery: aws ``` ### Eureka Enable clustering using Eureka discovery with distributed projections ```yaml vyne: projection: distributionMode: DISTRIBUTED hazelcast: discovery: eureka eurekaUri: http://eureka-server:8761/eureka/ useMetadataForHostAndPort: true ``` -------------------------------------------------------------------------------- # Custom functions Source: https://orbitalhq.com/docs/extending/custom-functions Section: Extending Orbital -------------------------------------------------------------------------------- Orbital lets you extend the platform with **custom functions**, written in [Kotlin](https://kotlinlang.org/) or Java. These functions behave just like built-in functions: you can call them from Taxi queries, use them in transformations, and reuse them across projects. ## Why custom functions? While Taxi’s [standard library](https://taxilang.org/docs/language/stdlib) covers common operations, sometimes you need: - Custom domain-specific calculations - Utility functions for reuse across queries Orbital supports two ways of writing custom functions: - **Precompiled JARs** (Java or Kotlin) - **Kotlin scripts** (`.taxi.kts` files) **Note: Choosing between JAR files vs Kotlin scripts** Use JARs if you want strong CI/CD integration, test coverage, and debugger support. Use scripts if you want to iterate quickly, without a compile step. ## Writing custom functions The process for writing a function is the same, regardless if you're using Kotlin or Java, or which method you choose for pacakging your functions. The steps are as follows: 1. Implement the `TaxiFunctionProvider` interface (`com.orbitalhq.functions.TaxiFunctionProvider`) - `TaxiFunctionProvider` is a marker interface, with no methods, but is used to find the classes that have custom functions - If you're using [Kotlin Scrpting](#writing-functions-in-kotlin-scripts) the you can skip this - the scripting engine handles this for you. 2. Annotate your functions with `@TaxiFunction` 3. Annotate parameters with `@TaxiParam` Here's an example: ```kotlin // Always put your functions in a package - this will // dictate the namespace that the corresponding Taxi function is declared under package com.orbitalhq.example import com.orbitalhq.functions.TaxiFunction import com.orbitalhq.functions.TaxiFunctionProvider import com.orbitalhq.functions.TaxiParam class ExampleFunctions : TaxiFunctionProvider { // <-- implement TaxiFunctionProvider @TaxiFunction // <-- annotation fun sayHello(@TaxiParam name: String):String { return "Hello, $name" } } ``` The function declarations are fairly vanilla. Orbital handles the marshalling into and out of your function for you, as well as generating the corresponding Taxi [function definitions](https://taxilang.org/docs/taxiql/functions#declaring-external-functions) for your functions. ### TaxiFunction Functions must be annotated with `@TaxiFunction` (don't forget to `import com.orbitalhq.functions.TaxiFunction`) The following parameters are available on the annotation | Parameter | Type | Meaning | Default | |---------------|--------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------| | `name` | String | Optional explicit name of the function in Taxi. | Defaults to the Kotlin/Java method name if not provided | | `description` | String | Optional description - You can use Markdown within the string. Is converted into docs which are exposed via Orbital's data catalog | - | | `returnType` | String | Allows customizing the return type. Provide the fully qualified name of the **Taxi** type this function returns. If your function returns either a `TypedInstance`, or a non-scalar type, this must be defined | The Taxi Primitive type that corresponds to the declared Kotlin / Java return type of your function | ## Input binding Custom functions can access the declared parameters, along with various parts of the Orbital execution context, simply by declaring them in the function signature. **Note: TaxiParam is all you need (probably)** Under most scenarios, `@TaxiParam` is all you need. Other parameter types are documented here for completeness, but are only needed in advanced situations ### Parameters Your function can accept as many input parameters as required. The TaxiQL engine will provide these as passed by callers. Marshalling to/from Taxi instances [is handled by Orbital](#type-marshalling), allowing you to work directly with the native JVM types. Parameters must be annotation with `@TaxiParam`, which accepts the following parameters: | Parameter | Type | Default | |------------|------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------| | `name` | The name of the parameter as published in the generated Taxi function. | Defaults to the name of the method parameter if avaialble, otherwise `p0`, `p1`, etc. | | `type` | The fully qualified Taxi Type name. | Defaults to the corresponding Taxi primitive of the JVM type declared | | `nullable` | Indicates if it is acceptable for callers to pass null | If not provided, and using Kotlin, then will match the nullability of the Kotlin parameter. Otherwise, defaults to `false`. | For example: ```java Java import com.orbitalhq.functions.TaxiParam @TaxiFunction String sayHello( @TaxiParam String name, // suffix will be defined of type con.foo.NameSuffix // and will be marked as nullable @TaxiParam(type = "com.foo.NameSuffix", nullable = true) String suffix ) { return "Hello " + name } ``` ```kotlin Kotlin import com.orbitalhq.functions.TaxiParam @TaxiFunction fun sayHello( @TaxiParam name: String, // suffix will be defined of type con.foo.NameSuffix // and will be marked as nullable, to match the Kotlin definition @TaxiParam(type = "com.foo.NameSuffix") suffix: String? ):String { return "Hello, $name" } ``` ### Schema Callers can request the Schema be injected, by declaring an input parameter of type `Schema` (`com.orbitalhq.schemas.Schema`). This provides access to the full compiled Taxi schema, which allows for introspection, and is required when manually constructing TypedInstances (though, you normally don't need to do this.) ```java Java import com.orbitalhq.functions.TaxiParam import com.orbitalhq.schemas.Schema @TaxiFunction String sayHello( @TaxiParam String name, Schema schema, ) { // ... } ``` ```kotlin Kotlin import com.orbitalhq.functions.TaxiParam import com.orbitalhq.schemas.Schema @TaxiFunction fun sayHello( @TaxiParam name: String, schema: Schema ):String { // ... } ``` ### EvaluationValueSupplier The EvaluationValueSupplier provides access to the current context under which functions are executing, and provides ability to do things like: - Look up values in scope by type - Look up values in scope by their declared name - Evaluate functions or Taxi queries (including queries that load other data) - Construct Taxi objects This is a powerful - but advanced - supplier. It's unlikely to be needed for most use-cases ```java Java import com.orbitalhq.functions.TaxiParam import com.orbitalhq.models.EvaluationValueSupplier @TaxiFunction String sayHello( @TaxiParam String name, EvaluationValueSupplier valueSupplier, ) { // ... } ``` ```kotlin Kotlin import com.orbitalhq.functions.TaxiParam import com.orbitalhq.models.EvaluationValueSupplier @TaxiFunction fun sayHello( @TaxiParam name: String, valueSupplier: EvaluationValueSupplier ):String { // ... } ``` ## Returning values In it's simplest form, you can just return a Scalar type from your function, and Orbital will handle everything for you. Values can be returning in the following ways: * Plain values (e.g. `String`, `Int`) * `TypedInstance` (if you want full control of wrapping values) * `Result` (Kotlin) — success or failure * `Either` (Vavr, for Java) — right = success, left = failure * `null` → converted into a `TypedNull` JVM scalar types are marshalled to/from Taxi [TypedInstances](#typed-instance) (the internal way that values are stored at runtime by the Taxi engine) as described in [Type Marshalling](#type-marshalling) ```java Java import com.orbitalhq.functions.TaxiParam import com.orbitalhq.models.EvaluationValueSupplier @TaxiFunction > // Orbital will automatically handle conversion between the JVM String, and a Taxi TypedInstance > String sayHello( @TaxiParam String name, ) { // ... } ``` ```kotlin Kotlin import com.orbitalhq.functions.TaxiParam import com.orbitalhq.models.EvaluationValueSupplier @TaxiFunction fun sayHello( @TaxiParam name: String, valueSupplier: EvaluationValueSupplier > // Orbital will automatically handle conversion between the JVM String, and a Taxi TypedInstance > ):String { // ... } ``` ### Returning failures Sometimes functions can fail, and you want to indicate that to your callers. Custom functions can model failures using special return types - `Either<>` (Java) or `Result<>` (Kotlin) **Warning: Avoid throwing exceptions** Throwing exceptions on the JVM comes with a performance penalty. Within Orbital's function runtime, all exceptions are converted to special nulls To indicate that a function may fail, Orbital supports functions returning a special return type: | Language | Return type | |----------|-------------------------------------------------------------------------| | Java | [Either](https://docs.vavr.io/#_either) from [vavr](https://vavr.io/) | | Kotlin | [Result](https://kotlinlang.org/api/core/kotlin-stdlib/kotlin/-result/) | When a function returns a failure, the result is mapped to a `TypedNull` - a special kind of `TypedInstance`, which carries details of what the failure was. ```java Java public class SafeDivideFunctions implements TaxiFunctionProvider { @TaxiFunction(description = "Safely divide two integers. Returns Right(result) or Left(error message).") public Either safeDivide( @TaxiParam(name = "numerator") int numerator, @TaxiParam(name = "denominator") int denominator) { if (denominator == 0) { // Left = failure return Either.left("Cannot divide by zero"); } else { // Right = success return Either.right(numerator / denominator); } } } ``` ```kotlin Kotlin class SafeDivideFunctions : TaxiFunctionProvider { @TaxiFunction(description = "Safely divide two integers. Returns Result.success or Result.failure.") fun safeDivide( @TaxiParam(name = "numerator") numerator: Int, @TaxiParam(name = "denominator") denominator: Int ): Result { return if (denominator == 0) { Result.failure(IllegalArgumentException("Cannot divide by zero")) } else { Result.success(numerator / denominator) } } } ``` ## Providing functions via a JAR file You can package your functions as a JAR file - authored in either Kotlin or Java - which is copied to your Taxi project. ### Dependencies Add the following Maven dependencies: ```xml com.orbitalhq functions-api ${orbital.version} ``` You'll also need to declare the Orbital maven repository: ```xml orbital-releases https://repo.orbitalhq.com/release {/* optional - for snapshots */} orbital-snapshots https://repo.orbitalhq.com/snapshot true ``` ### Writing functions Create a class that implements `TaxiFunctionProvider`, and annotate methods with `@TaxiFunction`. Function parameters must be annotated with `@TaxiParam`. ```kotlin package com.acme.functions class GreetingFunctions : TaxiFunctionProvider { @TaxiFunction(description = "Greets a user by name") fun greet(@TaxiParam(name = "name") name: String): String = "Hello, $name" } ```` The function can then be used in a query: ```taxi find { greeting : String = greet("Jimmy") } // returns: { greeting : "Hello, Jimmy" } ``` ### Service Loader Orbital uses a Java Service Loader to discover the actual functions at runtime, which must be configured. #### Manually To manually declare the services, create a file at `src/main/resources/META-INF/services/com.orbitalhq.functions.TaxiFunctionProvider`, with a list of the names of your provider classes: ```text META-INF/services/com.orbitalhq.functions.TaxiFunctionProvider com.acme.functions.GreetingFunctions ``` #### Using Google's Auto Service Alternatively, you can use Google's [Auto Service](https://github.com/google/auto/tree/main/service) to handle the boilerplate for you. Add the following Maven dependencies: ```xml pom.xml com.google.auto.service auto-service-annotations 1.1.1 provided ``` And wire it into your maven build (note the different configurations, depending on if your code is in Kotlin or Java) ```xml Kotlin org.jetbrains.kotlin kotlin-maven-plugin kapt kapt com.google.auto.service auto-service 1.1.1 ``` ```xml Java maven-compiler-plugin com.google.auto.service auto-service ${auto-service.version} ``` Then, in your function class, add the `@AutoService` annotation: ```kotlin Kotlin import com.google.auto.service.AutoService; @AutoService(TaxiFunctionProvider.class) class ExampleFunctions : TaxiFunctionProvider { // ...snip... } ``` ```java Java import com.google.auto.service.AutoService @AutoService(TaxiFunctionProvider::class) public class ExampleFunctions implements TaxiFunctionProvider { // ...snip... } ``` When Orbital loads your project, it will discover the JAR, register your functions into the schema, and make them available to call from Taxi. ### Taxi project configuration The JAR files must be added to your Taxi project, and the location defined in your `taxi.conf`'s `additionalSources` section, using the key `@orbital/function-jar` ```hocon taxi.conf name: orbital/function-jar-example version: 0.3.0 sourceRoot: src/ additionalSources: { > "@orbital/function-jar" : "functions/*.jar" } ``` ## Writing functions in Kotlin scripts Kotlin Scripting provides a lightweight alternative to compiling functions into JAR's. Scripts authored in plain Kotlin, and avoids the build steps required with JARs. * ✅ Fast iteration, no compilation step * ✅ Easier for quick prototyping * ✅ Avoids having to deal with ServiceLoader * ✅ Avoids having to deal with ServiceLoader * ✅ [Testable](#testing) via Preflight * ❌ No IDE debugger support * ❌ Limited CI/CD support outside of Preflight * ❌ Unable to use 3rd party libraries * ❌ Compilation happens at runtime Kotlin scripting functions are compiled at runtime within Orbital, whenever the schema changes. However, there's minimal performance difference in terms of actual function execution. ### Writing a kotlin script For quick iteration, you can define functions inline in `.taxi.kts` scripts. Scripts are compiled dynamically at runtime. ```kotlin CustomFunctions.taxi.kts import com.orbitalhq.functions.* @TaxiFunction fun greet(@TaxiParam name: String): String = "Hello, $name" ``` When authoring a Kotlin Taxi script, you do not declare a class - simply write the functions directly in the script file. The `@TaxiFunction`, `@TaxiParam` annotations are automatically added to the classpath at runtime. Scripts must use the `.taxi.kts` extension. Orbital compiles them into providers automatically. ### Taxi config for scripts Kotlin script files should be added to your Taxi project, and the location defined in your `taxi.conf`'s `additionalSources` section, using the key `@orbital/function-kts` ```hocon taxi.conf name: orbital/function-jar-example version: 0.3.0 sourceRoot: src/ additionalSources: { > "@orbital/function-kts" : "functions/*.taxi.kts" } ``` ### Enabling completion support in IntelliJ settings IntelliJ provides completion support for Kotlin scripting, including Taxi Kotlin scripts. First, fetch the DSL JAR from the Orbital repository: If working against a released version: ```bash mvn dependency:get -Dartifact=com.orbitalhq:functions-kotlin-scripting-api:0.36.0 -DremoteRepositories=https://repo.orbitalhq.com/release ``` If working against a snapshot version: ```bash mvn dependency:get -Dartifact=com.orbitalhq:functions-kotlin-scripting-api:0.36.0-SNAPSHOT -DremoteRepositories=https://repo.orbitalhq.com/snapshot ``` Then, inside IntelliJ: * Go to Preferences -> Build, Execution, Deployment -> Compiler -> Kotlin Compiler * At the bottom you will find a section Kotlin Scripting * Complete the field: `Script definition template classes to load explicitly`: `com.orbitalhq.functions.kotlin.KotlinTaxiFunctionScript` * Complete the field: `Classpath required for loading script definition template classes`: `/.m2/repository/com/orbitalhq/functions-kotlin-scripting-api/0.36.0/functions-kotlin-scripting-api.jar` Then: * Go to Preferences -> Language & Frameworks -> Kotlin -> Kotlin Scripting * Make sure the script template Taxi Custom Functions is active and above the default Kotlin Script * Apply changes ## Choosing between JARs and Scripts There are a number of consdierations when choosing between JARs and Kotlin Scripts - though the choice ultimately depends on the preference of the team. | Feature | JARs | Kotlin scripts | |---------------------------------|----------------------------------------------------------------|--------------------------------------------------------------------------------------------| | Debuggable | ✅ Via IDE when authoring (not available at runtime in Orbital) | ❌ IDE's have limited support for debugging Kotlin Scripts, no runtime debugging in Orbital | | Unit-testable | ✅ Standard JVM testing approaches | ⚠️ (Preflight only) | | Compilation | At build time, as part of CI/CD | At runtime, on the Orbital serve | | Support for 3rd party libraries | ✅ | ❌ | | Loading | Requires configuring ServiceLoader | Just write the script, Orbital handles everything else | | Packaging | Requires seperate build and copy step - not handled by Orbital tooling | Not required - script file is deployed as-is | ## Appendix ### Namespaces Always place your custom functions inside a **namespace** to avoid collisions: ```taxi namespace com.acme.functions declare function greet(name: String): String ``` Orbital infers the namespace from the package name (for JARs) or script path (for `.taxi.kts`). Explicit namespaces keep your functions organized and predictable. ### Testing Custom functions are available for use in [Preflight tests](/docs/testing/unit-testing), so can be used as part of your project's testing strategy. Additionally, if you're bundling your functions as a JAR file, you can make full use of CI/CD and standard JVM test frameworks ### TypedInstance `TypedInstance` is the main way that a Taxi value is held in memory in the JVM. It's a combination of the following: - The actual value - The Taxi type - It's DataSource (which powers Lineage) Inputs and Outputs of Functions are always a TypedInstance. However, Orbital will handle marshalling between the Taxi TypedInstance and the JVM value for you. You generally don't need to interact with a TypedInstance, unless you're trying to do some advanced things ### Type Marshalling Values are automatically marshalled between their Taxi and JVM counterparts | JVM Type | Taxi Type | |-----------------------------------|----------------| | `java.lang.Boolean` | `Boolean` | | `java.lang.String` | `String` | | `java.lang.Character` | `String` | | `java.lang.Integer` | `Integer` | | `kotlin.Int` | `Integer` | | `kotlin.Short` | `Integer` | | `java.lang.Short` | `Integer` | | `java.math.BigInteger` | `Long` | | `kotlin.Long` | `Long` | | `java.lang.Long` | `Long` | | `java.math.BigDecimal` | `Decimal` | | `kotlin.Double` | `Decimal` | | `kotlin.Float` | `Decimal` | | `java.lang.Double` | `Decimal` | | `java.lang.Float` | `Decimal` | | `kotlin.Double` | `Double` | | `kotlin.Float` | `Double` | | `java.lang.Double` | `Double` | | `java.lang.Float` | `Double` | | `java.lang.Object` | `Any` | | `java.time.LocalDate` | `LocalDate` | | `kotlinx.datetime.LocalDate` | `LocalDate` | | `java.time.LocalTime` | `Time` | | `kotlinx.datetime.LocalTime` | `Time` | | `java.time.LocalDateTime` | `DateTime` | | `kotlinx.datetime.LocalDateTime` | `DateTime` | | `java.time.Instant` | `Instant` | | `kotlinx.datetime.Instant` | `Instant` | ## Best practices and considerations ### Consider if you can use an existing function instead Taxi has a really rich [StdLib](https://taxilang.org/docs/language/stdlib) of common functions, and strong support for [composing existing functions](https://taxilang.org/docs/taxiql/functions#composed-functions) together to form new functions. You really only need to declare a custom function if you require capabilities not currently exposed via the StdLib, or if you wish to encapsulate complex business logic. ### Performance Unlike most things in Orbital, function evaluation is blocking - it blocks the current thread that the function evaluation is occurring on. Therefore, you **must** make sure you keep functions fast. While it is technically possible to execute sub-queries within a function, you should avoid doing so, as it risks slowing down the entire Orbital server while waiting for a response. However, fetching of arguments to pass into a function is non-blocking. Therefore, rather than running a query to fetch data from within your function, define an argument for the data you need as an input to your function. ### Avoid throwing an exception Throwing exceptions has a performance impact on the JVM, and should be avoided. Orbital catches all exceptions thrown from a custom function, and marshals it back to a `TypedNull`, with data source of `FailedEvaluatedExpression`, which contains details of the failure. Rather than throwing an exception, return one of the following types to signal failure: - A [`Result<>`](https://kotlinlang.org/api/core/kotlin-stdlib/kotlin/-result/) (on Kotlin) (via `Result.failure<>(MyException("Boom"))` ) - A [vavr Either](https://github.com/vavr-io/vavr) (on Java) via `Either.left()` -------------------------------------------------------------------------------- # Introduction to semantic integration Source: https://orbitalhq.com/docs/background/intro-to-semantic-integration Section: Background -------------------------------------------------------------------------------- ## Overview Orbital is a data integration & querying platform, built on top of the Taxi schema language. The idea behind Taxi (and semantic types in general) is to allow producers of data to define both the structural contract (the labels assigned to fields & shapes of objects), and the semantic contract (what each field actually means). A schema plays multiple roles: - **Labels** define how information is addressed within a message - eg., `firstName`, `lastName` - **Structure** defines how a message will be laid out - what's nested where. Sometimes data is flat (eg., tables or CSV files), othertimes there's nested structure (such as JSON) - **Semantics** define the contract of what information is in place at a certain location. - **Encoding** defines how data is written to the wire. (Not all schemas cover this) Most schema languages focus entirely on addressing Labelling and Structure, but offer little to model the semantics of a contract. And, when you're trying to consume data, the semantics are the most important thing. **Note: Core principal** Producers of data need to be careful about Labels, Structure and Semantics of the data they emit. For consumers - the only thing that really matters is the content. Labelling and Structure are just addresses to access the information consumers need. ## Structure-first schemas Most data schemas (and subsequent integration techniques) focus on describing and connecting data based on the structure of information. Here are some examples of linking data based on structure, shown in a couple of different languages: ```typescript // Server-side contracts that the producers of data have defined: interface Actor { name: string id : number agentId: number } interface Agent { name: string id : number } // As a consumer - load an actor, then fetch it's agent using the agentId function loadAgentsAndActors() { const actor = await loadActor(); // Here, we're also coupled to another API call const agent = await loadAgent(actor.agentId) const castMember = { actorName : actor.name, agentName : agent.name } ``` In this example, we wanted to consume data with two attributes - an actors name, and their agents name. We also wanted to apply labels that were meaningful to us as consumers (`actorName` and `agentName`) that differed from the way producers originally labelled the data (simply - `name`). The only way we could build this was becoming tightly coupled to the producers contract. That's a problem, because tight coupling makes it hard for producers to evolve their APIs, and brittle for consumers when they do. **Note: Core principal** Integration based on structure is brittle - changes by producers have expensive knock-on effects to consumers. ## Semantic Integration Taxi and Orbital provide a different style of schema and integration - by allowing schemas to define their semantic contract as well. Semantic Types describe the meaning of content in a field, rather than just how it's written out by computers. | Field Name | System Type | Semantic Type | | ------------ | ----------- | ------------------- | | `id` | `String` | `CustomerId` | | `name` | `String` | `CustomerFirstName` | | `houseNbr` | `Int` | `HouseNumber` | | `ageInYears` | `Int` | `CustomerAge` | The column on the left - `Field name` - describes how a producer has named the field in one specific system. The column in the middle - `System Type` - describes how a system might read or write a field. The column on the right - `Semantic Type` - describes the actual meaning of the text in a business sense. By defining a semantic contract, we introduce a way of producers and consumers to discuss data, that's not coupled to structure. **Note: Core principal** Semantic contracts provide a shared language for producers and consumers to discuss data, that's not coupled to structure, so not as brittle when structural contracts change. Semantic contracts allow requesting data based on it's meaning, irrespective of the shape producers are emitting it in. Let's look at our example query again, this time run through TaxiQL - a semantic query language: ```taxi given { ActorId = 1 } find { actorName : ActorName agentName : AgentName } ``` It's important to call out a few details: ### There's no `FROM` There's no FROM clause here - nothing that specifies which producer, and where, the data should come from. That's left to the query layer to determine, which means that consumers remain protected when producers inevitably change. ### There's no `JOIN` Likewise, we haven't specified how to navigate infrastructure and data contracts to describe how to link two ideas (Actor and Agent) together. In fact, for all we know - this could be coming from one table, or some complex lookup linking several entities together. As a consumer, understanding the specifics of how data has been structured is an implementation detail they shouldn't be concerned with. ### Field names are defined by the consumer The producing systems named these fields something else. The consumer has been able to define the shape of the data they need, and left it to the querying layer to solve. There's enough semantic metadata present for the query layer to understand what data is needed. This is possible because we've separated the Structural concept from the Semantic concept. Field names are now entirely up to the consumer to decide - the way it should be. **Note: Core principal** Semantic querying is centered around what data the consumer needs, how the consumer needs it. Producer concepts are abstracted away. ## Enhancing Structural contracts with Semantic metadata It's not a question of Structure OR Semantics - or choosing one schema language to rule them all. Taxi is a great schema language, but it's not intended as a replacement to your existing schemas./ Modern data architectures are polyglot - with a mix of APIs, databases, streaming sources, and flat files. The key for making semantic schemas work across this ecosystem is to enhance existing schema structure with semantic metadata. Taxi supports this, with a series of extensions. For example, in Swagger / OpenAPI: ```yaml // Partial OpenAPI example: paths: /pets: get: description: | Returns all pets from the system that the user has access to Nam sed condimentum est. Maecenas tempor sagittis sapien, nec rhoncus sem sagittis sit amet. Aenean at gravida augue, ac iaculis sem. Curabitur odio lorem, ornare eget elementum nec, cursus id lectus. Duis mi turpis, pulvinar ac eros ac, tincidunt varius justo. In hac habitasse platea dictumst. Integer at adipiscing ante, a sagittis ligula. Aenean pharetra tempor ante molestie imperdiet. Vivamus id aliquam diam. Cras quis velit non tortor eleifend sagittis. Praesent at enim pharetra urna volutpat venenatis eget eget mauris. In eleifend fermentum facilisis. Praesent enim enim, gravida ac sodales sed, placerat id erat. Suspendisse lacus dolor, consectetur non augue vel, vehicula interdum libero. Morbi euismod sagittis libero sed lacinia. Sed tempus felis lobortis leo pulvinar rutrum. Nam mattis velit nisl, eu condimentum ligula luctus nec. Phasellus semper velit eget aliquet faucibus. In a mattis elit. Phasellus vel urna viverra, condimentum lorem id, rhoncus nibh. Ut pellentesque posuere elementum. Sed a varius odio. Morbi rhoncus ligula libero, vel eleifend nunc tristique vitae. Fusce et sem dui. Aenean nec scelerisque tortor. Fusce malesuada accumsan magna vel tempus. Quisque mollis felis eu dolor tristique, sit amet auctor felis gravida. Sed libero lorem, molestie sed nisl in, accumsan tempor nisi. Fusce sollicitudin massa ut lacinia mattis. Sed vel eleifend lorem. Pellentesque vitae felis pretium, pulvinar elit eu, euismod sapien. operationId: findPets parameters: - name: tags in: query description: tags to filter by required: false style: form schema: type: array items: //------Below is a taxi extension for semantic types----------- x-taxi-type: name: petstore.Tag //----------End the extension---------------------------------- type: string ``` Or in Protobuf: ```protobuf // An example of enriching a structural contract with semantic metadata syntax = "proto3"; import "taxilang/dataType.proto"; // Import the DataType extension package tutorial; message Person { string name = 1 [(dataType='PersonName')]; int32 id = 2 [(dataType='PersonId')]; string email = 3 [(dataType='PersonEmail')]; } ``` -------------------------------------------------------------------------------- # Tips on building taxonomies Source: https://orbitalhq.com/docs/background/tips-on-taxonomies Section: Background -------------------------------------------------------------------------------- ## Overview Orbital is a data integration & querying platform, built on top of the Taxi schema language. The idea behind Taxi (and semantic types in general) is to allow producers of data to define both the structural contract (the labels assigned to fields & shapes of objects), and the semantic contract (what each field actually means). The key to making this work well is in defining a shared taxonomy - a set of common terms with small scope and well-defined meaning. In this guide we'll take a look tips and pitfalls when crafting a taxonomy. ## Structuring your taxonomy ### Avoiding common domain models Trying to enforce standardisation of contracts between multiple systems is really hard, and leads to a lot of time spent forming consensus of design between teams that are working on different goals. This creates complex processes for agreeing to making changes, which makes innovation hard. Many of the reasons for choosing a shared model across teams (such as lower cost of integration) are solved out-of-the box with Orbital - so you get all the benefit of shared models, without the overhead associated with enterprise domain models. ### Types are intended for sharing Types don't have structure. In software terms, Types are referred to as Scalar - i.e., they don't contain any attributes or fields. Things like `String` or `Number` are typical examples of scalars. In Orbital and Taxi, Types take on a semantic meaning, i.e., `CustomerFirstName` instead of `String`. **Note: Best practice recommendation** Types shouldn't have structure or fields, as this makes it harder to share them between systems and teams. Instead, favour lots of small types with well clearly defined meanings. Getting teams to agree on the meaning of a field is easier than getting teams to agree on how to structure or name it. **Note: Best practice recommendation** Build a set of well defined types that form the basis of your glossary. This set of types is commonly called a *Taxonomy*. These types should change infrequently, so spend the time to ensure they're well documented, with clear definitions of meaning ### Models are intended for systems Models represent a strict contract that a system exposes. As such, the team that designs the system are best placed to design the contract that makes sense for them. Avoid trying to design models via consensus. As [discussed above](#avoiding-common-domain-models), getting consensus on shared models is tough, and most of the reasons for adopting shared models are solved using semantic types. -------------------------------------------------------------------------------- # Different strategies for publishing schemas Source: https://orbitalhq.com/docs/connecting-data-sources/schema-publication-methods Section: Background -------------------------------------------------------------------------------- ## Overview Orbital works by understanding where and how data is exposed across your organisation. Orbital is technology-agnostic, and is designed for organisations with a wide variety of technology stacks. As a result, there's lots of different ways to make schemas available to Orbital, each with it's own set of pros and cons. In this guide, we'll explore the different techniques, so you can choose the right mix that works across your organisation. We'll also provide recommendations. These are a mixture of principals that we've kept in mind when designing Orbital, and techniques we've seen customers implement. These aren't hard rules - like any advice, it's up to you to choose what works. ## Publishing schemas to Orbital Each system needs to decide how it's schema information will be made available to Orbital, and in what form. In a large organisation, it's typical to mix-and-match these approaches, to fit in what works for each team. Orbital is designed to support multiple different methods at the same time. A schema needs to describe a few important elements: ### The structural contract of data What's the shape of the data that the system is exposing (output) or expecting (input)? This includes field names, nested objects, and expected parameters for operations. Most schema languages (eg., OpenAPI / Protobuf / SQL) describe this really well. ```protobuf // An example of a clear structural contract, taken from the protobuf docs syntax = "proto3"; package tutorial; message Person { string name = 1; int32 id = 2; string email = 3; } ``` ### The semantic contract of data What's the meaning of each field that's being exposed (output) or received (input)? This is really critical when mapping data between systems, as it's how we ensure we're passing the correct information into the correct field. There's much less support for semantic contracts in standard schema languages. As a result it's common for someone to build adhoc maps in Word Documents, Wikis, or Spreadsheets. This is where languages like [Taxi](https://taxilang.org) really shine, as they let you bake in the semantic contract as well as the structural contract ```taxi // A contract that contains semantic data model Person { name : PersonName inherits String id : PersonId inherits Int email : EmailAddress inherits String } ``` ### Enriching structural contracts with semantics While you can use Taxi to define your schemas outright, this isn't a very common practice, given the prevolence of feature-rich, well supported schema languages. Therefore, a good practice is to combine the two, using taxi extensions inside existing schema languages. ```protobuf // An example of enriching a structural contract with semantic metadata syntax = "proto3"; import "taxilang/dataType.proto"; // Import the DataType extension package tutorial; message Person { string name = 1 [(dataType='PersonName')]; int32 id = 2 [(dataType='PersonId')]; string email = 3 [(dataType='PersonEmail')]; } ``` ## Systems push schemas to Orbital The preferred way of exposing schema data to Orbital is to have systems (or CI/CD tooling) publish the schema directly to Orbital. By making the system (and it's team) responsible for publishing its own definition, the schema documentation lives as close as possible to the system itself, so has the best chance of being up-to-date. The team that maintains the system can evolve any schema documentation along with the system itself. ### Automatically adapting to change Orbital automates the integration between services, by leveraging the metadata present in the schemas that are published to it. As schemas change, Orbital automatically adapts its integrations accordingly. In order to make the most of this capability, it's ideal to have systems automatically publishing their own schemas. The larger the separation between a change happening in a system, and the team responsible for updating the schema, the greater the chance of schemas being incorrect. Of course, this is no different from manual integration without Orbital - if documentation isn't maintained, then integration becomes error-prone. ### Generating schema definitions from code Generating schemas directly from code is a great way of ensuring that schemas evolve with the code, as they're generated at run-time. As a schema language, Taxi has great support for generating schemas directly from [Kotlin and Java](https://docs.taxilang.org/generating-taxi-from-source), with other framework support planned. Using this approach, services generate their own schemas, and publish them directly to Orbital on startup. ### Augmenting existing schemas with semantic metadata Many applications and systems already publish schemas using a rich schema language, such as Swagger / OpenAPI, Protobuf, JsonSchema, etc. Generally speaking, these schema languages only describe the structural contract of the data, but not the [semantic contract](#the-semantic-contract-of-data). Therefore, the ideal is to enhance existing schemas with this additional metadata. The Taxi project has growing support for embedding semantic metadata inside existing schema languages. | Schema Format | Taxi Support | |---------------|--------------| | OpenAPI | Supported | | Swagger | Supported | | Protobuf | Supported | | Avro | Supported | | JsonSchema | Supported | In these cases, a great solution is to simply enhance the existing schemas with additional metadata. ## Orbital polls systems for updates Orbital's schema server can be configured to poll sources for schemas, using a variety of back-end storages: * File systems * Git Repositories * OpenAPI endpoints * HTTP servers This is a strong option for scenarios where sytems can't publish their own schemas (eg., databases), or for data sources that are otherwise structureless (eg., CSV files). Additionally, using a git-backed repository for a shared glossary / taxonomy is a great way to allow decentralized authorship of the core set of glossary terms. ## Storing schemas separately from systems Sometimes it's not possible to have systems publish their own code - there's a variety of reasons for this: * Database schemas - which can't automatically be pushed * Legacy or external systems, which can't be modified to publish their own schemas * Schemaless content - such as CSV files In these cases, it's possible to store schemas in a git repository, and have Orbital's schema server manually poll the repository. The disadvantages here are that it's easy for the schema definition to drift from the actual schema as the system changes.