Orbital

Managing data sources

Configuring connections

Creating connections to databases, message brokers, and more

Overview

The connections.conf file defines connections for systems that Orbital understands how to connect to.

A connection is required for most - but not - all data sources (eg., Databases, AWS services, etc). HTTP connections (OpenAPI / SOAP, etc) are generally self-descriptive enough not to require additional metadata.

Declaring in your taxi project

Connections are defined in a connections.conf, which lives inside a taxi project within your workspace.

Inside your taxi.conf file in your project, declare where you’re keeping your config for Orbital.

By convention, this lives at orbital/config/connections.conf, but this is configurable.

taxi.conf
name: com.myproject/demoversion: 0.1.0sourceRoot: src/ additionalSources: {     "@orbital/config" : "orbital/config/*.conf"    }

Your connections config file is a HOCON file, specifying how to connect to various resources.

For example:

connections.conf
jdbc {  orders {    connectionName = orders    connectionParameters {      database = ordersdb      host = localhost      username = martypitt      password = ${superSecretPassword}      port = 5432
    }    jdbcDriver = POSTGRES  }}

Creating connections in the UI

Connections can also be created from the UI, which is often the easiest way to get started. When you add a data source that needs a connection - a database table, a Kafka topic, a MongoDB collection - the Connection dropdown offers Add new connection…. Pick the project to save the connection in, name it, choose the connection type and fill in its settings. Test connection checks Orbital can reach the system before you save.

Creating a Postgres connection from the Add data source page

Creating a Postgres connection from the Add data source page

Creating a Postgres connection from the Add data source page

Connections created this way are written to the project’s connections.conf - the same file described on this page - so they can be checked into version control like any other config. The Data sources page lists every connection in the workspace, with its current health.

The Data sources page, showing a database and a Kafka connection and the services they expose

The Data sources page, showing a database and a Kafka connection and the services they expose

The Data sources page, showing a database and a Kafka connection and the services they expose

When you’re scripting your deployment, or want reproducible environments, editing the config file directly is the way to go.

Using environment variables

Environment variables can be used in configuring connections. For example, you may wish to:

  • Provide secrets via environment variables
  • Parameterize topic names or table names between environments

In configuration files

It may not always be desirable to specify sensitive connection information directly in the config file - especially if these are being checked into source control.

Environment variables can be used anywhere in the config file, following the HOCON standards.

For example:

connections.conf
jdbc {   another-connection {      connectionName = another-connection      jdbcDriver = POSTGRES       connectionParameters {         // .. other params omitted for bevity ..         password = ${postgres_password} // Reads the environment variable "postgres_password"      }   }}

Correctly handling substitutions in Urls

Substitution of variable inside a Url using Hocon can be tricky.

In short, here’s how substitutions need to be defined:

query-server {   // The Url has specifal characters (:), so needs to be inside of quotes.   // However, variable substitution doesn't work inside of quotes,   // so the variable must be outside of quotes.   url="http://"${MY_VARIABLE}":9305"}

For more information, see this issue in the Hocon library

In annotations

You can use environment variables in any annotations, to allow you to parameterize things like topic names, or table names between environments.

Any annotation used in a Taxi schema may use environment variables. Environment variables are defined ${LikeThis} within the string.

For example:

@KafkaService( connectionName = "market-prices" )service MyKafkaService {      @KafkaOperation( topic = "${STOCK_PRICES_TOPIC}", offset = "earliest" )   stream streamStockPrices():Stream<StockPrice>}

Next steps

Next, you can edit your connections.conf file to add connections for your services.