Configuring connections

Overview

The connections.conf file defines connections for systems that Orbital understands how to connect to.

A connection is required for most - but not - all data sources (eg., Databases, AWS services, etc). HTTP connections (OpenAPI / SOAP, etc) are generally self-descriptive enough not to require additional metadata.

Declaring in your taxi project

Connections are defined in a connections.conf, which lives inside a taxi project within your workspace.

Inside your taxi.conf file in your project, declare where you’re keeping your config for Orbital.

By convention, this lives at orbital/config/connections.conf, but this is configurable.

taxi.conf

name: com.myproject/demo
version: 0.1.0
sourceRoot: src/
 additionalSources: {
     "@orbital/config" : "orbital/config/*.conf"   
 }

Your connections config file is a HOCON file, specifying how to connect to various resources.

For example:

connections.conf

jdbc {
  orders {
    connectionName = orders
    connectionParameters {
      database = ordersdb
      host = localhost
      username = martypitt
      password = ${superSecretPassword}
      port = 5432

    }
    jdbcDriver = POSTGRES
  }
}

Using environment variables

Environment variables can be used in configuring connections. For example, you may wish to:

Provide secrets via environment variables
Parameterize topic names or table names between environments

In configuration files

It may not always be desirable to specify sensitive connection information directly in the config file - especially if these are being checked into source control.

Environment variables can be used anywhere in the config file, following the HOCON standards.

For example:

connections.conf

jdbc {
   another-connection {
      connectionName = another-connection
      jdbcDriver = POSTGRES 
      connectionParameters {
         // .. other params omitted for bevity ..
         password = ${postgres_password} // Reads the environment variable "postgres_password"
      }
   }
}

Correctly handling substitutions in Urls

Substitution of variable inside a Url using Hocon can be tricky.

In short, here’s how substitutions need to be defined:

query-server {
   // The Url has specifal characters (:), so needs to be inside of quotes.
   // However, variable substitution doesn't work inside of quotes,
   // so the variable must be outside of quotes.
   url="http://"${MY_VARIABLE}":9305"
}

For more information, see this issue in the Hocon library

In annotations

You can use environment variables in any annotations, to allow you to parameterize things like topic names, or table names between environments.

Any annotation used in a Taxi schema may use environment variables. Environment variables are defined ${LikeThis} within the string.

For example:

@KafkaService( connectionName = "market-prices" )
service MyKafkaService {

   @KafkaOperation( topic = "${STOCK_PRICES_TOPIC}", offset = "earliest" )
   stream streamStockPrices():Stream<StockPrice>
}

Compilation errors

Orbital enables the substitution of environment variables in annotations, which is not supported by the standard Taxi compiler.

Annotations that reference environment variables can be included without needing the variables to be defined when compiling Taxi projects in CI/CD environments or within VSCode.

However, in the Orbital environment, any annotation that references an environment variable must have that variable defined on the Orbital server. Otherwise, an undefined environment variable will trigger a compilation error.

Database connections

Database connections are defined under the jdbc element within the connections.conf file.

jdbc { // The root element for database connections
   another-connection { // Defines a connection called "another-connection"
      connectionName = another-connection // The name of the connection.  Must match the key used above.
      jdbcDriver = POSTGRES // Defines the driver to use.  See below for the possible options
      connectionParameters { // A list of connection parameters.  The actual values here are defined by the driver selected.
         database = transactions // The name of the database
         host = our-db-server // The host of the database
         password = super-secret // The password
         port = "2003" // The port
         username = jack // The username to connect with
      }
   }
}