Configuring connections

The connections.conf file defines connections for systems that Orbital understands how to connect to.

A connection is required for most - but not - all data sources (eg., Databases, AWS services, etc). HTTP connections (OpenAPI / SOAP, etc) are generally self-descriptive enough not to require additional metadata.

​

Declaring in your taxi project

Connections are defined in a connections.conf, which lives inside a taxi project within your workspace.

Inside your taxi.conf file in your project, declare where you’re keeping your config for Orbital.

By convention, this lives at orbital/config/connections.conf, but this is configurable.

name: com.myproject/demo
version: 0.1.0
sourceRoot: src/
 additionalSources: {
     "@orbital/config" : "orbital/config/*.conf"   
 }

Your connections config file is a HOCON file, specifying how to connect to various resources.

For example:

jdbc {
  orders {
    connectionName = orders
    connectionParameters {
      database = ordersdb
      host = localhost
      username = martypitt
      password = ${superSecretPassword}
      port = 5432

    }
    jdbcDriver = POSTGRES
  }
}

​

Passing sensitive data

It may not always be desirable to specify sensitive connection information directly in the config file - especially if these are being checked into source control.

Environment variables can be used anywhere in the config file, following the HOCON standards.

For example:

jdbc {
   another-connection {
      connectionName = another-connection
      jdbcDriver = POSTGRES 
      connectionParameters {
         // .. other params omitted for bevity ..
         password = ${postgres_password} // Reads the environment variable "postgres_password"
      }
   }
}

​

Correctly handling substitutions in Urls

Substitution of variable inside a Url using Hocon can be tricky.

In short, here’s how substitutions need to be defined:

query-server {
   // The Url has specifal characters (:), so needs to be inside of quotes.
   // However, variable substitution doesn't work inside of quotes,
   // so the variable must be outside of quotes.
   url="http://"${MY_VARIABLE}":9305"
}

For more information, see this issue in the Hocon library

​

Database connections

Database connections are defined under the jdbc element within the connections.conf file.

jdbc { // The root element for database connections
   another-connection { // Defines a connection called "another-connection"
      connectionName = another-connection // The name of the connection.  Must match the key used above.
      jdbcDriver = POSTGRES // Defines the driver to use.  See below for the possible options
      connectionParameters { // A list of connection parameters.  The actual values here are defined by the driver selected.
         database = transactions // The name of the database
         host = our-db-server // The host of the database
         password = super-secret // The password
         port = "2003" // The port
         username = jack // The username to connect with
      }
   }
}

​

Supported drivers

​

Postgres

To configure a Postgres connection, specify jdbcDriver = POSTGRES

Connection parameters are as follows:

​

Example

jdbc { // The root element for database connections
   another-connection { // Defines a connection called "another-connection"
      connectionName = another-connection // The name of the connection.  Must match the key used above.
      jdbcDriver = POSTGRES // Defines the driver to use.  See below for the possible options
      connectionParameters { // A list of connection parameters.  The actual values here are defined by the driver selected.
         database = transactions // The name of the database
         host = our-db-server // The host of the database
         password = super-secret // The password
         port = "2003" // The port
         username = jack // The username to connect with
      }
   }
}

​

Redshift

To configure a Postgres connection, specify jdbcDriver = REDSHIFT

Connection parameters are as follows:

​

Example

jdbc { // The root element for database connections
   another-connection { // Defines a connection called "another-connection"
      connectionName = another-connection // The name of the connection.  Must match the key used above.
      jdbcDriver = REDSHIFT // Defines the driver to use.  See below for the possible options
      connectionParameters { // A list of connection parameters.  The actual values here are defined by the driver selected.
         database = transactions // The name of the database
         host = our-db-server // The host of the database
         password = super-secret // The password
         port = "2003" // The port
         username = jack // The username to connect with
      }
   }
}

​

Snowflake

To configure a Postgres connection, specify jdbcDriver = SNOWFLAKE

Connection parameters are as follows:

​

Example

jdbc { // The root element for database connections
   another-connection { // Defines a connection called "another-connection"
      connectionName = another-connection // The name of the connection.  Must match the key used above.
      jdbcDriver = SNOWFLAKE // Defines the driver to use.  See below for the possible options
      connectionParameters { // A list of connection parameters.  The actual values here are defined by the driver selected.
        account = mySnowflakeAccount123.eu-west-1
        schema = public
        db = demo_db
        warehouse = COMPUTE_WH
        schema = public
        role = QUERY_RUNNER
      }
   }
}

​

Kafka connections

Kafka connections are stored in the same config file, under the kafka element.

The connection specifies how to connect to a broker - but not the specifics of actual topics. Those are specified on the @KafkaOperation() metadata on a taxi operation definition. Read

kafka {
   my-kafka-connection {
      connectionName=my-kafka-connection
      connectionParameters {
         brokerAddress="localhost:29092,localhost:39092"
         groupId="orbital"
      }
   }
   another-kafka-connection {
      connectionName=another-kafka-connection
      connectionParameters {
         brokerAddress="localhost:29092,localhost:39092"
         groupId="orbital"
      }
   }
}

The following configuration options are supported under the connectionParameters

​

AWS connections

AWS connections are stored under the aws element.

Orbital uses AWS connections for example to connect to SQS for data pipelines and other services as part of query execution.

Orbital will use the AWS default credentials provider by default. This means you can configure the access credentials and region with environment variables (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY & AWS_DEFAULT_REGION). When running Orbital in AWS (e.g. ECS) it’ll also automatically pick up the role used to run the service and use that.

You can also configure the AWS connections manually which can be useful in cases where you need to connect to various different AWS accounts from a single installation of Orbital. As with all other config file value, you can either set the value explicitly, or read from an environment variable (as shown).

aws {
    my-aws-account {
        connectionName=my-aws-account
        connectionParameters {
            awsAccessKey=${AWS_ACCESS_KEY_ID}
            awsSecretKey=${AWS_SECRET_ACCESS_KEY}
            awsRegion=${AWS_REGION}

            // Optional parameter for development and testing purposes to point to a different endpoint such as a LocalStack installation.
            endPointOverride=${?AWS_ENDPOINT_OVERRIDE}
        }
    }
}

​

Testing with Localstack

You can point Orbital at an AWS mock running on Localstack by specifying the endPointOverride property in the connection.