Data formats

Avro format

Preview

Avro support is in preview, and expected to be released as part of 0.32. To use it ahead of release, pull the docker images tagged next

Reading and writing Avro

Orbital can read and write data using Avro.

To declare that a type should be read as Csv, add the lang.taxi.formats.AvroMessage annotation to a model:

import lang.taxi.formats.AvroMessage 

@AvroMessage
closed model People {
    name : Name
    id : sId
    email : Email?
    phones : Phones[]
    last_updated : LastUpdated
}

When writing Avro, serialization order is important. Therefore, you should also specify the field order using an AvroField annotation:

import lang.taxi.formats.AvroMessage 
import lang.taxi.formats.AvroField 

@AvroMessage
closed model People {
   @AvroField(ordinal = 0) name : Name
   @AvroField(ordinal = 1) id : Id
   @AvroField(ordinal = 2) email : Email?
   @AvroField(ordinal = 3) phones : Phones[]
   @AvroField(ordinal = 4) last_updated : LastUpdated
}

Any requests or responses sent to/from systems where the model type has been annotated with @AvroMessage will be serialized using Avro.

Adding Taxi Types to Avro

Just like all schemas in Orbital, Taxi types are required to indicate how data is linked and relates between systems.

To add taxi metadata to your Avro file, use the taxi.dataType attribute. Here’s an example assigning a Taxi type name to a record / object:

{
   "type": "record",
   "namespace": "simple",
   "name": "AddressBook",
   // without this annotation, a type of simple.AddressBook would've
   // been created, based off the namespace and name
   "taxi.dataType": "com.example.AddressBook",
   "fields": [
     //... continues ...

Or, to define a taxi type for a field:

{
   "type": "record",
   "name": "Person",
   "fields": [
      {
        "name": "name",
        "type": "string",
        "taxi.dataType": "foo.PersonName"
      },
      {
         "name": "id",
         "type": "int",
         "taxi.dataType": "foo.PersonId"
      },

If a schema does not define / expose Taxi types for fields, Orbital will assign them when automatically.

For records, the namespace and name are used to infer the type name. For fields, a type is created based off the field name, and the type, which defines the underling primitive type.

Adding Avro projects to your workspace

Schemas declared in Avro can be added to your Workspace by either editing your workspace file (normally workspace.conf), or through the UI.

Using the Workspace file

Local file projects

You can add Avro schemas that are present locally on the file system of the server by adding the following declaration to your workspace.conf file

Beta bits...

The project identifier is duplicated below. That's necessary today, but we'll fix that before this goes GA.
file {
    projects=[
        {
            loader {
                // All projects in Orbital require a unique identifier
                identifier {
                    id="com.marty/avro-test/1.0.0"
                    name=avro-test
                    organisation="com.marty"
                    unversionedId="com.marty/avro-test"
                    uriSafeId="com.marty:avro-test:1.0.0"
                    version="1.0.0"
                }
                // set the packageType to Avro
                packageType=Avro
            }
            packageIdentifier {
                name=avro-test
                organisation="com.marty"
                version="1.0.0"
            }
            // The path on your local machine to the avro schema
            path="/home/martypitt/dev/vyne/schema-server-core/src/test/resources/avro/addressBook.avsc"
        }
    ]
}

Git projects

You can add Avro schemas that are present within a git repository by adding the following declaration to your workspace.conf file:

The Git repo is monitored, so any changes are pulled locally and automatically applied inside of Orbital.

Beta bits...

The project identifier is duplicated below. That's necessary today, but we'll fix that before this goes GA.
git {
    repositories=[
        {
            branch=main
            loader {
                identifier {
                    id="com.avro.git/avro/1.2.3"
                    name=avro
                    organisation="com.avro.git"
                    unversionedId="com.avro.git/avro"
                    uriSafeId="com.avro.git:avro:1.2.3"
                    version="1.2.3"
                }
                packageType=Avro
            }
            name=test-project
            // The path to the avro file within your Git repository
            path="/addressBook.avsc"
            // The url of your git repo
            uri="https://gitlab.com/acne/my-avro-project.git"
        }
    ]
}

Using the UI

You can load Avro schemas into your workspace using the UI

Start by navigating:

  • Projects > Add Project
  • Then select to add a project from either a Git Repo or Local disk

Adding an Avro project from Git

This workflow lets you add a reference to an Avro schema that’s checked into a git repository.

The Git repo is monitored, so any changes are pulled locally and automatically applied inside of Orbital.

  • Provide the url of the Git repository, and click “Test Connection”
    • If the test was successful, repository name and branch are populated for you
  • Set the project type to “Avro”
  • Set the path to the location of your avro schema within the repo
  • Provide a unique package identifier for this avro schema

Adding an avro project from a file

This workflow lets you add a reference to an Avro schema that’s already local on the server.

Any changes made to the file are automatically detected, and updated inside of Orbital.

'Local' means local to the server

This workflow adds a reference to a Avro file that's on the disk of the server.

It's intended for developers who are running Orbital in a docker image on their local machine.

This workflow isn't intended for uploading an Avro schema to a remote server. Instead, use a Git repository
  • Set the project type to “Avro”
  • Provide the path on your local machine to where the avro file is
  • Provide a unique package identifier for this avro schema
Previous
Overview
Next
CSV