Avro format

​

Reading and writing Avro

Orbital can read and write data using Avro.

To declare that a type should be read as Csv, add the lang.taxi.formats.AvroMessage annotation to a model:

import lang.taxi.formats.AvroMessage 

@AvroMessage
closed model People {
    name : Name
    id : sId
    email : Email?
    phones : Phones[]
    last_updated : LastUpdated
}

When writing Avro, serialization order is important. Therefore, you should also specify the field order using an AvroField annotation:

import lang.taxi.formats.AvroMessage 
import lang.taxi.formats.AvroField 

@AvroMessage
closed model People {
   @AvroField(ordinal = 0) name : Name
   @AvroField(ordinal = 1) id : Id
   @AvroField(ordinal = 2) email : Email?
   @AvroField(ordinal = 3) phones : Phones[]
   @AvroField(ordinal = 4) last_updated : LastUpdated
}

Any requests or responses sent to/from systems where the model type has been annotated with @AvroMessage will be serialized using Avro.

​

Adding Taxi Types to Avro

Just like all schemas in Orbital, Taxi types are required to indicate how data is linked and relates between systems.

To add taxi metadata to your Avro file, use the taxi.dataType attribute. Here’s an example assigning a Taxi type name to a record / object:

{
   "type": "record",
   "namespace": "simple",
   "name": "AddressBook",
   // without this annotation, a type of simple.AddressBook would've
   // been created, based off the namespace and name
   "taxi.dataType": "com.example.AddressBook",
   "fields": [
     //... continues ...

Or, to define a taxi type for a field:

{
   "type": "record",
   "name": "Person",
   "fields": [
      {
        "name": "name",
        "type": "string",
        "taxi.dataType": "foo.PersonName"
      },
      {
         "name": "id",
         "type": "int",
         "taxi.dataType": "foo.PersonId"
      },

If a schema does not define / expose Taxi types for fields, Orbital will assign them when automatically.

For records, the namespace and name are used to infer the type name. For fields, a type is created based off the field name, and the type, which defines the underling primitive type.

​

Adding Avro projects to your workspace

Schemas declared in Avro can be added to your Workspace by either editing your workspace file (normally workspace.conf), or through the UI.

​

Using the Workspace file

​

Local file projects

You can add Avro schemas that are present locally on the file system of the server by adding the following declaration to your workspace.conf file

file {
    projects=[
        {
            loader {
                // All projects in Orbital require a unique identifier
                identifier {
                    id="com.marty/avro-test/1.0.0"
                    name=avro-test
                    organisation="com.marty"
                    unversionedId="com.marty/avro-test"
                    uriSafeId="com.marty:avro-test:1.0.0"
                    version="1.0.0"
                }
                // set the packageType to Avro
                packageType=Avro
            }
            packageIdentifier {
                name=avro-test
                organisation="com.marty"
                version="1.0.0"
            }
            // The path on your local machine to the avro schema
            path="/home/martypitt/dev/vyne/schema-server-core/src/test/resources/avro/addressBook.avsc"
        }
    ]
}

​

Git projects

You can add Avro schemas that are present within a git repository by adding the following declaration to your workspace.conf file:

The Git repo is monitored, so any changes are pulled locally and automatically applied inside of Orbital.

git {
    repositories=[
        {
            branch=main
            loader {
                identifier {
                    id="com.avro.git/avro/1.2.3"
                    name=avro
                    organisation="com.avro.git"
                    unversionedId="com.avro.git/avro"
                    uriSafeId="com.avro.git:avro:1.2.3"
                    version="1.2.3"
                }
                packageType=Avro
            }
            name=test-project
            // The path to the avro file within your Git repository
            path="/addressBook.avsc"
            // The url of your git repo
            uri="https://gitlab.com/acne/my-avro-project.git"
        }
    ]
}

​

Using the UI

You can load Avro schemas into your workspace using the UI

Start by navigating:

• Projects > Add Project

• Then select to add a project from either a Git Repo or Local disk

​

Adding an Avro project from Git

This workflow lets you add a reference to an Avro schema that’s checked into a git repository.

The Git repo is monitored, so any changes are pulled locally and automatically applied inside of Orbital.

• Provide the url of the Git repository, and click “Test Connection”
  • If the test was successful, repository name and branch are populated for you
• Set the project type to “Avro”
• Set the path to the location of your avro schema within the repo
• Provide a unique package identifier for this avro schema

​

Adding an avro project from a file

This workflow lets you add a reference to an Avro schema that’s already local on the server.

Any changes made to the file are automatically detected, and updated inside of Orbital.

• Set the project type to “Avro”
• Provide the path on your local machine to where the avro file is
• Provide a unique package identifier for this avro schema