Avro format

​Reading and writing Avro

Orbital can read and write data using Avro.

To declare that a type should be read as Csv, add the lang.taxi.formats.AvroMessage annotation to a model:

import lang.taxi.formats.AvroMessage    @AvroMessageclosed model People {    name : Name    id : sId    email : Email?    phones : Phones[]    last_updated : LastUpdated}

When writing Avro, serialization order is important. Therefore, you should also specify the field order using an AvroField annotation:

import lang.taxi.formats.AvroMessage import lang.taxi.formats.AvroField 
@AvroMessageclosed model People {   @AvroField(ordinal = 0) name : Name   @AvroField(ordinal = 1) id : Id   @AvroField(ordinal = 2) email : Email?   @AvroField(ordinal = 3) phones : Phones[]   @AvroField(ordinal = 4) last_updated : LastUpdated}

Any requests or responses sent to/from systems where the model type has been annotated with @AvroMessage will be serialized using Avro.

​Adding Taxi Types to Avro

Just like all schemas in Orbital, Taxi types are required to indicate how data is linked and relates between systems.

To add taxi metadata to your Avro file, use the taxi.dataType attribute. Here’s an example assigning a Taxi type name to a record / object:

{   "type": "record",   "namespace": "simple",   "name": "AddressBook",   // without this annotation, a type of simple.AddressBook would've   // been created, based off the namespace and name   "taxi.dataType": "com.example.AddressBook",   "fields": [     //... continues ...

Or, to define a taxi type for a field:

{   "type": "record",   "name": "Person",   "fields": [      {        "name": "name",        "type": "string",        "taxi.dataType": "foo.PersonName"      },      {         "name": "id",         "type": "int",         "taxi.dataType": "foo.PersonId"      },

If a schema does not define / expose Taxi types for fields, Orbital will assign them when automatically.

For records, the namespace and name are used to infer the type name. For fields, a type is created based off the field name, and the type, which defines the underling primitive type.

​Adding Avro projects to your workspace

​Different ways to work with Avro specs

There are a few different ways to expose your Avro spec to Orbital, each which suits different team workflows.

Schemas declared in Avro can be added to your Workspace by either editing your workspace file (normally workspace.conf), or through the UI.

​Using the Workspace file

Local file projects

You can add Avro schemas that are present locally on the file system of the server by adding the following declaration to your workspace.conf file

file {    projects=[        {            loader {                // All projects in Orbital require a unique identifier                identifier {                    id="com.marty/avro-test/1.0.0"                    name=avro-test                    organisation="com.marty"                    unversionedId="com.marty/avro-test"                    uriSafeId="com.marty:avro-test:1.0.0"                    version="1.0.0"                }                // set the packageType to Avro                packageType=Avro            }            packageIdentifier {                name=avro-test                organisation="com.marty"                version="1.0.0"            }            // The path on your local machine to the avro schema            path="/home/martypitt/dev/vyne/schema-server-core/src/test/resources/avro/addressBook.avsc"        }    ]}

Git projects

You can add Avro schemas that are present within a git repository by adding the following declaration to your workspace.conf file:

The Git repo is monitored, so any changes are pulled locally and automatically applied inside of Orbital.

git {    repositories=[        {            branch=main            loader {                identifier {                    id="com.avro.git/avro/1.2.3"                    name=avro                    organisation="com.avro.git"                    unversionedId="com.avro.git/avro"                    uriSafeId="com.avro.git:avro:1.2.3"                    version="1.2.3"                }                packageType=Avro            }            name=test-project            // The path to the avro file within your Git repository            path="/addressBook.avsc"            uri="https://gitlab.com/acne/my-avro-project.git"        }    ]}

​Using the UI

You can load Avro schemas into your workspace using the UI

Start by navigating:

• Projects > Add Project

• Then select to add a project from either a Git Repo or Local disk

Adding an Avro project from Git

This workflow lets you add a reference to an Avro schema that’s checked into a git repository.

The Git repo is monitored, so any changes are pulled locally and automatically applied inside of Orbital.

• Provide the url of the Git repository, and click “Test Connection”
  • If the test was successful, repository name and branch are populated for you
• Set the project type to “Avro”
• Set the path to the location of your avro schema within the repo
• Provide a unique package identifier for this avro schema

Adding an avro project from a file

This workflow lets you add a reference to an Avro schema that’s already local on the server.

Any changes made to the file are automatically detected, and updated inside of Orbital.

• Set the project type to “Avro”
• Provide the path on your local machine to where the avro file is
• Provide a unique package identifier for this avro schema