Orbital

Managing data sources

Working with Protobuf

Orbital supports reading protobuf messages - either as a message format for HTTP endpoints or from a streaming query originating from Kafka.

You can annotate protobuf sources directly with Taxi metadata, or you can import a .proto file through the UI.

Embedding semantic types in protobuf

Within a protobuf, fields can be annotated with taxi metadata to indicate the semantic type used on a field.

Fields are annotated using the taxi.dataType extension (which must be imported in the protobuf file).

An example is shown:

// Import paths resolve from the Taxi project root, not from the// folder that holds the .proto files.import "proto/taxi/dataType.proto";
message CafeDrink {  optional string customer_name = 1 [(taxi.dataType)="foo.CustomerName"];  optional int32 customer_id = 2 [(taxi.dataType)="foo.CustomerId"];
  enum Foam {    NOT_FOAMY_AND_QUITE_BORING = 1;    ZOMG_SO_FOAMY = 3;  }}

Datatype proto extension

Protobufs do not currently support importing external type definitions, so you need to include the source of the dataType proto definition directly in your protobuf project.

By convention, the protobuf definition should be written to taxi/dataType.proto inside the folder your project’s @orbital/protobuf source set points at (for example proto/taxi/dataType.proto when the source set is proto/**/*.proto).

Import paths in .proto files are resolved relative to the Taxi project root, not the proto folder, so the import is import "proto/taxi/dataType.proto";.

import "google/protobuf/descriptor.proto";
package taxi;
extend google.protobuf.FieldOptions {  optional string dataType = 50002;}

Generated taxi

When a protobuf with taxi metadata is imported by Orbital, a taxi representation is automatically generated.

@lang.taxi.formats.ProtobufMessage(packageName = "" , messageName = "CafeDrink")model CafeDrink {   @lang.taxi.formats.ProtobufField(tag = 1 , protoType = "string")   customer_name : foo.CustomerName?
   @lang.taxi.formats.ProtobufField(tag = 2 , protoType = "int32")   customer_id : foo.CustomerId?}

Importing a Protobuf through the UI

You can import a protobuf schema directly through the UI

Import the protobuf file

The data sources that can be added through the UI

The data sources that can be added through the UI

The data sources that can be added through the UI

To provide the protobuf you can:

  • Upload a single protobuf schema (drop it on the form, or click Browse)
  • Specify a URL to a single protobuf file
Uploading a .proto file

Uploading a .proto file

Uploading a .proto file

The .proto file must declare a package, as it’s used to name the generated types.

Then click Configure

Preview the generated types

A preview is shown, containing the models, fields and types that were imported from the Protobuf schema. Select a model in the tree on the left to see its attributes.

The preview of an imported protobuf, with the Person message selected

The preview of an imported protobuf, with the Person message selected

The preview of an imported protobuf, with the Person message selected

You can change any of the types (eg., swapping a primitive type with a more specific semantic type), by clicking on the pencil icon next to the type name, and searching for the desired type.

Once you’re satisfied, click Save, and the schema will be updated.