Hazelcast as a data source

In addition to using Hazelcast as a cache, Orbital can use Hazelcast as a source for reading and writing data, as well as a streaming data source.

​

Adding a Hazelcast data source

To enable Hazelcast as a data source, first define a connection in your connections.conf file.

The simplest configuration simply needs a list of addresses to connect to:

hazelcast {
   myHazelcast {
      connectionName = myHazelcast
      addresses = ["localhost:5701"]
   }
}

​

Advanced configuration

Hazelcast has a number of connection parameters, which you may wish to tune. You can specify a full Hazelcast client configuration file, and provide the path.

For XML based configuration:

hazelcast {
   prodHazelcastCluster {
      connectionName = myHazelcast
      xmlConfig="hazelcast-config.xml"
   }
}

If your Hazelcast client config is in yaml format, you can define it your connection as:

hazelcast {
   prodHazelcastCluster {
      connectionName = myHazelcast
      yamlConfig="hazelcast-config.yaml"

   }
}

Paths can be either absolute (located on the server where Orbital is deployed), or relative (resolved relative to the conncections.conf file declaring the Hazelcast connection).

The Hazelcast docs provide more information on declarative configuration settings and provide full examples.

​

Supported formats

When values are written to Hazelcast, they must be serialized.

The following formats are supported. Note that because Orbital does not have access to custom Java classes, serialization formats that require classloading (such as Serializable, Externalizable and Custom Serialization) are not supported

​

Compact (preferred)

Compact Serialization is the preferred format when using Orbital to read/write to Hazelcast.

To enable Compact serialization, add a @CompactObject annotation to your Taxi definition:

import com.orbitalhq.hazelcast.CompactObject
import com.orbitalhq.hazelcast.HazelcastMap

@HazelcastMap(name = "films")
@CompactObject
closed model Film {
   @Id
   filmId : FilmId inherits Int
   // ...omitted...
}

​

JsonValue

Hazelcast’s JsonValue is supported for working with existing caches that have stored data using JsonValue.

Note - if you are both writing and reading the data using Orbital, it is recommended to use Compact, which has better performance.

​

Writing data to Hazelcast

To write data into a Hazelcast Map, define a Hazelcast Service in your taxi project, with a write operation:

import com.orbitalhq.hazelcast.HazelcastService
import com.orbitalhq.hazelcast.HazelcastMap
import com.orbitalhq.hazelcast.UpsertOperation
import com.orbitalhq.hazelcast.CompactObject

// The HazelcastService annotation should specify the 
// name of the connection defined in your connections.conf
@HazelcastService(connectionName = "myHazelcastConnection")
service HazelcastService {

   // Expose a write operation, and annotate it 
   // with UpsertOperation, to define the writing behaviour
   @UpsertOperation
   write operation upsert(Film):Film
}

type Language inherits String

// Define the name of the map to use. If not present in Hazelcast,
// the map wil be created
@HazelcastMap(name = "films")
// specify the serialization format 
@CompactObject
closed model Film {
   // A field annotated with @Id becomes the key.
   // Composite keys are not supported
   @Id
   filmId : FilmId inherits Int
   title : Title inherits String
   languages: Language[]
}

With your map declared, use a mutation query to insert data:

​

Examples

​

Inserting a static value into Hazelcast

// inserting a static value into Hazelcast
given { film : Film = 
  {
    filmId : 123,
    title : "Star Wars",  
    languages : ["English" , "American" ]
  } 
}
call HazelcastService::upsert

​

Storing a Query Result in Hazelcast

To write the outcome of a query to Hazelcast, it’s not necessary for the query’s result format to align with the format of the persisted value.

Orbital will automatically adapt the query result to the required persistence format, which may involve projections and even calling additional services if needed.

find { NetflixFilms[] } as {
  // projection not shown
}
call HazelcastService::upsert

​

Streaming data into Hazelcast

stream { FilmUploadedEvent }
// Each FilmUploadedEvent is projected into a Film
// which is the input parameter to `upsert`
call HazelcastService::upsert

​

Querying data from Hazelcast

Orbital supports querying from Hazelcast using both direct key lookups, and rich query criteria.

​

Defining a map to query

To query a map, you first define a service that exposes your Hazelcast Map.

Maps are exposed using taxi’s table declaration, as this indicates a data source that supports rich querying.

Here’s a complete example:

import com.orbitalhq.hazelcast.HazelcastService
import com.orbitalhq.hazelcast.HazelcastMap
import com.orbitalhq.hazelcast.CompactObject

// The HazelcastService annotation should specify the 
// name of the connection defined in your connections.conf
@HazelcastService(connectionName = "myHazelcastConnection")
service HazelcastService {

    // Table is a shorthand to declare
    // a data source that supports rich querying.
    table films : Film[]
}

// Define the name of the map to query.
@HazelcastMap(name = "films")
// specify the serialization format
@CompactObject
closed model Film {
   // A field annotated with @Id becomes the key, which 
   // is used when performing key lookups
   @Id
   filmId : FilmId inherits Int
   title : Title inherits String
   languages: Language[]
}

​

Writing queries

Once a Hazelcast map is exposed, it can be queried as a standard data source, including used as a data source when projecting and joining data from other sources (such as APIs, Kafka topics, or Databases).

Here are some sample queries:

​

Fetching everything from a map

find { Film[] }

​

Fetching a value with a specific key

If criteria is defined against the key (as defined using an @Id annotation), then a key lookup is performed:

// Assuming FilmId is annotated as @Id in the Film model
// as shown...
model Film {
   @Id
   filmId : FilmId inherits Int
   // ..snip..
}

// Elsewhere, writing a query...
find { Film( FilmId === 123 ) }

​

Fetching values using criteria

// find all films with a FilmId < 105
find { Film[]( FilmId < 105 ) }

// find all films released after 2019 with the title Star Wars
find { Film[]( ReleaseYear < 2019 && Title == "Star Wars" ) }

​

Streaming data from Hazelcast

Hazelcast Map’s can be treated as data streams, where inserts or updates are created as streams of events which can be queried using Orbital.

The following events trigger the current state of the record to be written to the event stream:

• Entry Added
• Entry Updated

​

Declaring a Map as a Stream

To stream updates from a map, you first define a service that exposes your Hazelcast Map as stream

Here’s a complete example:

import com.orbitalhq.hazelcast.HazelcastService
import com.orbitalhq.hazelcast.HazelcastMap
import com.orbitalhq.hazelcast.CompactObject

// The HazelcastService annotation should specify the 
// name of the connection defined in your connections.conf
@HazelcastService(connectionName = "myHazelcastConnection")
service HazelcastService {
   stream films : Stream<Film>
}

// Define the name of the map to query.
@HazelcastMap(name = "films")
// specify the serialization format
@CompactObject
closed model Film {
   // A field annotated with @Id becomes the key, which 
   // is used when performing key lookups
   @Id
   filmId : FilmId inherits Int
   title : Title inherits String
   languages: Language[]
}

​

Writing streaming queries

Once a Hazelcast map is exposed as a stream, it can be queried as a standard data source, including used as a data source when projecting and joining data from other sources (such as APIs, Kafka topics, or Databases).

Below are some sample queries. It’s useful when testing to combine this with writing data to Hazelcast to trigger change events which produce values on the data stream.

Here are some sample queries:

​

Stream all updates from a map

stream { Film }

​

Stream all updates from a map, and enrich with data from other sources

stream { Film } as {
  id : FilmId
  reviewScore : FilmReviewScore // not present in the map, will be looked up from another data source
}[]

​

Stream only specific events from a map

// Only provide updates on Films whose FilmId is less than 300
stream { Film.filterEach( (FilmId) -> FilmId < 300  ) }