Global Observer (aka Message Queue)

The Global Observer is an integration method that allows sending an external system all the data in the application as a series of messages as data changes, using a generic integration message queue.

If enabled, Researcher Solutions generates a message every time any object is created, updated or deleted in the system. This includes workflow transitions, submission of a form, editing a record, adding a record, etc.

Message format

Messages are formatted as JSON documents which contain zero or more messages, and a token which identifies the set of messages. If you use the polling API, this token is used to mark the messages as read. 

Inspect the queue

The message queue can be inspected in the UI. It is found in Tools → Integrations → Integration message queues → Global observation.

From here, you can see all messages ever generated by the system and which ones have been consumed by the integration layer. By clicking on the Info... link, you can also see the content of any message.

Errors in integrations

Any errors, such as HTTP transport errors, can be viewed and acknowledged in the admin interface: Tools → Integrations → Errors from integrations.

Default message format

The default format for messages is a JSON document containing zero or more messages, along with a token which can be used to mark messages as read.

The top level properties are:

Within the messages array, each entry will include the properties:

The messages will include other properties, but these will all be specific to your application and the integration.

Message IDs may not be sequential, and you can’t use them to check that all messages have been received.

Message delivery

Messages are offered up for consumption by the customer’s integration layer in one of two ways:

1. Customer polls for new messages on a schedule for new messages and marks them as read (default)
   1. See documentation: Message queue polling API
2. Researcher Solutions push messages to a customer-defined endpoint
   1. See documentation: Message queue push to remote HTTPS endpoint

Message queue polling API

The polling API enables your systems to pull messages from the Haplo application. This is our recommended method.

The Global observation page (referred to in section “Inspect the queue” above)  will give you two URLs: Poll URL and Mark URL.

The Poll URL is used to fetch the unread messages from the queue. This includes a token, which is then POSTed to the Mark URL to mark the messages as read and ensure they are not sent again.

This two step process ensures that transport errors will not result in messages being lost.

Create an API key

Polling requests are authenticated with an API key. Click Create API key... on the queue information page.

To authenticate the request, use HTTP Basic authentication with the username haplo and the API key as the password.

Request unread messages

To fetch a small number of the oldest unread messages, you should make a simple authenticated GET request to the Poll URL. Here’s an example script using a curl command:

#!/bin/sh
set -e
API_KEY=01234567890123456789
SERVER=example.haplo.train-au.cayuse.com
curl -X GET --user haplo:${API_KEY} https://${SERVER}/api/haplo-integration-messages/poll/{QUEUE}/fetch

Important: Replace QUEUE in the URL with the name of the queue. You can find the exact URL to use from the queue information page.

To read all messages in a queue, repeat the requests to the poll and mark endpoints until the unreadCount is zero.

Mark messages as read

The response to the Poll URL will include a token. This must be posted back to the server to mark the messages received as read, so they will not be included the next time the application is polled.

The token is opaque and should not be interpreted by your integration code, as its format may change at any time.

Your integration should extract the token from the response, then POST to the Mark URL with the token as the token parameter. An example curl script is:

#!/bin/sh
set -e
API_KEY=01234567890123456789
SERVER=example.haplo.train-au.cayuse.com
curl -X POST --data-urlencode "token=MSwy8w099=" --user haplo:${API_KEY} https://${SERVER}/api/haplo-integration-messages/poll/{QUEUE}/mark

Important: Replace QUEUE in the URL with the name of the queue. You can find the exact URL to use from the queue information page.

The response to the Mark request will be JSON with the structure below:

{
    "markedCount":2,
    "unreadCount":1
}

Configuration for different environments

You will probably need to write your integration so that it can be used with test and live environments.

The only things which will differ between environments will be the hostname in the URLs and the API key.

Message queue push to remote HTTPS endpoint

If your application pushes messages to an HTTPS endpoint on your systems, the configuration is specific to your application. We’ll provide details on how to configure the HTTPS endpoint.

Options include:

• Sending messages immediately they’re generated
• Sending batches of messages on a schedule
• HTTP Basic authentication
• SSL client certificates

Note that only https: endpoints are supported. Cayuse does not support unencrypted protocols.

Reliability

When the remote endpoint returns a 200 OK response, the messages will be marked as sent.

Any other status code or transport error will not mark them as sent, and the messages will be resent at some point in the future.

Admin user interface

The admin user interface will include a button to send all messages to your endpoint, rather than waiting for a schedule. This is useful when you have marked a message for resending.