Data Gator blog

To content | To menu | To search


Entries feed - Comments feed

Sunday 22 January 2017

Create a Topic

The curl command below will create a new Topic that will aggregate the IoT data of the supplied topicURI. Below is the data portion of the response.

curl -X POST -d '{ "topicURI": "" }' ''

The "errorWait" and "errorRetryMax" are just default values and the "brokerCreate" date will be calculated on the backend. The "id" is used to uniquely reference the Topic on DataGator. 

"topicURI": "",
"errorWait": 20,
"errorRetryMax": 3,
"brokerCreate": "1970-01-01T00:00:00.000Z",
"id": "587f7c45aa784d4a3743667d",
"created": "2017-01-18T14:31:33.520Z",
"modified": "2017-01-18T14:31:33.520Z"

Verify Topic Configuration

A get request to can be used to verify the values that the Topic was configured with on the backend.

curl ''

Below is the data portion of the response.

"topicURI": "",
"topicName": "WeatherStation",
"topicDescription": "MathWorks Weather Station, West Garage, Natick, MA 01760, USA",
"topicUnit": "Temperature (F)",
"latitude": "42.299676",
"longitude": "-71.350525",
"elevation": "60",
"errorWait": 20,
"errorRetryMax": 3,
"brokerCreate": "2014-05-20T21:50:32.000Z",
"id": "587f7c45aa784d4a3743667d",
"created": "2017-01-18T02:55:41.091Z",
"modified": "2017-01-18T02:55:47.000Z"

All the optional values for the Topic were filled by the backend. To really leave a value empty just supply an empty string "" for it when creating the Topic.


The URLs that were shown for the DataGator service are not yet being made publicly available and are subject to change. They were just used to provide a clearer explanation of about how interaction with the service would look like.

I plan to create integrations to more IoT data brokers in the future also to allow direct bulk uploading of data. Also planned is to have a bridge to CoAP and implement a direct uploading of IoT data.

Saturday 21 January 2017

Configure a Topic

Here is the JSON configuration data structure with some brief descriptions for a Topic.

Topic {
topicURI, (string): The location on the internet to make requests for Topic data.
topicKey, (string, optional): An API key if needed to make requests for Topic data.
topicName, (string, optional): A name for this Topic
topicDescription, (string, optional): A description for this Topic
topicUnit, (string, optional): The units of measure of this Topic data
latitude, (string, optional): The latitude geo location of this Topic.
longitude, (string, optional): The longitude geo location of this Topic.
elevation, (string, optional): The elevation geo location of this Topic.
timeTag, (string, optional): A path to locate the time stamp component in the data structure of this Topic.
valueTag, (string, optional): A path to locate the value of interest in the data structure of this Topic.
sepTag, (string, optional): A single character used to separate the path components into the data structure of this Topic.
errorWait, (number, optional): The number of seconds to wait between failed requests the broker for Topic data.
errorRetryMax, (number, optional): The number of total retries of failed attempts to request Topic data from the broker.
brokerCreate, (string, optional): The starting time and date to use for retrieving Topic data from the broker.

Most all values are optional and the only required value is topicURI for access to data. The DataGator service will detect the broker from the topicURI and deal with the specifics of how to request data from the different brokers. The DataGator service attempt to be as user friendly and simple to use as possible. 

Friday 20 January 2017

HTTP IoT broker intregation

I have implemented an HTTP data broker integration with ThingSpeak. There seems to be a large amount of public IoT data available and it should be possible for anyone who wants to use their service. There is a slightly different naming convention on ThingSpeak from what I have noticed some other brokers using. ThingSpeak seems to use "channels" and "fields" where other brokers may use "devices" and "topics" respectively. Where the DataGator service just makes different aggregated topic level data available.

Some broker integration has also been done with OpenSensors. This integration has proved to be more challenging as there is more variety in the way the data is structured on the broker. This variety in the data storage structure made it necessary to construct different parse paths into the structures to allow describing where exactly the data of interest is.

I think the DataGator will complement with what the brokers offer. The visualizations that are provided by ThingSpeak seem to use client side JavaScript and what the DataGator service would offer are rendered server side. Using server side rendering should provide an advantage for constrained clients and make it easier to directly display visualizations on these devices.

Thursday 5 January 2017

A new year and a change to the project proposal

This project was submitted to the Eclipse IoT Challenge 3.0 and is number 29 in the public list of proposals. I have decided to make a change from the original proposal that was submitted. Instead of making the service available primarily only on CoAP I will be using HTTP instead. CoAp could still be supported using a protocol bridge, so there is no lost in this. This should make the service available to a bigger audience.

I am now researching my integration options for the project.