Data Gator blog

To content | To menu | To search


Entries feed - Comments feed

Thursday 23 February 2017

Visualizing Aggregated Data (part 1)

IMPORTANT: The services described below are not currently available online. If anyone is seriously interested in testing this proof of concept demo, just let me know in the comments and I will set up a live secure (https) tunnel connection to the service. Sat Mar 11 2017 02:36:10

To create a Candlestick chart of the aggregated data all that is needed is to make a request to the chart end point. But, before that will work first the basic details of the chart need to be configured. This is very similar to how the other services work that I have written about in previous posts.

As shown in the example below the mutable parameters can be set in the URL request for the chart. For this example the URL would simply be pasted into the address bar of a web browser. The link below is functional and can be pasted directly into an address box of any web browser. The parameters can also be edited to try out different things.

Visualizing Aggregated Data (part 2)

The only required configuration keys for the Candlestick chart are the topicId (id of the Topic) and period (time period of the aggregated data). For this to work correctly the chosen Topic must have four different Aggregates of the same period that belong to the chosen Topic. The four different aggregate methods needed are Open, High, Low, and Close that are aggregated to the same period specified in the configuration key for the chart.

Only the topicId and period keys are needed, because the service will automatically search for the needed Aggregates (Open, High, Low and Close) using the period specified that belong to the Topic.

The different configuration keys available for a Candlestick chart are shown below.

Candlestick {
topicId (string),
period (number),
candlestickName (string, optional),
candlestickDesc (string, optional),
candlestickUnit (string, optional),
ohlcAggIds (Array[string], optional),
startingDate (string, optional),
width (number, optional),
height (number, optional),
imageFormat (string, optional),
imageDPI (number, optional),
imageQuality (number, optional),
theme (string, optional),
showInfo (boolean, optional),
timeZone (string, optional),
locale (string, optional)

I haven't had time to implement all the above parameters, but most should work.

Visualizing Aggregated Data (part 3)

Next to create an instance of a Candlestick chart configuration using curl.

curl -X POST --header 'Content-Type: application/json' --header 'Accept: application/json' -d '{ \
"topicId": "58a80d7d9b20b239b3378e6e", \
"period": 3600 \
}' ''

The response below showing the Aggregate objects automatically found for the Topic. Also there are no default values set for candlestickName, candlestickDesc or candlestickUnit as these are dynamically looked up from the Topic each time a chart image is generated on the server.

"topicId": "58a80d7d9b20b239b3378e6e",
"period": 3600,
"ohlcAggIds": [
"width": 1280,
"height": 720,
"imageFormat": "svg",
"imageDPI": 120,
"imageQuality": 80,
"theme": "dark",
"showInfo": true,
"timeZone": "utc",
"locale": "en_US.UTF-8",
"id": "58a85f5c9b20b239b3378e7d"

Visualizing Aggregated Data (part 4)

I have a tunnel connection setup to an old laptop that has been configured like a server. The data connection is only mobile internet from a very remote location and it will be very slow. Spaces can be insert into parameters that accept strings by using "%20". Using the special access_token=TOKEN will provide read access to 3 different Topics to create Candlestick charts. The data goes back to 2014 if I remember correct, but there are many spots where the data from the broker gets rough (missing or bad observations). The data seems to be getting worse as time goes on, maybe some maintenance is needed on the IoT device.

The 3 Candlestick object ids available shown first followed by there originating data URL are shown below.

  1. 58a85f5c9b20b239b3378e7d
  2. 58a860759b20b239b3378e7e
  3. 58a861009b20b239b3378e7f

The service uses many different VM containers and it would not be worth it to move everything onto a big server somewhere just for this small proof of concept demo, but I'm sure the rendered images would be much much faster. Locally on the laptop HD SVG images are rendered in about 150ms, other vector formats take about the same time. Raster formats (PNG, JPG, etc) take about 3 times as long as they are currently trans-coded from an SVG. These times are measured at the last stage entering and leaving the laptop and include verifying the token, gathering all the data, etc. The code that renders the images is currently all synchronous and single threaded. It is better to first get things implemented and running correctly before working on speed optimizations.

The data for the Topics are requested from the broker once about every 5 minutes, so there will be no new data to render any sooner than that. Historic data can be requested by using the startingDate parameter using just a date (startingDate=2015-03-23) or a fuller more precise format down to seconds (startingDate=2015-03-23T07:23:09). Different image formats can be returned by using the imageFormat parameter and the types currently supported should be bitmap, gif, jpeg, pdf, png, ps, svg, tiff and webp. I have limited the size of mostly the raster formats as they use a lot of the limited data currently available when they get large.

*** Update *** I have slowed down the interval for requesting new Topic data from the broker from every 5 minutes to about 20 minutes. This was done because the behavior of the broker seems different after the horizon has been reached in requesting Topic data. I am now busy with the Final Report and do not have time to look into it.

There is probably much I haven't mentioned, so if there are any questions just ask in the replies of any of the posts.