How It Works

APIs Streaming For Efficient Data distribution

 

cloud-architecture

 

Streamdata.io acts as a reverse proxy that translates REST API polling into a stream of data.
As a result, load and bandwidth can be reduced by up to x66

What is polling?

polling

Polling is a mechanism used to keep data up to date.

To get fresh data on the web, you need to press F5 key or hit the reload button
on your browser. By doing that, the browser then performs a request to get the latest version of the web page.
Polling is a background mechanism pressing this F5 key for you at a fixed rate: the polling frequency.
This way, data displayed is always up to date.

What’s wrong with that?

Each polling performs a request to receive the latest version of the data, but there is a big chance
that this latest version of the data is exactly the same as the previous one. That means the response of
the polling request contains data that you already have. While nothing has changed, the process
did waste significant resources such as CPU and bandwidth to transfer useless data.

This effect is multiplied by the number of clients accessing the data and, therefore, waste is experienced by each client device.

Polling example

Let’s take a more in-depth look at the polling mechanism, and what happens when an application keeps its data current by using this method.

Poll response #1

  [
      { "name": "foo" },
      { "name": "bar" },
      { "name": "foo foo" },
      { "name": "bar bar" }
  ]

Poll response #2

Nothing has changed on the server, so the client once again downloads the same content.

  [
      { "name": "foo" },
      { "name": "bar" },
      { "name": "foo foo" },
      { "name": "bar bar" }
  ]

Poll response #3

This time, the third item has been updated from “foo foo” to “FOO BAR”. Note how the whole document is again received even if only the third item has changed.

  [
      { "name": "foo" },
      { "name": "bar" },
      { "name": "FOO BAR" },
      { "name": "bar bar" }
  ]

While the volume of updated data is pretty small, it is enough to understand how much bandwidth is wasted by the polling mechanism.

Solution: Incremental push

push

streamdata.io allows the polling mechanism to move closer to the server where it costs less, and provides efficient incremental updates (with Server-Sent Events) to the client, hence minimizing both bandwidth usage and costs.

Poll response #1

A first snapshot is sent to the client.

  { 
    "id": "f81d4fae-7dec-11d0-a765-00a0c91e6bf6",
    "event": "snapshot",
    "data": [
      { "name": "foo" },
      { "name": "bar" },
      { "name": "foo foo" },
      { "name": "bar bar" }
    ] 
  }

Poll response #2

No change detected, nothing is sent to the client.

Poll response #3

A JSON Patch Operation containing only the part that has changed to sent.

{ 
    "id": "f81d4fae-7dec-11d0-a765-00a0c91e6bf6",
    "event": "delta",
    "data": [
        { "op":"replace", "path": "/3/name", "value": "FOO BAR" } 
    ]
  }

Give a try!

Try streaming any JSON REST API within 30 sec

curl -v "https://proxy.streamdata.io/http://mysite.com/myJsonRestService?parameter1=[]&Parameter2=[]"