About

Streamdata.io is a real-time cache proxy allowing you to poll JSON REST APIs and push updates to clients. Wait, there is more: Streamdata.io keeps a history of the modifications which occured on the data between two pollings! This way, Streamdata.io is able to give you the list of modifications which happened since the last data request.

In other words, Streamdata.io is the perfect cache that significantly reduces the bandwidth consumption when streaming data that change both rarely and frequently. Check How it works for more details.

Features

  • Incremental data updates: Replacing polling by push is not enough to minimize data streams. That’s why Streamdata.io pushes only the portion which has changed.

  • Push: Updates are pushed to the client using Server-Sent Events (SSE). By providing fallback mechanisms, Streamdata.io can also work with older browsers.

  • API Cache: To reduce server load and latency, Streamdata.io has an integrated cache mechanism.

Read our blog post SSE vs. WebSockets to have a better understanding of why we chose SSE.

Testing your API

To start, you can check that your API is compatible with Streamdata.io by testing it through our demo environment.

curl

No development required. Just run the following curl command:

	#$ curl https://proxy.streamdata.io/<API-url>

Where:

	<API-url> = url of your JSON API.

For instance, a test with our simulated market data API can be done by running the following curl command:

	#$ curl https://proxy.streamdata.io/http://stockmarket.streamdata.io/prices

Access resources

Once you have validated that your API is compatible with Streamdata.io, you will be able to easily fetch data in real-time from your web or mobile apps. You can access all necessary resources by creating your free account on our portal. You should also have a look at our Getting started page.

JavaScript

Check out our Javascript SDK and demo apps on our GitHub repo. For more information, see our Javascript SDK documentation

Java

Check out our sample app for a server to server integration.

Android

You can find Android sample apps on our GitHub repo

iOS

We have published an iOS sample that shows feeds from the New York Times API throught Streamdata.io here

Environments

Streamdata.io provides two different environments:

  • A demo environment that is almost identical to the production environment. It is here to help you quickly experiment with our product and test the compatibility of your APIs.

  • A production environment that is secured, highly available and scalable. You can sign up for a Streamdata.io account here.

Demo vs. Production

There are a few key differences that you’ll want to keep in mind:

Sandbox Production

URL Prefix

https://proxy.streamdata.io/

https://streamdata.motwin.net/

Authentication

None

App Token

Encryption

TLS

TLS

Uptime guarantees

None

Highly Available & Scalable

Protocol

Incremental cache

To accept incremental updates of cached content, you must use:

Accept: application/json-patch+json
This is the default behavior. We recommend using it for content which is partially and frequently updated. Using the following headers will also activate incremental cache mode:
Accept: */*

or

Accept: text/event-stream

or

No Accept header

Cache

Accept: application/json

In this mode, Streamdata.io will send you the entire updated content, each time a change occurs in the source API.

Use this mode for content which is unfrequently and fully updated.

NOTE: using generic header supersedes all other header usage. The below example,

Accept: */*, application/json

will activate incremental cache as '*/*' supersedes 'application/json'.

Security

TLS Encryption

Streamdata.io uses Transport Layer Security (TLS) to encrypt messages while transmetting them through the Internet. Using TLS ensures that client messages are protected when being sent to and from Streamdata.io. This prevents intercepted messages from being viewed by unauthorized parties.

Application token

Streamdata.io provides every user’s application with a unique token that authenticates them on the platform. This enables Streamdata.io to tie a request to a specific account in order to monitor traffic, enforce quotas, handle billing and manage access control.

You must include your application token with all API requests that are passed through Streamdata.io.

If you believe that your token has been compromised, e.g. if you notice suspicious activity in your console traffic reports, then Streamdata.io recommends renewing your token.

In future versions, you will be able to decide whether or not to allow up to 24 hours to phase out your old token, during which time both tokens will be active. Using a phased deactivation allows you time to fully deploy your new token.

Acquiring and Changing application tokens

If you need to list or renew your application token, login to Streamdata.io, select your application and click on Security to list or change the application token.

Using application token

In order to provide your application token when connecting to Streamdata.io, you can use our Javascript SDK and have a look at our sample application which shows how to pass your token while connecting to Streamdata.io. If you prefer not using our JS SDK, you MUST provide your token using X-Sd-Token query parameter.

Example:

curl "https://streamdata.motwin.net/http://stockmarket.streamdata.io/prices?X-Sd-Token=[YOUR_APP_TOKEN]"
Application token is only required for production environment.

Request Signature

In order to provide an additional level of security you can activate Request Signature. It can be activated for each application independently by enabling the option in Settings > Security on your portal account. When Streamdata.io receives a signed request, it uses your private key to compute a signature for the message it received. It then compares the signature it calculated against the signature presented by the requester. If it matches the signature presented by the requester, the system grants access to it. If the two signatures do not match, the request is dropped and the system responds with an error message.

Please be aware that if you activate this option, all requests coming to Streamdata.io for the current application must be signed. As each client application requires a private key to sign its requests, it is strongly recommended that you use appropriate security measures when storing your private keys. In order to simplify the signature process on the client side, you can use our Auth Javascript library.

HTTP Authentication

Streamdata.io supports the following Authentication types:

  • Query Parameters

  • HTTP Headers

  • HTTP Basic

  • OAuth 2.0

Query Parameters

The Query Parameters type lets you add several authentication parameters to your URL.

https://api.yourapi.com/v1/{endpoint}?QueryParameter=Authentication

HTTP Headers

The HTTP Headers setting lets you add several header parameters to your API.

Auth-Header: 'Some Value Parsed by the Server'

HTTP Basic

The HTTP Basic setting follows the HTTP Basic Authentication protocol of Username:Password.

Authorization: Basic EWxhUGRpbjpvbGVuIHNlh3FtZQ==

This setting will automatically require Username and Password in the Authentication section of the test console.

OAuth 2.0

The OAuth 2.0 setting lets you set the Authorization URL, Access Token URL, and Scopes.

Compression

Server-Sent Events flow can be compressed on demand using gzip and/or deflate methods.

An example of how to use the X-Sd-Compress header:

curl "https://proxy.streamdata.io/http://stockmarket.streamdata.io/prices" -H "Accept-Encoding: gzip, deflate" -H "X-Sd-Compress: true" --compress

If this header is not provided, the default behavior is NOT to compress the data flow.

Configuration

Cache control configuration is not available in current version, but will be coming soon!

Streamdata.io caching behavior can be fully configured through HTTP Cache-Control header held in your server response. By default, Streamdata.io proxy will configure itself automatically using HTTP Cache-Control header held in server response.

Cache-Control directives

Cache-Control: public | private | no-cache (default: private)

This header determines whether a response data may be cached and shared among all users (public) or if the response contains data which is user-specific, and must not be cached (private & no-cache).

Streamdata.io default behavior regarding directives:

public

private

no-cache

Cache

Yes

No

No

Poll

Yes

Yes

Yes

  • These settings cannot be overridden by file configuration.

Private and no-cache are treated the same way in the Streamdata.io proxy. The difference only sits on the client side. Private allows data to be cached on the client while no-cache forbids it.
Setting public cache-control for private user specific content might have serious security impacts. All requesters would, therefore, receive the cached data; i.e., sending the private user’s banking information to all requesters.
Cache-Control: max-age=<int value> (default: none)

This header can be set to specify the maximum time in seconds that the data should remain in cache. After this period and if at least one client remains subscribed to the URL, Streamdata.io will fetch a new version of the data from the server and push the difference to user agents. If max-age is missing or set to 0, Streamdata.io will not poll the resource.

By default, the previously cached response continues to be sent to the requester during the time needed to refresh the cache. To prevent such a behavior the response must include a revalidation directive.
Cache-Control: must-revalidate | proxy-revalidate

The above directive forces Streamdata.io to wait for the server response before replying to the user agent even if a cached version is available.

Getting started

To quickly get started with Streamdata.io, follow our Getting Started Guide, or have a look at our sample projects on GitHub