I was playing with streaming different APIs, and I found myself playing with the Twitter API
. Which is kind of senseless as Twitter has their own streaming API,
but it can be a little more investment to get up and running with both technically, and in a businesses sense if you are needing higher volumes, than Streamdata.io. Technically Twitter is just using a long-lived HTTP request, establishing a persistent HTTP request
, and making data available in a regular stream of HTTP responses. I've integrated with the Twitter Streaming API before, but I wanted to understand the technical details of how its different than using Streamdata.io, and HTTP Server-Sent Events (SSE) and JSON Patch.
To stream a Twitter list using Streamdata.io
all I did was use the URL for the Twitter lists statuses API https://api.twitter.com/1.1/lists/statuses.json, with two query parameters.
Then you just make sure and include in a Authorization: Bearer [token]
with each request. At first I passed the header in with call I made to Streamdata.io.
curl -v "https://streamdata.motwin.net/https://api.twitter.com/1.1/lists/statuses.json?slug=api&owner_screen_name=kinlane&X-Sd-Token=[streamdata api key]" -H "Authorization: Bearer [token]"
However, once I saw that it worked, I added the authorization header to my Streamdata.io settings so I can keep it out of the code, simplifying each request I am making to Streamdata.io.
curl -v "https://streamdata.motwin.net/https://api.twitter.com/1.1/lists/statuses.json?slug=api&owner_screen_name=kinlane&X-Sd-Token=[streamdata api key]"
The result is a pretty efficient stream of Tweets from any accounts on the Twitter list I maintain for the API industry. At first I get the entire response from the API, but then I just get an incremental JSON Patch update with any additional Tweets. Providing a pretty simple stream of Tweets without the overhead of using the Twitter Streaming API. I just need to adjust the polling frequency in my Streamdata.io settings to make sure I am not going to go over my Twitter rate limits, then I have myself a pretty robust streaming and cached source of my Twitter List.
Now, why would I want to do this? I'm not 100%, which is why I'm writing this story. First, I guess it is quicker and easier to implement than the full Twitter Streaming API. Second, it provides a nice cached endpoint that you could reuse across applications, systems, and teams I guess, making efficient use of your Twitter API rate limits. Beyond that, I guess if you have serious volume needs for Twitter data, you probably would want to go with their solution--as you get volume rate limits. I'd say Twitter's Streaming API isn't as advanced as using Server-Sent Events (SSE) and JSON Patch--using a persistent connection in HTTP/1.1 just isn't as efficient as HTTP/2, or Streamdata.io's approach.
Regardless, my approach provides an interesting look at how you can stream using a pretty well known API resource--Twitter. I'm going to mimic my Tweetdeck client using just Twitter lists, and try to bypass the Twitter algorithm a little bit. I'm kind of over the full stream of my Twitter account(s), and prefer to rely upon the Twitter lists I've created. I'm also kind of over Tweetdeck as a client, and would like to see how I can create a bare bones client using Streamdata.io, and take charge over my social streams a little more. Then I am going to try streaming some other Twitter endpoints and see what is possible there, maybe taking streaming via the Twitter platform in a different direction that where the Twitter Streaming API goes.