Tag Archives: Streaming API

Couchbase : Create a large dataset using Twitter and Java


via Tug’s Blog: Couchbase : Create a large dataset using Twitter and Java.

An easy way to create large dataset when playing/demonstrating Couchbase -or any other NoSQL engine- is to inject Twitter feed into your database.

For this small application I am using:

In this example I am using Java to inject Tweets into Couchbase, you can obviously use another langage if you want to.

The sources of this project are available on my Github repository  Twitter Injector for Couchbase you can also download the Binary version here, and execute the application from the command line, see Run The Application paragraph. Do not forget to create your Twitter oAuth keys (see next paragraph)

DEBUGGING NODEJS MEMORY LEAKS


via Debugging Node.js memory leaks – The Flowdock Blog.

Node.js is a big part of the Flowdock infrastructure. It’s what powers, among other things, our push notifications and Streaming API, and has been pretty solid for both purposes.

About a month ago, we rolled out major changes to our online status tracking service. Very quickly, our public Streaming API started running out of memory and needed to be constantly restarted. Here’s a recap of what we learned while tracking down the bugs.

Catching Salesforce Outbound Messages with NodeJS


via Catching Salesforce Outbound Messages with NodeJS – topcoder.

When building applications that are integrated with Salesforce, one of the choices you have to make is howyou get data out of Salesforce and into your app. You can use one of the many ETL tools on the market, you can poll for changed records, use the Force.com Streaming API, use Apex HTTP Callouts from Salesforce or Outbound Messages.

I’ve covered most of these approaches in the past but have purposefully overlooked Outbound Messages. Why? Because, in general, I hate working with XML. Perhaps it harkens back to my SAP days but every time I have to traverse an XML structure, I think a kitten dies somewhere.

However, with that said, Outbound Messages are quite magical. You hook them up as an action to yourWorkflow Rule, so that whenever a record is, for example, created or updated in some manner, the platform will fire off some record data to the endpoint specified in the Outbound Message. My only issue with them is that they only support XML and most web languages like Ruby, JavaScript, Go, etc. prefer JSON (and so do I). So here’s a simple NodeJS app that will receive the XML from your Outbound Message and convert it into a JavaScript object that you can then use to do all sorts of awesome stuff!

*You can find all of the code for this application at my github repo. What follows is the interesting part that catches the XML and parses it in routes/obm.js.

Twitter API Client for node (REST & Streaming API)


README.md

twit

Twitter API Client for node

Supports both the REST and Streaming API.

Installing

npm install twit

Usage:

var Twit = require('twit')

var T = new Twit({
    consumer_key:         '...'
  , consumer_secret:      '...'
  , access_token:         '...'
  , access_token_secret:  '...'
})

//
//  tweet 'hello world!'
//
T.post('statuses/update', { status: 'hello world!' }, function(err, data, response) {
  console.log(data)
})

//
//  search twitter for all tweets containing the word 'banana' since Nov. 11, 2011
//
T.get('search/tweets', { q: 'banana since:2011-11-11', count: 100 }, function(err, data, response) {
  console.log(data)
})

//
//  get the list of user id's that follow @tolga_tezel
//
T.get('followers/ids', { screen_name: 'tolga_tezel' },  function (err, data, response) {
  console.log(data)
})

//
//  retweet a tweet with id '343360866131001345'
//
T.post('statuses/retweet/:id', { id: '343360866131001345' }, function (err, data, response) {
  console.log(data)
})

//
//  destroy a tweet with id '343360866131001345'
//
T.post('statuses/destroy/:id', { id: '343360866131001345' }, function (err, data, response) {
  console.log(data)
})

//
// get `funny` twitter users
//
T.get('users/suggestions/:slug', { slug: 'funny' }, function (err, data, response) {
  console.log(data)
})

//
//  stream a sample of public statuses
//
var stream = T.stream('statuses/sample')

stream.on('tweet', function (tweet) {
  console.log(tweet)
})

//
//  filter the twitter public stream by the word 'mango'.
//
var stream = T.stream('statuses/filter', { track: 'mango' })

stream.on('tweet', function (tweet) {
  console.log(tweet)
})

//
// filter the public stream by the latitude/longitude bounded box of San Francisco
//
var sanFrancisco = [ '-122.75', '36.8', '-121.75', '37.8' ]

var stream = T.stream('statuses/filter', { locations: sanFrancisco })

stream.on('tweet', function (tweet) {
  console.log(tweet)
})

//
// filter the public stream by english tweets containing `#apple`
//
var stream = T.stream('statuses/filter', { track: '#apple', language: 'en' })

stream.on('tweet', function (tweet) {
  console.log(tweet)
})

twit API:

T.get(path, [params], callback)

GET any of the REST API endpoints.

path

The endpoint to hit. When specifying path values, omit the ‘.json’ at the end (i.e. use ‘search/tweets’instead of ‘search/tweets.json’).

params

(Optional) parameters for the request.

callback

function (err, data, response)

  • data is the parsed data received from Twitter.
  • response is the http.IncomingMessage received from Twitter.

T.post(path, [params], callback)

POST any of the REST API endpoints. Same usage as T.get().

T.getAuth()

Get the client’s authentication tokens.

T.setAuth(tokens)

Update the client’s authentication tokens.

T.stream(path, [params])

Use this with the Streaming API.

path

Streaming endpoint to hit. One of:

  • ‘statuses/filter’
  • ‘statuses/sample’
  • ‘statuses/firehose’
  • ‘user’
  • ‘site’

For a description of each Streaming endpoint, see the Twitter API docs.

params

(Optional) parameters for the request. Any Arrays passed in params get converted to comma-separated strings, allowing you to do requests like:

//
// I only want to see tweets about my favorite fruits
//

// same result as doing { track: 'bananas,oranges,strawberries' }
var stream = T.stream('statuses/filter', { track: ['bananas', 'oranges', 'strawberries'] })

stream.on('tweet', function (tweet) {
  //...
})

Using the Streaming API

T.stream(path, [params]) keeps the connection alive, and returns an EventEmitter.

The following events are emitted:

event: ‘tweet’

Emitted each time a status (tweet) comes into the stream.

stream.on('tweet', function (tweet) {
  //...
})

event: ‘delete’

Emitted each time a status (tweet) deletion message comes into the stream.

stream.on('delete', function (deleteMessage) {
  //...
})

event: ‘limit’

Emitted each time a limitation message comes into the stream.

stream.on('limit', function (limitMessage) {
  //...
})

event: ‘scrub_geo’

Emitted each time a location deletion message comes into the stream.

stream.on('scrub_geo', function (scrubGeoMessage) {
  //...
})

event: ‘disconnect’

Emitted when a disconnect message comes from Twitter. This occurs if you have multiple streams connected to Twitter’s API. Upon receiving a disconnect message from Twitter, Twit will close the connection and emit this event with the message details received from twitter.

stream.on('disconnect', function (disconnectMessage) {
  //...
})

event: ‘connect’

Emitted when a connection attempt is made to Twitter. The http request object is emitted.

stream.on('connect', function (request) {
  //...
})

event: ‘connected’

Emitted when the response is received from Twitter. The http response object is emitted.

stream.on('connected', function (response) {
  //...
})

event: ‘reconnect’

Emitted when a reconnection attempt to Twitter is scheduled. If Twitter is having problems or we get rate limited, we schedule a reconnect according to Twitter’s reconnection guidelines. The last http requestand response objects are emitted, along with the time (in milliseconds) left before the reconnect occurs.

stream.on('reconnect', function (request, response, connectInterval) {
  //...
})

event: ‘warning’

This message is appropriate for clients using high-bandwidth connections, like the firehose. If your connection is falling behind, Twitter will queue messages for you, until your queue fills up, at which point they will disconnect you.

stream.on('warning', function (warning) {
  //...
})

event: ‘status_withheld’

Emitted when Twitter sends back a status_withheld message in the stream. This means that a tweet was withheld in certain countries.

stream.on('status_withheld', function (withheldMsg) {
  //...
})

event: ‘user_withheld’

Emitted when Twitter sends back a user_withheld message in the stream. This means that a Twitter user was withheld in certain countries.

stream.on('user_withheld', function (withheldMsg) {
  //...
})

event: ‘friends’

Emitted when Twitter sends the “friends” preamble when connecting to a user stream. This message contains a list of the user’s friends, represented as an array of user ids.

stream.on('friends', function (friendsMsg) {
  //...
})

event: ‘direct_message’

Emitted when a direct message is sent to the user. Unfortunately, Twitter has not documented this event for user streams.

stream.on('direct_message', function (directMsg) {
  //...
})

event: ‘user_event’

Emitted when Twitter sends back a User stream event. See the Twitter docs for more information on each event’s structure.

stream.on('user_event', function (eventMsg) {
  //...
})

In addition, the following user stream events are provided for you to listen on:

  • blocked
  • unblocked
  • favorite
  • unfavorite
  • follow
  • unfollow
  • user_update
  • list_created
  • list_destroyed
  • list_updated
  • list_member_added
  • list_member_removed
  • list_user_subscribed
  • list_user_unsubscribed
  • unknown_user_event (for an event that doesn’t match any of the above)

Example:

stream.on('favorite', function (event) {
  //...
})

event: ‘error’

Emitted when an API request or response error occurs. An Error object is emitted, with properties:

{
  message:      '...',  // error message
  statusCode:   '...',  // statusCode from Twitter
  code:         '...',  // error code from Twitter
  twitterReply: '...',  // raw response data from Twitter
  allErrors:    '...'   // array of errors returned from Twitter
}

stream.stop()

Call this function on the stream to stop streaming (closes the connection with Twitter).

stream.start()

Call this function to restart the stream after you called .stop() on it. Note: there is no need to call.start() to begin streaming. Twit.stream calls .start() for you.


What do I have access to?

Anything in the Twitter API:


Go here to create an app and get OAuth credentials (if you haven’t already):https://dev.twitter.com/apps/new

How do I run the tests?

Create two files: config1.js and config2.js at the root of the twit folder. They should contain two different sets of oauth credentials for twit to use (two accounts are needed for testing interactions). They should both look something like this:

module.exports = {
    consumer_key: '...'
  , consumer_secret: '...'
  , access_token: '...'
  , access_token_secret: '...'
}

Then run the tests:

npm test

You can also run the example:

node examples/rtd2.js

iRTD2

The example is a twitter bot named RTD2 written using twit. RTD2 tweets about github and curates its social graph.


License

(The MIT License)

Copyright (c) by Tolga Tezel tolgatezel11@gmail.com

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the “Software”), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

Changelog

1.1.14

  • Emit connected event upon receiving the response from twitter

1.0.0

  • now to stop and start the stream, use stream.stop() and stream.start() instead of emitting thestart and stop events
  • If twitter sends a disconnect message, closes the stream and emits disconnect with the disconnect message received from twitter

0.2.0

  • Updated twit for usage with v1.1 of the Twitter API.

0.1.5

  • BREAKING CHANGE to twit.stream(). Does not take a callback anymore. It returns immediately with the EventEmitter that you can listen on. The Usage section in the Readme.md has been updated. Read it.

0.1.4

  • twit.stream() has signature function (path, params, callback)