Consuming the streaming API with PyCurl
The easiest way to handle HTTP streaming in Python is to use PyCurl, the Python bindings for the well-known Curl network library. PyCurl allows you to provide a callback function that will be executed every time a new block of data is available. The following code is a simple demonstration of HTTP streaming with PyCurl:
import pycurl, json
STREAM_URL = “http://chirpstream.twitter.com/2b/user.json”
USER = “segphault”
PASS = “XXXXXXXXX”
conn = pycurl.Curl()
conn.setopt(pycurl.USERPWD, “%s:%s” % (USER, PASS))
The code example above shows how to instantiate a Curl object, set the URL, provide login credentials, and send the data to a callback function. The callback function in the example simply echoes the received data to the terminal. If you put in your own Twitter credentials and run that code in a Python script at the command line, you will see the stream of JSON data transmitted by the Twitter service.
When the connection is idle and there is no other data to send, the streaming API will emit an empty line every 30 seconds. The empty line is a keep-alive signal that is intended to prevent client applications from timing out and dropping the connection. PyCurl doesn’t require any special configuration, but other network libraries might require the user to set a custom timeout duration for idle connections. You should make sure that it is set to something that is higher than the streaming API’s 30-second keep-alive interval so that the connection isn’t dropped.