elastictweetmap

A scalable tweet map using AWS Elastic Beanstalk and Elasticsearch

Components:

• Tweet stream: Stream tweets with geo-location in push-message fashion using Tweepy.
• AWS Elasticsearch: Persistent storage and index for tweets.
• Amazon SQS: Message queuing service that alerts webapp when a new tweet has beefo1und.
• Django backend: Fetches tweets in realtime as well as those stored persistently and displays them onto the front-end.
• Front-end: Maps API and Ajax calls.
• AWS Elastic Beanstalk: For load balancing.

AWS Elasticsearch instance: URL. Elasticsearch Python API is used to connect with the instance. Use the Sense plugin to query the Elasticsearch instance for debug. Link to JSON tutorial for querying Elasticsearch.

Screenshot of app:

Filtering and displaying live tweets:

The following are skimmed from the twitter stream and inserted into elastic search:

• id
• text
• twitter handle
• timestamp
• coordinates
• place

Example of JSON object stored in Elasticsearch as well as the message body of SQS notification:

	{
	   "id": 787531772296769500,
	   "handle": "martha1445",
	   "text": "I truly have the best family!! I love that you guys took the time… https://t.co/DjArPPiHdf",
	   "time": "1476597180805",
	   "coordinates": {
	      "type": "Point",
	      "coordinates": [
	         -117.87728023,
	         34.1181941
	      ]
	   },
	   "place": {
	      "country": "United States",
	      "bounding_box": {
	         "type": "Polygon",
	         "coordinates": [
	            [
	               [
	                  -117.890263,
	                  34.10549
	               ],
	               [
	                  -117.890263,
	                  34.165551
	               ],
	               [
	                  -117.809111,
	                  34.165551
	               ],
	               [
	                  -117.809111,
	                  34.10549
	               ]
	            ]
	         ]
	      },
	      "country_code": "US",
	      "attributes": {},
	      "place_type": "city",
	      "url": "https://api.twitter.com/1.1/geo/id/eb1bb64775708bc1.json",
	      "full_name": "Glendora, CA",
	      "id": "eb1bb64775708bc1",
	      "name": "Glendora"
	   }
    }

Python packages needed for installation (pip install packagename):

• tweepy
• elasticsearch
• boto
• requests_aws4auth
• django