Skip to content

Elastic tweet map with real time tweet fetching

Latest
Latest
Compare
Choose a tag to compare
@litesaber15 litesaber15 released this 28 Oct 03:51
· 11 commits to master since this release
2
8c1833f

elastictweetmap

A scalable tweet map using AWS Elastic Beanstalk and Elasticsearch

Components:

  • Tweet stream: Stream tweets with geo-location in push-message fashion using Tweepy.
  • AWS Elasticsearch: Persistent storage and index for tweets.
  • Amazon SQS: Message queuing service that alerts webapp when a new tweet has been found.
  • Back-end: Fetches tweets in realtime as well as those stored persistently and displays them onto the front-end.
  • Front-end: Maps API.
  • AWS Elastic Beanstalk: For load balancing.

AWS Elasticsearch instance: URL

Elasticsearch Python API is used to connect with the instance. Use the Sense plugin to query the Elasticsearch instance for debug. Link to JSON tutorial for querying Elasticsearch.

The following are skimmed from the twitter stream and inserted into elastic search:

  • id
  • text
  • timestamp
  • coordinates
  • place

Example of JSON object stored in Elasticsearch as well as the message body of SQS notification:

    {
       "id": 787531772296769500,
       "text": "I truly have the best family!! I love that you guys took the time… https://t.co/DjArPPiHdf",
       "time": "1476597180805",
       "coordinates": {
          "type": "Point",
          "coordinates": [
             -117.87728023,
             34.1181941
          ]
       },
       "place": {
          "country": "United States",
          "bounding_box": {
             "type": "Polygon",
             "coordinates": [
                [
                   [
                      -117.890263,
                      34.10549
                   ],
                   [
                      -117.890263,
                      34.165551
                   ],
                   [
                      -117.809111,
                      34.165551
                   ],
                   [
                      -117.809111,
                      34.10549
                   ]
                ]
             ]
          },
          "country_code": "US",
          "attributes": {},
          "place_type": "city",
          "url": "https://api.twitter.com/1.1/geo/id/eb1bb64775708bc1.json",
          "full_name": "Glendora, CA",
          "id": "eb1bb64775708bc1",
          "name": "Glendora"
       }
    }

Python packages needed for installation (pip install packagename):

  • tweepy
  • elasticsearch
  • boto
Assets 2
Loading