https://github.com/3RiverDev/TwitterReporter

TwitterReporter: Breaking News Detection and Visualization through the Geo-Tagged Twitter Network
https://github.com/3RiverDev/TwitterReporter

Last synced: about 1 month ago
JSON representation

TwitterReporter: Breaking News Detection and Visualization through the Geo-Tagged Twitter Network

• Host: GitHub
• URL: https://github.com/3RiverDev/TwitterReporter
• Owner: 3RiverDev
• Created: 2012-07-08T15:50:41.000Z (almost 12 years ago)
• Default Branch: master
• Last Pushed: 2013-09-30T21:20:17.000Z (over 10 years ago)
• Last Synced: 2024-03-26T15:07:01.278Z (3 months ago)
• Language: Java
• Size: 13.8 MB
• Stars: 5
• Watchers: 3
• Forks: 2
• Open Issues: 0
• Metadata Files:
  • Readme: README.txt

Lists

• awesome-stars - TwitterReporter - Tagged Twitter Network | 3RiverDev | 5 | (Java)

README

        
TwitterReporter: Breaking News Detection and Visualization through the Geo-Tagged Twitter Network

Twitter provides a constant stream of concise data, useful within both geospatial and temporal domains.  This project attempts to accomplish a useful and interesting task: using live Twitter data to automatically identify breaking news events in near real-time.

Please see TwitterReporter.pdf for more details.

TwitterReporter was granted increased access to the location-based streaming filter Twitter API.  The removed rate-limiting effectively gives us all live, geotagged tweets in the continental US by filtering on a grid system.  Although this is important for data quality, the library can still be used on any account.

Currently, the bot processes and cleanses incoming data through the following steps:

1.) the tweet must be geotagged

2.) skip any accounts flagged with non-English languages.

3.) replace whitespace with single spaces (easier to parse)

4.) skip anything with non-printable ASCII characters

5.) remove URLs

6.) remove replies ("RT @foo" syntax)

7.) remove hashtags

8.) remove XHTML encoded characters

9.) remove non-alphanumeric characters

10.) remove stopwords

The stopwords are an aggregated list from 9 different sources, as well as a full geological database.  All words are compiled into a single stopwords/generated.txt file that is used by Lucene.

Eventually (work-in-progress), resulting, usable tweets are run through a simple document-frequency (DF) algorithm in geospatial chunks.  Other algorithms were experiemented with (IDF, etc.), but thrown out for various reasons (see the paper for explanations).  If a topic is found, it is stored along with the tweet that composed it.

The hope is that stored topics can be displayed on a geographic visualization.  Google Maps apps, etc. are envisioned.

The Java/XML system is built on the following:

- Twitter4J

- Hibernate ORM

- Apache Lucene

CONTRIBUTIONS ARE WELCOME!  Feel free to contact me with any questions!

Licensed under the GNU Lesser General Public License (LGPL) v3.0