{"id":13446425,"url":"https://github.com/harshulj/tweetlake","last_synced_at":"2025-03-21T06:31:21.886Z","repository":{"id":142649989,"uuid":"44400300","full_name":"harshulj/tweetlake","owner":"harshulj","description":"Tweet Lake is a commandline interface to Twitter Streaming API and big data project that extracts interesting stats out of tweet corpus.","archived":false,"fork":false,"pushed_at":"2022-07-01T22:02:55.000Z","size":19,"stargazers_count":20,"open_issues_count":3,"forks_count":0,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-03-05T02:36:07.035Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://blog.harshulja.in/tweetlake-twitter-streaming-data/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/harshulj.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2015-10-16T17:26:22.000Z","updated_at":"2021-10-22T09:18:23.000Z","dependencies_parsed_at":"2024-01-18T17:08:38.920Z","dependency_job_id":null,"html_url":"https://github.com/harshulj/tweetlake","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/harshulj%2Ftweetlake","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/harshulj%2Ftweetlake/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/harshulj%2Ftweetlake/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/harshulj%2Ftweetlake/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/harshulj","download_url":"https://codeload.github.com/harshulj/tweetlake/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244750744,"owners_count":20504133,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-07-31T05:00:53.029Z","updated_at":"2025-03-21T06:31:21.563Z","avatar_url":"https://github.com/harshulj.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"# TweetLake\nTweet Lake is a commandline interface to Twitter Streaming API and big data project that extracts interesting stats out of tweet corpus.\n\nTwitter API is a gold mine for big data lovers. People can do all sort of analysis and find some cool stats using twitter data. Whether it is sentiment analysis on products/topics or demographics. All this is possible on twitter data. I often like playing with machine learning algorithms. And one of the challenges that I face is the availability of data set. Most of the times I am browsing the internet and searching for data. Sometimes I find what I need and sometimes I have to create my own data set. After struggling a lot, I thought may be I can create a flexible tool that can do the job of data collection. So I started building [TweetLake](https://github.com/harshulj/tweetlake). \n\n**TweetLake (v 0.1.dev1)** is a modular command line tool that makes twitter data collection easier. This is written in Python and makes use of [Tweepy](http://tweepy.org/). It makes the job of data collection simpler. \n\n##### Installation\nYou can install TweetLake via pip, but this is not advised as TweetLake is currently in active development. This not very flexible and more over you would want to change the source code as per your convenience.\n```\n# Installing via pip\n$ [sudo] pip install tweetlake\n```\nThe **best way** is to clone the repo.  You can download or clone it from [Github (https://github.com/harshulj/tweetlake)]( https://github.com/harshulj/tweetlake/). Once cloned run the following commands.\n```\n$ git clone https://github.com/harshulj/tweetlake.git\n$ cd tweetlake\n$ virtualenv env\n$ source env/bin/activate\n$ python setup.py develop\n```\nThese commands basically creates a virtual environment and install TweetLake within that environment. This is useful because you can directly edit the source and modify the commands as per your convenience.\n\n##### Usage\nUsing TweetLake is pretty easy. First step is to create `data/` folder in project root.\nSecondly, update `config.py`. Enter your config in tweetlake config\n```\n$ tl --help\nUsage: tl [OPTIONS] COMMAND [ARGS]...\n\nOptions:\n  --help  Show this message and exit.\n\nCommands:\n  feature  This command takes raw tweets as input and...\n  filter   This command either filter tweets based on...\n  sample   This command opens a stream to twitter and...\n  stats    Use this command to filter on the tweets and...\n```\n##### Commands\nCurrently there are 4 commands as mentioned above. 3 of them can be used to collect twitter data.\n\n**First one, *filter*** is for collecting tweets based on keywords.\n```\n$ tl filter --help\nUsage: tl filter [OPTIONS] TWEET_TYPE FILE\n\n  This command either filter tweets based on happy or sad sentiments. OR You\n  can also pass the keywords to filter the tweets. User tweet type as\n  'custom' to support keywords Either of the 2 will work.\n\nOptions:\n  --keywords TEXT  Keywords to filter the tweets as CSV\n  --place TEXT     Place to filter the public tweets. Make sure this list is\n                   less than 25.\n  --follow TEXT    List of user ids to filter the public tweets from. Make\n                   sure this list is less than 5000 in CSV format\n  --help           Show this message and exit.\n```\n\n**Second *sample***, is for collecting sample tweets.\n```\n$ tl  sample --help\nStarting Tweet Lake\n\nUsage: tl sample [OPTIONS]\n\n  This command opens a stream to twitter and fetches all the sample tweets.\n\nOptions:\n  --help  Show this message and exit.\n```\n**Third one *stats***, prints out number of total tweets collected also the aggregate number of tweets based on language.\n```\n$ tl stats --help\nStarting Tweet Lake\n\nUsage: tl stats [OPTIONS] FILE\n\n  Use this command to filter on the tweets and get the aggregate numbers.\n\nOptions:\n  --lang TEXT  Aggregate based on the language of the tweet.\n  --help       Show this message and exit.\n\n```\n##### Data Format\nCurrently TweetLake supports file for storing tweets. Each tweet is saved as json in a file. This file can be read and processed as per the requirement.\n\n##### Supervisord Support\nTweetLake has support for running the commands using supervosord. Currently there are 3 commands that are there in supervisord.conf by default. You can see `supervisor.conf` for more details.\n\nUsing supervisor you can collect Twitter's sample data as we ll as you can collect tweets classified as *happy* and *sad*. \n\n##### Road Ahead\nThis is just a beginning for TweetLake. I want to convert TweetLake into a big data framework for twitter data. A framework like [scrapy](scrapy.org), which can be extended with easy. Currently I am adding support for Support vector Machine and Naive Bayes Classifier. I'll also add some example commands and algorithms. \n\nIf you have any suggestions or comments please open a ticket. I would love to get a feedback on what features to add.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fharshulj%2Ftweetlake","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fharshulj%2Ftweetlake","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fharshulj%2Ftweetlake/lists"}