{"id":49072423,"url":"https://github.com/garybake/twitstream","last_synced_at":"2026-04-20T08:06:51.469Z","repository":{"id":72035582,"uuid":"445328860","full_name":"garybake/twitstream","owner":"garybake","description":"realtime hashtag analytics","archived":false,"fork":false,"pushed_at":"2024-10-15T18:59:55.000Z","size":56,"stargazers_count":0,"open_issues_count":2,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2024-12-06T20:38:28.097Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/garybake.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-01-06T22:32:33.000Z","updated_at":"2024-07-15T20:30:45.000Z","dependencies_parsed_at":null,"dependency_job_id":"6f18eb89-fa7f-442c-89c3-ee340860c3ce","html_url":"https://github.com/garybake/twitstream","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/garybake/twitstream","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/garybake%2Ftwitstream","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/garybake%2Ftwitstream/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/garybake%2Ftwitstream/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/garybake%2Ftwitstream/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/garybake","download_url":"https://codeload.github.com/garybake/twitstream/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/garybake%2Ftwitstream/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32038482,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-20T00:18:06.643Z","status":"online","status_checked_at":"2026-04-20T02:00:06.527Z","response_time":94,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-04-20T08:06:49.670Z","updated_at":"2026-04-20T08:06:51.453Z","avatar_url":"https://github.com/garybake.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# TwitStream\nRealtime hashtag analytics\n\n### Setup\nGet api keys  \n`https://developer.twitter.com/en/portal/dashboard`\n\nCreate and add to .env file in project root folder\n```\nPROJECT_NAME = 'hashtag_aggregator'\n\nKAFKA_HOST_NAME = 'localhost'\nKAFKA_PORT = 9092\nKAFKA_TWEETS_TOPIC = 'tweets'\nKAFKA_HASHTAG_TOPIC = 'hashtagcount'\n\nSPARK_CHECKPOINT_DIR = 'xxxx'\n\nTWIT_ACCESS_TOKEN = 'xxxx'\nTWIT_ACCESS_SECRET = 'xxxx'\nTWIT_CONSUMER_KEY = 'xxxx'\nTWIT_CONSUMER_SECRET = 'xxxx'\n\nTWIT_STREAM_URL = 'https://stream.twitter.com/1.1/statuses/filter.json'\nTWIT_STREAM_LOCATION = '-2.768555,53.121229,-2.305756,53.353010'\n\nWEBSERVER_HOSTNAME = '0.0.0.0'\nWEBSERVER_PORT = 8000\n```\n### 1. Start kafka container\n`docker-compose up -d`\n\n### 2. Start twitter feed\n`python ./twitter_feed/twitter_app.py`\n\n### 3. Start spark stream app\n`python ./spark_stream/spark_stream_app.py`\n\n### 4. Start web server\n`uvicorn main:app --reload`\n\nServer on http://localhost:8000/  \n\nDocumentation on http://127.0.0.1:8000/docs#/  \nTODO: add the consumer message to the docs\n\n### Dev notes\n- You need a spark installation with the kafka jars installed (see todo)\n- There are 2 scripts in the dev directory for testing that feeds/consumers are working.\n\n### Notes links \nhttps://iwpnd.pw/articles/2020-03/apache-kafka-fastapi-geostream  \n\nListen for twitter stream  \n`telnet localhost 9009`\n\nFind a bounding box  \nhttp://bboxfinder.com/\n\n### TODO\n- Lots\n- Move feeder to aiokafka\n- Use spark docker container","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgarybake%2Ftwitstream","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgarybake%2Ftwitstream","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgarybake%2Ftwitstream/lists"}