{"id":23072510,"url":"https://github.com/doches/ego-twitter","last_synced_at":"2025-04-03T10:43:49.428Z","repository":{"id":66510554,"uuid":"198380347","full_name":"doches/ego-twitter","owner":"doches","description":"A place to experiment with Kafka, Stanford CoreNLP, and Twitter","archived":false,"fork":false,"pushed_at":"2019-07-23T17:37:51.000Z","size":1033,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-02-09T00:17:51.350Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/doches.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-07-23T07:51:23.000Z","updated_at":"2019-07-23T17:37:52.000Z","dependencies_parsed_at":null,"dependency_job_id":"46eede14-f2b5-414f-b555-960693cc1435","html_url":"https://github.com/doches/ego-twitter","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/doches%2Fego-twitter","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/doches%2Fego-twitter/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/doches%2Fego-twitter/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/doches%2Fego-twitter/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/doches","download_url":"https://codeload.github.com/doches/ego-twitter/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246989512,"owners_count":20865305,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-12-16T07:20:00.459Z","updated_at":"2025-04-03T10:43:49.387Z","avatar_url":"https://github.com/doches.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Twitter Pipeline\n\n`ego-twitter` is just a place for me to experiment with building high-throughput pipelines for natural language processing. \n\n# Getting Started\n\n#### 1. Kafka\n\nFirst, you'll need to start up a local Kafka + Zookeeper cluster for local dev\n\n    ./scripts/start.sh\n\nYou can stop this cluster at any time using:\n\n    ./scripts/stop.sh\n\n#### 2. Twitter\n\nIn order to obtain data to process, you'll probably want to run the twitter producer. \nTo do that, first fill in your Twitter API credentials in `pipeline/twitter-producer/src/dist/config.yml` (start with the template in `pipeline/twitter-producer/src/dist/config.yml.template`)\n\nThe producer runs forever until manually killed. You can run it from Intellij for local dev, or from the cli with `./gradlew twitter-producer:run`. While running, it will periodically poll twitter for search results based on the configured keywords and write the resulting raw tweets into Kafka for processing.\n\n#### 3. Entity \u0026 Sentiment Extraction\n\nAs a proof-of-concept, this pipeline implements a consumer that reads tweets from Kafka and runs them through a multi-stage NLP pipeline built with the Stanford CoreNLP libraries. This consumer extracts named entities and annotates each with an estimated sentiment rating (-1 for most negative, +1 for most positive) based on the sentiment of the surrounding sentence.\n\nNote that this processing assumes the text is in English, for now.\n\nYou can run the consumer via Intellij for local dev, or from the CLI with `./gradlew twitter-consumer:run`. It will poll Kafka for incoming tweets, batch-process them, and write out a running count of the mean sentiment for each found named entity into (by default) a `results.txt` file.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdoches%2Fego-twitter","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdoches%2Fego-twitter","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdoches%2Fego-twitter/lists"}