{"id":19075794,"url":"https://github.com/lpsm-dev/twitter-realtime-processing-covid","last_synced_at":"2025-04-30T00:12:53.107Z","repository":{"id":37049449,"uuid":"271552515","full_name":"lpsm-dev/twitter-realtime-processing-covid","owner":"lpsm-dev","description":"✔️ Twitter Realtime Processing about Covid-19 + Docker + Docker  Compose + Apache Kafka + Elastic Stack (Elasticsearch + Kibana)","archived":false,"fork":false,"pushed_at":"2025-04-29T09:00:47.000Z","size":371,"stargazers_count":8,"open_issues_count":10,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-30T00:12:47.027Z","etag":null,"topics":["alpine","covid","covid-19","docker","docker-compose","elasticsearch","kafka","kafka-consumer","kafka-producer","kibana","python","streaming","twitter","twitter-api","twitter-streaming-api"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"wtfpl","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lpsm-dev.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2020-06-11T13:25:15.000Z","updated_at":"2025-03-04T15:11:03.000Z","dependencies_parsed_at":"2025-04-18T07:43:47.254Z","dependency_job_id":"b3f14b2c-8cb2-4a02-8d73-7ffbc5827153","html_url":"https://github.com/lpsm-dev/twitter-realtime-processing-covid","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lpsm-dev%2Ftwitter-realtime-processing-covid","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lpsm-dev%2Ftwitter-realtime-processing-covid/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lpsm-dev%2Ftwitter-realtime-processing-covid/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lpsm-dev%2Ftwitter-realtime-processing-covid/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lpsm-dev","download_url":"https://codeload.github.com/lpsm-dev/twitter-realtime-processing-covid/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":251602831,"owners_count":21615969,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["alpine","covid","covid-19","docker","docker-compose","elasticsearch","kafka","kafka-consumer","kafka-producer","kibana","python","streaming","twitter","twitter-api","twitter-streaming-api"],"created_at":"2024-11-09T01:56:01.631Z","updated_at":"2025-04-30T00:12:53.083Z","avatar_url":"https://github.com/lpsm-dev.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cp align=\"center\"\u003e\n  \u003cimg alt=\"python-kafka\" src=\"https://miro.medium.com/max/1400/1*SevEMhvGvosd6am1u0Qqqg.png\" width=\"250px\" float=\"center\"/\u003e\n\u003c/p\u003e\n\n\u003ch1 align=\"center\"\u003eWelcome to Twitter Realtime Processing Repository\u003c/h1\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003cstrong\u003ePython Realtime Processing Tweets COVID-19 using Kafka + Elasticsearch + Kibana + Docker + Docker-Compose\u003c/strong\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"https://github.com/lpmatos/twitter-realtime-processing-covid\"\u003e\n    \u003cimg alt=\"Open Source\" src=\"https://badges.frapsoft.com/os/v1/open-source.svg?v=102\"\u003e\n  \u003c/a\u003e\n\n  \u003ca href=\"https://github.com/lpmatos/twitter-realtime-processing-covid/graphs/contributors\"\u003e\n    \u003cimg alt=\"GitHub Contributors\" src=\"https://img.shields.io/github/contributors/lpmatos/twitter-realtime-processing-covid\"\u003e\n  \u003c/a\u003e\n\n  \u003ca href=\"https://github.com/lpmatos/twitter-realtime-processing-covid\"\u003e\n    \u003cimg alt=\"GitHub Language Count\" src=\"https://img.shields.io/github/languages/count/lpmatos/twitter-realtime-processing-covid\"\u003e\n  \u003c/a\u003e\n\n  \u003ca href=\"https://github.com/lpmatos/twitter-realtime-processing-covid\"\u003e\n    \u003cimg alt=\"GitHub Top Language\" src=\"https://img.shields.io/github/languages/top/lpmatos/twitter-realtime-processing-covid\"\u003e\n  \u003c/a\u003e\n\n  \u003ca href=\"https://github.com/lpmatos/twitter-realtime-processing-covid/stargazers\"\u003e\n    \u003cimg alt=\"GitHub Stars\" src=\"https://img.shields.io/github/stars/lpmatos/twitter-realtime-processing-covid?style=social\"\u003e\n  \u003c/a\u003e\n\n  \u003ca href=\"https://github.com/lpmatos/twitter-realtime-processing-covid/commits/master\"\u003e\n    \u003cimg alt=\"GitHub Last Commit\" src=\"https://img.shields.io/github/last-commit/lpmatos/twitter-realtime-processing-covid\"\u003e\n  \u003c/a\u003e\n\n  \u003ca href=\"https://github.com/lpmatos/twitter-realtime-processing-covid\"\u003e\n    \u003cimg alt=\"Repository Size\" src=\"https://img.shields.io/github/repo-size/lpmatos/twitter-realtime-processing-covid\"\u003e\n  \u003c/a\u003e\n\n  \u003ca href=\"https://github.com/lpmatos/twitter-realtime-processing-covid/issues\"\u003e\n    \u003cimg alt=\"Repository Issues\" src=\"https://img.shields.io/github/issues/lpmatos/twitter-realtime-processing-covid\"\u003e\n  \u003c/a\u003e\n\n  \u003ca href=\"https://github.com/lpmatos/twitter-realtime-processing-covid/blob/master/LICENSE\"\u003e\n    \u003cimg alt=\"MIT License\" src=\"https://img.shields.io/github/license/lpmatos/twitter-realtime-processing-covid\"\u003e\n  \u003c/a\u003e\n\u003c/p\u003e\n\n## ➤ Getting Started\n\nIf you want use this repository you need to make a **git clone**:\n\n```bash\ngit clone --depth 1 https://github.com/lpmatos/twitter-realtime-processing-covid.git -b master\n```\n\nThis will give access on your **local machine**.\n\n## ➤ Pre-Requisites\n\nTo this project you yeed:\n\n* Python 3.8.\n* Docker and Docker Compose.\n* Kafka ecosystem.\n* Elasticsearch.\n* Kiabana\n\n## ➤ How to use it?\n\n#### Locale\n\n1. Set the application environment variables.\n2. Install python packages in requirements.txt.\n3. Run docker-compose.yml to deploy all kafka and elastic ecosystem.\n4. Profit.\n\n#### Docker\n\n1. Set all environment variables in dot-env files.\n2. Creathe a docker network.\n3. Run docker-compose.yml to deploy all kafka and elastic ecosystem.\n4. Run docker-compose-tools.to run the application.\n5. Profit.\n\nThis system is fully containerised. You will need [Docker](https://docs.docker.com/install/) and [Docker Compose](https://docs.docker.com/compose/) to run it.\n\nYou simply need to create a Docker network called `kafka-network` to enable communication between the Kafka cluster and the apps:\n\n```bash\n$ docker network create kafka-network\n```\n\nAll set!\n\n## ➤ Description \u003ca name = \"description\"\u003e\u003c/a\u003e\n\n### Sending Data to Elasticsearch\n\n```bash\ncurl -X POST kafka-connect:8083/connectors -H \"Content-Type: application/json\" -d '{\n\t\"name\": \"elasticsearch-sink-kafka\",\n\t\"config\": {\n\t\t\"connector.class\": \"io.confluent.connect.elasticsearch.ElasticsearchSinkConnector\",\n\t\t\"type.name\": \"kafka-connect\",\n\t\t\"key.converter.schemas.enable\": \"false\",\n\t\t\"tasks.max\": \"1\",\n\t\t\"topics\": \"dados-tweets\",\n\t\t\"value.converter.schemas.enable\": \"false\",\n\t\t\"key.ignore\": \"true\",\n\t\t\"connection.url\": \" http://elasticsearch:9200\",\n\t\t\"value.converter\": \"org.apache.kafka.connect.json.JsonConverter\",\n\t\t\"key.converter\": \"org.apache.kafka.connect.storage.StringConverter\",\n\t\t\"schema.ignore\": \"true\"\n\t}\n}'\n```\n\n### Environment variables\n\n**Name**  |  **Description**\n:---:  |  :---:\n**TWITTER_CONSUMER_KEY**  |  Twitter Consumer Key\n**TWITTER_CONSUMER_SECRET**  |  Twitter Consumer Secret\n**TWITTER_ACCESS_TOKEN**  |  Twitter Access Token\n**TWITTER_ACCESS_TOKEN_SECRET**  |  Twitter Access Token Secret\n**LOG_PATH**  |  Just the Log Path\n**LOG_FILE**  |  Just the Log File\n**LOG_LEVEL**  |  Just the Log Level\n**LOGGER_NAME**  |  Just the Logger name\n**KAFKA_BROKER_URL**  |  Kafka Broker URL\n**KAFKA_TOPIC**  |  Kafka Topic Name\n\n### Environment file\n\nWe use decouple for strict separation of settings from code. It helps us with to store parameters in .env file and properly convert values to correct data type.\n\nCopy the file .env-example to a .env file and replace the values inside of it.\n\n## ➤ Usage \u003ca name = \"usage\"\u003e\u003c/a\u003e\n\nWays to run and use this project.\n\n\u003cdetails\u003e\u003csummary\u003eDocker\u003c/summary\u003e\n\u003cp\u003e\n\nSteps to build the Docker Image.\n\n#### Build\n\n```bash\ndocker image build -t \u003cIMAGE_NAME\u003e -f \u003cPATH_DOCKERFILE\u003e \u003cPATH_CONTEXT_DOCKERFILE\u003e\ndocker image build -t \u003cIMAGE_NAME\u003e . (This context)\n```\n\n#### Run\n\nSteps to run the Docker Container.\n\n* **Linux** running:\n\n```bash\ndocker container run -d -p \u003cLOCAL_PORT:CONTAINER_PORT\u003e \u003cIMAGE_NAME\u003e \u003cCOMMAND\u003e\ndocker container run -it --rm --name \u003cCONTAINER_NAME\u003e -p \u003cLOCAL_PORT:CONTAINER_PORT\u003e \u003cIMAGE_NAME\u003e \u003cCOMMAND\u003e\n```\n\n* **Windows** running:\n\n```\nwinpty docker.exe container run -it --rm \u003cIMAGE_NAME\u003e \u003cCOMMAND\u003e\n```\n\nFor more information, access the [Docker](https://docs.docker.com/) documentation or [this](docs/docker.md).\n\u003c/p\u003e\n\u003c/details\u003e\n\n\u003cdetails\u003e\u003csummary\u003eDocker-Compose\u003c/summary\u003e\n\u003cp\u003e\n\nBuild and run a docker-compose.\n\n```bash\ndocker-compose up --build\n```\n\nDown all services deployed by docker-compose.\n\n```bash\ndocker-compose down\n```\n\nDown all services and delete all images.\n\n```bash\ndocker-compose down --rmi all\n```\n\u003c/p\u003e\n\u003c/details\u003e\n\n## ➤ Visuals \u003ca name = \"visuals\"\u003e\u003c/a\u003e\n\nDepending on what you are making, it can be a good idea to include screenshots or even a video (you'll frequently see GIFs rather than actual videos). Tools like ttygif can help, but check out Asciinema for a more sophisticated method.\n\n## ➤ Author \u003ca name = \"author\"\u003e\u003c/a\u003e\n\n👤 **Lucca Pessoa**\n\nHey!! If you like this project or if you find some bugs feel free to contact me in my channels:\n\n\u003e\n\u003e * Email: lpsm-dev@protonmail.com\n\u003e * Website: https://github.com/lpmatos\n\u003e * GitHub: [@lpmatos](https://github.com/lpmatos)\n\u003e * GitLab: [@lpmatos](https://gitlab.com/lpmatos)\n\u003e\n\n## ➤ Versioning \u003ca name = \"versioning\"\u003e\u003c/a\u003e\n\nTo check the change history, please access the [**CHANGELOG.md**](CHANGELOG.md) file.\n\n## ➤ Troubleshooting \u003ca name = \"troubleshooting\"\u003e\u003c/a\u003e\n\nIf you have any problems, please contact [me](https://github.com/lpmatos).\n\n## ➤ Project status \u003ca name = \"project-status\"\u003e\u003c/a\u003e\n\nThis project is currently undergoing a reorganization 👾.\n\n## ➤ Show your support \u003ca name = \"show-your-support\"\u003e\u003c/a\u003e\n\n\u003cdiv align=\"center\"\u003e\n\nGive me a ⭐️ if this project helped you!\n\n\u003cp\u003e\n  \u003cimg alt=\"gif-header\" src=\"https://www.icegif.com/wp-content/uploads/baby-yoda-bye-bye-icegif.gif\" width=\"350px\" float=\"center\"/\u003e\n\u003c/p\u003e\n\nMade with 💜 by [me](https://github.com/lpmatos) :wave: inspired on [readme-md-generator](https://github.com/kefranabg/readme-md-generator)\n\n\u003c/div\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flpsm-dev%2Ftwitter-realtime-processing-covid","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flpsm-dev%2Ftwitter-realtime-processing-covid","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flpsm-dev%2Ftwitter-realtime-processing-covid/lists"}