{"id":16407576,"url":"https://github.com/musabbozkurt/data-collection-framework","last_synced_at":"2026-01-24T21:17:49.810Z","repository":{"id":176736978,"uuid":"63420029","full_name":"musabbozkurt/data-collection-framework","owner":"musabbozkurt","description":"Twitter Data Collection Framework","archived":false,"fork":false,"pushed_at":"2024-10-05T08:19:08.000Z","size":807,"stargazers_count":1,"open_issues_count":0,"forks_count":1,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-04-22T04:19:20.656Z","etag":null,"topics":["docker","docker-compose","mariadb","mongodb","php","phpmyadmin","python3","twitter-api"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/musabbozkurt.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2016-07-15T12:26:33.000Z","updated_at":"2024-10-05T08:19:12.000Z","dependencies_parsed_at":"2024-11-09T21:44:27.165Z","dependency_job_id":"83975467-c36d-4f31-9a5f-82712f0b4315","html_url":"https://github.com/musabbozkurt/data-collection-framework","commit_stats":null,"previous_names":["musabbozkurt/data-collection-framework"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/musabbozkurt/data-collection-framework","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/musabbozkurt%2Fdata-collection-framework","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/musabbozkurt%2Fdata-collection-framework/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/musabbozkurt%2Fdata-collection-framework/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/musabbozkurt%2Fdata-collection-framework/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/musabbozkurt","download_url":"https://codeload.github.com/musabbozkurt/data-collection-framework/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/musabbozkurt%2Fdata-collection-framework/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28737069,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-24T19:23:36.361Z","status":"ssl_error","status_checked_at":"2026-01-24T19:23:28.966Z","response_time":89,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["docker","docker-compose","mariadb","mongodb","php","phpmyadmin","python3","twitter-api"],"created_at":"2024-10-11T06:14:17.099Z","updated_at":"2026-01-24T21:17:49.805Z","avatar_url":"https://github.com/musabbozkurt.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"### Prerequisites\n\n- `Python 3.8+` should be installed\n- `Docker` should be installed\n- `Pycharm IDE, IntelliJ IDEA, etc.` can be installed to open the project (Recommended)\n- Update the following `Twitter API` credentials with your configs (DON'T SHARE THEM)\n  in [ConfigFile.config](data_collection_framework/config/ConfigFile.config) file\n    * `consumer_key` =\n    * `consumer_secret` =\n    * `access_key` =\n    * `access_secret` =\n\n---\n\n### How to Run the Application\n\n1. Run via `Terminal` (OPTION 1)\n    1. Run `docker-compose up --build` command\n    2. `cd data_collection_framework/`\n    3. `python3 -m pip install -r requirements.txt`\n    4. `chmod +x Server.py`\n    5. `./Server.py`\n2. Run via `Pycharm IDE, IntelliJ IDEA, etc.` (OPTION 2)\n    1. Run `docker-compose up --build` command via `Terminal`\n    2. Right-click on the [Server.py](data_collection_framework/Server.py) and click `Run`\n\n---\n\n### How to Test the Application\n\n1. http://localhost:3000/\n2. Sign up\n3. Sign in\n\n---\n\n### [docker-compose.yml](docker-compose.yml) contains the followings\n\n- MongoDB\n    - `url`: http://localhost:8081/ or http://0.0.0.0:8081/\n    - `username`: `mexpress`\n    - `password`: `mexpress`\n- MariaDB\n    - `MARIADB_DATABASE`: `root_db`\n    - `MARIADB_USER`: `user`\n    - `MARIADB_PASSWORD`: `user`\n- phpMyAdmin\n    - http://localhost:8080/\n- php-app\n    - http://localhost:3000/\n    - Sign up\n    - Sign in\n\n---\n\n### Other variables in [ConfigFile.config](data_collection_framework/config/ConfigFile.config) file can be reviewed\n\n- `word_list_for_streaming` = `['python', 'java', '#java', 'javascript']` modify the list according to the needs\n- `file_path_for_timeline_outputs` = Path for timeline i.e. user path/timeline/\n- `file_path_for_followers_outputs` = Path for followers i.e user path/followers/\n- `file_path_for_list_of_username` = Path for userList file can be text file contains Twitter usernames\n- `file_path_for_mongo` = filepath/of/text/files/to/store/mongodb/*.txt\n- `file_path_for_tokenization` = File path contains text files\n- `file_path_for_cross_val` = File path must have at least 2 folders contains text files\n\n---\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmusabbozkurt%2Fdata-collection-framework","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmusabbozkurt%2Fdata-collection-framework","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmusabbozkurt%2Fdata-collection-framework/lists"}