{"id":23165656,"url":"https://github.com/maxpoi/oz-twitter-analysis","last_synced_at":"2026-05-09T04:32:36.840Z","repository":{"id":91030149,"uuid":"361694430","full_name":"maxpoi/Oz-Twitter-Analysis","owner":"maxpoi","description":"Assignment 2 for COMP90024, Cluster and Cloud Computing. ","archived":false,"fork":false,"pushed_at":"2021-05-26T01:47:48.000Z","size":4127,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-04T21:34:05.095Z","etag":null,"topics":["ansible","aurin","backend","cloud","couchdb","docker","docker-swarm","frontend","restful-api","twitter-api","visualization"],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/maxpoi.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-04-26T09:36:54.000Z","updated_at":"2021-05-26T01:47:51.000Z","dependencies_parsed_at":"2023-04-29T23:30:47.716Z","dependency_job_id":null,"html_url":"https://github.com/maxpoi/Oz-Twitter-Analysis","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/maxpoi/Oz-Twitter-Analysis","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/maxpoi%2FOz-Twitter-Analysis","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/maxpoi%2FOz-Twitter-Analysis/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/maxpoi%2FOz-Twitter-Analysis/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/maxpoi%2FOz-Twitter-Analysis/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/maxpoi","download_url":"https://codeload.github.com/maxpoi/Oz-Twitter-Analysis/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/maxpoi%2FOz-Twitter-Analysis/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32807203,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-08T08:22:46.396Z","status":"online","status_checked_at":"2026-05-09T02:00:06.633Z","response_time":123,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ansible","aurin","backend","cloud","couchdb","docker","docker-swarm","frontend","restful-api","twitter-api","visualization"],"created_at":"2024-12-18T01:28:14.122Z","updated_at":"2026-05-09T04:32:36.823Z","avatar_url":"https://github.com/maxpoi.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# COMP90024-Assignment-2\n\u003c!-- ALL-CONTRIBUTORS-BADGE:START - Do not remove or modify this section --\u003e\n[![All Contributors](https://img.shields.io/badge/all_contributors-5-orange.svg?style=flat-square)](#contributors-)\n\u003c!-- ALL-CONTRIBUTORS-BADGE:END --\u003e\n## Contributors ✨\n\n\u003c!-- ALL-CONTRIBUTORS-LIST:START - Do not remove or modify this section --\u003e\n\u003c!-- prettier-ignore-start --\u003e\n\u003c!-- markdownlint-disable --\u003e\n\u003ctable\u003e\n  \u003ctr\u003e\n    \u003ctd align=\"center\"\u003e\u003ca href=\"https://github.com/maxpoi\"\u003e\u003cimg src=\"https://avatars.githubusercontent.com/u/27168274?v=4?s=100\" width=\"100px;\" alt=\"\"/\u003e\u003cbr /\u003e\u003csub\u003e\u003cb\u003eJiacheng Ye\u003c/b\u003e\u003c/sub\u003e\u003c/a\u003e\u003cbr /\u003e\u003ca href=\"https://github.com/maxpoi/COMP90024-Assignment-2/commits?author=maxpoi\" title=\"Code\"\u003e💻\u003c/a\u003e\u003c/td\u003e\n    \u003ctd align=\"center\"\u003e\u003ca href=\"https://github.com/Ma-Yuyao\"\u003e\u003cimg src=\"https://avatars.githubusercontent.com/u/69780852?v=4?s=100\" width=\"100px;\" alt=\"\"/\u003e\u003cbr /\u003e\u003csub\u003e\u003cb\u003eMa-Yuyao\u003c/b\u003e\u003c/sub\u003e\u003c/a\u003e\u003cbr /\u003e\u003ca href=\"https://github.com/maxpoi/COMP90024-Assignment-2/commits?author=Ma-Yuyao\" title=\"Code\"\u003e💻\u003c/a\u003e\u003c/td\u003e\n    \u003ctd align=\"center\"\u003e\u003ca href=\"https://github.com/cc1032802\"\u003e\u003cimg src=\"https://avatars.githubusercontent.com/u/62432272?v=4?s=100\" width=\"100px;\" alt=\"\"/\u003e\u003cbr /\u003e\u003csub\u003e\u003cb\u003ecc1032802\u003c/b\u003e\u003c/sub\u003e\u003c/a\u003e\u003cbr /\u003e\u003ca href=\"https://github.com/maxpoi/COMP90024-Assignment-2/commits?author=cc1032802\" title=\"Code\"\u003e💻\u003c/a\u003e\u003c/td\u003e\n    \u003ctd align=\"center\"\u003e\u003ca href=\"https://github.com/jxstar11\"\u003e\u003cimg src=\"https://avatars.githubusercontent.com/u/73589480?v=4?s=100\" width=\"100px;\" alt=\"\"/\u003e\u003cbr /\u003e\u003csub\u003e\u003cb\u003ejxstar11\u003c/b\u003e\u003c/sub\u003e\u003c/a\u003e\u003cbr /\u003e\u003ca href=\"https://github.com/maxpoi/COMP90024-Assignment-2/commits?author=jxstar11\" title=\"Code\"\u003e💻\u003c/a\u003e\u003c/td\u003e\n    \u003ctd align=\"center\"\u003e\u003ca href=\"https://github.com/YUJGUAN\"\u003e\u003cimg src=\"https://avatars.githubusercontent.com/u/64241998?v=4?s=100\" width=\"100px;\" alt=\"\"/\u003e\u003cbr /\u003e\u003csub\u003e\u003cb\u003eYUJGUAN\u003c/b\u003e\u003c/sub\u003e\u003c/a\u003e\u003cbr /\u003e\u003ca href=\"https://github.com/maxpoi/COMP90024-Assignment-2/commits?author=YUJGUAN\" title=\"Code\"\u003e💻\u003c/a\u003e\u003c/td\u003e\n  \u003c/tr\u003e\n\u003c/table\u003e\n\n\u003c!-- markdownlint-restore --\u003e\n\u003c!-- prettier-ignore-end --\u003e\n\n\u003c!-- ALL-CONTRIBUTORS-LIST:END --\u003e\n\n## Project structure\n\n\u003e The ansible folder uses the Ansible playbook folder strucure. \n\u003e If a *templates* folder exists (like in *ansible/roles/deploy/couchdb*), then a J2 template is used to generate required files.\n\u003e J2 template is required because some files need to use the Ansible inventory variables.\n\n\n```\n.\n├── ansible                              # The folder of all ansible scripts (for setting up \u0026 deploy server)\n│   ├── roles                            # The ansible roles folder, listing all the tasks\n│   │   ├── deploy                       # where all the deploy tasks are listed \n│   │   │   └── couchdb                  #\n│   │   │   │   ├── tasks                # \n│   │   │   │   │   └── main.yaml        # \n│   │   │   │   └── templates            #\n│   │   │   │       └── xxx.xxx.j2       # \n|   |   |   ├── app                      # \n|   |   |   |   └── ...                  #\n|   |   |   ├── copy-directory           #\n|   |   |   |   └── ...                  #\n|   |   |   ├── harvester                #\n|   |   |   |   └── ...                  #\n│   │   ├── openstack                    # where all the setting up MRC tasks are listed \n│   │   │   ├── ...                      # ⬆\n│   │   │   └── remove                   # where all the uninstall server tasks are listed\n│   │   │   │   ├── ...                  # ⬆\n│   │   │   │   └── ...                  # ⬆\n│   │   └── set-up                       # where all the setting up each individual instance server tasks are listed\n│   │   │   ├── ...                      # ⬆\n│   │   │   └── ...                      # ⬆\n│   ├── vars                             # A folder listing all used Ansible environment variables\n│   ├── hosts                            # A customized Ansible inventory file; passed into playbook by using -i command\n│   ├── main.yaml                        # The main Ansible playbook file. It uses all the roles except the ones in the remove folder\n│   └── uninstall_server.yaml            # If this playbook is run, all MRC instances, security groups, volumes will be removed\n├── app                                  # The folder for the actual application\n│   ├── backend                          # The folder containing all back-end codes\n│   │   ├── api                          # The folder containing all api provided to the front-end\n|   |   |   ├── get_aurin.py             # Define \u0026 realize APIs for getting AURIN data\n|   |   |   └── get_map_reduce_result.py # Define \u0026 realize APIs for getting map_reduce data\n│   │   ├── crawler                      # The folder containing Twitter Harvester codes\n|   |   |   ├── crawl_by_keyword         # The folder containing Twitter Harvester codes for crawling by keyword\n|   |   |   |   ├── 5G                   # The folder containing Twitter Harvester codes for crawling for 5G scenario\n|   |   |   |   |   ├── ...              # ⬆\n|   |   |   |   |   └── ...              # ⬆\n|   |   |   |   ├── AFL                  # The folder containing Twitter Harvester codes for crawling for AFL scenario\n|   |   |   |   |   ├── ...              # ⬆ \n|   |   |   |   |   └── ...              # ⬆\n|   |   |   |   ├── food                 # The folder containing Twitter Harvester codes for crawling for food scenario\n|   |   |   |   |   ├── ...              # ⬆\n|   |   |   |   |   └── ...              # ⬆\n|   |   |   |   └── vaccine              # The folder containing Twitter Harvester codes for crawling for vaccine scenario\n|   |   |   |   |   ├── ...              # ⬆\n|   |   |   |   |   └── ...              # ⬆\n|   |   |   ├── crawl_by_raw_data        # The folder containing Twitter Harvester codes for crawling any keywords\n|   |   |   |   ├── node_1               # The folder containing Twitter Harvester codes for crawling any keywords hosted on node_1\n|   |   |   |   |   ├── Dockerfile       # ⬆\n|   |   |   |   |   └── ...              # ⬆\n|   |   |   |   ├── node_2               # The folder containing Twitter Harvester codes for crawling any keywords hosted on node_2\n|   |   |   |   |   ├── Dockerfile       # ⬆\n|   |   |   |   |   └── ...              # ⬆\n|   |   |   |   └── node_3               # The folder containing Twitter Harvester codes for crawling any keywords hosted on node_3\n|   |   |   |   |   ├── Dockerfile       # ⬆\n|   |   |   |   |   └── ...              # ⬆\n|   |   |   └── twitter_api_config.py    # The file is to set Twitter API configuration information\n│   │   └── mapreduce                    # The folder containing CouchDB map_reduce codes\n|   |   |   └── map_reduce.py            # The file is to set map_reduce for CouchDB\n|   |   |── upload_data                  # The folder containing Uploading data to CouchDB codes\n|   |   |   |── AURIN                    # The folder containing AURIN data (.json) we needed for this project\n|   |   |   |   └── ...                  # ⬆\n|   |   |   |── AURIN-CSV                # The folder containing AURIN data (.csv) we needed for this project\n|   |   |   |   └── ...                  # ⬆\n|   |   |   |── Dockerfile               #\n|   |   |   |── input_data_from_files.py # The file is to input data into CouchDB\n|   |   |   └── requirements.txt         # \n|   |   |── utils                        # The folder containing until functions\n|   |   |   |── get_path.py              # The file is to get the path\n|   |   |   └── sentiment_analysis.py    # The file is to perform sentiment analysis\n|   |   |── couchdb_config.py            # The file is to set CouchDB configuration information\n|   |   |── requirements.txt             #\n|   |   |── run_node_1.sh                #\n|   |   |── run_node_2.sh                #\n|   |   |── run_node_3.sh                #\n|   |   └── run_upload_data.sh           #\n│   └── frontend                         # \n|   |   |── Dockerfile                   #\n|   |   └── ...                          #\n├── .all-contributorsec                  # Automate generated file by all-contributor plugin\n├── openrc.sh                            # An environment set up bash file; used in run.sh\n├── run_first.sh                         # The main shell script. Must be run at the very start\n├── run_last.sh                          # The main shell script. Must be run at the very end\n└── README.md\n```\n\n## How to run?\nBefore running the shell scripts, there are couple preparations must be done first.\n1.\tGo to your MRC dashboard, download the OpenRC file by clicking your profile icon and choose download.\n2.\tRename the downloaded file to “*openrc.sh*” and move it under the root project folder.\n3.\tNavigate to “*Key Pairs*” under “*Project – Compute*” and create a new key pair by clicking “Create Key Pair”. \n4.\tFill in key name and choose key type as “*SSH Key*”. \n5.\tSave the downloaded *.pem* file to a directory where you can easily navigate to. (Warning! **DO NOT** share this key pair with anyone or public it!!!)\n6.\tOpen file “*hosts*” under *./ansible/*, after “*ansible_ssh_private_key_file*” append the *absolute path* to this .pem file you just downloaded.\n7.\tOpen file “*mrc.yaml*” under *./ansible/vars/*, after field “*instance: key_name:*”, replace the old key name by the key name you just created.\n8.\tClick “User” on the top right and click “Setting”\n9.\tNavigate to “Reset Password”\n10.\tClick “Reset Password” and write down the new password somewhere safe. You will need to enter this password later when running the shell scripts.\n11.\t*[Optional]* Open “*mrc.yaml*” file again, and decrease the “*vol_size*” if there no 500 GB storage available in your project space.\n\nAfter that, you can run the shell scripts following the following instructions to run Ansible.\n1)\tcd to the project folder\n2)\tin your terminal, enter “*sh ./run_first.sh*” (or simply double-click the “*run_first.sh*” file)\n3)\tTerminal will ask you to enter password, which is the password from step 9 previously.\n4)\tCopy the smallest IP address. By smallest we need numerically smallest, for example, 172.0.0.0 \u003c 172.0.0.1\n5)\tThen open file “*couchdb_config.py*” under *./app/backend/*, replace all strings “172.xxx.xxx.xxx” with the smallest IP address you just copied. \n6)\t(Make sure you are still at the project root folder in your terminal) enter “*sh ./run_last.sh*” and enter the same password again.\n7)\tWhen this script finishes, app is up on the cloud and can be accessed by entering the smallest IP address in step 4 followed by “:8003”. For example, “172.168.123.43:8003”. Make sure you are using The University of Melbourne’s network as well.\n\nNote: Here assume the project space in MRC is empty. If not, uncomment the second line in “*run_first.sh*” first before running it. \n\n## Project specification\nLook at the specification pdf.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmaxpoi%2Foz-twitter-analysis","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmaxpoi%2Foz-twitter-analysis","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmaxpoi%2Foz-twitter-analysis/lists"}