{"id":24145224,"url":"https://github.com/caesariodito/bangkit-dashboard-scraper","last_synced_at":"2025-08-08T15:51:38.739Z","repository":{"id":232961220,"uuid":"785545843","full_name":"caesariodito/bangkit-dashboard-scraper","owner":"caesariodito","description":"A side project to scrape bangkit dashboard website to make my life better; am also trying out puppeteer;","archived":false,"fork":false,"pushed_at":"2024-04-28T04:42:01.000Z","size":1392,"stargazers_count":0,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-06-22T06:36:45.396Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/caesariodito.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2024-04-12T05:21:49.000Z","updated_at":"2024-04-28T04:42:04.000Z","dependencies_parsed_at":"2024-04-15T08:55:15.715Z","dependency_job_id":null,"html_url":"https://github.com/caesariodito/bangkit-dashboard-scraper","commit_stats":null,"previous_names":["caesariodito/bangkit-dashboard-scraper"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/caesariodito/bangkit-dashboard-scraper","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/caesariodito%2Fbangkit-dashboard-scraper","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/caesariodito%2Fbangkit-dashboard-scraper/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/caesariodito%2Fbangkit-dashboard-scraper/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/caesariodito%2Fbangkit-dashboard-scraper/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/caesariodito","download_url":"https://codeload.github.com/caesariodito/bangkit-dashboard-scraper/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/caesariodito%2Fbangkit-dashboard-scraper/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":269447899,"owners_count":24418754,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-08T02:00:09.200Z","response_time":72,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-01-12T06:50:11.660Z","updated_at":"2025-08-08T15:51:38.673Z","avatar_url":"https://github.com/caesariodito.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Bangkit Dashboard Scraper\n\nA side project to scrape bangkit dashboard website to make my life less burdening. kekw.\n\nUsed typescript, bun, puppeteer and exceljs.\n\nBelow is a brief intro and the reasons for the tech stack that I chose. **_It's just some personal notes, you don't need to read it tbh._**\n\nSooo, it's been a busy time for me doing mentoring on bangkit along with other stuff. Unfortunately, I wouldn't be able to keep up with the spreadsheet tracker that I made and shared to my mentees. It's pretty burdening for me to view all the data one by one and make sure everything is up-to-date weekly or even daily – it's just not possible for me.\n\nThen this project came through my mind, I just want to automate things that I can automate and also help my mentees with the tracker (since I share it with them). At the time I developed this side project, I had been neglecting the spreadsheet tracker for almost a month 💀. So yes, I think this is the right move for me, and fortunately, I got some small spare time to develop this project at Eid holiday.\n\nIn this side project, I wanted to try new things, hence the stack was created. It was an _alien_ stack for me. **I had never gotten the time to use bun** _(well I once tried it to run laravel breeze project and everything went well so I chose to use it again, since it was fast)_ and **had never used puppetee**r – heck, even I don't know the syntax.\n\nI used to scrape things with python selenium, and moving to puppeteer was pretty uncomfortable for me. But hey, it was an interesting experience for me. I also feel like puppeteer is much more faster compared to selenium _(pleb thoughts, since I haven't dug deep into selenium that much)_. Puppeteer feels like a adult teenager, while selenium feels like an adult LMAOAO.\n\n**Anyways, a small disclaimer:** I used some of AI-generated code to develop this project since I don't know the syntax and mechanism of puppeteer that much. But ofc I read the puppeteer docs, but they were also vague for me, so I need further assistance. **Pardon me if the code is ugly as hell**. I only spent ~a day developing this.\n\nIf you want to contribute to this side project as the demands of the spreadsheet tracker grow, feel free to open a PR! But make sure you document it well, tho currently I don't have any guidelines for it.\n\n## The Demo\n\n### Scraper Video\n\n[![Drive Video](/docs/thumbnail.png)](https://drive.google.com/file/d/1Vm6LOc4BTR1BNAmyknrGwFYgjky16HK2/preview)\n\n### Convert Video\n\n[![Drive Video](/docs/thumbnail_2.png)](https://drive.google.com/file/d/1xyuJiBIRteDHicauCl6sDtzlNWUvJ_jI/preview)\n\n## How to Run\n\nThis is a step-by-step guide to run this project. Even tho I documented how to use this project starting from the empty folder (via the demo video), but anyways having a text documentation would be convenient too.\n\n### Steps\n\n1. clone the project and cd into it\n2. install the package via\n\n   ```bash\n   npm install\n   # or\n   pnpm install\n   # or\n   ...\n   bun install # \u003c-- I use this code, but everything should be working as expected too\n   ```\n\n3. run the script via\n\n   ```bash\n   bun run start\n   # or\n   npm run start\n   ```\n\n   you can always view or edit the script in the [`package.json`](package.json) file.\n\n4. after you run it, it will open the login page, and asked you to login **manually** via google. you can just type your email and password as usual.\n\n5. after u finish logging in, make sure you wait the page to fully load. it will show the list of available mentees in your class. if everything's good you need to press **Enter** on the terminal to continue the script.\n\n6. it will scrape some of the important data related to the spreadsheet tracker and after it finishes, it will save the output to [`profile_data.json`](example_profile_data.json) file.\n\n7. you are then free to tweak and preprocess the data further. for example, importing the .json file directly to spreadsheet.\n\n8. Optionally, I created a script to convert the JSON file into XLSX (Excel), you can access it via `bun run convert` or if you want to modify the script, it is available on the [`convert.ts`](convert.ts) and [`package.json`](package.json) file. The result of the script can be viewed in [`example_profile.xlsx`](example_profile.xlsx)\n\n## Future Improvements\n\n- [x] export to excel/csv (soon)\n- [ ] integrate with spreadsheet directly via API\n- [ ] bypass login with indexedDB (idk really know much about this)\n- [ ] create a weekly cron-job to fully automate the tasks\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcaesariodito%2Fbangkit-dashboard-scraper","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcaesariodito%2Fbangkit-dashboard-scraper","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcaesariodito%2Fbangkit-dashboard-scraper/lists"}