{"id":21314671,"url":"https://github.com/rohit1kumar/internsync","last_synced_at":"2025-03-15T21:22:53.680Z","repository":{"id":204894939,"uuid":"711624439","full_name":"rohit1kumar/internsync","owner":"rohit1kumar","description":"A Python web scraper to scrape internships from Internshala , with the option to integrate Google Sheets and sync it with GitHub Actions","archived":false,"fork":false,"pushed_at":"2024-09-09T04:17:24.000Z","size":29,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-01-22T10:23:14.483Z","etag":null,"topics":["python","scraping","webscraping"],"latest_commit_sha":null,"homepage":"https://docs.google.com/spreadsheets/d/1vK_4_jWJ4H1HJJr_FWxPNavddi5AU-HLqc3UCRC9hnM/edit?usp=sharing","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/rohit1kumar.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-10-29T20:27:57.000Z","updated_at":"2025-01-07T03:39:46.000Z","dependencies_parsed_at":"2023-12-21T21:00:16.645Z","dependency_job_id":null,"html_url":"https://github.com/rohit1kumar/internsync","commit_stats":null,"previous_names":["rohit1kumar/internshala-scraper-py","rohit1kumar/internsync"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rohit1kumar%2Finternsync","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rohit1kumar%2Finternsync/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rohit1kumar%2Finternsync/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rohit1kumar%2Finternsync/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/rohit1kumar","download_url":"https://codeload.github.com/rohit1kumar/internsync/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243791136,"owners_count":20348423,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["python","scraping","webscraping"],"created_at":"2024-11-21T18:14:49.164Z","updated_at":"2025-03-15T21:22:53.657Z","avatar_url":"https://github.com/rohit1kumar.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# InternSync\nA Playwright based web scraper to scrape internships from [Internshala](https://internshala.com/), written in Python. Data is stored in a CSV file.\n\n## Disclaimer\n*This is for educational purposes only. I am not responsible for any misuse of this code.*\n\n## Features\n- Store data in a CSV file\n- Store data in a Google Sheet (optional)\n- Keeps the Google Sheet data synced using GitHub Actions (optional)\n\n## Prerequisites\nMake sure you have the following dependencies installed:\n\n- [Playwright](https://playwright.dev/python/docs/library/) library for Python \n- Python 3.x\n\n*you can install them using the following commands too:* \n```bash\npip install playwright \u0026\u0026 playwright install chromium\n```\n\n## Usage\n1. Clone the repository\n2. Install the dependencies using `pip install -r requirements.txt`\n3. Run the script using `python main.py`\n\nOptional steps (for Google Sheets mode only):\n1. Create a new Google Sheet\n2. Create a new project in [Google Cloud Platform](https://console.cloud.google.com/)\n3. Follow this guide for setting up the [Google Sheets API](https://docs.gspread.org/en/v5.12.0/oauth2.html#for-bots-using-service-account)\n4. Download the JSON file and add all the credentials to the `.env` file (refer to `.env.example`)\n5. Get the Google Sheet ID from the URL e.g `https://docs.google.com/spreadsheets/d/GOOGLE_SHEET_ID/edit`\n6. Add to `GOOGLE_SHEET_ID` to the `.env` file\n\nOptional steps (for syncing Google Sheets using GitHub Actions):\n1. GitHub Actions are already setup in the repository\n2. Download GitHub CLI or add secrets manually to the repository from `.env` file\n3. With GitHub CLI run `gh secret set -R \u003cyour-username/your-repo\u003e -f .env`\n\n## Options\n- `--headful`: Run the script in non-headless mode (show the browser)\n    ```bash\n    python main.py --headful\n    ```\n- `--gs`: Run the script in Google Sheets mode (store data in Google Sheets)\n    ```bash\n    python main.py --gs\n    ```\n[//]: # (- `--limit`: Limit the number of internships to scrape \u0026#40;default: 10\u0026#41;)\n\n[//]: # (- `--output`: Output file name \u0026#40;default: internships.csv\u0026#41; )\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frohit1kumar%2Finternsync","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frohit1kumar%2Finternsync","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frohit1kumar%2Finternsync/lists"}