{"id":25885156,"url":"https://github.com/blackvoidx/web-scraping","last_synced_at":"2025-03-02T17:28:59.082Z","repository":{"id":274626994,"uuid":"923526144","full_name":"blackvoidx/web-scraping","owner":"blackvoidx","description":"Some of my scraping tools and scripts","archived":false,"fork":false,"pushed_at":"2025-02-15T13:36:37.000Z","size":270,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-02-16T17:50:38.959Z","etag":null,"topics":["badges","medium","medium-article","medium-premium","tryhackme","web-scraper","web-scraping","web-scrapping","web-screenshot","web-screenshot-bot"],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/blackvoidx.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2025-01-28T12:06:40.000Z","updated_at":"2025-02-15T13:36:41.000Z","dependencies_parsed_at":"2025-01-28T13:34:48.685Z","dependency_job_id":"d30257d4-c394-4e7b-aaa3-4e883f11361e","html_url":"https://github.com/blackvoidx/web-scraping","commit_stats":null,"previous_names":["mamad-1999/web-scraping","blackvoidx/web-scraping"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/blackvoidx%2Fweb-scraping","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/blackvoidx%2Fweb-scraping/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/blackvoidx%2Fweb-scraping/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/blackvoidx%2Fweb-scraping/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/blackvoidx","download_url":"https://codeload.github.com/blackvoidx/web-scraping/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":241544012,"owners_count":19979641,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["badges","medium","medium-article","medium-premium","tryhackme","web-scraper","web-scraping","web-scrapping","web-screenshot","web-screenshot-bot"],"created_at":"2025-03-02T17:28:58.349Z","updated_at":"2025-03-02T17:28:59.069Z","avatar_url":"https://github.com/blackvoidx.png","language":"JavaScript","readme":"# TryHackMe Badge Screenshot Automation\n\nThis project automates the process of capturing a high-quality screenshot of a TryHackMe profile badge and updating it in a GitHub repository every 12 hours using GitHub Actions.\n\n## Preview\n\n![2025-02-02_16-15](https://github.com/user-attachments/assets/f295bff7-0a34-41f2-b016-cd732a5dd113)\n\n\n## Features\n- Uses Puppeteer to capture a high-resolution screenshot of the TryHackMe badge.\n- Automates the update process with a GitHub Action.\n- Runs every 12 hours via a scheduled cron job.\n\n## Prerequisites\nEnsure you have the following installed:\n- [Node.js](https://nodejs.org/) (Version 18 recommended)\n- [Puppeteer](https://pptr.dev/)\n- Dependencies required by Puppeteer:\n  ```sh\n  sudo apt-get install -y libnss3 libxss1 fonts-liberation libatk1.0-0 \\\n  libatk-bridge2.0-0 libcups2 libdrm2 libgbm1 libnspr4 libxcomposite1 \\\n  libxdamage1 libxrandr2 libxtst6 xdg-utils\n  ```\n\n## Installation \u0026 Setup\n\n1. Clone this repository:\n   ```sh\n   git clone https://github.com/yourusername/yourrepository.git\n   cd yourrepository\n   ```\n\n2. Install Node.js dependencies:\n   ```sh\n   npm install puppeteer\n   ```\n\n3. Create a new file named `generateBadge.js` and paste the following script:\n   ```javascript\n   const puppeteer = require('puppeteer');\n\n   (async () =\u003e {\n       try {\n           const browser = await puppeteer.launch({\n               args: ['--no-sandbox', '--disable-setuid-sandbox'],\n           });\n           const page = await browser.newPage();\n\n           await page.setViewport({\n               width: 1920,\n               height: 1080,\n               deviceScaleFactor: 2,\n           });\n\n           const url = 'https://tryhackme.com/api/v2/badges/public-profile?userPublicId=3153096';\n           await page.goto(url, { waitUntil: 'networkidle2' });\n\n           const badgeSelector = '#thm-badge';\n           await page.waitForSelector(badgeSelector);\n\n           await page.addStyleTag({\n               content: '#thm-badge { transform: scale(1.5); transform-origin: top left; }',\n           });\n\n           const badgeElement = await page.$(badgeSelector);\n           if (badgeElement) {\n               await badgeElement.screenshot({ path: 'tryhackme-badge-high-quality.png' });\n               console.log('High-quality screenshot saved as tryhackme-badge-high-quality.png');\n           } else {\n               console.log('Badge element not found!');\n           }\n\n           await browser.close();\n       } catch (error) {\n           console.error('Error generating screenshot:', error);\n       }\n   })();\n   ```\n\n4. Run the script manually to test it:\n   ```sh\n   node generateBadge.js\n   ```\n   This should generate a `tryhackme-badge-high-quality.png` file.\n\n5. Move the generated image to the `assets/` folder:\n   ```sh\n   mv tryhackme-badge-high-quality.png assets/\n   ```\n\n## Automating with GitHub Actions\n\nTo automate the badge update process, create a GitHub Action workflow:\n\n1. Inside your repository, navigate to `.github/workflows/` (create these folders if they don't exist).\n2. Create a new file named `update_badge.yml` and paste the following content:\n\n   ```yaml\n   name: Update TryHackMe Badge\n\n   on:\n     schedule:\n       - cron: \"0 */12 * * *\"  # Runs every 12 hours\n\n   jobs:\n     update-badge:\n       runs-on: ubuntu-latest\n\n       steps:\n         - name: Checkout repository\n           uses: actions/checkout@v3\n\n         - name: Set up Node.js\n           uses: actions/setup-node@v3\n           with:\n             node-version: \"18\"\n\n         - name: Install Puppeteer dependencies\n           run: sudo apt-get install -y libnss3 libxss1 fonts-liberation libatk1.0-0 libatk-bridge2.0-0 libcups2 libdrm2 libgbm1 libnspr4 libxcomposite1 libxdamage1 libxrandr2 libxtst6 xdg-utils\n\n         - name: Install Puppeteer\n           run: npm install puppeteer\n\n         - name: Generate TryHackMe badge\n           run: node generateBadge.js\n\n         - name: Move badge to assets folder\n           run: mv tryhackme-badge-high-quality.png assets/tryhackme-badge-high-quality.png\n\n         - name: Commit and push changes\n           run: |\n             git config --global user.name \"github-actions[bot]\"\n             git config --global user.email \"github-actions[bot]@users.noreply.github.com\"\n             git add assets/tryhackme-badge-high-quality.png\n             git commit -m \"Update TryHackMe badge\"\n             git push\n   ```\n\n## Final Steps\n1. Commit and push your changes to GitHub:\n   ```sh\n   git add .\n   git commit -m \"Added TryHackMe badge automation\"\n   git push origin main\n   ```\n2. The GitHub Action will now run automatically every 12 hours to update your badge.\n----\n\n# Medium Premium Checker (Golang)\n\nThis is a Go-based tool that checks whether a given Medium article URL is a premium (member-only) story. It uses the `chromedp` package to automate a headless Chrome browser, navigate to the provided URL, and determine if the article is premium by checking for specific indicators such as \"Member-only story\" text or a golden star icon.\n\n## Installation\n\n1. **Install Go**: Ensure you have Go installed on your system. You can download it from [here](https://golang.org/dl/).\n\n2. **Clone the Repository**:\n   ```bash\n   git clone https://github.com/yourusername/medium-premium-checker.git\n   cd medium-premium-checker\n   ```\n\n3. **Install Dependencies**:\n   ```bash\n   go mod tidy\n   ```\n\n4. **Build the Tool**:\n   ```bash\n   go build -o medium-premium-checker\n   ```\n\n## Usage\n\nRun the tool from the command line by providing a Medium article URL as an argument:\n\n```bash\n./medium-premium-checker https://medium.com/your-article-url\n```\n\n### Example Output\n\n- If the article is premium:\n  ```bash\n  🔒 https://medium.com/your-article-url is PREMIUM\n  ```\n\n- If the article is not premium:\n  ```bash\n  🔒 https://medium.com/your-article-url is NOT premium\n  ```\n\n# Medium Premium Checker (Node.js Puppeteer)\n\nThis is a Node.js-based tool that checks whether a given Medium article URL is a premium (member-only) story. It uses the `puppeteer` package to automate a headless Chrome browser, navigate to the provided URL, and determine if the article is premium by checking for specific indicators such as \"Member-only story\" text or a golden star icon.\n\n## Installation\n\n1. **Install Node.js**: Ensure you have Node.js installed on your system. You can download it from [here](https://nodejs.org/).\n\n2. **Clone the Repository**:\n   ```bash\n   git clone https://github.com/yourusername/medium-premium-checker-puppeteer.git\n   cd medium-premium-checker-puppeteer\n   ```\n\n3. **Install Dependencies**:\n   ```bash\n   npm install puppeteer\n   ```\n\n## Usage\n\nRun the tool from the command line by providing a Medium article URL as an argument:\n\n```bash\nnode index.js https://medium.com/your-article-url\n```\n\n### Example Output\n\n- If the article is premium:\n  ```bash\n  🔒 https://medium.com/your-article-url is PREMIUM\n  ```\n\n- If the article is not premium:\n  ```bash\n  🔒 https://medium.com/your-article-url is NOT premium\n  ```\n\n# TryHackMe Streak Notifier\n\nThis project monitors your TryHackMe streak and sends an alert to a Telegram channel if the streak does not increase.\n\n## Requirements\n\nInstall dependencies using:\n```bash\npip install -r requirements.txt\n```\n\n## Setup\n\n1. Create a `.env` file and add your bot token and channel ID:\n   ```ini\n   BOT_TOKEN=your_bot_token\n   CHANNEL_ID=your_channel_id\n   ```\n\n2. Run the script:\n   ```bash\n   python script.py\n   ```\n\n## How It Works\n- Fetches your current TryHackMe streak from the API.\n- Compares it with the previously stored streak in `streak.txt`.\n- If the streak has not increased, it sends an alert to the configured Telegram channel.\n\n## Notes\n- Ensure `streak.txt` exists or the script will create it.\n- The script should be run periodically (e.g., via a cron job) to check streak updates.\n\nGithub Action usage of this script:\n```yml\nname: Check THM Streak\n\non:\n  schedule:\n    - cron: \"30 16 * * *\"\n  workflow_dispatch:\n\njobs:\n  check-streak:\n    runs-on: ubuntu-latest\n\n    steps:\n      - name: Checkout repository\n        uses: actions/checkout@v4\n        with:\n          fetch-depth: 0 # Required to allow committing changes\n\n      - name: Set up Python\n        uses: actions/setup-python@v5\n        with:\n          python-version: \"3.x\"\n\n      - name: Install dependencies\n        run: |\n          python -m pip install --upgrade pip\n          pip install requests beautifulsoup4 python-dotenv\n\n      - name: Create .env file\n        run: |\n          echo \"BOT_TOKEN=${{ secrets.BOT_TOKEN }}\" \u003e .env\n          echo \"CHANNEL_ID=${{ secrets.CHANNEL_ID }}\" \u003e\u003e .env\n\n      - name: Run streak check\n        run: python script.py\n\n      - name: Commit streak changes\n        id: commit\n        run: |\n          git config --global user.name \"github-actions[bot]\"\n          git config --global user.email \"github-actions[bot]@users.noreply.github.com\"\n          git add streak.txt\n          git diff --quiet \u0026\u0026 git diff --staged --quiet || (git commit -m \"Update streak.txt [skip ci]\" \u0026\u0026 git push)\n\n```\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fblackvoidx%2Fweb-scraping","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fblackvoidx%2Fweb-scraping","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fblackvoidx%2Fweb-scraping/lists"}