{"id":27278973,"url":"https://github.com/toofancodes/scrapperathleticscontacts","last_synced_at":"2026-04-25T21:34:21.583Z","repository":{"id":287342120,"uuid":"964405805","full_name":"toofanCodes/ScrapperAthleticsContacts","owner":"toofanCodes","description":"StaffScrapper is a smart and flexible web scraper built for one job: collecting staff contact details from athletics department websites. Whether you're pulling emails, job titles, or phone numbers — even from JavaScript-heavy or obfuscated sites — this tool handles it with ease.  Designed for marketing teams, outreach coordinators, and data analys","archived":false,"fork":false,"pushed_at":"2025-04-11T07:44:17.000Z","size":12,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-11T09:40:51.542Z","etag":null,"topics":["beautifulsoup","beautifulsoup4","python","requests","scrapping-python","selenium","selenium-webdriver","webdriver-manager"],"latest_commit_sha":null,"homepage":"https://www.linkedin.com/in/saranpavuluri/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/toofanCodes.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-04-11T06:57:11.000Z","updated_at":"2025-04-11T07:47:54.000Z","dependencies_parsed_at":"2025-04-11T09:51:11.664Z","dependency_job_id":null,"html_url":"https://github.com/toofanCodes/ScrapperAthleticsContacts","commit_stats":null,"previous_names":["toofancodes/scrapperathleticscontacts"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/toofanCodes%2FScrapperAthleticsContacts","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/toofanCodes%2FScrapperAthleticsContacts/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/toofanCodes%2FScrapperAthleticsContacts/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/toofanCodes%2FScrapperAthleticsContacts/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/toofanCodes","download_url":"https://codeload.github.com/toofanCodes/ScrapperAthleticsContacts/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248453484,"owners_count":21106265,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["beautifulsoup","beautifulsoup4","python","requests","scrapping-python","selenium","selenium-webdriver","webdriver-manager"],"created_at":"2025-04-11T17:46:29.510Z","updated_at":"2026-04-25T21:34:21.555Z","avatar_url":"https://github.com/toofanCodes.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"### ✅ `README.md` for `staffScrapper_Apr2025.py`\n\n```markdown\n# 🏫 Staff Directory Scraper for Athletic Websites\n\nThis Python script automates the extraction of staff contact information (such as name, email, title, department, and phone number) from college athletics staff directories. It is designed to handle diverse and dynamic HTML structures, including those rendered with JavaScript.\n\n---\n\n## 📌 Features\n\n- ✅ Scrapes data from multiple URLs using a `.csv` input list\n- ✅ Handles complex HTML structures (including JavaScript-rendered content with Selenium)\n- ✅ Extracts:\n  - Full name\n  - Position / title\n  - Email address (even from obfuscated JS formats)\n  - Phone number\n  - Associated department or sport (when available)\n  - Source URL\n- ✅ Gracefully handles errors and logs them for debugging\n- ✅ Supports headless scraping for automation pipelines\n\n---\n\n## 📂 File Structure\n\n| File | Description |\n|------|-------------|\n| `staffScrapper_Apr2025.py` | Main scraper script |\n| `target_urls.csv` | Input CSV with URLs (one per line) |\n| `staff_directory.csv` | Output file with scraped data |\n| `scrape_errors.txt` | Error Output log showing failed URLs and parsing issues |\n\n---\n\n## 🛠 Requirements\n\nInstall dependencies using:\n\n```bash\npip install -r requirements.txt\n```\n\n### `requirements.txt` content:\n\n```\nrequests\nbeautifulsoup4\nselenium\nwebdriver-manager\n```\n\n---\n\n## 📥 Usage\n\n1. Prepare a CSV file named `target_urls.csv` with this structure:\n\n```\nhttps://example.edu/staff-directory\nhttps://another.edu/staff-directory\n...\n```\n\n\u003e **Note**: No header row is required.\n\n2. Run the script:\n\n```bash\npython staffScrapper_Apr2025.py\n```\n\n3. Output will be saved as:\n   - `staff_directory.csv` — extracted contact info\n   - `scrape_errors.txt` — any URLs that couldn’t be processed\n\n---\n\n## 🧠 How It Works\n\n- Tries multiple parsing strategies (table, definition list, generic row matching)\n- Uses `Selenium` headless Chrome if the page is JavaScript-heavy\n- Identifies email patterns even when obfuscated with JS `document.write` or `innerText` replacement\n- Categorizes staff into departments based on headings where possible\n\n---\n\n## ⚠️ Known Limitations\n\n- Pages with extreme JavaScript complexity may not be 100% compatible\n- Obfuscated email formats beyond standard patterns may be missed\n- Sites using CAPTCHAs or anti-bot protection are unsupported\n\n\n## 👤 Author\n\n**Jaya Saran Teja Pavuluri**  \n[GitHub](https://github.com/toofanCodes)  \n📧 saran.in.usa@gmail.com\n\n---\n\n## 📝 License\n\nMIT License – do what you want, just don't spam the scrapped contacts 😉\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftoofancodes%2Fscrapperathleticscontacts","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftoofancodes%2Fscrapperathleticscontacts","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftoofancodes%2Fscrapperathleticscontacts/lists"}