{"id":23652059,"url":"https://github.com/mostafahima/law-firm-data-extractor","last_synced_at":"2025-07-17T08:39:16.812Z","repository":{"id":265318970,"uuid":"895771954","full_name":"MostafaHima/Law-Firm-Data-Extractor","owner":"MostafaHima","description":"A project to collect detailed law firm data from \"Law.com\" using Selenium, storing it in a structured, easily analyzable format.","archived":false,"fork":false,"pushed_at":"2024-11-28T22:24:20.000Z","size":29,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-07-06T04:50:54.100Z","etag":null,"topics":["scraping-python"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/MostafaHima.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-11-28T21:48:16.000Z","updated_at":"2024-12-13T20:50:12.000Z","dependencies_parsed_at":"2024-11-28T22:31:55.361Z","dependency_job_id":"1a5d44a1-d2a9-47a2-9596-d991855599e2","html_url":"https://github.com/MostafaHima/Law-Firm-Data-Extractor","commit_stats":null,"previous_names":["mostafahima/lawfirmscraper","mostafahima/law-firm-data-extractor"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/MostafaHima/Law-Firm-Data-Extractor","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MostafaHima%2FLaw-Firm-Data-Extractor","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MostafaHima%2FLaw-Firm-Data-Extractor/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MostafaHima%2FLaw-Firm-Data-Extractor/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MostafaHima%2FLaw-Firm-Data-Extractor/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/MostafaHima","download_url":"https://codeload.github.com/MostafaHima/Law-Firm-Data-Extractor/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MostafaHima%2FLaw-Firm-Data-Extractor/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":265586089,"owners_count":23792851,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["scraping-python"],"created_at":"2024-12-28T16:50:20.012Z","updated_at":"2025-07-17T08:39:16.793Z","avatar_url":"https://github.com/MostafaHima.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# **LawFirmScraper**\n\n**LawFirmScraper** is a web scraping project that gathers detailed information about law firms from the \"Law.com\" website. The project uses **Selenium** to automate the process of collecting data from multiple pages and stores it in an organized format.\n\n## **Data Link on Google Sheets**\nThe data has been uploaded to **Google Sheets**. You can access the sheet from [here](https://docs.google.com/spreadsheets/d/16N9k-3NBUPVeVtVHINMSlX6Rnl8rOzkSheF2_BwFvXg/edit?gid=0#gid=0).\n\n\n## **Features**\n\n- **Web scraping**: Collect data about law firms such as names, ratings, descriptions, and more.\n- **Data processing**: Process the collected data to extract relevant details.\n- **Data uploading**: Upload the processed data to an external database (like Sheety).\n- **Browser automation**: Uses Selenium and ChromeDriver to automate browser interaction.\n\n## **Technologies Used**\n\n- **Selenium**: For extracting data from web pages.\n- **Python**: The primary programming language for the project.\n- **Sheety API**: For uploading the data to an external database.\n- **ChromeDriver**: To automate interaction with the Chrome browser.\n\n## **Project Setup**\n\n### 1. **Install Dependencies**\n\nFirst, clone the project:\n\n```bash\ngit clone https://github.com/your_username/your_project_name.git\ncd your_project_name\n\npython -m venv venv\nsource venv/bin/activate  # For Linux/Mac systems\nvenv\\Scripts\\activate     # For Windows systems\npip install -r requirements.txt\nAPI_KEY=your_api_key_here\nSHEET_URL=your_sheety_url_here\npython main.py\n\nLawFirmScraper/\n│\n├── .gitignore             # .gitignore file (includes .env and other unnecessary files)\n├── .env                   # File for storing environment variables like API keys\n├── .idea/                 # IDE project files (can be ignored)\n├── __pycache__/           # Python bytecode files\n├── get_data.py            # Script to extract data from law firm pages\n├── main.py                # Main script to run the project\n├── prepere_data.py        # Script to prepare the data and load more pages\n├── procces_data.py        # Script to process and clean the data\n├── upload_data.py         # Script to upload the processed data\n├── requirements.txt       # List of dependencies for the project\n└── README.md              # Project documentation (this file)\n```\n\n\n# How It Works\n1. Extraction: The project first navigates to the law firm listing page on Law.com and clicks the \"Load More\" button to load all available firms.\n2. Data Extraction: After the page is fully loaded, the script collects the law firm links and other important details for each firm.\n3. Data Processing: The collected data is cleaned and structured to be ready for uploading.\n4. Data Uploading: The processed data is uploaded to an external database using the Sheety API.\n\n## License\n\nThis project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.\n##\n##\n\n##\n\n# **LawFirmScraper**\n\n**LawFirmScraper** هو مشروع لاستخراج البيانات من الإنترنت (ويب سكرابينغ) يجمع معلومات تفصيلية عن شركات المحاماة من موقع \"Law.com\". يستخدم المشروع **Selenium** لأتمتة عملية جمع البيانات من عدة صفحات وتخزينها في تنسيق منظم.\n## **رابط جدول البيانات**\nتم رفع البيانات إلى **Google Sheets**. يمكنك الوصول إلى الجدول من [هنا](https://docs.google.com/spreadsheets/d/16N9k-3NBUPVeVtVHINMSlX6Rnl8rOzkSheF2_BwFvXg/edit?gid=0#gid=0).\n\n\n## **المميزات**\n\n- **استخراج البيانات من الويب**: جمع بيانات عن شركات المحاماة مثل الأسماء، التقييمات، الأوصاف، والمزيد.\n- **معالجة البيانات**: معالجة البيانات المجمعة لاستخراج التفاصيل ذات الصلة.\n- **رفع البيانات**: رفع البيانات المعالجة إلى قاعدة بيانات خارجية (مثل Sheety).\n- **أتمتة المتصفح**: يستخدم Selenium وChromeDriver لأتمتة التفاعل مع المتصفح.\n\n## **التقنيات المستخدمة**\n\n- **Selenium**: لاستخراج البيانات من صفحات الويب.\n- **Python**: اللغة الأساسية للمشروع.\n- **Sheety API**: لرفع البيانات إلى قاعدة بيانات خارجية.\n- **ChromeDriver**: لأتمتة تفاعل المتصفح Chrome.\n\n## **إعداد المشروع**\n\n### 1. **تثبيت الاعتمادات**\n\nأولاً، قم باستنساخ المشروع:\n\n```bash\ngit clone https://github.com/اسم_المستخدم/اسم_المشروع.git\ncd اسم_المشروع\n\npython -m venv venv\nsource venv/bin/activate  # في حالة نظام Linux/Mac\nvenv\\Scripts\\activate     # في حالة نظام Windows\npip install -r requirements.txt\nAPI_KEY=your_api_key_here\nSHEET_URL=your_sheety_url_here\npython main.py\n\n\nLawFirmScraper/\n│\n├── .gitignore             # ملف .gitignore (يحتوي على .env وملفات أخرى غير ضرورية)\n├── .env                   # ملف لتخزين المتغيرات البيئية مثل مفاتيح API\n├── .idea/                 # ملفات مشروع IDE (يمكن تجاهلها)\n├── __pycache__/           # ملفات Python البايت\n├── get_data.py            # سكربت لاستخراج البيانات من صفحات شركات المحاماة\n├── main.py                # السكربت الرئيسي لتشغيل المشروع\n├── prepere_data.py        # سكربت لإعداد البيانات وتحميل المزيد من الصفحات\n├── procces_data.py        # سكربت لمعالجة وتنظيف البيانات\n├── upload_data.py         # سكربت لرفع البيانات المعالجة\n├── requirements.txt       # قائمة بالاعتمادات للمشروع\n└── README.md              # وثائق المشروع (هذا الملف)\n```\n\n## **كيفية العمل**\n1. الاستخراج: يقوم المشروع أولاً بالانتقال إلى صفحة شركات المحاماة في Law.com ويضغط على زر \"تحميل المزيد\" لتحميل جميع الشركات المتاحة.\n2. استخراج البيانات: بعد تحميل الصفحة بالكامل، يقوم السكربت بجمع روابط الشركات والتفاصيل الأخرى المهمة لكل شركة.\n3. معالجة البيانات: يتم تنظيف البيانات المجمعة وهيكلتها بحيث تكون جاهزة للرفع.\n4. رفع البيانات: يتم رفع البيانات المعالجة إلى قاعدة بيانات خارجية باستخدام Sheety API.\n\n\n\n\n## License\n\nThis project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmostafahima%2Flaw-firm-data-extractor","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmostafahima%2Flaw-firm-data-extractor","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmostafahima%2Flaw-firm-data-extractor/lists"}