{"id":27072781,"url":"https://github.com/hong539/acgbox_crawler","last_synced_at":"2025-04-05T23:33:47.025Z","repository":{"id":153103178,"uuid":"623788037","full_name":"hong539/acgbox_crawler","owner":"hong539","description":"An web-crawler for gamer.com.tw/acgbox","archived":false,"fork":false,"pushed_at":"2024-02-08T04:29:11.000Z","size":158,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2024-02-08T05:28:36.700Z","etag":null,"topics":["beautifulsoup4","crawler","pandas","python","requests","scrapy","sqlalchemy","web-crawler"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/hong539.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2023-04-05T04:56:57.000Z","updated_at":"2023-07-20T17:42:53.000Z","dependencies_parsed_at":"2024-01-07T11:42:32.872Z","dependency_job_id":null,"html_url":"https://github.com/hong539/acgbox_crawler","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hong539%2Facgbox_crawler","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hong539%2Facgbox_crawler/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hong539%2Facgbox_crawler/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hong539%2Facgbox_crawler/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/hong539","download_url":"https://codeload.github.com/hong539/acgbox_crawler/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247415824,"owners_count":20935384,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["beautifulsoup4","crawler","pandas","python","requests","scrapy","sqlalchemy","web-crawler"],"created_at":"2025-04-05T23:33:46.485Z","updated_at":"2025-04-05T23:33:47.017Z","avatar_url":"https://github.com/hong539.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# acgbox_crawler\n\nAn Python bot for doing ETL(extract, transform, load) personal favorite lists from gamer.com.tw/acgbox.\n\n## Prerequisites\n\n* Setup all on My Arch Linux VM\n    * Python version ==3.8 and all dependencies will be managed via pyenv+pipenv\n        * [Usage with pyenv+pipenv](https://github.com/hong539/setup_dev_environment/blob/main/programming_languages/python/python.md#usage-with-pyenvpipenv)\n    * PostgreSQL 15\n\n## to_do_list\n\n* ~~Migration from MySQL to PostgreSQL~~\n* Find Last Updated Date of a Web Page\n* [TritonHo/RDBMS course](https://github.com/TritonHo/slides/blob/master/Taipei%202019-04%20course/lesson0.pdf)\n* ~~pandas to_sql method if_exists='append' implementation function update method for only update new ACG collects?~~\n* implementation function finding method for find last pages?\n* research or implementation function for html tags div to table\n* check which files will be stored via podman when not executing MySQL container\n* implementation function CRUD API/query method for MySQL\n* implementation function load_data\n* implementation function modfy_data with advanced string replace in pandas.DataFrame\n* refactor some parts codes to class acgbox_crawler(object)\n* podman-compose up with docker-selenium\n    * WARN[0011] aardvark-dns binary not found, container dns will not be enabled\n\n## quick start\n\n### setup on Arch Linux\n\n```shell\n#setup on Arch Linux\n#update package databases\nsudo pacman -Syy\n\n#install podman\nsudo pacman -S podman\n#podman-docker\nsudo pacman -S podman-docker\n#podman-compose\nsudo pacman -S podman-compose\n#fuse-overlayfs\nsudo pacman -S fuse-overlayfs\n\n#podman: /usr/lib/libc.so.6: version `GLIBC_2.38' not found (required by podman)\n#upgrading packages\nsudo pacman -Syu\n\n#check podman\npodman --version\n\n#create a MySQL container with podman-compose\ncd db_settingup/\n\n#check out the db_settingup.md\n```\n\n### start this project and do development\n\n```shell\n#After Setting UP with Usage with your python projects\n#Spawns a shell within the virtualenv.\npipenv shell\n\n#if no packages installed\npipenv install\n\n#add some Packages\npipenv install diagrams\npipenv install \"psycopg[binary,pool]\"\npipenv install requests\npipenv install beautifulsoup4\npipenv install pandas\npipenv install lxml\npipenv install SQLAlchemy\npipenv install PyYAML\npipenv install pymysql\npipenv install fake-useragent\npipenv install user_agent\npipenv install tornado\n\n#Generate a requirements.txt from Pipfile.lock. to requirements.txt\npipenv requirements \u003e requirements.txt\n\n#Becareful your execute PATH! XD \n#Test\npipenv shell\ncd src\npython main.py\n#time a simple command or give resource usage\ntime python main.py\n# real    2m55.699s\n# user    0m3.973s\n# sys     0m0.858s\n\n#podman\nsudo pacman -S aardvark-dns\n\n#docker/podman with selenium\ncd docker-selenium\n\n#podman\npodman-compose up -d\npodman-compose stop\n\n#docker\ndocker-compose up -d\n```\n\n## misc(miscellaneous)\n\n* html generator    \n    * [htmlgenerator](https://pypi.org/project/htmlgenerator/)\n    * [Yattag/ a generator for html/xml/pythonic ](https://www.yattag.org/)\n    * [HTML div/table tag TABLE GENERATOR](https://divtable.com/generator/)\n\n* html parser\n    * [Beautiful Soup](https://www.crummy.com/software/BeautifulSoup/bs4/doc/)\n        * [searching-by-css-class](https://www.crummy.com/software/BeautifulSoup/bs4/doc/#searching-by-css-class)\n    * [requests](https://github.com/psf/requests)\n    * [pandas](https://pandas.pydata.org/)\n    * [lxml](https://lxml.de/)\n* [PyYAML](https://pyyaml.org/)\n* Automation test\n    * [selenium](https://pypi.org/project/selenium/)\n        * [Selenium Documentation](https://www.selenium.dev/selenium/docs/api/py/api.html)\n    * [Scrapy](https://scrapy.org/)\n    * [Playwright/python](https://playwright.dev/python/)\n        * enables reliable end-to-end testing for modern web apps.\n    * [MechanicalSoup](https://mechanicalsoup.readthedocs.io/en/stable/)\n\n## Important!!!\n\n== We're Using GitHub Under Protest ==\n\nThis project is currently hosted on GitHub.  This is not ideal; GitHub is a\nproprietary, trade-secret system that is not Free and Open Souce Software\n(FOSS).  We are deeply concerned about using a proprietary system like GitHub\nto develop our FOSS project.  We have an\n[open {bug ticket, mailing list thread, etc.} ](INSERT_LINK) where the\nproject contributors are actively discussing how we can move away from GitHub\nin the long term.  We urge you to read about the\n[Give up GitHub](https://GiveUpGitHub.org) campaign from\n[the Software Freedom Conservancy](https://sfconservancy.org) to understand\nsome of the reasons why GitHub is not a good place to host FOSS projects.\n\nIf you are a contributor who personally has already quit using GitHub, please\n[check this resource](INSERT_LINK) for how to send us contributions without\nusing GitHub directly.\n\nAny use of this project's code by GitHub Copilot, past or present, is done\nwithout our permission.  We do not consent to GitHub's use of this project's\ncode in Copilot.\n\n![Logo of the GiveUpGitHub campaign](https://sfconservancy.org/img/GiveUpGitHub.png)","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhong539%2Facgbox_crawler","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhong539%2Facgbox_crawler","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhong539%2Facgbox_crawler/lists"}