{"id":28576154,"url":"https://github.com/liron-li/livaspider","last_synced_at":"2026-03-04T21:02:47.272Z","repository":{"id":108305438,"uuid":"90349048","full_name":"liron-li/LivaSpider","owner":"liron-li","description":"使用python编写的异步io爬虫，编写少量的代码即可轻松的爬取目标页面","archived":false,"fork":false,"pushed_at":"2018-05-25T07:15:13.000Z","size":11,"stargazers_count":6,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"master","last_synced_at":"2025-08-25T08:06:33.379Z","etag":null,"topics":["asyncio","python-3-6","spider"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/liron-li.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2017-05-05T07:26:15.000Z","updated_at":"2021-01-14T03:08:54.000Z","dependencies_parsed_at":"2023-08-30T21:19:39.964Z","dependency_job_id":null,"html_url":"https://github.com/liron-li/LivaSpider","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/liron-li/LivaSpider","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/liron-li%2FLivaSpider","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/liron-li%2FLivaSpider/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/liron-li%2FLivaSpider/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/liron-li%2FLivaSpider/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/liron-li","download_url":"https://codeload.github.com/liron-li/LivaSpider/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/liron-li%2FLivaSpider/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30092877,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-04T20:42:30.420Z","status":"ssl_error","status_checked_at":"2026-03-04T20:42:30.057Z","response_time":59,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["asyncio","python-3-6","spider"],"created_at":"2025-06-10T23:08:07.254Z","updated_at":"2026-03-04T21:02:47.249Z","avatar_url":"https://github.com/liron-li.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003e 最近学到了python的asyncIO，于是便利用空余时间写了这个小爬虫\n\n##### 环境\n- python3.6\n\n##### 依赖：\n- beautifulsoup4==4.5.3\n- requests==2.13.0\n- alembic==0.9.1\n- SQLAlchemy==1.1.9\n\n##### 目录结构\n```angular2html\n├── alembic                  # alembic 目录\n│   ├── env.py               # alembic env配置文件\n│   ├── README\n│   ├── script.py.mako       # alembic 模板文件\n│   └── versions             # 表迁移文件\n│       └── 404fa70bcf2c_create_tables.py\n├── alembic.ini              # alembic 配置文件\n├── core\n│   ├── crawling.py          # 爬虫基类\n│   ├── __init__.py\n│   └── models.py            # sqlAlchemy模型\n├── example_crawl_baike.py   # example 爬取百度百科\n└── README.md\n```\n\n##### 如何使用？\n\n- 数据库配置\n修改`alembic.ini`文件中的`sqlalchemy.url`\n```angular2html\nsqlalchemy.url = driver://user:pass@localhost/dbname\n```\n- 生成表迁移文件\n```angular2html\nalembic revision --autogenerate -m \"your desc\"\n```\n\n- 执行迁移\n```angular2html\nalembic upgrade head\n```\n- 爬虫配置\n```angular2html\nconfig = {\n        # 请求头\n        \"headers\": headers,\n        # cookies\n        \"cookies\": cookies,\n        # 根url\n        \"base_url\": \"http://baike.baidu.com/\",\n        # 起始url\n        \"start_url\": \"http://baike.baidu.com/item/%E9%93%81%E6%A0%91/110475\",\n        # 抓取的网站正则\n        \"url_rule\": r'^http://baike.baidu.com/item/',\n    }\n```\n爬虫运行时会抓取`start_url`中的符合`url_rule`正则的所有url存入数据库做为爬取的目标url，\n直至`url_pool`表中的所有记录都爬取完，爬虫结束\n\n\n- 运行爬虫\n```angular2html\npython example_crawl_baike.py\n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fliron-li%2Flivaspider","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fliron-li%2Flivaspider","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fliron-li%2Flivaspider/lists"}