{"id":18577427,"url":"https://github.com/pjt3591oo/spider-base_crawler","last_synced_at":"2025-05-16T01:32:28.685Z","repository":{"id":100333440,"uuid":"230043549","full_name":"pjt3591oo/spider-base_crawler","owner":"pjt3591oo","description":"scrapy 기반 크롤러 만들기","archived":false,"fork":false,"pushed_at":"2020-04-24T00:42:01.000Z","size":21,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-02-17T15:26:04.235Z","etag":null,"topics":["crawler","python","scrapy","spider"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/pjt3591oo.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-12-25T04:53:48.000Z","updated_at":"2020-04-24T00:42:04.000Z","dependencies_parsed_at":"2023-05-13T22:15:33.490Z","dependency_job_id":null,"html_url":"https://github.com/pjt3591oo/spider-base_crawler","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pjt3591oo%2Fspider-base_crawler","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pjt3591oo%2Fspider-base_crawler/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pjt3591oo%2Fspider-base_crawler/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pjt3591oo%2Fspider-base_crawler/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/pjt3591oo","download_url":"https://codeload.github.com/pjt3591oo/spider-base_crawler/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254453434,"owners_count":22073608,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["crawler","python","scrapy","spider"],"created_at":"2024-11-06T23:29:05.645Z","updated_at":"2025-05-16T01:32:28.636Z","avatar_url":"https://github.com/pjt3591oo.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Run\n\n* regular\n\n```\n$ scrapy crawl [크롤러 이름]\n```\n\n크롤러 이름은 **`spiders`**에 있는 파일명을 적는다. 전달 옵션에 따라 다양한 형태로 저장가능하며 pipelines를 통해서도 저장할 수 있다.\n\n## 스타일닷컴\n\n* 구동\n\n```bash\n$ scrapy crawl shop_sta\n```\n\n* json 저장\n\n```bash\n$ scrapy crawl shop_sta -o result.json\n```\n\n* csv 저장\n\n```bash\n$ scrapy crawl shop_sta -o result.csv -t csv\n```\n\n스타일 닷컴에서 쇼핑몰 리스트 수집\n\n\n```json\n{\n  \"storeSeq\": \"24\", \n  \"ageNames\": \"30대\", \n  \"appHit\": 896, \n  \"cateNames\": \"미시스타일,심플베이직\", \n  \"description\": \"세련되고 엘레강스한 여성의류 쇼핑몰. 깔끔한 핏의 하이퀄리티 아이템 판매\", \n  \"webUrl\": \"http://www.annanblue.com/\", \n  \"storeName\": \"안나앤블루\", \n  \"storeGender\": \"F\", \n  \"webProdUrl\": \"/shop/shopdetail.html\"\n}\n```\n\n## 업비트\n\n* 구동\n\n```bash\n$ scrapy crawl exchange_upbit_by_chart\n```\n\n* json 저장\n\n```bash\n$ scrapy crawl exchange_upbit_by_chart -o result.json\n```\n\n* csv 저장\n\n```bash\n$ scrapy crawl exchange_upbit_by_chart -o result.csv -t csv\n```\n\n## 코인원\n\n* 구동\n\n```bash\n$ scrapy crawl exchange_coinone\n```\n\n* json 저장\n\n```bash\n$ scrapy crawl exchange_coinone -o result.json\n```\n\n* csv 저장\n\n```bash\n$ scrapy crawl exchange_coinone -o result.csv -t csv\n```\n\n## 전자공시 시스템\n\n* 구동\n\n```bash\n$ scrapy crawl dart\n```\n\n* json 저장\n\n```bash\n$ scrapy crawl dart -o dart.json\n```\n\n* csv 저장\n\n```bash\n$ scrapy crawl dart -o dart.csv -t csv\n```\n\n## 네이버 웹툰 리스트\n\n* 구동\n\n```bash\n$ scrapy crawl naver_webtoon\n```\n\n* json 저장\n\n```bash\n$ scrapy crawl naver_webtoon -o webtoon.csv\n```\n\n* csv 저장\n\n```bash\n$ scrapy crawl naver_webtoon -o webtoon.csv -t csv    \n```\n\n\n# crawler generator\n\n```bash\n$ scrapy genspider [크롤러 이름] [링크]\n```\n\n**`spiders`** 아래에 [크롤러 이름] 파일이 생성되며 다음과 같이 내용을 채움\n\n```py\nimport scrapy\n\n\nclass ExchangeUpbitSpider(scrapy.Spider):\n    name = 'exchange_upbit'\n    allowed_domains = ['https://www.upbit.com/exchange']\n    start_urls = ['http://https://www.upbit.com/exchange/']\n\n    def parse(self, response):\n        pass\n\n```\n\n전달된 링크가 **`allowed_domains`**, **`start_urls`** 값으로 전달된다.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpjt3591oo%2Fspider-base_crawler","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpjt3591oo%2Fspider-base_crawler","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpjt3591oo%2Fspider-base_crawler/lists"}