{"id":15859375,"url":"https://github.com/nakabonne/staticcollector","last_synced_at":"2026-05-19T07:34:41.911Z","repository":{"id":108451747,"uuid":"101463784","full_name":"nakabonne/StaticCollector","owner":"nakabonne","description":"Application to analyze static files of competing sites","archived":false,"fork":false,"pushed_at":"2017-11-05T11:53:30.000Z","size":28141,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-09-12T06:56:05.707Z","etag":null,"topics":["crawler","go","golang"],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/nakabonne.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2017-08-26T05:05:27.000Z","updated_at":"2020-08-16T09:50:16.000Z","dependencies_parsed_at":"2023-04-12T16:46:59.344Z","dependency_job_id":null,"html_url":"https://github.com/nakabonne/StaticCollector","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/nakabonne/StaticCollector","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nakabonne%2FStaticCollector","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nakabonne%2FStaticCollector/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nakabonne%2FStaticCollector/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nakabonne%2FStaticCollector/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/nakabonne","download_url":"https://codeload.github.com/nakabonne/StaticCollector/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nakabonne%2FStaticCollector/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":275312968,"owners_count":25442564,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-15T02:00:09.272Z","response_time":75,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["crawler","go","golang"],"created_at":"2024-10-05T21:20:20.021Z","updated_at":"2025-09-15T20:08:10.577Z","avatar_url":"https://github.com/nakabonne.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"## Overview\n\nApplication to analyze static files of competing sites.\nYou can do the following.\n\n- Confirm the change in ranking of competing websites\n- Compare the two HTML before and after the rank change\n\n![result](https://github.com/ryonakao/StaticCollector/blob/media_for_demo/media/demo_ver2.gif)\n\n\n- Register keywords\n\n\n![resul](https://github.com/ryonakao/StaticCollector/blob/media_for_demo/media/keyword_insert.png)\n\n\n- Crawl on the web\n\n\n![resu](https://github.com/ryonakao/StaticCollector/blob/media_for_demo/media/crawl.png)\n\n## Installation\n\n```\n$ go get github.com/ryonakao/StaticCollector\n```\n\n## SetUP\n\n### Setup mongoDB\n\nStart\n\n```\n$ sudo mongod --dbpath /var/lib/mongodb --logpath /var/log/moodb.log\n```\n\nCreate collection\n\n```\n$ mongo\n\u003e use web_crawler\n\u003e db.createCollection('static_files');\n```\n\nInsert tmp data\n\n```\n\u003e db.static_files.insert({word_id:1, page_id:1, title:'tmp title', html:\"\u003chtml\u003e\u003c/html\u003e\", rank:2, target_day:ISODate(\"2017-08-24T04:54:00.697Z\")});\n```\n\n### Setup Mysql\n\nStart\n\n```\n$ mysql.server restart\n```\n\nCreate tables\n\n```\n$ mysql -u root -p\nmysql\u003e CREATE DATABASE web_crawler;\nmysql\u003e use web_crawler\nmysql\u003e CREATE TABLE keywords (id int AUTO_INCREMENT PRIMARY KEY, word varchar(100) NOT NULL);\nmysql\u003e CREATE TABLE pages (id int AUTO_INCREMENT PRIMARY KEY, url varchar(300) UNIQUE NOT NULL);\n```\n\n# License\n\n`StaticCollector` source code is available under the MIT [License](https://github.com/ryonakao/StaticCollector/blob/master/LICENSE).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnakabonne%2Fstaticcollector","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnakabonne%2Fstaticcollector","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnakabonne%2Fstaticcollector/lists"}