{"id":22119984,"url":"https://github.com/maxpleaner/tagger","last_synced_at":"2025-08-31T19:40:47.077Z","repository":{"id":84577159,"uuid":"51794487","full_name":"MaxPleaner/tagger","owner":"MaxPleaner","description":"scrapes descriptions \u0026 tags for LinkedIn companies","archived":false,"fork":false,"pushed_at":"2016-09-04T17:39:49.000Z","size":1008,"stargazers_count":1,"open_issues_count":0,"forks_count":1,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-03-24T06:30:52.119Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/MaxPleaner.png","metadata":{"files":{"readme":"readme.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2016-02-15T23:54:40.000Z","updated_at":"2016-03-03T19:50:31.000Z","dependencies_parsed_at":"2023-03-12T23:42:51.653Z","dependency_job_id":null,"html_url":"https://github.com/MaxPleaner/tagger","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/MaxPleaner/tagger","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MaxPleaner%2Ftagger","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MaxPleaner%2Ftagger/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MaxPleaner%2Ftagger/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MaxPleaner%2Ftagger/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/MaxPleaner","download_url":"https://codeload.github.com/MaxPleaner/tagger/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MaxPleaner%2Ftagger/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":273032934,"owners_count":25034067,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-31T02:00:09.071Z","response_time":79,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-12-01T14:19:41.938Z","updated_at":"2025-08-31T19:40:47.063Z","avatar_url":"https://github.com/MaxPleaner.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"## Tagger\n\nscrapes descriptions \u0026 tags for LinkedIn companies \n\n------\n\n#### Setup\n\nIt requires one environment variable to be set, `MONKEY_LEARN_TOKEN` which is an api key from\n[monkeylearn.com](http://monkeylearn.com). MonkeyLearn is used to extract tags from text.\n\nOther than that, it's standard Rails:\n\n`clone`, `bundle` `rake db:create db:migrate`, `localhost:3000`\n\n#### basic usage\n\n  - Use the HTML interface at `localhost:3000`\n  - Scraper command (call with `SelfScraper.begin(\"google\")` in `rails c` when the server is running)\n\n#### details on scraper\n\nYou can load some seed data by running `rake db:data:load`\n\nYou can run the crawler by running `SelfScraper.begin(\"google\")` in `rails console`. This example will start\nat the Google  page and move to \"related\" pages from there.\n- This automatically interacts with the server using `Mechanize`.\n- Eventually it will loop and stop finding new companies.\n- This is because of bidirectional linking in  LinkedIn's 'people also clicked' sections.\n- There are often small groups of companies which all link back to each other.\n- When this happens, the scraper will need to be restarted with a new company name. \n\nNote the 'clear cache' button on the HTML site actually wipes the\nentire database.  \n\nThis is a HTML scaper, not an authenticated API application, so it probably has\nmore severe rate limits. `999` errors means the IP address is being throttled. \n\nThe bulk of the code is in [`application_controller.rb`](https://github.com/MaxPleaner/tagger/blob/master/app/controllers/application_controller.rb),\n[`pages_controller.rb`](https://github.com/MaxPleaner/tagger/blob/master/app/controllers/pages_controller.rb),\n[`pages/root.html.erb`](https://github.com/MaxPleaner/tagger/blob/master/app/views/pages/root.html.erb),\nand [`application.rb`](https://github.com/MaxPleaner/tagger/blob/master/config/application.rb).","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmaxpleaner%2Ftagger","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmaxpleaner%2Ftagger","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmaxpleaner%2Ftagger/lists"}