{"id":16115286,"url":"https://github.com/iafan/hc","last_synced_at":"2025-03-18T10:30:30.742Z","repository":{"id":146408015,"uuid":"121493099","full_name":"iafan/hc","owner":"iafan","description":"Run headless Chrome in discardable Docker containers for reproducible and safe automation","archived":false,"fork":false,"pushed_at":"2019-06-15T01:47:31.000Z","size":23,"stargazers_count":6,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2023-03-24T00:21:52.630Z","etag":null,"topics":["automation","cli","docker","headless-chrome"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/iafan.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-02-14T09:34:35.000Z","updated_at":"2024-06-19T08:55:02.249Z","dependencies_parsed_at":"2024-06-19T08:55:01.489Z","dependency_job_id":"10bab500-f63e-415e-8d75-5a2bac552d1d","html_url":"https://github.com/iafan/hc","commit_stats":null,"previous_names":[],"tags_count":0,"template":null,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/iafan%2Fhc","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/iafan%2Fhc/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/iafan%2Fhc/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/iafan%2Fhc/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/iafan","download_url":"https://codeload.github.com/iafan/hc/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244201627,"owners_count":20415113,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["automation","cli","docker","headless-chrome"],"created_at":"2024-10-09T20:18:26.142Z","updated_at":"2025-03-18T10:30:30.343Z","avatar_url":"https://github.com/iafan.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# About `hc`\r\n\r\n`hc` is a command-line tool that runs headless Chrome in isolated\r\nDocker containers (which are automatically created and destroyed)\r\nfor safe and reproducible browser automation and data extraction tasks:\r\n\r\n1. Generating HTML snapshots for static and dynamically rendered pages;\r\n2. Downloading XHR resources;\r\n3. Evaluating and capturing the output of arbitrary JavaScript code;\r\n4. Generating screenshots.\r\n\r\n**Consider this utility EXPERIMENTAL. The list of commands,\r\ntheir behavior and invocation syntax may change in the future.**\r\n\r\n## Quick examples\r\n\r\nOutput the rendered HTML of the page (as the browser sees it after building the page):\r\n\r\n```sh\r\nhc html \"http://example.com/\"\r\n```\r\n\r\nOutput the number of paragraphs on a page:\r\n\r\n```sh\r\nhc eval \"http://example.com/\" \"return document.getElementsByTagName('p').length\"\r\n```\r\n\r\nMake a screenshot:\r\n\r\n```sh\r\nhc screenshot \"http://example.com/\" \u003eout.png\r\n```\r\n\r\nSee more examples below.\r\n\r\n## Advantages\r\n\r\n1. Ease of deployment: `hc` compiles into a binary with no runtime dependencies\r\n   apart from Docker;\r\n\r\n2. Ease of use in shell scripts or other scripting languages;\r\n\r\n3. Unix way of working with the data: pipe the fetched HTML, JSON or binary\r\n   resources through other streaming tools. `hc` maintains a clean separation\r\n   between data (STDOUT) and logging / error reporting (STDERR),\r\n   and uses meaningful process exit codes;\r\n\r\n4. Security and reproducibility: `hc` uses Docker to temporarily spin up and\r\n   shut down the container for each command invocation. This guarantees that\r\n   Chrome starts in the same clean state when running a command (think of it\r\n   as a per-command incognito mode);\r\n\r\n5. Resiliency: if the script doesn't finish execution within a given deadline,\r\n   it is shut down automatically, and the container is killed, so your scripts\r\n   never get stuck;\r\n\r\n6. Headless Chrome container runs only for the duration of the command execution,\r\n   so when you don't need it, it doesn't waste your system resources.\r\n\r\n# Installation\r\n\r\n```sh\r\n$ go get github.com/iafan/hc\r\n```\r\n\r\n# Prerequisites\r\n\r\n[Docker](https://www.docker.com/community-edition) and [Go](https://golang.org/dl/).\r\nAs for the actual Docker image, `hc` uses [justinribeiro/chrome-headless](https://hub.docker.com/r/justinribeiro/chrome-headless/) by default\r\n(which will be installed automatically). If you prefer some other image,\r\nuse the `--docker-image` command-line flag.\r\n\r\n**Note:** The first time you run some `hc` command that requires headless Chrome,\r\nDocker will download and install the missing image. Please be patient.\r\n\r\n# Examples\r\n\r\n## Evaluating JavaScript on a page\r\n\r\nSave the list of href attributes of all the links on the page to a file:\r\n\r\n```sh\r\n$ hc eval \\\r\n    --output-file \"links-{TIMESTAMP}.txt\" \\\r\n    \"https://httpbin.org/\" \\\r\n    \"return Array(...document.getElementsByTagName('a')).map(el =\u003e el.getAttribute('href')).join('\\n')\"\r\n```\r\n\r\nHere the output is redirected to a file with `links-{TIMESTAMP}.txt` name template;\r\n`{TIMESTAMP}` will be replaced automatically with the current date and time\r\nin `YYYY-MM-DD-hh-mm-ss` format, so the final file name will look like this:\r\n`links-2018-02-12-15-34-59.txt`\r\n\r\n## Get the contents of a web page\r\n\r\nOutput the rendered HTML document:\r\n\r\n```sh\r\n$ hc html \"https://httpbin.org/status/418\"\r\n```\r\n\r\nThe command above is equivalient to:\r\n\r\n```sh\r\n$ hc eval \"https://httpbin.org/status/418\" \"return document.documentElement.outerHTML\"\r\n```\r\n\r\nIf you need just the contents of the \u003cbody\u003e tag, use:\r\n\r\n```sh\r\n$ hc eval \"https://httpbin.org/status/418\" \"return document.body.innerHTML\"\r\n```\r\n\r\n## Load a resource in the context of a web page\r\n\r\nNote how this method is different from loading the resource URL directly:\r\nthe resource is loaded by the host page itself, with proper headers and\r\ncookies, and `hc` just captures its content. This allows for easy capturing\r\nof XHR resources.\r\n\r\nOutput the value of the resource with the exact URL match:\r\n\r\n```sh\r\n$ hc resource \"http://example.com/\" \"http://example.com/xhr/someData.js\"\r\n```\r\n\r\nOutput the value of the first resource with the URL starting with a given prefix:\r\n\r\n```sh\r\n$ hc resource --match contains \"https://httpbin.org/\" \"tracker.js\"\r\n```\r\n\r\nOutput the value of the first resource with the URL matching a given\r\nregular expression (here the resource is a binary file, so the best option is\r\nto redirect the output to a file, or use the `--output-file` flag as described\r\nin one of the previous examples):\r\n\r\n```sh\r\n$ hc resource --match regexp https://httpbin.org/ \"forkme.*?\\.png\" \u003e ~out.png\r\n```\r\n\r\n## Save a screenshot of a web page\r\n\r\nMake screenshot of a web page and save it to `out.png`:\r\n\r\n```sh\r\nhc screenshot \"http://example.com/\" \u003eout.png\r\n```\r\n\r\nWhen rendering the page, viewport size is set to 1024x768 by default. The final\r\ndimensions of the screenshot are determined by the page content, but you can\r\ncontrol the initial size to imitate different devices:\r\n\r\n```sh\r\nhc screenshot --initial-width 800 --initial-height 600 \"http://example.com/\" \u003eout.png\r\n```\r\n\r\nIn the command above the initial viewport size is set to 800x600 prior to\r\nrendering the page.\r\n\r\nIn addition to limiting the initial viewport size, there's an option to limit\r\nthe maximum viewport size:\r\n\r\n```sh\r\nhc screenshot --max-width 1000 --max-height 1000 \"http://example.com/\" \u003eout.png\r\n```\r\n\r\nHere the maximum viewport size is limited to 1000x1000px. If the content doesn't fit\r\nin this viewport, scrollbars will appear on the screenshot.\r\n\r\nWhen no maximum height or width are defined, the viewport size will be adjusted\r\nto accommodate the content so that an entire page is captured without scrollbars.\r\n\r\n# Feedback\r\n\r\nFeel free to provide your feedback, suggestions or bug reports here in the \u003ca href=\"https://github.com/iafan/hc/issues\"\u003ebug tracker\u003c/a\u003e, or message [@afan](https://gophers.slack.com/messages/@afan/) in the [Gophers Slack channel](https://gophersinvite.herokuapp.com/).\r\n\r\n# Credits\r\n\r\n1. `godet` library (Remote client for Chrome DevTools): Copyright (c) 2017 Raffaele Sena [[link](https://github.com/raff/godet)]\r\n2. `justinribeiro/chrome-headless` (Headless Chrome Docker image): Copyright (c) 2015 Justin Ribeiro [[link](https://hub.docker.com/r/justinribeiro/chrome-headless/)]\r\n3. `chrome.json` seccomp descriptor file: Copyright (c) 2015 Jessie Frazelle\r\n   [[link](https://github.com/jessfraz/dotfiles/blob/master/etc/docker/seccomp/chrome.json)]\r\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fiafan%2Fhc","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fiafan%2Fhc","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fiafan%2Fhc/lists"}