{"id":13580962,"url":"https://github.com/holsee/chroxy","last_synced_at":"2025-04-05T00:08:36.725Z","repository":{"id":38428512,"uuid":"131710102","full_name":"holsee/chroxy","owner":"holsee","description":"Headless Chrome as a Service","archived":false,"fork":false,"pushed_at":"2021-01-20T10:51:59.000Z","size":340,"stargazers_count":258,"open_issues_count":8,"forks_count":26,"subscribers_count":6,"default_branch":"master","last_synced_at":"2025-03-28T23:07:06.572Z","etag":null,"topics":["browser-automation","chrome","elixir","headless-chrome"],"latest_commit_sha":null,"homepage":"https://hex.pm/packages/chroxy","language":"Elixir","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/holsee.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-05-01T12:22:26.000Z","updated_at":"2025-02-11T14:28:17.000Z","dependencies_parsed_at":"2022-09-02T05:23:19.020Z","dependency_job_id":null,"html_url":"https://github.com/holsee/chroxy","commit_stats":null,"previous_names":[],"tags_count":13,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/holsee%2Fchroxy","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/holsee%2Fchroxy/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/holsee%2Fchroxy/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/holsee%2Fchroxy/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/holsee","download_url":"https://codeload.github.com/holsee/chroxy/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247266564,"owners_count":20910836,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["browser-automation","chrome","elixir","headless-chrome"],"created_at":"2024-08-01T15:01:56.813Z","updated_at":"2025-04-05T00:08:36.707Z","avatar_url":"https://github.com/holsee.png","language":"Elixir","funding_links":[],"categories":["Elixir"],"sub_categories":[],"readme":"![](logo.png)\n\n# Chroxy [![Build Status](https://travis-ci.org/holsee/chroxy.svg?branch=master)](https://travis-ci.org/holsee/chroxy)\n\nA proxy service to mediate access to Chrome that is run in headless mode,\nfor use in high-frequency application load testing, end-user behaviour\nsimulations and programmatic access to Chrome Devtools.\n\nEnables automatic initialisation of the underlying chrome browser pages upon the\nrequest for a connection, as well as closing the page once the WebSocket\nconnection is closed.\n\nThis project was born out of necessity, as we needed to orchestrate a large\nnumber of concurrent browser scenario executions, with low-level control and\nadvanced introspection capabilities.\n\n## Versions\n\nElixir: 1.8+\nOTP: 21.3+\n\nSee `.travis.yml` for complete list of supported versions.\n\n## Features\n\n* Direct WebSocket connections to chrome pages, speaking [Chrome Remote Debug\nprotocol](https://chromedevtools.github.io/devtools-protocol/).\n* Provides connections to Chrome Browser Pages via WebSocket connection.\n* Manages Chrome Browser process via Erlang processes using `erlexec`\n  * OS Process supervision and resiliency through automatic restart on crash.\n* Uses Chrome Remote Debugging Protocol for optimal client compatibility.\n* Transparent Dynamic Proxy provides automatic resource cleanup.\n\n## Cowboy Compatibility\n\nCowboy is a major dependency of Phoenix, as such here is a little notice as to\nwhich versions of cowboy are hard dependencies of Chroxy. This notice will be\nremoved at version 1.0 of Chroxy.\n\n`Cowboy 1.x` \u003c= version 0.5.1\n`Cowboy 2.x` \u003e version 0.6.0\n`Cowboy 2.8+` \u003e version 0.7.0\n\n## Project Goals\n\nThe objective of this project is to enable connections to headless chrome\ninstances with *minimal overhead and abstractions*.  Unlike browser testing\nframeworks such as `Hound` and `Wallaby`, Chroxy aims to provide direct\nunfettered access to the underlying browser using the [Chrome Debug\nprotocol](https://chromedevtools.github.io/devtools-protocol/) whilst\nenabling many 1000s of concurrent connections channelling these to an underlying\nchrome browser resource pool.\n\n### Elixir Supervision of Chrome OS Processes - Resiliency\n\nChroxy uses Elixir processes and OTP supervision to manage the chrome instances,\nas well as including a transparent proxy to facilitate automatic initialisation\nand termination of the underlying chrome page based on the upstream connection\nlifetime.\n\n## Getting Started\n\n_Get dependencies and compile:_\n```\n$ mix do deps.get, compile\n```\n\n_Run the Chroxy Server:_\n```\n$ mix run --no-halt\n```\n\n_Run with an attached session:_\n```\n$ iex -S mix\n```\n\n_Run Docker image_\n\nNote: Chrome required a bump in shared memory allocation when running within\ndocker in order to function in a stable manner.\n\nExposes 1330, and 1331 (default ports for connection api and chrome proxy endpoint).\n```\ndocker build . -t chroxy\ndocker run --shm-size 2G -p 1330:1330 -p 1331:1331 chroxy\n```\n\n## Operation Examples:\n\n### Using [Chroxy Client](https://github.com/holsee/chroxy_client) \u0026 `ChromeRemoteInterface`\n\n_Establish 100 Browser Connections:_\n``` elixir\nclients = Enum.map(1..100, fn(_) -\u003e\n  ChroxyClient.page_session!(%{host: \"localhost\", port: 1330})\nend)\n```\n\n_Run 100 Asynchronous browser operations:_\n``` elixir\nTask.async_stream(clients, fn(client) -\u003e\n  url = \"https://github.com/holsee\"\n  {:ok, _} = ChromeRemoteInterface.RPC.Page.navigate(client, %{url: url})\nend, timeout: :infinity) |\u003e Stream.run\n```\n\nYou can then use any `Page` related functionality using\n`ChromeRemoteInterface`.\n\n### Use any client that speaks Chrome Debug Protocol:\n\n_Get the address for a connection:_\n```\n$ curl http://localhost:1330/api/v1/connection\n\nws://localhost:1331/devtools/page/2CD7F0BC05863AB665D1FB95149665AF\n```\nWith this address you can establish the connection to the chrome instance (which\nis routed via a transparent proxy).\n\n## Configuration\n\nThe configuration is designed to be friendly for containerisation as such uses\nenvironment variables\n\n### Chroxy as a Library\n\n``` elixir\ndef deps do\n  [{:chroxy, \"~\u003e 0.3\"}]\nend\n```\n\nIf using Chroxy as a dependency of another mix projects you may wish to leverage\nthe configuration implementation of Chroxy by replication the configuration in\n`\"../deps/chroxy/config/config.exs\"`.\n\nExample: Create a Page Session, Registering for Event and Navigating to URL\n\n``` elixir\nws_addr = Chroxy.connection()\n{:ok, page} = ChromeRemoteInterface.PageSession.start_link(ws_addr)\nChromeRemoteInterface.RPC.Page.enable(page)\nChromeRemoteInterface.PageSession.subscribe(page, \"Page.loadEventFired\", self())\nurl = \"https://github.com/holsee\"\n{:ok, _} = ChromeRemoteInterface.RPC.Page.navigate(page, %{url: url})\n# Message Received by self() =\u003e {:chrome_remote_interface, \"Page.loadEventFired\", _}\n```\n\n### Configuration Variables\n\nPorts, Proxy Host and Endpoint Scheme are managed via Env Vars.\n\n| Variable                             | Default       | Desc.                                                      |\n| :------------------------            | :------------ | :--------------------------------------------------------- |\n| CHROXY_CHROME_PORT_FROM              | 9222          | Starting port in the Chrome Browser port range             |\n| CHROXY_CHROME_PORT_TO                | 9223          | Last port in the Chrome Browser port range                 |\n| CHROXY_PROXY_HOST                    | \"127.0.0.1\"   | Host which is substituted to route connections via proxy   |\n| CHROXY_PROXY_PORT                    | 1331          | Port which proxy listener will accept connections on       |\n| CHROXY_ENDPOINT_SCHEME               | :http         | `HTTP` or `HTTPS`                                          |\n| CHROXY_ENDPOINT_PORT                 | 1330          | HTTP API will register on this port                        |\n| CHROXY_CHROME_SERVER_PAGE_WAIT_MS    | 200           | Milliseconds to wait after asking chrome to create a page  |\n| CHROME_CHROME_SERVER_CRASH_DUMPS_DIR | \"/tmp\"        | Directory to which chrome will write crash dumps           |\n\n## Components\n\n### Proxy\n\nAn intermediary TCP proxy is in place to allow for monitoring of the _upstream_\nclient and _downstream_ chrome RSP web socket connections, in order to clean up\nresources after connections are closed.\n\n`Chroxy.ProxyListener` - Incoming Connection Management \u0026 Delegation\n* Listens for incoming connections on `CHROXY_PROXY_HOST`:`CHROXY_PROXY_PORT`.\n* Exposes `accept/1` function which will accept the next _upstream_ TCP connection and\n  delegate the connection to a `ProxyServer` process along with the `proxy_opts`\n  which enables the dynamic configuration of the _downstream_ connection.\n\n`Chroxy.ProxyServer` - Dynamically Configured Transparent Proxy\n* A dynamically configured transparent proxy.\n* Manages delegated connection as the _upstream_ connection.\n* Establishes _downstream_ connection based on `proxy_opts` or\n  `ProxyServer.Hook.up/2` hook modules response, at initialisation.\n\n`Chroxy.ProxyServer.Hook` - Behaviour for `ProxyServer` hooks. Example: `ChromeProxy`\n* A mechanism by which a module/server can be invoked when a `ProxyServer`\n  process is coming _up_ or _down_.\n* Two _optional_ callbacks can be implemented:\n  * `@spec up(indentifier(), proxy_opts()) :: proxy_opts()`\n    * provides the registered process with the option to add or change proxy\n      options prior to downstream connection initialisation.\n  * `@spec down(indentifier(), proxy_state) :: :ok`\n    * provides the registered process with a signal that the proxy connection\n       is about to terminate, due to either _upstream_ or _downstream_\n       connections closing.\n\n### Chrome Browser Management\n\nChrome is the first browser supported, and the following server processes manage\nthe communication and lifetime of the Chrome Browsers and Tabs.\n\n`Chroxy.ChromeProxy` - Implements `ProxyServer.Hook` for Chrome resource management\n* Exposes function `connection/1` which returns the websocket connection\n    to the browser tab, with the proxy host and port substituted in order to\n   route the connection via the underlying `ProxyServer` process.\n* Registers for callbacks from the underlying `ProxyServer`, implementing the\n  `down/2` callback in order to clean up the Chrome resource when connections\n  close.\n\n`Chroxy.ChromeServer` - Wraps Chrome Browser OS Process\n* Process which manages execution and control of a Chrome Browser OS process.\n* Provides basic API wrapper to manage the required browser level functionality\n  around page creation, access and closing.\n* Translates browser logging to elixir logging, with correct levels.\n\n`Chroxy.BrowserPool` - Inits \u0026 Controls access to pool of browser processes\n* Exposes `connection/0` function which will return a WebSocket connection to a\n  browser tab, from a random browser process in the managed pool.\n\n`Chroxy.BrowerPool.Chrome` - Chrome Process Pool\n* Manages `ChromeServer` process pool, responsible for spawning a browser\n  process for each defined PORT in the port range configured.\n\n### HTTP API - `Chroxy.Endpoint`\n\n`GET /api/v1/connection`\n\nReturns WebSocket URI `ws://` to a Chrome Browser Page which is routed via the\nProxy.  This is the first port of call for an external client connecting to the service.\n\nRequest:\n```\n$ curl http://localhost:1330/api/v1/connection\n```\nResponse:\n```\nws://localhost:1331/devtools/page/2CD7F0BC05863AB665D1FB95149665AF\n```\n\n## Kubernetes\n\nThe following is an example configuration which can be used to run Chroxy on\nKubernetes.\n\ndeployment.yaml\n```yaml\napiVersion: extensions/v1beta1\nkind: Deployment\nmetadata:\n  name: crawler\n  namespace: default\n  labels:\n    app: myApp\n    tier: crawler\n\nspec:\n  replicas: 1\n  revisionHistoryLimit: 1\n  strategy:\n    type: RollingUpdate\n    rollingUpdate:\n      maxUnavailable: 0\n      maxSurge: 1\n  selector:\n    matchLabels:\n      app: myApp\n      tier: crawler\n  template:\n    metadata:\n      labels:\n        app: myApp\n        tier: crawler\n    spec:\n      containers:\n        - image: eu.gcr.io/..../...:latest # your consumer\n          name: api\n          imagePullPolicy: Always\n          resources:\n            requests:\n              cpu: 30m\n              memory: 100Mi\n          ports:\n            - containerPort: 4000\n          env:\n          - name: USER_AGENT\n            value: ...\n          - name: INSTANCE_NAME\n            valueFrom:\n              fieldRef:\n                fieldPath: metadata.name\n\n        # [START chroxy]\n        - name: headless-chrome\n          image: eu.gcr.io/..../chroxy:latest # chroxy\n          imagePullPolicy: Always\n          resources:\n            requests:\n              cpu: 30m\n              memory: 100Mi\n          env:\n            - name: CHROXY_CHROME_PORT_FROM\n              value: \"9222\"\n            - name: CHROXY_CHROME_PORT_TO\n              value: \"9223\"\n          ports:\n            - containerPort: 1331\n            - containerPort: 1330\n        # [END chroxy]\n```\n\nservice.yaml\n```yaml\napiVersion: v1\nkind: Service\nmetadata:\n  namespace: default\n  name: crawler-api\n  labels:\n    app: myApp\n    tier: crawler\nspec:\n  selector:\n    app: myApp\n    tier: crawler\n  ports:\n  - port: 4000\n    protocol: TCP\n```\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fholsee%2Fchroxy","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fholsee%2Fchroxy","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fholsee%2Fchroxy/lists"}