{"id":21006208,"url":"https://github.com/derlin/get-html","last_synced_at":"2025-07-13T23:34:57.152Z","repository":{"id":57433991,"uuid":"244017645","full_name":"derlin/get-html","owner":"derlin","description":"python GET raw or rendered HTML (for humans)","archived":false,"fork":false,"pushed_at":"2020-07-17T09:10:28.000Z","size":696,"stargazers_count":13,"open_issues_count":0,"forks_count":1,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-04-18T13:38:16.409Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/derlin.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-02-29T18:09:52.000Z","updated_at":"2024-06-22T02:46:04.000Z","dependencies_parsed_at":"2022-08-27T22:31:16.508Z","dependency_job_id":null,"html_url":"https://github.com/derlin/get-html","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/derlin%2Fget-html","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/derlin%2Fget-html/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/derlin%2Fget-html/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/derlin%2Fget-html/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/derlin","download_url":"https://codeload.github.com/derlin/get-html/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254256750,"owners_count":22040351,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-19T08:49:55.592Z","updated_at":"2025-05-15T01:33:46.526Z","avatar_url":"https://github.com/derlin.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# get-html: get raw or rendered HTML (for humans)\n\n**\u003cp align=center\u003eRead all the details on how I started implementing this at ⇝\n[Rendering-HTML_a-journey](https://github.com/derlin/get-html/blob/master/blog/Rendering-HTML_a-journey.md) ⇜ \u003c/p\u003e**\n\nThis module is made for anyone needing to scrape HTML (i.e. scrape the web).\n\nIt knows how to do only one thing, but does it well: getting HTML from a web page. The kind of HTML is up to you. Either:\n\n* the **raw HTML**, directly from the server or,\n* the **rendered HTML**, after JS/AJAX calls.\n\nMoreover, it is made such that you can use a unique method throughout your project, \nand switch between the two behaviors at launch by setting an environment variable.\n\n## HtmlRenderer: a class to seamlessly render a page\n\n`HtmlRenderer` handles all the specific `pyppeteer` stuff for you. It is also thread-safe.\n\n### \"sync\" usage \n\nHere is a **typical usage**:\n\n```python\nfrom get_html import HtmlRenderer\n\nrenderer = HtmlRenderer()\ntry:  \n    # use the renderer. The underlying browser will be instantiated on first call to render.\n    response = renderer.render(url='https://xkcd.com')\n    html = response.text # or resposne.content to get the raw bytes\n    # ... etc.\nfinally:\n    # close the underlying browser\n    renderer.close()\n```\n\nOr simply use a **context manager**:\n```python\nfrom get_html import create_renderer\n\nwith create_renderer() as renderer:  \n    # use the renderer. The underlying browser will be instanciated on first call to render\n    response = renderer.render(url='https://xkcd.com')\n    html = response.text # or resposne.content to get the raw bytes\n\n# here, the underlying browser will be closed\n```\n\nIf you need to **manipulate the page** before getting the content, pass an *async* function to `render`.\nIt will be called after the page loaded, but before the HTML content is fetched.\nFor example:\n```python\nfrom get_html import create_renderer\n\nasync def scroll_to_end(page):\n    # https://github.com/miyakogi/pyppeteer/issues/205#issuecomment-470886682\n    await page.evaluate('{window.scrollBy(0, document.body.scrollHeight);}')\n\nwith create_renderer() as renderer: \n    response = renderer.render('https://9gag.com', manipulate_page_func=scroll_to_end)\n```\n\n### \"async\" usage \n\nAll public methods have an *async* counterpart. When using *async*, however, you need to ensure that\n\n1. the browser is created once,\n2. the browser is closed once.\n \nBy default, the browser is launched upon first use, usually on the first call of `render`. \nWhen using *async*, this may be a problem, as multiple coroutine will try to create the browser multiple times.\nTo avoid this, ensure you trigger the browser creation *before* launching the other tasks (same for closing, wait for all tasks to complete).\n\nA concrete example is available in\n[examples/async_example.py](https://github.com/derlin/blob/master/examples/async_example.py). Here is the gist:\n\n```python\nimport asyncio\nfrom get_html import HtmlRenderer\n\nloop = asyncio.get_event_loop()\nrenderer = HtmlRenderer()\n\n# trigger the browser creation only once, before the tasks\nloop.run_until_complete(renderer.async_browser)\n\n# .. TASKS WITH RENDERING CALLS ... \n#    e.g. loop.run_until_complete(someRenderingTask())\n\n# finally close the browser once the tasks completed\nloop.run_until_complete(renderer.async_close())\n```\n\n## do_get: seemlessly switch between behaviors\n\n```python\nfrom get_html.env_defined_get import do_get\n\nresponse = do_get('https://xkcd.com')\nassert response.status_code == 200\n\nhtml_bytes = response.content\nhtml_string = response.text\n```\n\nThe actual behavior of `do_get` will depend on the environment variable `RENDER_HTML`:\n\n* `RENDER_HTML=[1|y|true|on]`: `do_get` will launch a chromium instance under the hood and render the page (rendered HTML)\n* `RENDER_HTML=\u003canything BUT 2\u003e` (default): `do_get` will forward the call to `requests.get` (raw HTML). \n  Do **NOT** use `2` before reading through the multi-threading section.\n\nIf rendering support is on, a browser instance will be launched **on module load**, and will be kept alive throughout the life of the application.\nKeep that in mind if you have low-memory (chromium !!).\n\n## Multi-threading\n\n`HtmlRenderer` is thread-safe.\n\nFor `do_get` with rendering support, there are two possibilities.\n\n1. Create **only one browser**, shared by all threads. In this case, only one thread can execute `render` at a time (locking mechanism);\n2. Create **one browser per thread**. In this case, threads can render in parallel. But be careful, each time a new thread calls `do_get`,\n  *a new browser is launched*, that will keep running until the end of the program (or until you call `get_html.env_defined_get.close()`).\n\nEnable mode (2) by setting `RENDER_HTML=2`. But again, ensure you don't have too many threads, since chromium needs a lot of memory.\n\n## Running tests\n\nOn Windows/Linux:\n```bash\npip install tox\ntox\n```\n\nOn Mac (see https://github.com/tox-dev/tox/issues/1485):\n```bash\npip install \"tox\u003c3.7\"\ntox\n```\n\n## Common errors\n\n**`libXX not found` (Linux)**\n\nSee [issue #290 of puppeteer](https://github.com/puppeteer/puppeteer/issues/290#issuecomment-480247488)\n\n```bash\nsudo apt-get install gconf-service libasound2 libatk1.0-0 libatk-bridge2.0-0 libc6 libcairo2 libcups2 libdbus-1-3 libexpat1 libfontconfig1 libgcc1 libgconf-2-4 libgdk-pixbuf2.0-0 libglib2.0-0 libgtk-3-0 libnspr4 libpango-1.0-0 libpangocairo-1.0-0 libstdc++6 libx11-6 libx11-xcb1 libxcb1 libxcomposite1 libxcursor1 libxdamage1 libxext6 libxfixes3 libxi6 libxrandr2 libxrender1 libxss1 libxtst6 ca-certificates fonts-liberation libappindicator1 libnss3 lsb-release xdg-utils wget\n``` ","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fderlin%2Fget-html","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fderlin%2Fget-html","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fderlin%2Fget-html/lists"}