{"id":13937708,"url":"https://github.com/cantabular/scraperwiki-python","last_synced_at":"2026-01-21T13:30:55.631Z","repository":{"id":3027807,"uuid":"4047850","full_name":"cantabular/scraperwiki-python","owner":"cantabular","description":"ScraperWiki Python library for scraping and saving data; in maintenance mode","archived":false,"fork":false,"pushed_at":"2026-01-12T14:04:33.000Z","size":451,"stargazers_count":158,"open_issues_count":4,"forks_count":69,"subscribers_count":20,"default_branch":"main","last_synced_at":"2026-01-14T00:36:43.099Z","etag":null,"topics":["scraper","scraperwiki"],"latest_commit_sha":null,"homepage":"https://scraperwiki.com","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-2-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/cantabular.png","metadata":{"files":{"readme":"README.rst","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2012-04-17T01:39:05.000Z","updated_at":"2026-01-12T14:04:36.000Z","dependencies_parsed_at":"2026-01-12T13:03:50.708Z","dependency_job_id":null,"html_url":"https://github.com/cantabular/scraperwiki-python","commit_stats":null,"previous_names":["scraperwiki/scraperwiki-python","cantabular/scraperwiki-python","sensiblecodeio/scraperwiki-python"],"tags_count":6,"template":false,"template_full_name":null,"purl":"pkg:github/cantabular/scraperwiki-python","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cantabular%2Fscraperwiki-python","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cantabular%2Fscraperwiki-python/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cantabular%2Fscraperwiki-python/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cantabular%2Fscraperwiki-python/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/cantabular","download_url":"https://codeload.github.com/cantabular/scraperwiki-python/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cantabular%2Fscraperwiki-python/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28633760,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-21T04:47:28.174Z","status":"ssl_error","status_checked_at":"2026-01-21T04:47:22.943Z","response_time":86,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["scraper","scraperwiki"],"created_at":"2024-08-07T23:03:47.951Z","updated_at":"2026-01-21T13:30:55.624Z","avatar_url":"https://github.com/cantabular.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"ScraperWiki Python library\n==========================\n\nThis is a Python library for scraping web pages and saving data.\n\n**Warning: This library is now in maintenance mode.**\n\n**The library has been updated to work with Python 3.14 but there are no guarantees on future maintenance.**\n\nInstalling\n----------\n\n::\n\n   pip install scraperwiki\n\nScraping\n--------\n\nscraperwiki.scrape(url[, params][,user_agent])\n  Returns the downloaded string from the given url.\n\n  ``params`` are sent as a POST if set.\n\n  ``user_agent`` sets the user-agent string if provided.\n\nSaving data\n-----------\n\nHelper functions for saving and querying an SQL database. Updates\nthe schema automatically according to the data you save.\n\nCurrently only supports SQLite. It will make a local SQLite database.\nIt is based on `SQLAlchemy \u003chttps://pypi.python.org/pypi/SQLAlchemy\u003e`_.\n\nscraperwiki.sql.save(unique_keys, data[, table_name=\"swdata\"])\n  Saves a data record into the datastore into the table given by ``table_name``.\n\n  ``data`` is a dict object with field names as keys; ``unique_keys`` is a subset of data.keys() which determines when a record is overwritten. For large numbers of records `data` can be a list of dicts.\n\n  ``scraperwiki.sql.save`` is entitled to buffer an arbitrary number of\n  rows until the next read via the ScraperWiki API, an exception is hit,\n  or until process exit. An effort is made to do a timely periodic flush.\n  Records can be lost if the process experiences a hard-crash, power outage\n  or SIGKILL due to high memory usage during an out-of-memory condition. The\n  buffer can be manually flushed with ``scraperwiki.sql.flush()``.\n\nscraperwiki.sql.execute(sql[, vars])\n  Executes any arbitrary SQL command. For example CREATE, DELETE, INSERT or DROP.\n\n  ``vars`` is an optional list of parameters, inserted when the SQL command contains ‘?’s. For example::\n\n    scraperwiki.sql.execute(\"INSERT INTO swdata VALUES (?,?,?)\", [a,b,c])\n\n  The ‘?’ convention is like \"paramstyle qmark\" from `Python's DB API 2.0 \u003chttp://www.python.org/dev/peps/pep-0249/\u003e`_ (but note that the API to the datastore is nothing like Python's DB API).  In particular the ‘?’ does not itself need quoting, and can in general only be used where a literal would appear. (Note that you cannot substitute in, for example, table or column names.)\n\nscraperwiki.sql.select(sqlfrag[, vars])\n  Executes a select command on the datastore.  For example::\n\n    scraperwiki.sql.select(\"* FROM swdata LIMIT 10\")\n\n  Returns a list of dicts that have been selected.\n\n  ``vars`` is an optional list of parameters, inserted when the select command contains ‘?’s.  This is like the feature in the ``.execute`` command, above.\n\nscraperwiki.sql.commit()\n  Functionality now removed. It is retained, but does nothing. (sql.save auto-commits after every action).\n\nscraperwiki.sql.show_tables([dbname])\n  Returns an array of tables and their schemas in the current database.\n\nscraperwiki.sql.save_var(key, value)\n  Saves an arbitrary single-value into a table called ``swvariables``. Intended to store scraper state so that a scraper can continue after an interruption.\n\nscraperwiki.sql.get_var(key[, default])\n  Retrieves a single value that was saved by ``save_var``. Only works for string, float, or int types. For anything else, use the `pickle library \u003chttp://docs.python.org/library/pickle.html\u003e`_ to turn it into a string.\n\nMiscellaneous\n-------------\n\nscraperwiki.status(type, message=None)\n  Functionality now removed since it was only for the now defunct ScraperWiki platform. It is retained but always returns None.\n\nscraperwiki.pdftoxml(pdfdata)\n  Convert a byte string containing a PDF file into an XML file containing the coordinates and font of each text string (see `the pdftohtml documentation \u003chttp://linux.die.net/man/1/pdftohtml\u003e`_ for details). This requires ``pdftohtml`` which is part of ``poppler-utils``. \n\nEnvironment Variables\n---------------------\n\nSCRAPERWIKI_DATABASE_NAME\n  default: ``scraperwiki.sqlite`` - name of database\n\nSCRAPERWIKI_DATABASE_TIMEOUT\n  default: ``300`` - number of seconds database will wait for a lock\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcantabular%2Fscraperwiki-python","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcantabular%2Fscraperwiki-python","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcantabular%2Fscraperwiki-python/lists"}