{"id":13464683,"url":"https://github.com/matiasb/demiurge","last_synced_at":"2025-03-25T11:32:03.961Z","repository":{"id":16177246,"uuid":"18923652","full_name":"matiasb/demiurge","owner":"matiasb","description":"PyQuery-based scraping micro-framework.","archived":false,"fork":false,"pushed_at":"2022-01-14T01:34:49.000Z","size":30,"stargazers_count":116,"open_issues_count":2,"forks_count":19,"subscribers_count":11,"default_branch":"master","last_synced_at":"2025-03-04T15:05:07.927Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"http://demiurge.readthedocs.org","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/matiasb.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2014-04-18T19:47:29.000Z","updated_at":"2025-01-21T16:53:02.000Z","dependencies_parsed_at":"2022-09-01T20:40:44.191Z","dependency_job_id":null,"html_url":"https://github.com/matiasb/demiurge","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/matiasb%2Fdemiurge","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/matiasb%2Fdemiurge/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/matiasb%2Fdemiurge/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/matiasb%2Fdemiurge/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/matiasb","download_url":"https://codeload.github.com/matiasb/demiurge/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245454118,"owners_count":20617982,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-07-31T14:00:48.583Z","updated_at":"2025-03-25T11:31:58.888Z","avatar_url":"https://github.com/matiasb.png","language":"Python","funding_links":[],"categories":["Web Crawling \u0026 Web Scraping","All","资源列表","Python","Web Crawling","Awesome Python"],"sub_categories":["HTML 处理","Web Crawling \u0026 Web Scraping"],"readme":"demiurge\n========\n\nPyQuery-based scraping micro-framework.\nSupports Python 2.x and 3.x.\n\n[![Build Status](https://travis-ci.org/matiasb/demiurge.png?branch=master)](https://travis-ci.org/matiasb/demiurge)\n\nDocumentation: http://demiurge.readthedocs.org\n\n\nInstalling demiurge\n-------------------\n\n```\n$ pip install demiurge\n```\n\nQuick start\n-----------\n\nDefine items to be scraped using a declarative (Django-inspired) syntax:\n\n```python\nimport demiurge\n\nclass TorrentDetails(demiurge.Item):\n    label = demiurge.TextField(selector='strong')\n    value = demiurge.TextField()\n\n    def clean_value(self, value):\n        unlabel = value[value.find(':') + 1:]\n        return unlabel.strip()\n\n    class Meta:\n        selector = 'div#specifications p'\n\nclass Torrent(demiurge.Item):\n    url = demiurge.AttributeValueField(\n        selector='td:eq(2) a:eq(1)', attr='href')\n    name = demiurge.TextField(selector='td:eq(2) a:eq(2)')\n    size = demiurge.TextField(selector='td:eq(3)')\n    details = demiurge.RelatedItem(\n        TorrentDetails, selector='td:eq(2) a:eq(2)', attr='href')\n\n    class Meta:\n        selector = 'table.maintable:gt(0) tr:gt(0)'\n        base_url = 'http://www.mininova.org'\n\n\n\u003e\u003e\u003e t = Torrent.one('/search/ubuntu/seeds')\n\u003e\u003e\u003e t.name\n'Ubuntu 7.10 Desktop Live CD'\n\u003e\u003e\u003e t.size\nu'695.81\\xa0MB'\n\u003e\u003e\u003e t.url\n'/get/1053846'\n\u003e\u003e\u003e t.html\nu'\u003ctd\u003e19\\xa0Dec\\xa007\u003c/td\u003e\u003ctd\u003e\u003ca href=\"/cat/7\"\u003eSoftware\u003c/a\u003e\u003c/td\u003e\u003ctd\u003e...'\n\n\u003e\u003e\u003e results = Torrent.all('/search/ubuntu/seeds')\n\u003e\u003e\u003e len(results)\n116\n\u003e\u003e\u003e for t in results[:3]:\n...     print t.name, t.size\n...\nUbuntu 7.10 Desktop Live CD 695.81 MB\nSuper Ubuntu 2008.09 - VMware image 871.95 MB\nPortable Ubuntu 9.10 for Windows 559.78 MB\n...\n\n\u003e\u003e\u003e t = Torrent.one('/search/ubuntu/seeds')\n\u003e\u003e\u003e for detail in t.details:\n...     print detail.label, detail.value\n... \nCategory: Software \u003e GNU/Linux\nTotal size: 695.81 megabyte\nAdded: 2467 days ago by Distribution\nShare ratio: 17 seeds, 2 leechers\nLast updated: 35 minutes ago\nDownloads: 29,085\n```\n\nSee documentation for details: http://demiurge.readthedocs.org\n\n\nWhy *demiurge*?\n---------------\n\nPlato, as the speaker Timaeus, refers to the Demiurge frequently in the Socratic\ndialogue Timaeus, c. 360 BC. The main character refers to the Demiurge as the\nentity who \"fashioned and shaped\" the material world. Timaeus describes the\nDemiurge as unreservedly benevolent, and hence desirous of a world as good as\npossible. The world remains imperfect, however, because the Demiurge created\nthe world out of a chaotic, indeterminate non-being.\n\nhttp://en.wikipedia.org/wiki/Demiurge\n\n\nContributors\n------------\n\n - Martín Gaitán (@mgaitan)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmatiasb%2Fdemiurge","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmatiasb%2Fdemiurge","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmatiasb%2Fdemiurge/lists"}