{"id":26650558,"url":"https://github.com/gaojiuli/toapi","last_synced_at":"2025-03-25T02:01:43.867Z","repository":{"id":39761331,"uuid":"112169994","full_name":"elliotgao2/toapi","owner":"elliotgao2","description":"Every web site provides APIs.","archived":false,"fork":false,"pushed_at":"2022-07-05T22:11:04.000Z","size":1777,"stargazers_count":3499,"open_issues_count":9,"forks_count":235,"subscribers_count":77,"default_branch":"master","last_synced_at":"2024-10-29T17:39:37.396Z","etag":null,"topics":["api","crawler","flask","html","json","python","spider","toapi","web"],"latest_commit_sha":null,"homepage":"https://gaojiuli.github.io/toapi/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/elliotgao2.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-11-27T08:40:55.000Z","updated_at":"2024-10-19T11:37:17.000Z","dependencies_parsed_at":"2022-08-09T15:25:34.681Z","dependency_job_id":null,"html_url":"https://github.com/elliotgao2/toapi","commit_stats":null,"previous_names":["elliotgao2/toapi","gaojiuli/toapi"],"tags_count":4,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/elliotgao2%2Ftoapi","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/elliotgao2%2Ftoapi/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/elliotgao2%2Ftoapi/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/elliotgao2%2Ftoapi/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/elliotgao2","download_url":"https://codeload.github.com/elliotgao2/toapi/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245383037,"owners_count":20606265,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["api","crawler","flask","html","json","python","spider","toapi","web"],"created_at":"2025-03-25T02:01:42.136Z","updated_at":"2025-03-25T02:01:43.843Z","avatar_url":"https://github.com/elliotgao2.png","language":"Python","readme":"# Toapi\n\n[![Build](https://travis-ci.org/gaojiuli/toapi.svg?branch=master)](https://travis-ci.org/gaojiuli/toapi)\n[![Coverage](https://codecov.io/gh/gaojiuli/toapi/branch/master/graph/badge.svg)](https://codecov.io/gh/gaojiuli/toapi)\n[![Python](https://img.shields.io/pypi/pyversions/toapi.svg)](https://pypi.python.org/pypi/toapi/)\n[![Version](https://img.shields.io/pypi/v/toapi.svg)](https://pypi.python.org/pypi/toapi/)\n[![License](https://img.shields.io/pypi/l/toapi.svg)](https://pypi.python.org/pypi/toapi/)\n\n## Overview\n\nToapi give you the ability to make every web site provides APIs.\n\n- v1.0.0 Documentation: [http://www.toapi.org](http://www.toapi.org)\n- Awesome: [https://github.com/toapi/awesome-toapi](https://github.com/toapi/awesome-toapi)\n- Organization: [https://github.com/toapi](https://github.com/toapi)\n\n## Features\n\n- Automatic converting HTML web site to API service.\n- Automatic caching every page of source site.\n- Automatic caching every request.\n- Support merging multiple web sites into one API service. \n\n## Get Started\n\n### Installation\n\n```text\n$ pip install toapi\n```\n\n### Usage\n\ncreate `app.py` and copy the code:\n\n```python\nfrom flask import request\nfrom htmlparsing import Attr, Text\nfrom toapi import Api, Item\n\napi = Api()\n\n\n@api.site('https://news.ycombinator.com')\n@api.list('.athing')\n@api.route('/posts?page={page}', '/news?p={page}')\n@api.route('/posts', '/news?p=1')\nclass Post(Item):\n    url = Attr('.storylink', 'href')\n    title = Text('.storylink')\n\n\n@api.site('https://news.ycombinator.com')\n@api.route('/posts?page={page}', '/news?p={page}')\n@api.route('/posts', '/news?p=1')\nclass Page(Item):\n    next_page = Attr('.morelink', 'href')\n\n    def clean_next_page(self, value):\n        return api.convert_string('/' + value, '/news?p={page}', request.host_url.strip('/') + '/posts?page={page}')\n\n\napi.run(debug=True, host='0.0.0.0', port=5000)\n```\n\nrun `python app.py`\n\nthen open your browser and visit `http://127.0.0.1:5000/posts?page=1` \n\nyou will get the result like:\n\n```json\n{\n  \"Page\": {\n    \"next_page\": \"http://127.0.0.1:5000/posts?page=2\"\n  }, \n  \"Post\": [\n    {\n      \"title\": \"Mathematicians Crack the Cursed Curve\", \n      \"url\": \"https://www.quantamagazine.org/mathematicians-crack-the-cursed-curve-20171207/\"\n    }, \n    {\n      \"title\": \"Stuffing a Tesla Drivetrain into a 1981 Honda Accord\", \n      \"url\": \"https://jalopnik.com/this-glorious-madman-stuffed-a-p85-tesla-drivetrain-int-1823461909\"\n    }\n  ]\n}\n```\n\n\n## Contributing\n\nWrite code and test code and pull request.\n\n\n\n","funding_links":[],"categories":["Web Content Extracting","资源列表","Web内容提取","Python","Python (1887)","网络","Web Content Extracting [🔝](#readme)","Awesome Python"],"sub_categories":["网络","Web Content Extracting"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgaojiuli%2Ftoapi","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgaojiuli%2Ftoapi","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgaojiuli%2Ftoapi/lists"}