{"id":17748515,"url":"https://github.com/purarue/url_cache","last_synced_at":"2025-10-29T02:02:53.052Z","repository":{"id":43091062,"uuid":"300495152","full_name":"purarue/url_cache","owner":"purarue","description":"A file system cache which saves URL metadata and summarizes content","archived":false,"fork":false,"pushed_at":"2024-11-14T07:07:42.000Z","size":3184,"stargazers_count":11,"open_issues_count":3,"forks_count":1,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-02-28T10:44:36.094Z","etag":null,"topics":["cache","metadata","opengraph","subtitles","url","youtube"],"latest_commit_sha":null,"homepage":"https://pypi.org/project/url-cache/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/purarue.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-10-02T03:58:18.000Z","updated_at":"2025-02-08T05:19:48.000Z","dependencies_parsed_at":"2023-02-13T20:17:27.452Z","dependency_job_id":"6e7d5298-4312-4fda-8b48-b1bbf73240b8","html_url":"https://github.com/purarue/url_cache","commit_stats":{"total_commits":119,"total_committers":1,"mean_commits":119.0,"dds":0.0,"last_synced_commit":"627225f3dd0d91f8ee564c028ca23e645bd7646c"},"previous_names":["seanbreckenridge/url_metadata","seanbreckenridge/url_cache"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/purarue%2Furl_cache","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/purarue%2Furl_cache/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/purarue%2Furl_cache/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/purarue%2Furl_cache/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/purarue","download_url":"https://codeload.github.com/purarue/url_cache/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243940058,"owners_count":20372044,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cache","metadata","opengraph","subtitles","url","youtube"],"created_at":"2024-10-26T10:01:58.867Z","updated_at":"2025-10-29T02:02:53.047Z","avatar_url":"https://github.com/purarue.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"[![PyPi version](https://img.shields.io/pypi/v/url_cache.svg)](https://pypi.python.org/pypi/url_cache) [![Python3.10|3.11|3.12|3.13](https://img.shields.io/pypi/pyversions/url_cache.svg)](https://pypi.python.org/pypi/url_cache) [![PRs Welcome](https://img.shields.io/badge/PRs-welcome-brightgreen.svg?style=flat-square)](http://makeapullrequest.com)\n\nThis is currently very alpha and in development, so expect changes to the API/interface. It aims to walk the line between extracting enough text/data for it to be useful, but no so much that it takes enormous amounts of space.\n\nAs it stands I'm sort of pessimistic this would ever be a silver bullet, getting useful info out of arbitrary HTML is hard, so you're sort of stuck writing parsers for each website you're interested in. However, I still use this frequently, especially as a cache for API information like described [below](#api-cache-examples)\n\nCurrent TODOs:\n\n- [ ] Add more sites using the [abstract interface](https://github.com/purarue/url_cache/blob/master/src/url_cache/sites/abstract.py), to get more info from sites I use commonly. Ideally, should be able to reuse common scraper/parsers/API interface libraries in python, instead of recreating everything from scratch\n- [ ] Create a (separate repo/project) daemon which handles configuring this and slowly requests things in the background as they become available through given sources; allow user to provide generators/inputs define include/exclude lists/regexes. Probably just integrate with [promnesia](https://github.com/karlicoss/promnesia) so avoid duplicating the work of searching for URLs on disk\n\n## Installation\n\nRequires `python3.10+`\n\nTo install with pip, run:\n\n    python3 -m pip install url_cache\n\nAs this is still in development, for the latest changes install from git: `python3 -m pip install git+https://github.com/purarue/url_cache`\n\n## Rationale\n\nA file system cache which saves URL metadata and summarizes content\n\nThis is meant to provide more context to any of my tools which use URLs. If I [watched some youtube video](https://github.com/purarue/mpv-history-daemon) and I have a URL, I'd like to have the subtitles for it, so I can do a text-search over all the videos I've watched. If I [read an article](https://github.com/purarue/browserexport), I want the article text! This requests, parses and abstracts away that data for me locally, so I can do something like:\n\n```python\n\u003e\u003e\u003e from url_cache.core import URLCache\n\u003e\u003e\u003e u = URLCache()\n\u003e\u003e\u003e data = u.get(\"https://purarue.xyz/\")\n\u003e\u003e\u003e data.metadata[\"images\"][-1]\n{'src': 'https://raw.githubusercontent.com/purarue/glue/master/assets/screenshot.png', 'alt': 'screenshot', 'type': 'body_image', 'width': 600}\n\u003e\u003e\u003e data.metadata[\"description\"]\n\"purarue's Home Page\"\n```\n\nIf I ever request that URL again, the information is grabbed from a local cache instead.\n\nGenerally, this uses:\n\n- [`lassie`](https://github.com/michaelhelmick/lassie) to get generic metadata; the title, description, opengraph information, links to images/videos on the page.\n- [`readability`](https://github.com/buriy/python-readability) to convert/compress HTML to a summary of the HTML content.\n\nSite-Specific Extractors:\n\n- [Youtube](./docs/url_cache/sites/youtube/subtitles_downloader.md): to get manual/auto-generated captions (converted to a `.srt` file) from Youtube URLs\n- Stackoverflow (Just a basic URL preprocessor to reduce the possibility of conflicts/duplicate data)\n- MyAnimeList (using [Jikan v4](https://docs.api.jikan.moe/))\n\nThis is meant to be extendible -- so its possible for you to write your own extractors/file loaders/dumpers (for new formats (e.g. `srt`)) for sites you use commonly and pass those to `url_cache.core.URLCache` to extract richer data for those sites. Otherwise, it saves the information from `lassie` and the summarized HTML using `readability` for each URL.\n\nTo avoid scope creep, this probably won't support:\n\n- Converting the HTML summary to text (use something like the `lynx` command below)\n- Minimizing HTML - run something like `find ~/.local/share/url_cache/ -name '*.html' -exec \u003csome tool/script that minimizes in place\u003e \\;` instead -- the data is just stored in individual files in the data directory\n\n### Usage:\n\nIn Python, this can be configured by using the `url_cache.core.URLCache` class: For example:\n\n```python\nimport logging\nfrom url_cache.core import URLCache\n\n# make requests every 2 seconds\n# debug logs\n# save to a folder in my home directory\ncache = URLCache(loglevel=logging.DEBUG, sleep_time=2, cache_dir=\"~/Documents/urldata\")\nc = cache.get(\"https://github.com/purarue\")\n# just request information, don't read/save to cache\ndata = cache.request_data(\"https://www.wikipedia.org/\")\n```\n\nFor more information, see [the docs](./docs/url_cache/core.md)\n\nThe CLI interface provides some utility commands to get/list information from the cache.\n\n```\nUsage: url_cache [OPTIONS] COMMAND [ARGS]...\n\nOptions:\n  --cache-dir PATH                Override default cache directory location\n  --debug / --no-debug            Increase log verbosity\n  --sleep-time INTEGER            How long to sleep between requests\n  --summarize-html / --no-summarize-html\n                                  Use readability to summarize html. Otherwise\n                                  saves the entire HTML document\n\n  --skip-subtitles / --no-skip-subtitles\n                                  Skip downloading Youtube Subtitles\n  --subtitle-language TEXT        Subtitle language for Youtube Subtitles\n  --help                          Show this message and exit.\n\nCommands:\n  cachedir  Prints the location of the local cache directory\n  export    Print all cached information as JSON\n  get       Get information for one or more URLs Prints results as JSON\n  in-cache  Prints if a URL is already cached\n  list      List all cached URLs\n```\n\nAn environment variable `URL_CACHE_DIR` can be set, which changes the default cache directory.\n\n### API Cache Examples\n\nI've also successfully used this to cache responses from API results in some of my projects, by subclassing and overriding the `request_data` function. I just make a request and return a summary, and it transparently caches the rest. See:\n\n- [`albums/discogs_cache`](https://github.com/purarue/albums/blob/9d296c4abb8e9e16c8dd410aeae8e5bb760008de/nextalbums/discogs_cache.py)\n- [`my_feed/tmdb`](https://github.com/purarue/my_feed/blob/master/src/my_feed/sources/trakt/tmdb.py)\n- [`dbsentinel/metadata`](https://github.com/purarue/dbsentinel/blob/accfc70485644d8966a582204c6c47839d2d874e/mal_id/metadata_cache.py)\n\n### CLI Examples\n\nThe `get` command emits `JSON`, so it could with other tools (e.g. [`jq`](https://stedolan.github.io/jq/)) used like:\n\n```shell\n$ url_cache get \"https://click.palletsprojects.com/en/7.x/arguments/\" | \\\n  jq -r '.[] | .html_summary' | lynx -stdin -dump | head -n 5\nArguments\n\n   Arguments work similarly to [1]options but are positional. They also\n   only support a subset of the features of options due to their\n   syntactical nature. Click will also not attempt to document arguments\n```\n\n```shell\n$ url_cache export | jq -r '.[] | .metadata | .title'\npurarue - Overview\nArguments — Click Documentation (7.x)\n```\n\n```shell\nurl_cache list --location\n/home/username/.local/share/url_cache/data/2/c/7/6284b2f664f381372fab3276449b2/000\n/home/username/.local/share/url_cache/data/7/5/1/70fc230cd88f32e475ff4087f81d9/000\n```\n\n```shell\n# to make a backup of the cache directory\n$ tar -cvzf url_cache.tar.gz \"$(url_cache cachedir)\"\n```\n\nAccessible through the `url_cache` script and `python3 -m url_cache`\n\n### Implementation Notes\n\nThis stores all of this information as individual files in a cache directory. In particular, it `MD5` hashes the URL and stores information like:\n\n```\n.\n└── a\n    └── a\n        └── e\n            └── cf0118bb22340e18fff20f2db8abd\n                └── 000\n                    ├── data\n                    │   └── subtitles.srt\n                    ├── key\n                    ├── metadata.json\n                    └── timestamp.datetime.txt\n```\n\nIn other words, this is a file system hash table which implements separate chaining.\n\nYou're free to delete any of the directories in the cache if you want, this doesn't maintain a strict index, it uses a hash of the URL and then searches for a matching `key` file.\n\nBy default this waits 5 seconds between requests. Since all the info is cached, I use this by requesting all the info from one data source (e.g. my bookmarks, or videos I've watched recently) in a loop in the background, which saves all the information to my computer. The next time I do that same loop, it doesn't have to make any requests and it just grabs all the info from local cache.\n\nOriginally created for [`HPI`](https://github.com/purarue/HPI).\n\n---\n\n### Testing\n\n```\ngit clone 'https://github.com/purarue/url_cache'\ncd ./url_cache\npip install '.[testing]'\nmypy ./src/url_cache\nflake8 ./src/url_cache\npytest\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpurarue%2Furl_cache","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpurarue%2Furl_cache","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpurarue%2Furl_cache/lists"}