{"id":13557354,"url":"https://github.com/dogsheep/dogsheep-photos","last_synced_at":"2025-08-09T16:21:42.198Z","repository":{"id":41376535,"uuid":"256834907","full_name":"dogsheep/dogsheep-photos","owner":"dogsheep","description":"Upload your photos to S3 and import metadata about them into a SQLite database","archived":false,"fork":false,"pushed_at":"2024-01-05T18:52:26.000Z","size":68,"stargazers_count":177,"open_issues_count":24,"forks_count":15,"subscribers_count":10,"default_branch":"master","last_synced_at":"2025-04-02T02:38:17.169Z","etag":null,"topics":["datasette","datasette-io","datasette-tool","dogsheep","sqlite"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dogsheep.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2020-04-18T19:22:13.000Z","updated_at":"2025-02-28T21:18:04.000Z","dependencies_parsed_at":"2024-04-20T04:33:31.199Z","dependency_job_id":"94fac6c0-fcaa-4058-9c49-3538022fb430","html_url":"https://github.com/dogsheep/dogsheep-photos","commit_stats":{"total_commits":41,"total_committers":2,"mean_commits":20.5,"dds":"0.024390243902439046","last_synced_commit":"325aa38cb23d0757bb1335ee2ea94a082475a66e"},"previous_names":["dogsheep/photos-to-sqlite"],"tags_count":5,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dogsheep%2Fdogsheep-photos","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dogsheep%2Fdogsheep-photos/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dogsheep%2Fdogsheep-photos/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dogsheep%2Fdogsheep-photos/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dogsheep","download_url":"https://codeload.github.com/dogsheep/dogsheep-photos/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248018060,"owners_count":21034048,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["datasette","datasette-io","datasette-tool","dogsheep","sqlite"],"created_at":"2024-08-01T12:04:17.947Z","updated_at":"2025-04-09T10:07:46.669Z","avatar_url":"https://github.com/dogsheep.png","language":"Python","funding_links":[],"categories":["Python","others"],"sub_categories":[],"readme":"# dogsheep-photos\n\n[![PyPI](https://img.shields.io/pypi/v/dogsheep-photos.svg)](https://pypi.org/project/dogsheep-photos/)\n[![Changelog](https://img.shields.io/github/v/release/dogsheep/dogsheep-photos?include_prereleases\u0026label=changelog)](https://github.com/dogsheep/dogsheep-photos/releases)\n[![CircleCI](https://circleci.com/gh/dogsheep/dogsheep-photos.svg?style=svg)](https://circleci.com/gh/dogsheep/dogsheep-photos)\n[![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/dogsheep/dogsheep-photos/blob/master/LICENSE)\n\nSave details of your photos to a SQLite database and upload them to S3.\n\nSee [Using SQL to find my best photo of a pelican according to Apple Photos](https://simonwillison.net/2020/May/21/apple-photos-sqlite/) for background information on this project.\n\n## What these tools do\n\nThese tools are a work-in-progress mechanism for taking full ownership of your photos. The core idea is to help implement the following:\n\n* Every photo you have taken lives in a single, private Amazon S3 bucket\n* You have a single SQLite database file which stores metadata about those photos - potentially pulled from multiple different places. This may include EXIF data, Apple Photos, the results of running machine learning APIs against photos and much more besides.\n* You can then use [Datasette](https://github.com/simonw/datasette) to explore your own photos.\n\nI'm a heavy user of Apple Photos so the initial releases of this tool will have a bias towards that, but ideally I would like a subset of these tools to be useful to people no matter which core photo solution they are using.\n\n## Installation\n\n    $ pip install dogsheep-photos\n\n## Authentication (if using S3)\n\nIf you want to use S3 to store your photos, you will need to first create S3 credentials for a new, dedicated bucket.\n\nYou may find the [s3-credentials tool](https://github.com/simonw/s3-credentials) useful for this.\n\nRun this command and paste in your credentials. You will need three values: the name of your S3 bucket, your Access key ID and your Secret access key.\n\n    $ dogsheep-photos s3-auth\n\nThis will create a file called `auth.json` in your current directory containing the required values. To save the file at a different path or filename, use the `--auth=myauth.json` option.\n\n## Uploading photos\n\nRun this command to upload every photo in a specific directory to your S3 bucket:\n\n    $ dogsheep-photos upload photos.db \\\n        ~/Pictures/Photos\\ Library.photoslibrary/original\n\nThe command will only upload photos that have not yet been uploaded, based on their sha256 hash.\n\n`photos.db` will be created with an `uploads` table containing details of which files were uploaded.\n\nTo see what the command would do without uploading any files, use the `--dry-run` option.\n\nThe sha256 hash of the photo contents will be used as the name of the file in the bucket, with an extension matching the type of file. This is an implementation of the [Content addressable storage](https://en.wikipedia.org/wiki/Content-addressable_storage) pattern.\n\n## Importing Apple Photos metadata\n\nThe `apple-photos` command imports metadata from your Apple Photos library.\n\n    $ photo-to-sqlite apple-photos photos.db\n\nImported metadata includes places, people, albums, quality scores and machine learning labels for the photo contents.\n\n## Creating a subset database\n\nYou can create a new, subset database of photos using the `create-subset` command.\n\nThis is useful for creating a shareable SQLite database that only contains metadata for a selected set of photos.\n\nSince photo metadata contains latitude and longitude you may not want to share a database that includes photos taken at your home address.\n\n`create-subset` takes three arguments: an existing database file created using the `apple-photos` command, the name of the new, shareable database file you would like to create and a SQL query that returns the `sha256` hash values of the photos you would like to include in that database.\n\nFor example, here's how to create a shareable database of just the photos that have been added to albums containing the word \"Public\":\n\n    $ dogsheep-photos create-subset \\\n        photos.db \\\n        public.db \\\n        \"select sha256 from apple_photos where albums like '%Public%'\"\n\n## Serving photos locally with datasette-media\n\nIf you don't want to upload your photos to S3 but you still want to browse them using Datasette you can do so using the [datasette-media](https://github.com/simonw/datasette-media) plugin. This plugin adds the ability to serve images and other static files directly from disk, configured using a SQL query.\n\nTo use it, first install Datasette and the plugin:\n\n    $ pip install datasette datasette-media\n\nIf any of your photos are `.HEIC` images taken by an iPhone you should also install the optional `pyheif` dependency:\n\n    $ pip install pyheif\n\nNow create a `metadata.yaml` file configuring the plugin:\n\n```yaml\nplugins:\n  datasette-media:\n    thumbnail:\n      sql: |-\n        select path as filepath, 200 as resize_height from apple_photos where uuid = :key\n    large:\n      sql: |-\n        select path as filepath, 1024 as resize_height from apple_photos where uuid = :key\n```\nThis will configure two URL endpoints - one for 200 pixel high thumbnails and one for 1024 pixel high larger images.\n\nCreate your `photos.db` database using the `apple-photos` command, then run Datasette like this:\n\n    $ datasette -m metadata.yaml\n\nYour photos will be served on URLs that look like this:\n\n    http://127.0.0.1:8001/-/media/thumbnail/F4469918-13F3-43D8-9EC1-734C0E6B60AD\n    http://127.0.0.1:8001/-/media/large/F4469918-13F3-43D8-9EC1-734C0E6B60AD\n\nYou can find the UUIDs for use in these URLs by running `select uuid from photos_with_apple_metadata`.\n\n### Displaying images using datasette-json-html\n\nIf you are using `datasette-media` to serve photos you can include images directly in Datasette query results using the [datasette-json-html](https://github.com/simonw/datasette-json-html) plugin.\n\nRun `pip install datasette-json-html` to install the plugin, then use queries like this to view your images:\n\n```sql\nselect\n    json_object(\n        'img_src',\n        '/-/media/thumbnail/' || uuid\n    ) as photo,\n    uuid,\n    date\nfrom\n    apple_photos\norder by\n    date desc\nlimit 10;\n```\nThe `photo` column returned by this query should render as image tags that display the correct images.\n\n### Displaying images using custom template pages\n\nDatasette's [custom pages](https://datasette.readthedocs.io/en/stable/custom_templates.html#custom-pages) feature lets you create custom pages for a Datasette instance by dropping HTML templates into a `templates/pages` directory and then running Datasette using `datasette --template-dir=templates/`.\n\nYou can combine that ability with the [datasette-template-sql](https://github.com/simonw/datasette-template-sql) plugin to create custom template pages that directly display photos served by `datasette-media`.\n\nInstall the plugin using `pip install datasette-template-sql`.\n\nCreate a `templates/pages` folder and add the following files:\n\n`recent-photos.html`\n```html+jinja\n\u003ch1\u003eRecent photos\u003c/h1\u003e\n\n\u003cdiv\u003e\n{% for photo in sql(\"select * from apple_photos order by date desc limit 20\") %}\n    \u003cimg src=\"/-/media/photo/{{ photo['uuid'] }}\"\u003e\n{% endfor %}\n\u003c/div\u003e\n```\n`random-photos.html`\n```html+jinja\n\u003ch1\u003eRandom photos\u003c/h1\u003e\n\n\u003cdiv\u003e\n{% for photo in sql(\"with foo as (select * from apple_photos order by date desc limit 5000) select * from foo order by random() limit 20\") %}\n    \u003cimg src=\"/-/media/photo/{{ photo['uuid'] }}\"\u003e\n{% endfor %}\n\u003c/div\u003e\n```\nNow run Datasette like this:\n\n    $ datasette photos.db -m metadata.yaml --template-dir=templates/\n\nVisiting `http://localhost:8001/recent-photos` will display 20 recent photos. Visiting `http://localhost:8001/random-photos` will display 20 photos randomly selected from your 5,000 most recent.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdogsheep%2Fdogsheep-photos","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdogsheep%2Fdogsheep-photos","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdogsheep%2Fdogsheep-photos/lists"}