{"id":15680934,"url":"https://github.com/jonnor/ipld-image","last_synced_at":"2025-05-07T11:21:23.429Z","repository":{"id":138841175,"uuid":"60356815","full_name":"jonnor/ipld-image","owner":"jonnor","description":"Images expressed as Interplanetary Linked Data","archived":false,"fork":false,"pushed_at":"2016-07-14T22:33:35.000Z","size":605,"stargazers_count":11,"open_issues_count":2,"forks_count":2,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-03-31T09:37:19.566Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"CoffeeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jonnor.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2016-06-03T15:07:27.000Z","updated_at":"2021-07-02T14:51:00.000Z","dependencies_parsed_at":"2023-03-13T10:52:44.000Z","dependency_job_id":null,"html_url":"https://github.com/jonnor/ipld-image","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jonnor%2Fipld-image","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jonnor%2Fipld-image/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jonnor%2Fipld-image/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jonnor%2Fipld-image/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jonnor","download_url":"https://codeload.github.com/jonnor/ipld-image/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252866118,"owners_count":21816396,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-03T16:46:36.095Z","updated_at":"2025-05-07T11:21:23.413Z","avatar_url":"https://github.com/jonnor.png","language":"CoffeeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"\nipld-image is an attempt to create a structured representation of images on [IPFS](http://ipfs.io/),\nthat allows to do image processing operations directly on this structure, instead of\nhaving to operate on opaque blobs of serialized images (like a PNG or JPEG).\n\nThe datastructure allows to address parts of the image in X/Y (for cropping, collages),\non multiple levels-of-detail (for downsampling, thumbnails). \nThis is achieved by representing the image as a tiled mipmap pyramid.\n\n## Status\n\n**0.0.1: Initial proof of concept works**\n\n* `ipld-image-fromblob` can take an image file (PNG), and upload to IPFS as an IPLD image\n* `ipld-image-toblob` can take an hash of IPLD image, and render an image file (PNG) from it\n* ipld-image is just a working name\n* **data format is not stable yet**\n\nSee [TODO](#todo) for more details\n\n## Motivation\n\nImages are a huge part of web content today.\nTheir primarily (only) representation is that of a file, a blob of bytes, which we know nothing about\napart from its [MIME-type](https://en.wikipedia.org/wiki/Media_type).\nThe file typically contains compressed pixel data, and sometimes some metadata.\n\n### Inefficient processing\n\nSo if we want to display the image, we have to download and process the whole file.\nFor some formats one can stream only the beginning of a file, and from that get a lower-quality\nimage from it. This is intended to allow [progressive rendering](https://blog.codinghorror.com/progressive-image-rendering/).\nTheoretically one could cancel the stream when one deems the quality high-enough, but no web browsers available does this - and unassisted it cannot know what quality is considered good-enough.\n\nThis means that there is no space savings possible.\nThis is inefficient, and painful - especially on slow pay-per-MB connections as is typical on mobile.\nFurthermore due to responsive design, the same image (semantically) may be presented at many different screen sizes,\ndepending on the layout of the page it is included o.\nWith smart-cropping the image might be not just rescaled, but also show only a subset of the image.\n\nTo solve this today, one typically uses an image processing server which\nautomatically creates (multiple) down-scaled versions of an image.\nExamples include [imgflo-server](https://github.com/imgflo/imgflo-server).\n\nHowever, the processing server must also download the entire image, even if it knows\nthat only a downscaled cropped part would be needed.\n\n### Lack of metadata\n(addressing this might be out-of-scope for v1)\n\nWhen receiving a down-scaled image blob, there is (in general) no way to find back the original source image.\nThis means that for instance author attribution must be side-channeled (and usually is not).\n\nMost processing services strip all metadata in the process of creating versions for display.\nIn a few cases this can be a benefit, as privacy-invading metadata like geographic location is not present.\nBut mostly it limits usefulness, like one cannot know which camera settings where used,\nso one cannot do after-the-fact projection/lens correction.\n\n## Background\n\n[IPLD](https://github.com/ipfs/specs/tree/master/ipld) is the Inter Planetary Linked Data format.\nIt will form the base of [IPFS](http://ipfs.io/), an effort to re-architect Internet protocols\nto be peer2peer based on content-addressing. IPLD thus serves a similar role to that of\nIP packets in the conventional [Internet protocol stack](https://en.wikipedia.org/wiki/Internet_protocol_suite).\n\nNote: IPFS 0.5 will transition to IPLD as the underlying protocol,\nwhereas IPFS 0.4 (and earlier) use [MerkleDAG](https://github.com/ipfs/specs/tree/master/merkledag),\na less generic version of the same basic idea.\n\n## Dataformat\n\nA [mipmap](https://en.wikipedia.org/wiki/Mipmap) is an structure for efficiently storing\nimages at different levels of detail. At the lowest level are the original image in full resolution,\nthen at each level up the image resolution is halved in both width and height.\nSo 2x2=4 tiles at level N becomes 1 tile at level N+1.\nWith square images, A fully mipmapped structure is at most 33% larger than the original.\n\nThis also means that an image at level=2 (1/4 width and height) is\n1/16 the number of pixels that needs to be downloaded and processed.\n\nipld-image uses a mipmapped structure, but instead of each level being a continous buffer,\nit is a set of tiles, with each tile containing a encoded piece of the pixel data.\n\nThe spec here is given as a pseudo-YAML structure.\n\n```yaml\n## Image\n\n# IPLD-image protocol version\n'ipld-image-version': 1\n\n# The Image this data was derived from, if any\n# It SHOULD be used when processing an image, say when overlaying text, changing colors etc\nderivedfrom: { '/': Image }\n\n# If lossy compressed, this SHOULD be set to a losslessly compressed version\n# If processing an image, and this is set, the client SHOULD use canonicalversion instead of this one\ncanonicalversion: { '/': Image }\n\n# size of each tile\n# The tile size should be such that each Tile is less than the IPFS block size\n# Currently IPFS block size is 256KB. An 256x256 px image with 3-4 bytes per pixel should almost always be below this\ntilesize: { x: 256, y: 128 }\n# number of tiles spanned.\ntiles: { x: 10, y: 15 }\n\n# The size of the image spanned by the tiles (in pixels) is:\n# [tilesize.X*tiles.x , tilesize*tiles.y]\n# here, 2560px by 1920\n\n# which part of the spanned data is visible\n# this allows re-using tiles even doing crops/views which don't\n# If not specified, the boundary is implicitly\n# x: 0, y: 0, width: tilesize.x*tiles.x, height: tilesize.y*tiles.y\nboundary:\n  x: 10\n  y: 10\n  width: 1000\n  height: 1000\n\n# mipmap structure containing the image data\nlevel0: { '/': TileList } # n=tiles.x*tiles.y\nlevel1: { '/': TileList } [ .. ] # n/=4\nlevel2: { '/': TileList } # n/=4\n...\nlevelH: { '/': TileList } n=1\n```\n\n```yaml\n## Tile\n# format of data\nformat: \"png-idat\"\n# size of data in pixels\nsize: { x: 256, y: 128 }\n# link to the chunk of image data\ndata: {\"/\",  }\n```\n\n```yaml\n## TileList\n# stored in a scanline fashion\n# ie: the first tile is at x=0,y=0, then follows y=0, x=1,2,3,4,5 -\u003e (tiles.x-1)\n# then everything in y=1. Repeat until all rows are included\n[ { '/': Tile,} { '/': Tile } ... ]\n```\n\nAn advantage of this initial spec is that the `Tile`, containing the image data,\nis self-describing yet has no dependencies on its surroundings.\nThis should allow reusing the `Tile` also in other `TileList`s or `Image`s.\n\n### Open questions\n\n#### Non-square images\n\nHow to deal with fact that images can be non-square and have width/height which are not multiples of 2?\n\n* Mipmaps usually have these restrictions, and it makes it very simple because it is very clear how many\nlevels there are (`log2(size)`), and the shape of each level (always `previoussize/2`),\nand each tile is completely filled with meaningful data.\n* Should we allow different sized tiles, so that there is always meaningful data in them?\nOnly on left/bottom edge? Or also along right/top edge - as that would allow reusing the tiles there,\nfor having a layout with multiple images in them, with no gaps/seamless...\nThough at that time, why not allow also in the middle?\n* How to deal with fact that non-square images will not reduce down to a level with single. \nMust one render transparency into chunks then?\nKind-of a case where we get sparse-ness further up in the mipmaps\n\n#### Alternative tiles\n\nIs storing all levels as part of image the best approach?\n\nSome alternatives are:\n\n* a) Let each level link to level above and/or above.\n* b) let each tile link to the tiles on the level under. This means a lot of indirection\n\n#### Multiple representations for tiles\n\nShould one allow multiple representations for a tile? Say different compression/formats\n\n#### Sparse images?\n\nShould we allow sparse images (some areas not covered by tile).\n\n* Problem is then need to spec out what this should be filled, which could be limiting.  Transparent chunk?\n* Also, would not have much space-saving because likely a fully transparent tile will already be (de-duplication is builtin).\n* Non-rectangular images would have more to gain than... But then also need to be able to specify non-rectangular boundary (polygon etc)\n\n#### Non-uniform chunk sizes\n\nShould we allow non-uniform chunk sizes?\n\n* This heavily suggests sparse images also.\n* However it becomes really tricky to assemble a linear substream for a rectangular image, if chunks can be any size and any location...\n\n\n## Transformation on the dataformat\n\n`TODO: define usecases which we want the dataformat to support (and which ones are not so important).`\n\n`TODO: write how each of these would be performed on example data`\n\n## TODO\n\n### 0.0.2: Proof of concept\n\n* Actually support mipmap tile pyramid. Building the pyramid, and rendering low-resolution output using higher levels\n* Set and respect the `boundary` property\n* Support for rendering out a cropped version / area of interest\n* Write a little demo. Take an image blob, put it into IPFS as ipld-image, progress crop/downscaled, then display output\n\n### 0.1.0: Minimally useful\n\n* Figure out how to best support IPFS 0.4 (no native IPLD). Use a MerkleDAG object?\nFallback: serialize IPLD structure into database/IPFS as JSON blob?\n* Implement support in imgflo-server\n* Write tests\n\n### Later\n\n* Add a JSON Schema for the datastructure\n* Split out spec from implementation, put into https://github.com/ipfs/specs ?\n* Support browser/client-side\n* Support js-ipfs natively\n* Sketch out how this could be used to implement a GeglTileStore, for backing buffers in\n[GEGL](http://gegl.org), the image processing library used by imgflo-server and GIMP\n* Consider extending for video processing\n\n## Ideas\n\n### Perceptual encoding\nRight now, we can deducplicate parts of images when the encoded representation of tiles are identical.\nHowever even the tiniest, impercievable change, like a 1 bit quantification error, will invalidate deduplication.\nSome [existing discussion here](https://github.com/ipfs/faq/issues/15), with references to academic papers.\n\n### Direct streaming rendering\nRight now each tile is stored as a proper PNG image.\nIn order to construct an image file for rendering, we decode each of the neccesary tiles,\nblit it into an in-memory RGBA image representation, and then encode this as a new PNG file.\n\nWhat if instead we could store tiles as compressed data (without headers), then\nassemble an image file by concatating a new header with a set of such pre-encoded tiles.\nThis would skip both the decoding and re-encoding steps.\nThis would reducing neccesary computations and memory usage significantly.\n\nPNG might not be suitable for this, as the encoded stream seems to be in scanline ordering.\nIt may require storing each scanline of each tile as a separate chunk..\n\nThis is primarily of interest when IPLD is the core protocol for IPFS (0.5), and there is\nsupport for IPLD path/selectors including ordering. As that way, one could theoretically\nexpress the rendering of an output image file using only IPFS primitives.\n\n\n## Related projects\n\n* [IIIF](http://iiif.io/api/presentation/2.1/#status-of-this-document), standard for mipmapped images.\nAlso [ipfs-iiif](https://gist.github.com/edsilv/97759a93cb7c5f0fedb8178fee5e1dd3)\n\n## Contributors\n\n* [@Kubuxu](https://github.com/Kubuxu)\n* [@jonnor](https://github.com/jonnor)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjonnor%2Fipld-image","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjonnor%2Fipld-image","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjonnor%2Fipld-image/lists"}