{"id":20631187,"url":"https://github.com/solidlabresearch/derived-resources-component","last_synced_at":"2025-04-15T18:36:55.920Z","repository":{"id":218929994,"uuid":"747274387","full_name":"SolidLabResearch/derived-resources-component","owner":"SolidLabResearch","description":"Adds support for derived resources","archived":false,"fork":false,"pushed_at":"2024-08-20T13:28:34.000Z","size":699,"stargazers_count":4,"open_issues_count":4,"forks_count":0,"subscribers_count":5,"default_branch":"main","last_synced_at":"2025-03-28T23:43:41.942Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/SolidLabResearch.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-01-23T16:01:32.000Z","updated_at":"2024-08-20T13:28:37.000Z","dependencies_parsed_at":"2024-02-29T08:31:11.228Z","dependency_job_id":"6596066d-4891-4f6a-8d35-d0707b5708ff","html_url":"https://github.com/SolidLabResearch/derived-resources-component","commit_stats":null,"previous_names":["solidlabresearch/derived-resources-component"],"tags_count":4,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SolidLabResearch%2Fderived-resources-component","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SolidLabResearch%2Fderived-resources-component/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SolidLabResearch%2Fderived-resources-component/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SolidLabResearch%2Fderived-resources-component/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/SolidLabResearch","download_url":"https://codeload.github.com/SolidLabResearch/derived-resources-component/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":249129955,"owners_count":21217475,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-16T14:11:27.038Z","updated_at":"2025-04-15T18:36:55.905Z","avatar_url":"https://github.com/SolidLabResearch.png","language":"TypeScript","readme":"# Derived resources\n\n[![npm version](https://img.shields.io/npm/v/@solidlab/derived-resources-component)](https://www.npmjs.com/package/@solidlab/derived-resources-component)\n\nAdds support for derived resources to a\n[Community Solid Server](https://github.com/CommunitySolidServer/CommunitySolidServer) instance.\n\n## Install\n\n```\nnpm install\nnpm run build\n```\n\nRun `npm run start:example` to have a CSS instance with the new components implementing the template `templates/root/base`.\nSee 'Examples' below for a walkthrough of the demonstration.\n\n## What are derived resources?\n\nDerived resources are resources whose contents are determined by aggregating contents of other resources.\nThis allows you to apply different access control rules to resources that, for example,\nreveal a subset of certain data.\n\n## How does it work\n\nDerived resources, as implemented in this repository,\nconsist of 3 components:\n- The template, describing the URL through which the derived resource is accessible.\n- One or more selectors, that determine the resources that are an input for the derivation.\n- The filter, which is applied to the selector to determine the result.\n\nTo create a derived resource, put the above 3 components in the metadata of a container,\nas described in the [CSS metadata documentation](https://communitysolidserver.github.io/CommunitySolidServer/7.x/usage/metadata/).\nFor example:\n```turtle\n@prefix derived: \u003curn:npm:solid:derived-resources:\u003e .\n\n\u003chttp://localhost:3000/\u003e derived:derivedResource [\n    derived:template \"test\";\n    derived:selector \u003chttp://localhost:3000/data\u003e;\n    derived:filter \u003chttp://localhost:3000/filter\u003e\n  ].\n```\nThe above metadata, when placed in the metadata of container `http://localhost:3000`,\ndefines `http://localhost:3000/test` as a derived resource.\nThe contents are defined by using `http://localhost:3000/data`\nas an input for the query found at `http://localhost:3000/filter`.\n\nThe last modified date, and the resulting ETag, of the derived resource\nis that of the most recently changed selector.\n\nBelow are some more details on each of the fields.\n\n### Template\n\nThis string is concatenated with the URL of the container to determine the URL of the derived resource.\nURL templates such as `\"{testVar}\"` can be used, and their value is fed into the filter.\nWhen such a URL template is used, all URLs that match are accessible.\nUsing this value you would be able to access `http://localhost:3000/foo`,\n`http://localhost:3000/bar`, etc.\n\nYou can also define derived resources by editing the metadata of a document instead of a container.\nIn that case the template value needs to be the empty string.\nThe actual body of such a document is irrelevant and can be anything.\nAll metadata assigned to this resource will be copied to the corresponding derived resource.\n\nBesides the variables extracted from a URL template,\nthere are certain variables that will always be available.\nCurrently, there are two such variables:\n- `source`: The identifier of the resource that contains the metadata defining the corresponding derived resource.\n- `identifier`: The identifier of the resource being accessed.\n\n### Selectors\n\nThe data of all resources linked here is combined before executing the filter in the next step.\n\nInstead of a fixed URL, these can also contain glob patterns `*` and `**`\nFor example, if the object in the selector triple is `\"http://localhost:3000/*\"`,\nall resources found in the `http://localhost:3000/` container are used.\n\n### Filter\n\nThis is the identifier of a resource that contains a valid filter.\nOne type of filter is a SPARQL query,\nfor all types, see [here](documentation/filters.md).\n\nIn case the template contains a URL template, the resulting variable(s) can be used here.\nThis is done with a simple string replacement on the query.\nTo continue the example of before,\nall occurrences of `$testVar$` in the SPARQL query are replaced with the provided value.\n\n## Examples\n\nWhen you start the server with `npm run start:example`,\nit generates several sample resources which define multiple derived resources.\nHow these are defined can be seen at \u003chttp://localhost:3000/derived/.meta\u003e.\nThat metadata defines the following derived resources:\n- \u003chttp://localhost:3000/derived/test\u003e: A basic derived resource applying a query to a single selector.\n- `http://localhost:3000/derived/template/{var}.ttl`: A template for derived resources where `var` will be applied to the query.\n     You can try out \u003chttp://localhost:3000/derived/template/foaf:knows.ttl\u003e and \u003chttp://localhost:3000/derived/template/foaf:name.ttl\u003e.\n- `http://localhost:3000/derived/query`: Similar to the previous example, but instead uses a query parameter for the variable.\n     For example, \u003chttp://localhost:3000/derived/query?var=foaf:knows\u003e.\n  - **Note:** Currently not working due to https://github.com/CommunitySolidServer/CommunitySolidServer/issues/1889\n- \u003chttp://localhost:3000/derived/pattern\u003e: A derived resource whose selectors are all resources in a container.\n- \u003chttp://localhost:3000/derived/multiple\u003e: A derived resource with multiple selectors.\n\nThe metadata of \u003chttp://localhost:3000/dummy.txt\u003e also defines a derived resource,\ncausing that URl to return the results of a derivation instead of the contents of the template file.\n\nTwo containers, \u003chttp://localhost:3000/derived/\u003e and \u003chttp://localhost:3000/derived/template/\u003e,\nreturn `ldp:contains` triples as expected from a container.\nBut these are also derived resources,\ngenerated by performing a SPARQL query on the metadata that defines the derived resources.\nThe query can be seen at \u003chttp://localhost:3000/filters/container\u003e.\n\n## Derived index resources\n\nA new feature that was added at a later point was support for derived index resources.\nMore info on these and the relevant examples can be found [here](documentation/derived-index.md).\n\n## Caching\n\nA new `ResourceStore` is added in this component: the `CachedResourceStore`.\nThis store caches resources in an LRU cache and invalidates them when they are modified.\nThis is a memory-based cache, meaning this can not be used by servers with worker threads,\nand is interesting for servers using non-memory backends, such as a file backend.\nThis will make it so the input sources can be requested faster.\nBut this will make it so the input sources can be requested faster.\nThis class is independent of the derived resources so could move to the main CSS repo.\n\nFor derived resources caching is a bit more complicated as they have to be updated when any of their input sources change.\nThe amount of input sources might also change,\nif a wildcard selector is used.\nTo store the resulting representations in an LRU cache,\na key is generated which is based on\n1. the filter,\n2. the identifiers of all input resources,\n3. and the timestamps of all those input resources.\n\nThis way, if any of those changes, the stored cache entry will never be used again,\npreventing invalid data.\nThe disadvantage is that this data is never explicitly invalidated,\nso it stays in the cache until it gets dropped because the cache is full.\n\n## Known limitations/decisions\n\n- No locks are used when reading data from the selectors. This is to prevent potential deadlocks.\n- Authorization on the selector resources is ignored.\n  Note that this allows users to access data they do not have access to if they guess the URL of such data.\n  We might want to implement it so creating a derived resource requires read access on all selectors,\n  similar to notifications.\n  This can be enabled as seen [here](documentation/derived-index.md).\n- An extra conversion store is added to the config as the new `ResourceStore` needs to do content negotiation\n  on data it receives from the backend,\n  but also wants to allow content negotiation on the data stream it generates.\n- The metadata describing these resources is readable for anyone who can read the associated resource.\n  If necessary, the solution could be changed to remove that metadata on GET requests.\n- The filter triple could be extended to also allow the value directly in the object instead of being in a separate resource.\n- The filter could be extended to also support external URIs.\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsolidlabresearch%2Fderived-resources-component","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsolidlabresearch%2Fderived-resources-component","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsolidlabresearch%2Fderived-resources-component/lists"}