{"id":20631224,"url":"https://github.com/solidlabresearch/ldests","last_synced_at":"2026-04-20T12:02:14.157Z","repository":{"id":201144434,"uuid":"666020974","full_name":"SolidLabResearch/LDESTS","owner":"SolidLabResearch","description":null,"archived":false,"fork":false,"pushed_at":"2023-11-16T14:57:31.000Z","size":393,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-12-10T03:39:30.436Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Kotlin","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/SolidLabResearch.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-07-13T14:19:45.000Z","updated_at":"2023-11-23T13:23:17.000Z","dependencies_parsed_at":"2023-11-16T15:34:24.517Z","dependency_job_id":null,"html_url":"https://github.com/SolidLabResearch/LDESTS","commit_stats":null,"previous_names":["solidlabresearch/ldests"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/SolidLabResearch/LDESTS","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SolidLabResearch%2FLDESTS","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SolidLabResearch%2FLDESTS/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SolidLabResearch%2FLDESTS/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SolidLabResearch%2FLDESTS/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/SolidLabResearch","download_url":"https://codeload.github.com/SolidLabResearch/LDESTS/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SolidLabResearch%2FLDESTS/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32046393,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-20T11:35:06.609Z","status":"ssl_error","status_checked_at":"2026-04-20T11:34:48.899Z","response_time":94,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-16T14:11:35.475Z","updated_at":"2026-04-20T12:02:14.111Z","avatar_url":"https://github.com/SolidLabResearch.png","language":"Kotlin","readme":"# LDESTS: Linked Data Event Streams for Time Series\n## Introduction\nLDESTS aims to make time series data streams using RDF feasible on platforms such as Solid. It achieves this by using a data model (also referred to as \"shape\") defining the per-sample structure. By defining this structure in advance, repetition in structure can be avoided by filtering out the unique properties of individual samples. As only these unique properties are encoded in the resulting data, a **lossless, highly compressed** stream can be achieved.\n## Currently supported\nThis tool is developed using [Kotlin Multiplatform](https://kotlinlang.org/docs/multiplatform.html), allowing for multiple target platforms. Currently, **only JS/TS on Node.js is supported**. There are currently **no** plans to **support JS/TS on Web**. There are plans to add Java (and Android) integration **at a later stage**.\n## Getting started\n**Notice:** building from source has only been tested on **Linux**. While the build process should work for any platform capable of using Gradle, other platforms have not been tested. If you come across an issue building the library from source, be sure to let us know by creating an [issue](https://github.com/SolidLabResearch/LDESTS/issues/new).\n### JS/TS on Node.js\n#### NPM\nCurrently, **no NPM releases are available**, but are planned. If you want to integrate LDESTS in your own project right now, you have to build the library from source.\n#### From source\nThe build process requires the commands `git`, `yarn` and `npm` to function, so make sure these are properly installed and configured before proceeding.\\\nBegin by cloning the repository to a known location:\n\n```\n[user@host ~]$ git clone https://github.com/SolidLabResearch/LDESTS.git\n```\nNext, navigate to the new folder and start the building process (either `debug` if you want to diagnose a problem in the source code, or `release`):\n```\n[user@host ~]$ cd LDESTS\n[user@host LDESTS]$ ./gradlew js:release\n```\nThis process can take some time, as it fetches and configures all dependencies for the build to take place. After the process has finished, the folder `bin/js` should contain the resulting library. From here, you can add it as a dependency to your own project by running:\n```\n[user@host MyProject]$ npm install file:path/to/LDESTS/bin/js/ldests-$version.tgz\n```\nFinally, you should be able to include the library into your own project, with type definitions available in TS projects:\n```ts\nimport { LDESTS, Shape } from \"ldests\";\n```\n## How to use\n### Node.js\nAfter having [built and included LDESTS in your own project](#jsts-on-nodejs), you can take inspiration from the example found in `jsTest` to get started.\n#### Creating a stream\nIntegrating custom streams is typically a two steps process.\nCreate a data model/shape\n```ts\nconst shape: Shape = Shape.Companion.parse({\n    type: \"myDataType\",\n    identifier: \"myDataIdentifier\",\n    constants: {\n        \"myConstantProperty\": [\"myConstantValues\", \"...\"]\n    },\n    variables: {\n        \"myVariableProperty\": \"myVariableDatatype\"\n    }\n});\n```\nInitialise or continue an LDESTS stream using the shape from above\n```ts\nconst stream = await new LDESTS.Builder(\"myStream\")\n    .shape(shape)\n    .build();\n```\nThe stream created above can be configured to your needs: you can\n- customise the configuration to have fragments to your desired size;\n- attach different publishers (ranging from Solid pods to in-memory `N3Store`s);\n- configure how to fragment your stream.\n\nIf you no longer need your stream object, you can `close` the stream, allowing all last transactions to finalise and any connections to stop properly:\n```ts\nawait stream.close();\n```\n#### Inserting data\nA stream can append new data through (single) triple or entire file insertion:\n```ts\nstream.insertTriple(triple); // adds a single triple \"asynchronously\" to the input stream\nawait stream.insertFile(\"path/to/file.nt\"); // adds an entire RDF file to the stream and `await`s until finished\n```\nIt is possible for the resulting stream to not reflect new data *yet*. To make sure the stream has these new additions available to consumers, the operations have to be `await`ed and the stream has to be flushed:\n```ts\nawait stream.flush(); // ensures all additional data is processed and published before it returns\n```\n#### Consuming a stream\nFirst, the stream instance has to be created (as seen [here](#creating-a-stream)). Later, it will be possible to [automatically infer the stream's settings](#roadmapfuture-work) (including the shape) when using a single publisher. Currently, only Solid pods are compatible with querying.\n```ts\nconst pod = { type: PublisherType.Solid, url: \"http://solid.local\" } as SolidPublisherConfig;\nconst query = \"SELECT * WHERE { ?s ?p ?o . }\";\n// looks for \"myStream\" as defined by the creation of `stream` above on the pod \"solid.local\"\nawait stream.query(pod, query, (binding) =\u003e { callback(binding) });\n```\nLater, time and property constraints found in the provided query will be used to filter the available data so only relevant fragments are retrieved. `await`ing the result of `query` is not required, but can help with flow control.\\\n**Note:** as these triples are regenerated from the compressed stream, the exact subject URIs are lost. Every sample's subject is still unique throughout the query, however.\n## How it works\nThe stream's data model describes every possible variation of the incoming data. By providing a property or a set of properties the stream has to fragment the incoming data with, all possible fixed data models can be generated. Every resulting model gets its own data stream. These data streams are being appended to when incoming data matches that stream's model.\\\nMatching what data belongs to which data stream is being done through multiple SPARQL queries. As the individual data models represent a template for the individual samples of that stream, generic queries for these samples can be made, allowing for multiple SPARQL queries to occur over the input stream. By then analysing the resulting bindings generated from these queries, the data can be associated with the right fragment of that model's data stream.\\\nRereading this data also heavily uses the data models: by first analysing the requested data's constraints, a selection of data streams can be made. The fragments making up the various streams of interest are then filtered based on time. Every matching fragment is then read and parsed, which involves the recreation of all triples making up the original data sample, thanks to the presence of the data models.\n## Roadmap/Future work\nCurrent features and changes are either planned/possible (depending on requirements/demand, in no particular order):\n- Support for the JVM (\u0026 Android)\n- Automatic shape generation using samples from the data streams\n- Inferring stream properties\n- Supporting variable stream layout on a per-publisher basis\n- Manual publisher management, including more granular control over their configurations\n- Support for custom publisher implementations\n- Proper (automated) releases for Node.js JS/TS on NPM\n## Credits\nThis research was partly funded the Flemish Government under the “Onderzoeksprogramma\nArtificiële Intelligentie (AI) Vlaanderen” programme, the SolidLab Vlaanderen project (Flemish\nGovernment, EWI and RRF project VV023/10) and the FWO Project FRACTION (Nr. G086822N).\\\nThe Node.js JS/TS integration uses\n- [RDFJS/N3](https://github.com/rdfjs/N3.js/) for everything related to RDF storage (both in-memory and turtle files);\n- [Comunica](https://github.com/comunica/comunica/) for querying triple sources, such as in-memory stores and Solid pods, through SPARQL queries;\n- [Incremunica](https://github.com/maartyman/incremunica/) for querying ongoing RDF streams, created by manual insertion, through SPARQL queries\n\nunder the hood.\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsolidlabresearch%2Fldests","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsolidlabresearch%2Fldests","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsolidlabresearch%2Fldests/lists"}