Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/smartdataanalytics/rdfprocessingtoolkit
Command line interface based RDF processing toolkit to run sequences of SPARQL statements ad-hoc on RDF datasets, streams of bindings and streams of named graphs with support for processing JSON, CSV and XML using function extensions. Ships with an integrated SPARQL and GraphQL server.
https://github.com/smartdataanalytics/rdfprocessingtoolkit
debian-package docker-image jena named-graphs rdf rpm-package sparql sparql-extensions sparql-functions
Last synced: 6 days ago
JSON representation
Command line interface based RDF processing toolkit to run sequences of SPARQL statements ad-hoc on RDF datasets, streams of bindings and streams of named graphs with support for processing JSON, CSV and XML using function extensions. Ships with an integrated SPARQL and GraphQL server.
- Host: GitHub
- URL: https://github.com/smartdataanalytics/rdfprocessingtoolkit
- Owner: SmartDataAnalytics
- License: other
- Created: 2018-01-16T15:26:55.000Z (almost 7 years ago)
- Default Branch: develop
- Last Pushed: 2024-12-09T14:55:28.000Z (14 days ago)
- Last Synced: 2024-12-14T09:20:05.899Z (10 days ago)
- Topics: debian-package, docker-image, jena, named-graphs, rdf, rpm-package, sparql, sparql-extensions, sparql-functions
- Language: Java
- Homepage: https://smartdataanalytics.github.io/RdfProcessingToolkit/
- Size: 2.19 MB
- Stars: 40
- Watchers: 8
- Forks: 3
- Open Issues: 24
-
Metadata Files:
- Readme: README-NGS.md
- License: LICENSE
Awesome Lists containing this project
README
## Named Graph Streams (NGS)
```bash
➜ ngs -h
Usage: ngs [-h] [COMMAND]
Named Graph Stream Subcommands
-h, --help
Commands:
cat Output and optionally convert graph input
filter Yield items (not) satisfying a given condition
head List or skip the first n named graphs
tail Output the last named graphs (scans from the start)
map (flat-)Map each named graph to a new set of named graphs
probe Determine content type based on input
sort Sort named graphs by key
subjects Group triples with consecutive subjects into named graphs
until Yield items up to and including the first one that satisfies the condition
wc Mimicks the wordcount (wc) command; counts graphs or quads
while Yield items up to but excluding the one that satisfies the condition
```The command line swiss army knife for efficient processing of collections of named graphs using standard quad-based RDF syntaxes (n-quads, trig).
The tool suite is akin to well know shell scripting tools, such as head, tail, join, sort (including sort --merge), etc.Named graphs can be used to denote data records, i.e. sets of triples that form a logical unit for the task at hand.
### Building
NGS is part of the rdf-processing-toolkit build. Installing the debian package also makes the command `ngs` tool available.### Example Usage
* Output the first n named graphs of a trig file
* * `cat data.trig | ngs head`
* * `cat data.trig | ngs head -n 1000` (Get the first 1000 graphs)
* * `ngs head data.trig` (without useless cat)
* Run a SPARQL query (from a file or inline) on each graph. This is run in parallel for high efficiency.
* * `cat data.trig | ngs map --sparql script.sparql`
* Given some set of initial graphs, map them through a transformation, sort the resulting graphs by some key (merging consecutive ones together), sort the result randomly and take a sample of 1000
* * `cat data.trig | ngs map --sparql script.sparql | ngs sort -m -k '?o { ?s ?o }' | ngs sort -m -R | ngs head -n 1000`### Notes on ngs map
* When using `ngs map` with SPARQL queries that use a SERVICE clause the connection and query execution timeouts can be controlled using the `-t` flag:
```
echo " " | ngs subjects |\
ngs map -t '5000,5000' --sparql 'SELECT DISTINCT ?t { ?s ?p ?o SERVICE { ?s a ?t } }'
```
* Triples and quads can be mapped into the default graph or a specific named graph using the following variants
```
cat file.trig | ngs map --dg
ngs map --dg file.trig
ngs map --graph 'http://my.graph/' file.ttl
```### Rules
* Input must be quads where all quads of a graph must be consecutive
* `map`: Map an input named graph to a set of output graphs. Output graphs are created using CONSTRUCT queries. Thanks to Jena's extension, CONSTRUCT { GRAPH ?g { ... } } is allowed. All explicitly generated named graphs are returned as-is. CONSTRUCT'ed data in the default graph is wrapped in a graph with the same name as the input graph.### Implementation status
* Note: head and tail currently do not support the + flag as in e.g. ngs head -n+10| Command | Descripion | Status |
|------------|----------------------------------------------------------------------------------|--------|
| head | Output the first n graphs | done |
| cat | Output all graphs | done |
| probe | Probe a file or stdin for a quad-based format | done |
| map | Execute a SPARQL query on every named graph and yield the set of obtained graphs | done |
| until | Yield graphs up to and including the first one that satisfies a condition | done |
| while | Yield graphs as long as they satisfy a condition | done |
| wc | Count graphs or quads | done |
| join | Merge triples of named graphs based on a key | wip |
| sort | Sort named graphs based on a key | done |
| subjects | Group a stream of triples by subject to yield a stream of named graphs | done |
| filterKeep | Only yield graphs for which an ASK query yields true | wip |
| filterDrop | Omit graphs from output when an ASK query yields true | wip |
| ... | More to come | |