{"id":29763964,"url":"https://github.com/nebulastream/nebulastream","last_synced_at":"2026-01-26T06:03:13.731Z","repository":{"id":301339054,"uuid":"809730404","full_name":"nebulastream/nebulastream","owner":"nebulastream","description":"Data Management for the Internet of Things","archived":false,"fork":false,"pushed_at":"2026-01-17T19:50:06.000Z","size":583003,"stargazers_count":74,"open_issues_count":186,"forks_count":25,"subscribers_count":9,"default_branch":"main","last_synced_at":"2026-01-18T05:52:48.981Z","etag":null,"topics":["bigdata","streamprocessing"],"latest_commit_sha":null,"homepage":"https://www.nebula.stream","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/nebulastream.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2024-06-03T10:43:06.000Z","updated_at":"2026-01-16T19:07:03.000Z","dependencies_parsed_at":"2025-07-10T13:03:24.362Z","dependency_job_id":"01261c1e-9442-40e1-81fd-bcb616839630","html_url":"https://github.com/nebulastream/nebulastream","commit_stats":null,"previous_names":["nebulastream/nebulastream"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/nebulastream/nebulastream","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nebulastream%2Fnebulastream","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nebulastream%2Fnebulastream/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nebulastream%2Fnebulastream/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nebulastream%2Fnebulastream/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/nebulastream","download_url":"https://codeload.github.com/nebulastream/nebulastream/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nebulastream%2Fnebulastream/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28767998,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-26T03:54:34.369Z","status":"ssl_error","status_checked_at":"2026-01-26T03:54:33.031Z","response_time":59,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bigdata","streamprocessing"],"created_at":"2025-07-26T23:00:46.334Z","updated_at":"2026-01-26T06:03:13.726Z","avatar_url":"https://github.com/nebulastream.png","language":"C++","funding_links":[],"categories":["Table of Contents","Recently Updated"],"sub_categories":["Streaming Engine","[Jul 25, 2025](/content/2025/07/25/README.md)"],"readme":"\u003cdiv align=\"center\"\u003e\n  \u003cpicture\u003e\n    \u003csource media=\"(prefers-color-scheme: light)\" srcset=\"docs/resources/NebulaBanner.png\"\u003e\n    \u003csource media=\"(prefers-color-scheme: dark)\" srcset=\"docs/resources/NebulaBannerDarkMode.png\"\u003e\n    \u003cimg alt=\"NebulaStream logo\" src=\"docs/resources/NebulaBanner.png\" height=\"100\"\u003e\n  \u003c/picture\u003e\n  \u003cbr /\u003e\n  \u003c!-- Badges --\u003e\n  \u003ca href=\"https://github.com/nebulastream/nebulastream-public/actions/workflows/nightly.yml\"\u003e\n    \u003cimg src=\"https://github.com/nebulastream/nebulastream-public/actions/workflows/nightly.yml/badge.svg\"\n         alt=\"NES Nightly\" /\u003e\n  \u003c/a\u003e\n  \u003ca href=\"https://bench.nebula.stream/c-benchmarks/\"\u003e\n    \u003cimg src=\"https://img.shields.io/badge/Benchmark-Conbench-blue?labelColor=3D444C\"\n         alt=\"Conbench\" /\u003e\n  \u003c/a\u003e\n  \u003ca href=\"https://codecov.io/github/nebulastream/nebulastream\" \u003e \n    \u003cimg src=\"https://codecov.io/github/nebulastream/nebulastream/graph/badge.svg?token=ER83Nm1crF\" alt=\"Codecov\"/\u003e \n  \u003c/a\u003e  \n\u003c/div\u003e\n\n----\n\nNebulaStream is an end-to-end data-management system for cloud-edge-sensor deployments.\nThe platform combines ease of use, extensibility, and efficiency to let teams focus on business logic while the engine optimizes the data path.\n\n**Core Pillars**\n- **Ease of Use** - Rich, out-of-the-box functionality for a variety of stream processing queries.\n- **Extensibility** - Uniform plugin registries for custom connectors, formats, operators, and optimizations.\n- **Efficiency** - Hardware-tailored code, adaptive execution, and interleaved I/O to handle large workloads.\n\n## Vision\n\nNebulaStream is a joint research project at BIFOLD with contributors from the DIMA Group at TU Berlin and the DFKI IAM Group.\nIt aims to execute thousands of queries over millions of heterogeneous sources in massively distributed environments.\nWe advance this vision through five core technologies:\n- **Heterogeneous Hardware Support** - Targets diverse CPU architectures and accelerators such as GPUs and TPUs.\n- **Code Generation** - Compiles every query to efficient native code for low latency and energy usage.\n- **In-Network Processing** - Pushes operators toward data sources to reduce network traffic.\n- **Adaptive Resource Management** - Reacts to topology or workload changes without interrupting queries.\n\nThe system architecture spans sensor to cloud:\n1. **Sources \u0026 Sinks** - Built-in connectors (e.g., JDBC, MQTT, TCP) and formats (e.g., CSV, JSON) with plugin hooks for custom components; extend them via the plugin framework in the [Extensibility guide](docs/guide/extensibility.md).\n2. **I/O Handling** - Thread-shared source processing and asynchronous callbacks minimize waiting time.\n3. **Query Submission** - SQL-like language with prebuilt operators such as join and aggregation, plus user-defined operator plugins.\n4. **Query Optimization** - Rule-based optimizer generates hardware-aware plans; users can extend the rule engine.\n5. **Adaptive Runtime** - Task-based scheduling keeps execution responsive under dynamic workloads.\n\n## Quick Start\n\n### Build \u0026 Tooling\n\n#### Use the Nix flake\nInstall [Nix with flakes enabled](https://nixos.org/download/) and generate build links:\n\n```shell\nnix run .#clion-setup\n```\n\nConfigure, build, and test through the Nix-wrapped helpers:\n\n```shell\n./.nix/nix-cmake.sh \\\n  -DCMAKE_BUILD_TYPE=Debug \\\n  -G Ninja \\\n  -S . -B cmake-build-debug\n\n./.nix/nix-cmake.sh --build cmake-build-debug\n./.nix/ctest --test-dir cmake-build-debug -j\n```\n\n#### Use the development container\nBuild or reuse the local development image (installs the current user inside the container to avoid permission issues):\n\n```shell\n./scripts/install-local-docker-environment.sh\n```\n\nConfigure, build, and test by mounting the repository into the container:\n\n```shell\ndocker run \\\n  --workdir $(pwd) \\\n  -v $(pwd):$(pwd) \\\n  nebulastream/nes-development:local \\\n  cmake -B cmake-build-debug\n\ndocker run \\\n  --workdir $(pwd) \\\n  -v $(pwd):$(pwd) \\\n  nebulastream/nes-development:local \\\n  cmake --build cmake-build-debug -j\n\ndocker run \\\n  --workdir $(pwd) \\\n  -v $(pwd):$(pwd) \\\n  nebulastream/nes-development:local \\\n  ctest --test-dir cmake-build-debug -j\n```\n\n### Execute Queries \u0026 Tests\n\n#### Run systest for a query\nBuild the `systest` executable in your chosen build directory and execute a specific query. The example below targets the first query in the arithmetic function suite and stores artifacts in `cmake-build-debug/systest-run`:\n\n```shell\ncmake --build cmake-build-debug -j --target systest\ncmake-build-debug/systest/systest -t nes-systests/function/arithmetical/FunctionAdd.test:1\n```\n\nAppend worker configuration overrides after `--` (for example `-- --worker.default_query_execution.execution_mode=INTERPRETER`). See the [systest guide](docs/development/systests.md) for authoring custom `.test` files and the complete CLI reference.\n\n#### Start the client with a worker\nBuild the client and worker binaries, start the worker in one terminal, and submit a short-lived query from another terminal using `nebuli`:\n\n```shell\ncmake --build cmake-build-debug -j --target nes-single-node-worker nes-nebuli\n\n# Terminal 1: start the worker (listens on localhost:8080 by default)\ncmake-build-debug/nes-single-node-worker/nes-single-node-worker\n\n# Terminal 2: prepare a tiny CSV and submit a query via nebuli\nprintf '1\\n2\\n3\\n' \u003e demo-input.csv\n\ncat \u003e demo.sql \u003c\u003c'EOF'\nCREATE LOGICAL SOURCE demo(value UINT64);\nCREATE PHYSICAL SOURCE FOR demo TYPE File SET('./demo-input.csv' AS `SOURCE`.FILE_PATH, 'CSV' AS PARSER.`TYPE`, '\\n' AS PARSER.TUPLE_DELIMITER, ',' AS PARSER.FIELD_DELIMITER);\nCREATE SINK result(demo.value UINT64) TYPE File SET('./demo-output.csv' AS `SINK`.FILE_PATH, 'CSV' AS `SINK`.INPUT_FORMAT);\nSELECT value FROM demo INTO result;\nEOF\n\ncmake-build-debug/nes-nebuli/nes-nebuli -s localhost:8080 -w \u003c demo.sql\n```\n\nThe generated CSV appears at `demo-output.csv`. Inspect or retire the query with additional `nebuli` subcommands such as `show`, `stop`, or `unregister` as needed.\n\n## Documentation\n- Design proposals and architectural notes: [Design index](docs/design/README.md)\n- Developer workflows and environment setup: [Development environment](docs/development/development.md), [Run workflows locally](docs/running_workflows_locally.md)\n- Technical deep dives: [Dependency architecture](docs/technical/dependency.md), [Query engine task queue](docs/technical/QueryEngine_TaskQueue.md), [Watermarking trigger details](docs/technical/watermarking_progress_window_triggering.md)\n- Organizational guidelines and processes: [Meetings overview](docs/organizational/meetings.md), [Nightly CI process](docs/organizational/processes/nightly_ci.md)\n\n## Contributing \u0026 Quality\n- Follow the [development guide](docs/development/development.md) for environment setup and tooling expectations.\n- Code style is enforced with `clang-format`, `clang-tidy`, license checks, and pragma guards; run the `format` target before submitting patches.\n- Review the Git workflow aids in [Useful git commands](docs/git/useful_commands.md), [PR checklist](docs/git/checklist_pr.md), and IDE tips in [CLion tricks](docs/git/clion_tricks.md).\n- External contributors should work from personal forks and open pull requests referencing their forked branches; direct pushes to this repository are reserved for maintainers.\n- Adhere to our [Code of Conduct](CODE_OF_CONDUCT.md).\n\n### Build Types\n- Debug - Compiles all logging and keeps asserts enabled; best during active development.\n- RelWithDebInfo - Keeps warning-level logging and asserts for balanced debugging/performance.\n- Release - Optimizes for throughput with error-level logging. Asserts are enabled.\n- Benchmark - Strips logging and assertions for maximum performance; use only with well-tested queries because undefined behavior is not guarded.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnebulastream%2Fnebulastream","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnebulastream%2Fnebulastream","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnebulastream%2Fnebulastream/lists"}