{"id":13436507,"url":"https://github.com/pipelinedb/pipelinedb","last_synced_at":"2026-02-21T01:32:45.432Z","repository":{"id":12112843,"uuid":"14702444","full_name":"pipelinedb/pipelinedb","owner":"pipelinedb","description":"High-performance time-series aggregation for PostgreSQL","archived":false,"fork":false,"pushed_at":"2022-02-20T10:24:23.000Z","size":48997,"stargazers_count":2643,"open_issues_count":133,"forks_count":243,"subscribers_count":103,"default_branch":"master","last_synced_at":"2025-03-18T21:50:24.368Z","etag":null,"topics":["aggregation","analytics","pipelinedb","postgresql","push","realtime","sql","stream-processing","time-series"],"latest_commit_sha":null,"homepage":"https://www.pipelinedb.com","language":"C","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/pipelinedb.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2013-11-26T00:11:48.000Z","updated_at":"2025-03-13T00:41:16.000Z","dependencies_parsed_at":"2022-08-01T08:19:15.358Z","dependency_job_id":null,"html_url":"https://github.com/pipelinedb/pipelinedb","commit_stats":null,"previous_names":[],"tags_count":57,"template":false,"template_full_name":null,"purl":"pkg:github/pipelinedb/pipelinedb","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pipelinedb%2Fpipelinedb","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pipelinedb%2Fpipelinedb/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pipelinedb%2Fpipelinedb/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pipelinedb%2Fpipelinedb/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/pipelinedb","download_url":"https://codeload.github.com/pipelinedb/pipelinedb/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pipelinedb%2Fpipelinedb/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29670125,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-21T00:11:43.526Z","status":"ssl_error","status_checked_at":"2026-02-20T23:52:33.807Z","response_time":59,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aggregation","analytics","pipelinedb","postgresql","push","realtime","sql","stream-processing","time-series"],"created_at":"2024-07-31T03:00:49.301Z","updated_at":"2026-02-21T01:32:45.405Z","avatar_url":"https://github.com/pipelinedb.png","language":"C","readme":"PipelineDB [has joined Confluent](https://www.confluent.io/blog/pipelinedb-team-joins-confluent), read the blog post [here](https://www.pipelinedb.com/blog/pipelinedb-is-joining-confluent).\r\n\r\nPipelineDB will not have new releases beyond `1.0.0`, although critical bugs will still be fixed.\r\n\r\n# PipelineDB\r\n\r\n[![Gitter chat](https://img.shields.io/badge/gitter-join%20chat-brightgreen.svg?style=flat-square)](https://gitter.im/pipelinedb/pipelinedb)\r\n[![Twitter](https://img.shields.io/badge/twitter-@pipelinedb-55acee.svg?style=flat-square)](https://twitter.com/pipelinedb)\r\n\r\n## Overview\r\n\r\nPipelineDB is a PostgreSQL extension for high-performance time-series aggregation, designed to power realtime reporting and analytics applications.\r\n\r\nPipelineDB allows you to define [continuous SQL queries](http://docs.pipelinedb.com/continuous-views.html) that perpetually aggregate time-series data and store **only the aggregate output** in regular, queryable tables. You can think of this concept as extremely high-throughput, incrementally updated materialized views that never need to be manually refreshed.\r\n\r\nRaw time-series data is never written to disk, making PipelineDB extremely efficient for aggregation workloads.\r\n\r\nContinuous queries produce their own [output streams](http://docs.pipelinedb.com/streams.html#output-streams), and thus can be [chained together](http://docs.pipelinedb.com/continuous-transforms.html) into arbitrary networks of continuous SQL.\r\n\r\n## PostgreSQL compatibility\r\n\r\nPipelineDB runs on 64-bit architectures and currently supports the following PostgreSQL versions:\r\n\r\n* **PostgreSQL 10**: 10.1, 10.2, 10.3, 10.4, 10.5\r\n* **PostgreSQL 11**: 11.0\r\n\r\n## Getting started\r\n\r\nIf you just want to start using PipelineDB right away, head over to the [installation docs](http://docs.pipelinedb.com/installation.html) to get going.\r\n\r\nIf you'd like to build PipelineDB from source, keep reading!\r\n\r\n## Building from source\r\n\r\nSince PipelineDB is a PostgreSQL extension, you'll need to have the [PostgreSQL development packages](https://www.postgresql.org/download/) installed to build PipelineDB.\r\n\r\nNext you'll have to install [ZeroMQ](http://zeromq.org/) which PipelineDB uses for inter-process communication. [Here's](https://gist.github.com/derekjn/14f95b7ceb8029cd95f5488fb04c500a) a gist with instructions to build and install ZeroMQ from source.\r\nYou'll also need to install some Python dependencies if you'd like to run PipelineDB's Python test suite:\r\n\r\n```\r\npip install -r src/test/py/requirements.txt\r\n```\r\n\r\n#### Build PipelineDB:\r\n\r\nOnce PostgreSQL is installed, you can build PipelineDB against it:\r\n\r\n```\r\nmake USE_PGXS=1\r\nmake install\r\n```\r\n\r\n#### Test PipelineDB *(optional)*\r\nRun the following command:\r\n\r\n```\r\nmake test\r\n```\r\n\r\n#### Bootstrap the PipelineDB environment\r\nCreate PipelineDB's physical data directories, configuration files, etc:\r\n\r\n```\r\nmake bootstrap\r\n```\r\n\r\n**`make bootstrap` only needs to be run the first time you install PipelineDB**. The resources that `make bootstrap` creates may continue to be used as you change and rebuild PipeineDB.\r\n\r\n\r\n#### Run PipelineDB\r\nRun all of the daemons necessary for PipelineDB to operate:\r\n\r\n```\r\nmake run\r\n```\r\n\r\nEnter `Ctrl+C` to shut down PipelineDB.\r\n\r\n`make run` uses the binaries in the PipelineDB source root compiled by `make`, so you don't need to `make install` before running `make run` after code changes--only `make` needs to be run.\r\n\r\nThe basic development flow is:\r\n\r\n```\r\nmake\r\nmake run\r\n^C\r\n\r\n# Make some code changes...\r\nmake\r\nmake run\r\n```\r\n\r\n#### Send PipelineDB some data\r\n\r\nNow let's generate some test data and stream it into a simple continuous view. First, create the stream and the continuous view that reads from it:\r\n\r\n    $ psql\r\n    =# CREATE FOREIGN TABLE test_stream (key integer, value integer) SERVER pipelinedb;\r\n    CREATE FOREIGN TABLE\r\n    =# CREATE VIEW test_view WITH (action=materialize) AS SELECT key, COUNT(*) FROM test_stream GROUP BY key;\r\n    CREATE VIEW\r\n\r\nEvents can be emitted to PipelineDB streams using regular SQL `INSERTS`. Any `INSERT` target that isn't a table is considered a stream by PipelineDB, meaning streams don't need to have a schema created in advance. Let's emit a single event into the `test_stream` stream since our continuous view is reading from it:\r\n\r\n    $ psql\r\n    =# INSERT INTO test_stream (key, value) VALUES (0, 42);\r\n    INSERT 0 1\r\n\r\nThe 1 in the `INSERT 0 1` response means that 1 event was emitted into a stream that is actually being read by a continuous query. Now let's insert some random data:\r\n\r\n    =# INSERT INTO test_stream (key, value) SELECT random() * 10, random() * 10 FROM generate_series(1, 100000);\r\n    INSERT 0 100000\r\n\r\nQuery the continuous view to verify that the continuous view was properly updated. Were there actually 100,001 events counted?\r\n\r\n    $ psql -c \"SELECT sum(count) FROM test_view\"\r\n      sum\r\n    -------\r\n    100001\r\n    (1 row)\r\n\r\nWhat were the 10 most common randomly generated keys?\r\n\r\n    $ psql -c \"SELECT * FROM test_view ORDER BY count DESC limit 10\"\r\n\tkey  | count \r\n\t-----+-------\r\n\t 2   | 10124\r\n\t 8   | 10100\r\n\t 1   | 10042\r\n\t 7   |  9996\r\n\t 4   |  9991\r\n\t 5   |  9977\r\n\t 3   |  9963\r\n\t 6   |  9927\r\n\t 9   |  9915\r\n\t10   |  4997\r\n\t 0   |  4969\r\n\r\n\t(11 rows)\r\n","funding_links":[],"categories":["C","Table of Contents","Stream Processing","analytics"],"sub_categories":["Streaming SQL"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpipelinedb%2Fpipelinedb","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpipelinedb%2Fpipelinedb","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpipelinedb%2Fpipelinedb/lists"}