{"id":19458527,"url":"https://github.com/postgrespro/lsm","last_synced_at":"2025-04-25T06:30:29.050Z","repository":{"id":40427135,"uuid":"274192835","full_name":"postgrespro/lsm","owner":"postgrespro","description":"RocksDB FDW for PostgreSQL","archived":false,"fork":false,"pushed_at":"2022-05-17T07:52:21.000Z","size":89,"stargazers_count":27,"open_issues_count":2,"forks_count":3,"subscribers_count":10,"default_branch":"master","last_synced_at":"2025-04-24T10:48:41.342Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"C","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/postgrespro.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-06-22T16:46:27.000Z","updated_at":"2025-03-18T17:31:03.000Z","dependencies_parsed_at":"2022-08-09T20:10:39.425Z","dependency_job_id":null,"html_url":"https://github.com/postgrespro/lsm","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/postgrespro%2Flsm","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/postgrespro%2Flsm/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/postgrespro%2Flsm/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/postgrespro%2Flsm/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/postgrespro","download_url":"https://codeload.github.com/postgrespro/lsm/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250766958,"owners_count":21483894,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-10T17:27:26.503Z","updated_at":"2025-04-25T06:30:28.729Z","avatar_url":"https://github.com/postgrespro.png","language":"C","funding_links":[],"categories":["C"],"sub_categories":[],"readme":"\u003cimg style=\"width:100%;\" src=\"/github-banner.png\"\u003e\n\n# PostgresForeignDataWrapper\n\n[![Build Status](https://travis-ci.com/vidardb/PostgresForeignDataWrapper.svg?branch=master)](https://travis-ci.com/github/vidardb/PostgresForeignDataWrapper)\n\nThis PostgreSQL extension implements a Foreign Data Wrapper (FDW) for [RocksDB](https://rocksdb.org/). This repo has been listed in PostgreSQL [wiki](https://wiki.postgresql.org/wiki/Foreign_data_wrappers). \n\nRocksDB is a high performance key-value store based on a log-structured merge-tree (LSM tree). RocksDB can efficiently use many CPU cores and fast storage. This is the first foreign data wrapper that connects a LSM-tree-based storage engine to PostgreSQL. Because RocksDB is an embeddable key-value store, you do not need to run another server to use this extension.\n\nThis extension can also be used for other systems that have RocksDB-like APIs, but please check the compatibility before you use this extension for other systems.\n\nThis extension is developed and maintained by the VidarDB team. Feel free to report bugs or issues via Github.\n\n# Building\n\nWe test this foreign data wrapper on Ubuntu Server 18.04 using PostgreSQL-11 together with RocksDB-6.2.4 (built with GCC-7.4.0).\n\n- Install PostgreSQL and the dev library which is required by extensions:\n\n  ```sh\n  # add the repository\n  sudo tee /etc/apt/sources.list.d/pgdg.list \u003c\u003c END\n  deb http://apt.postgresql.org/pub/repos/apt/ bionic-pgdg main\n  END\n\n  # get the signing key and import it\n  wget https://www.postgresql.org/media/keys/ACCC4CF8.asc\n  sudo apt-key add ACCC4CF8.asc\n\n  # fetch the metadata from the new repo\n  sudo apt-get update\n\n  # install postgresql and the dev library\n  sudo apt-get install postgresql-11\n  sudo apt-get install postgresql-server-dev-11\n  ```\n\n- Install [RocksDB](https://github.com/facebook/rocksdb) from source code:\n\n  ```sh\n  git clone -b v6.2.4 https://github.com/facebook/rocksdb.git\n\n  cd rocksdb\n\n  sudo DEBUG_LEVEL=0 make shared_lib install-shared\n  \n  sudo sh -c \"echo /usr/local/lib \u003e\u003e /etc/ld.so.conf\"  \n \n  sudo ldconfig\n  ```\n\n- Build this foreign data wrapper:\n\n  ```sh\n  git clone git@github.com:postgrespro/lsm.git\n\n  cd lsm\n\n  make\n\n  sudo make install\n  ```\n\n- Before using this foreign data wrapper, we need to add it to `shared_preload_libraries` in the `postgresql.conf`:\n\n  ```sh\n  echo \"shared_preload_libraries = 'lsm'\" \u003e\u003e postgresql.conf\n  ```\n\n  and restart PostgreSQL:\n\n  ```sh\n  sudo service postgresql restart\n  ```\n\n- When uninstall this extension, first issue the following commands, and then delete the data by locating PostgreSQL data folder via `show data_directory;` in PostgreSQL terminal.\n\n  ```sh\n  cd PostgresForeignDataWrapper\n  \n  sudo make uninstall\n  ```\n\n# Limitations\n\n- The first attribute in the table definition must be the primary key.\n\n- Composite primary key is not supported for now.\n\n- ACID relies on the storage engine.\n\n- Data types of Postgres are not natively supported. \n\n# Usage\n\nThis extension does not have any parameter. After creating the extension and corresponding server, you can use RocksDB as a foreign storage engine for your PostgreSQL.\n\nA simple example is as follows (*you can run '`sudo -u postgres psql -U postgres`' to connect the local postgresql server*):\n\n\n```\n    CREATE DATABASE example;  \n    \\c example  \n\n    CREATE EXTENSION lsm;  \n    CREATE SERVER lsm_server FOREIGN DATA WRAPPER lsm_fdw;  \n\n    CREATE FOREIGN TABLE student(id INTEGER, name TEXT) SERVER lsm_server;  \n\n    INSERT INTO student VALUES(20757123, 'Rafferty');  \n    SELECT * FROM student;  \n\n    INSERT INTO student VALUES(20767234, 'Jones');  \n    SELECT * FROM student;  \n\n    DELETE FROM student WHERE name='Jones';  \n    SELECT * FROM student;  \n\n    UPDATE student SET name='Tom' WHERE id=20757123;  \n    SELECT * FROM student;  \n\n    DROP FOREIGN TABLE student;  \n\n    DROP SERVER lsm_server;  \n    DROP EXTENSION lsm_fdw;  \n  \n    \\c postgres  \n    DROP DATABASE example;  \n\n``` \n\n# Testing\n\nWe have tested certain typical SQL statements and will add more test cases later. The test scripts are in the test/sql folder which are recommended to be placed in a non-root directory. The corresponding results can be found in the test/expected folder. You can run the tests in the following way:\n\n\n```sh\n    sudo service postgresql restart  \n\n    cd PostgresForeignDataWrapper\n\n    sudo -u postgres psql -U postgres -a -f test/sql/create.sql \n\n    sudo -u postgres psql -U postgres -d lsmtest -a -f test/sql/test.sql \n\n    sudo -u postgres psql -U postgres -d lsmtest -a -f test/sql/clear.sql  \n```\n\n# Debug \n\nIf you want to debug the source code, you may need to start PostgreSQL in the debug mode:\n\n\n```sh\n    sudo service postgresql stop  \n\n    sudo -u postgres /usr/lib/postgresql/11/bin/postgres -d 0 -D /var/lib/postgresql/11/main -c config_file=/etc/postgresql/11/main/postgresql.conf\n```  \n\n# Docker\n\nWe can also run PostgreSQL with RocksDB in Docker container and you can refer to [here](docker_image/README.md).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpostgrespro%2Flsm","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpostgrespro%2Flsm","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpostgrespro%2Flsm/lists"}