{"id":20632467,"url":"https://github.com/citusdata/postgres_vectorization_test","last_synced_at":"2025-10-12T00:08:37.570Z","repository":{"id":21589097,"uuid":"24909153","full_name":"citusdata/postgres_vectorization_test","owner":"citusdata","description":"Vectorized executor to speed up PostgreSQL","archived":false,"fork":false,"pushed_at":"2015-03-31T10:50:49.000Z","size":394,"stargazers_count":332,"open_issues_count":1,"forks_count":24,"subscribers_count":58,"default_branch":"master","last_synced_at":"2025-04-01T13:53:21.802Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"C","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/citusdata.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2014-10-07T20:03:26.000Z","updated_at":"2025-02-20T07:51:45.000Z","dependencies_parsed_at":"2022-08-21T20:20:58.641Z","dependency_job_id":null,"html_url":"https://github.com/citusdata/postgres_vectorization_test","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/citusdata%2Fpostgres_vectorization_test","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/citusdata%2Fpostgres_vectorization_test/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/citusdata%2Fpostgres_vectorization_test/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/citusdata%2Fpostgres_vectorization_test/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/citusdata","download_url":"https://codeload.github.com/citusdata/postgres_vectorization_test/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248143962,"owners_count":21054854,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-16T14:16:17.257Z","updated_at":"2025-10-12T00:08:32.552Z","avatar_url":"https://github.com/citusdata.png","language":"C","readme":"Vectorized Executor\n===================\n\nI interned at Citus Data this summer, and implemented a vectorized executor for\nPostgreSQL. We observed performance improvements of 3-4x for simple SELECT\nqueries with vectorized execution, and decided to open source my project as a\nproof of concept.\n\nThis readme first describes the motivation behind my internship, and my journey\nwith PostgreSQL, database execution engines, and GProf. If you'd like to skip that,\nyou can also jump to [build instructions](README.md#building).\n\nMotivation\n----------\n\nI'm a second year student at Bogazici University in Istanbul, and I interned at\n[Citus](http://citusdata.com/). When I started my internship, my mentor Metin \ndescribed to me a common question they were hearing from customers: \"I can fit \nmy working set into memory, thanks to cheaper RAM, columnar stores, or scaling \nout of data to multiple machines. Now my analytic queries are bottlenecked on \nCPU. Can these queries go faster?\"\n\nSince this question's scope was too broad, we decided to pick a simple, yet\nuseful and representative query. My goal was to go crazy with this (class of)\nquery's performance. Ideally, my changes would also apply to other queries.\n\n    postgres=# SELECT l_returnflag,\n    \t       \t      sum(l_quantity) as sum_qty,\n    \t\t          count(*) as count_order\n    \t       FROM lineitem\n    \t       GROUP BY l_returnflag;\n\n\nTechnical Details\n-----------------\n\nI started my challenge by compiling PostgreSQL for performance profiling, and\nrunning it with a CPU-profiler called GProf. I then ran our example SELECT\nquery 25 times to make sure GProf collected enough samples, and looked at the\nprofile output.\n\nIn particular, I was looking for any \"hot spot\" functions whose behavior I could\nunderstand and change without impacting correctness. For this, I digged down the\nGProf call graph, and found the top three functions whose behavior looked\nself-contained enough for me to understand.\n\n    index  %time    self  children    called      name\n    ...\n    [7]     43.3    0.77    17.20    150030375    LookupTupleHashEntry [7]\n    [9]     24.0    1.37     8.60    150030375    advance_aggregates [9]\n    [17]    12.0    0.26     4.73    150030400    SeqNext [17]\n\nThese numbers discouraged me in three ways. First, I was hoping to find a single\nfunction that *was* the performance bottleneck. Instead, PostgreSQL was\nspending a proportional amount of time scanning over the lineitem table's tuples\n[17], projecting relevant columns from each tuple and grouping them [7], and\napplying aggregate functions on grouped values [9].\n\nSecond, I read the code for these functions, and found that they were already\noptimized. I also found out through profile results that Postgres introduced a\nper-tuple overhead. For each tuple, it stored and cleared tuples, dispatched to\nrelevant executor functions, performed MVCC checks, and so forth.\n\nThird, I understood at a higher level that PostgreSQL was scanning tuples,\nprojecting columns, and computing aggregates. What I didn't understand was the\ndependencies between the thousand other functions involved in query execution.\nIn fact, whenever I made a change, I spent more time crashing and debugging the\ndatabase than the change itself.\n\nTo mitigate these issues, we decided to redefine the problem one month into my\ninternship. To simplify the problem of understanding many internal PostgreSQL\nfunctions, we decided to apply my performance optimizations on the columnar\nstore extension. This decision had the additional benefit of slashing CPU usage\nrelated to column projections [7].\n\nThen, to speed up tuple scans and aggregate computations, and also to reduce the\nper-tuple CPU overhead, we decided to try an interesting idea called vectorized\nexecution.\n\n[Vectorized execution](http://www.cse.ust.hk/damon2011/proceedings/p5-sompolski.pdf)\nwas popularized by the [MonetDB/X100](http://oai.cwi.nl/oai/asset/16497/16497B.pdf)\nteam. This idea is based on the observation that most database engines follow an \niterator-based execution model, where each database operator implements a next() \nmethod. Each call to next() produces one new tuple that may in turn be passed to \nother operators. This \"tuple at a time\" model introduces an interpretation overhead\nand also adversely affects high performance features in modern CPUs. Vectorized\nexecution reduces these overheads by using bulk processing. In this new model,\nrather than producing one tuple on each call, next() operates on and produces a\nbatch of tuples (usually 100-10K).\n\nWith the vectorization idea in mind, I started looking into cstore_fdw to see\nhow I could implement aggregate functions. Sadly, PostgreSQL didn't yet have\naggregate push downs for foreign tables. On the bright side, it provided these\npowerful hooks that enabled developers to intercept query planning and execution\nlogic however they liked.\n\nI started simple this time. I initially overlooked groupings, and implemented\nvectorized versions of simple sum(), avg(), and count() on common data types. I\nthen grabbed the execution hook, and routed any relevant queries to my\nvectorized functions.\n\nNext, I generated TPC-H data with a scale factor of 1, loaded the data into the\ndatabase, and made sure that data was always *in-memory*. I then ran\nsimple \"Select sum(l_quantity) From lineitem\" type queries, and compared the\nregular executor to the vectorized version.\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"images/simple_aggregates.png?raw=true\" alt=\"Run-times for simple aggregates\"/\u003e\n\u003c/p\u003e\n\nThe results looked cheerful. Vectorized functions showed performance benefits of\n4-6x across different aggregate functions and data types. The simplest of these\nfunctions also had the greatest benefits, plain count(\\*). This wasn't all that\nsurprising. The standard executor was calling count(\\*) on one new tuple,\ncount(\\*) was incrementing a counter, and then onto the next tuple. The\nvectorized version was instead a simple for() loop over a group of values.\n\nFrom there, I started looking into queries that aggregated and grouped their\nresults on one dimension. This time, I made changes to implement vectorized\n\"hash aggregates\", and again routed any relevant queries to my vectorized\nfunctions. I then compared the two executors for simple group bys with\naggregates.\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"images/groupby_aggregates.png?raw=true\" alt=\"Run-times for group by aggregates\"/\u003e\n\u003c/p\u003e\n\nThese results showed performance benefits of around 3x. These improvements were\nsmall in comparison to plain aggregate vectorization because of the generic hash\nfunction's computational overhead. Also, I only had time to implement hash\nvectorization for Postgres' pass-by-value data types. In fact, it took me quite\na while to understand that Postgres even had pass-by-value types, and that those\ndiffered from pass-by-reference ones.\n\nIf I had the time, I'd look into making my hash aggregate logic more generic.\nI'd also try an alternate hashing function that's more specialized, one that\nonly operates on 1-2 columns, and as a result, that goes faster.\n\nLearnings\n---------\n\nIn the end, I feel that I learned a lot during my internship at Citus. My first\ntakeaway was that reading code takes much more time than writing it. In fact,\nonly after two months of reading code and asking questions, did I start to\nunderstand how the PostgreSQL aggregate logic worked.\n\nSecond, it's exciting to try out new ideas in databases. For beginners, the best\nway to start is to carve out a specific piece of functionality, find the related\n[PostgreSQL extension API](http://www.postgresql.org/docs/9.3/static/extend.html), \nand start implementing against it.\n\nFinally, I'm happy that we're open sourcing my work. I realize that the code\nisn't as robust or generic as PostgreSQL is. That said, I know a lot more about\nPostgreSQL and love it, and I can only hope that the ideas in here will\nstimulate others.\n\n\nBuilding\n--------\n\nThe vectorized execution logic builds on the\n[cstore\\_fdw](https://github.com/citusdata/cstore_fdw) extension. Therefore,\nthe dependencies and build steps are exactly the same between the two\nextensions. The difference is that cstore\\_fdw reduces the amount of disk I/O\nby only reading relevant columns and compression data. This extension helps more\nwhen the working set fits into memory. In that case, it reduces the CPU overhead\nby vectorizing simple SELECT queries.\n\nMy primary goal with this extension was to test vectorization's potential\nbenefits. As such, we'd love for you to try this out and give us any feedback.\nAt the same time, please don't consider this extension as generic and\nproduction-ready database code. PostgreSQL does set a high bar there.\n\ncstore\\_fdw depends on protobuf-c for serializing and deserializing table metadata.\nSo we need to install these packages first:\n\n    # Fedora 17+, CentOS, and Amazon Linux\n    sudo yum install protobuf-c-devel\n\n    # Ubuntu 10.4+\n    sudo apt-get install protobuf-c-compiler\n    sudo apt-get install libprotobuf-c0-dev\n\n    # Mac OS X\n    brew install protobuf-c\n\n**Note.** In CentOS 5 and 6, you may need to install or update EPEL 5 or EPEL 6\nrepositories. See [this page]\n(http://www.rackspace.com/knowledge_center/article/installing-rhel-epel-repo-on-centos-5x-or-6x)\nfor instructions.\n\n**Note.** In Amazon Linux, EPEL 6 repository is installed by default, but it is not\nenabled. See [these instructions](http://aws.amazon.com/amazon-linux-ami/faqs/#epel)\nfor how to enable it.\n\nOnce you have protobuf-c installed on your machine, you are ready to build\ncstore\\_fdw.  For this, you need to include the pg\\_config directory path in\nyour make command. This path is typically the same as your PostgreSQL\ninstallation's bin/ directory path. For example:\n\n    PATH=/usr/local/pgsql/bin/:$PATH make\n    sudo PATH=/usr/local/pgsql/bin/:$PATH make install\n\n**Note.** postgres_vectorization_test requires PostgreSQL 9.3. It doesn't support \nother versions of PostgreSQL.\n\nBefore using cstore\\_fdw, you also need to add it to ```shared_preload_libraries```\nin your ```postgresql.conf``` and restart Postgres:\n\n    shared_preload_libraries = 'cstore_fdw'    # (change requires restart)\n\nYou can use PostgreSQL's ```COPY``` command to load or append data into the table.\nYou can use PostgreSQL's ```ANALYZE table_name``` command to collect statistics\nabout the table. These statistics help the query planner to help determine the\nmost efficient execution plan for each query.\n\n\nExample\n-------\n\nAs an example, we demonstrate loading and querying data to/from a column store\ntable from scratch here. Let's start with downloading and decompressing the data\nfiles.\n\n    wget http://examples.citusdata.com/customer_reviews_1998.csv.gz\n    wget http://examples.citusdata.com/customer_reviews_1999.csv.gz\n\n    gzip -d customer_reviews_1998.csv.gz\n    gzip -d customer_reviews_1999.csv.gz\n\nThen, let's log into Postgres, and run the following commands to create a column\nstore foreign table:\n\n```SQL\n-- load extension first time after install\nCREATE EXTENSION cstore_fdw;\n\n-- create server object\nCREATE SERVER cstore_server FOREIGN DATA WRAPPER cstore_fdw;\n\n-- create foreign table\nCREATE FOREIGN TABLE customer_reviews\n(\n    customer_id TEXT,\n    review_date DATE,\n    review_rating INTEGER,\n    review_votes INTEGER,\n    review_helpful_votes INTEGER,\n    product_id CHAR(10),\n    product_title TEXT,\n    product_sales_rank BIGINT,\n    product_group TEXT,\n    product_category TEXT,\n    product_subcategory TEXT,\n    similar_product_ids CHAR(10)[]\n)\nSERVER cstore_server;\n```\n\nNext, we load data into the table:\n\n```SQL\nCOPY customer_reviews FROM '/home/user/customer_reviews_1998.csv' WITH CSV;\nCOPY customer_reviews FROM '/home/user/customer_reviews_1999.csv' WITH CSV;\n```\n\n**Note.** If you are getting ```ERROR: cannot copy to foreign table\n\"customer_reviews\"``` when trying to run the COPY commands, double check that you\nhave added cstore\\_fdw to ```shared_preload_libraries``` in ```postgresql.conf```\nand restarted Postgres.\n\nNext, we collect data distribution statistics about the table. This is optional,\nbut usually very helpful:\n\n```SQL\nANALYZE customer_reviews;\n```\n\nFinally, let's run some simple SQL queries and see how vectorized execution\nperforms. We also encourage you to load the same data into a regular PostgreSQL\ntable, and compare query performance differences.\n\n```SQL\n-- Number of customer reviews\nSELECT count(*) FROM customer_reviews;\n\n-- Average and total votes for all customer reviews\nSELECT avg(review_votes), sum(review_votes) FROM customer_reviews;\n\n-- Total number of helpful votes per product category\nSELECT\n    product_group, sum(review_helpful_votes) AS total_helpful_votes\nFROM\n    customer_reviews\nGROUP BY\n    product_group;\n\n-- Number of reviews by date (year, month, day)\nSELECT\n    review_date, count(review_date) AS review_count\nFROM\n    customer_reviews\nGROUP BY\n    review_date;\n```\n\n\nLimitations\n-----------\n\nThe vectorized executor intercepts PostgreSQL's query execution hook. If the\nextension can't process the current query, it hands the query over to the\nstandard Postgres executor. Therefore, if your query isn't going any faster,\nthen we currently don't support vectorization for it.\n\nThe current set of vectorized queries are limited to simple aggregates (sum,\ncount, avg) and aggregates with group bys. The next set of changes I wanted to\nincorporate into the vectorized executor are: filter clauses, functions or\nexpressions, expressions within aggregate functions, groups by that support \nmultiple columns or aggregates, and passing vectorized tuples from groupings to \norder by clauses.\n\nI think all except the last one are relatively easy, but I didn't have the time\nto work on them. The last one is harder as PostgreSQL's query planner follows\na recursive pull model. In this model, each relational operator is called\nrecursively to traverse the operator tree from the root downwards, with the\nresult tuples being pulled upwards. Such a recursion occurs with the aggregate\noperator, and I could intercept all operators that are below the aggregate\noperator in the query plan tree. If there was an operator on top of the\naggregate operator, such as an order by, I may have had to copy code from the\nexecutor to properly intercept the recursion.\n\n\nUsage with CitusDB\n--------------------\n\nThe example above illustrated how to load data into a PostgreSQL database running\non a single host. However, sometimes your data is too large to analyze effectively\non a single host. CitusDB is a product built by Citus Data that allows you to run\na distributed PostgreSQL database to analyze your data using the power of multiple\nhosts. CitusDB is based on a modern PostgreSQL version and allows you to easily\ninstall PostgreSQL extensions and foreign data wrappers, including cstore_fdw. For\nan example of how to use cstore\\_fdw with CitusDB see the\n[CitusDB documentation][citus-cstore-docs].\n\n\nUninstalling cstore_fdw\n-----------------------\n\nBefore uninstalling the extension, first you need to drop all the cstore tables:\n\n    postgres=# DROP FOREIGN TABLE cstore_table_1;\n    ...\n    postgres=# DROP FOREIGN TABLE cstore_table_n;\n\nThen, you should drop the cstore server and extension:\n\n    postgres=# DROP SERVER cstore_server;\n    postgres=# DROP EXTENSION cstore_fdw;\n\ncstore\\_fdw automatically creates some directories inside the PostgreSQL's data\ndirectory to store its files. To remove them, you can run:\n\n    $ rm -rf $PGDATA/cstore_fdw\n\nThen, you should remove cstore\\_fdw from ```shared_preload_libraries``` in\nyour ```postgresql.conf```:\n\n    shared_preload_libraries = ''    # (change requires restart)\n\nFinally, to uninstall the extension you can run the following command in the\nextension's source code directory. This will clean up all the files copied during\nthe installation:\n\n    $ sudo PATH=/usr/local/pgsql/bin/:$PATH make uninstall\n\n\nCopyright\n---------\n\nCopyright (c) 2014 Citus Data, Inc.\n\nThis module is free software; you can redistribute it and/or modify it under the\nApache v2.0 License.\n\nFor all types of questions and comments about the wrapper, please contact us at\nengage @ citusdata.com.\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcitusdata%2Fpostgres_vectorization_test","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcitusdata%2Fpostgres_vectorization_test","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcitusdata%2Fpostgres_vectorization_test/lists"}