{"id":19458463,"url":"https://github.com/postgrespro/vops","last_synced_at":"2025-04-04T12:06:57.869Z","repository":{"id":17696180,"uuid":"81828870","full_name":"postgrespro/vops","owner":"postgrespro","description":null,"archived":false,"fork":false,"pushed_at":"2025-02-10T10:00:43.000Z","size":547,"stargazers_count":163,"open_issues_count":1,"forks_count":23,"subscribers_count":39,"default_branch":"master","last_synced_at":"2025-03-28T10:02:07.489Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"C","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/postgrespro.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-02-13T13:37:56.000Z","updated_at":"2025-02-11T23:57:38.000Z","dependencies_parsed_at":"2024-04-27T21:26:34.840Z","dependency_job_id":"f907d2ee-2944-43a5-b7cb-4954268aa78c","html_url":"https://github.com/postgrespro/vops","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/postgrespro%2Fvops","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/postgrespro%2Fvops/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/postgrespro%2Fvops/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/postgrespro%2Fvops/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/postgrespro","download_url":"https://codeload.github.com/postgrespro/vops/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247174407,"owners_count":20896076,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-10T17:27:11.439Z","updated_at":"2025-04-04T12:06:57.847Z","avatar_url":"https://github.com/postgrespro.png","language":"C","funding_links":[],"categories":[],"sub_categories":[],"readme":"## \u003cspan id=\"motivation\"\u003eMotivation\u003c/span\u003e\n\nPostgreSQL looks very competitive with other mainstream databases on\nOLTP workload (execution of large number of simple queries). But on OLAP\nqueries, requiring processing of larger volumes of data, DBMS-es\noriented on analytic queries processing can provide an order of\nmagnitude better speed. Let's investigate why it happen and can we do\nsomething to make PostgreSQL efficient also for OLAP\nqueries.\n\n## \u003cspan id=\"profiling\"\u003eWhere DBMS spent most of the time during processing OLAP queries?\u003c/span\u003e\n\nProfiling execution of queries shows several main factors, limiting\nPostgres performance:\n\n1.  Unpacking tuple overhead (tuple\\_deform). To be able to access\n    column values, Postgres needs to deform the tuple. Values can be\n    compressed, stored at some other page (TOAST), ... Also, as far as\n    size of column can be varying, to extract N-th column we need to\n    unpack preceding N-1 columns. So deforming tuple is quite expensive\n    operation, especially for tables with large number of attributes. In\n    some cases rearranging columns in the table allows to significantly\n    reduce query execution time. Another universal approach is to split\n    table into two: one small with frequently accessed scalar columns\n    and another with rarely used large columns. Certainly in this case\n    we need to perform extra join but it allows to several times reduce\n    amount of fetched data. In queries like TPC-H Q6, tuple deform takes\n    about 40% of total query execution time.\n2.  Interpretation overhead. Postgres compiler and optimizer build tree\n    representing query execution plan. So query executor performs\n    recursive invocation of evaluate functions for nodes of this tree.\n    Implementation of some nodes also contain switches used to select\n    requested action. So query plan is interpreted by Postgres query\n    executor rather than directly executed. Usually interpreter is about\n    10 times slower than native code. This is why elimination of\n    interpretation overhead allows to several times increase query\n    speed, especially for queries with complex predicates where most\n    time is spent in expression evaluation.\n3.  Abstraction penalty. Support of abstract (user defined) types and\n    operations is one of the key features of Postgres. It's executor is\n    able to deal not only with built-in set of scalar types (like\n    integer, real, ...) but with any types defined by user (for example\n    complex, point,...). But the price of such flexibility is that each\n    operations requires function call. Instead of adding to integers\n    directly, Postgres executor invokes function which performs addition\n    of two integers. Certainly in this case function call overhead is\n    much larger then performed operation itself. Function call overhead\n    is also increased because of Postgres function call convention\n    requiring passing parameter values through memory (not using\n    register call convention).\n4.  Pull model overhead. Postgres executor is implementing classical\n    Volcano-style query execution model - pull model. Operand's values\n    are pulled by operator. It simplifies executor and operators\n    implementation. But it has negative impact on performance, because\n    leave nodes (fetching tuple from heap or index page) have to do a\n    lot of extra work saving and restoring their context.\n5.  MVCC overhead. Postgres provides multiversion concurrency control,\n    which allows multiple transactions to concurrently work with the\n    same record without blocking each other. It is goods for frequently\n    updated data (OLTP), but for read-only or append-only data in OLAP\n    scenarios it adds just extra overhead. Both space overhead (about 20\n    extra bytes per tuple) and CPU overhead (checking visibility of each\n    tuple).\n\nThere are many different ways of addressing this issues. For example we\ncan use JIT (Just-In-Time) compiler to generate native code for query\nand eliminate interpretation overhead and increase heap deform speed. We\ncan rewrite optimizer from pull to push model. We can try to optimize\ntuple format to make heap deforming more efficient. Or we can generate\nbyte code for query execution plan which interpretation is more\nefficient than recursive invocation of evaluate function for each node\nbecause of better access locality. But all this approaches require\nsignificant rewriting of Postgres executor and some of them also require\nchanges of all Postgres architecture.\n\nBut there is an approach which allows to address most of this issues\nwithout radical changes of executor. It is vector operations. It is\nexplained in next section.\n\n## \u003cspan id=\"vertical\"\u003eVertical storage\u003c/span\u003e\n\nTraditional query executor (like Postgres executor) deals with single\nrow of data at each moment of time. If it has to evaluate expression\n(x+y) then it fetches value of \"x\", then value of \"y\", performs\noperation \"+\" and returns the result value to the upper node. In\ncontrast vectorized executor is able to process in one operation\nmultiple values. In this case \"x\" and \"y\" represent not just a single\nscalar value, but vector of values and result is also a vector of\nvalues. In vector execution model interpretation and function call\noverhead is divided by size of vector. The price of performing function\ncall is the same, but as far as function proceeds N values instead of\njust one, this overhead become less critical.\n\nWhat is the optimal size for the vector? From the explanation above it\nis clear that the larger vector is, the less per-row overhead we have.\nSo we can form vector from all values of the correspondent table\nattribute. It is so called vertical data model or columnar store. Unlike\nclassical \"horizontal\" data model where the unit of storing data is row\n(tuple), here we have vertical columns. Columnar store has the following\nmain advantages:\n\n  - Reduce size of fetched data: only columns used in query need to be\n    fetched\n  - Better compression: storing all values of the same attribute\n    together makes it possible to much better and faster compress them,\n    for example using delta encoding.\n  - Minimize interpretation overhead: each operation is perform not for\n    single value, but for set of values\n  - Use CPU vector instruction (SIMD) to process data\n\nThere are several DBMS-es implementing columnar store model. Most\npopular are [Vertica](https://vertica.com/),\n[MonetDB](https://www.monetdb.org/Home). But actually performing\noperation on the whole column is not so good idea. Table can be very\nlarge (OLAP queries are used to work with large data sets), so vector\ncan also be very big and even doesn't fit in memory. But even if it fits\nin memory, working with such larger vectors prevent efficient\nutilization of CPU caches (L1, L2,...). Consider expression\n(x+y)\\*(x-y). Vector executor performs addition of two vectors : \"x\" and\n\"y\" and produces result vector \"r1\". But when last element of vector \"r\"\nis produced, first elements of vector \"r1\" are already thrown from CPU\ncache, as well as first elements of \"x\" and \"y\" vectors. So when we need\nto calculate (x-y) we once again have to load data for \"x\" and \"y\" from\nslow memory to fast cache. Then we produce \"r2\" and perform\nmultiplication of \"r1\" and \"r2\". But here we also need first to load\ndata for this vectors into the CPU cache.\n\nSo it is more efficient to split column into relatively small *chunks*\n(or *tiles* - there is no single notion for it accepted by everyone).\nThis chunk is a unit of processing by vectorized executor. Size of such\nchunk is chosen to keep all operands of vector operations in cache even\nfor complex expressions. Typical size of chunk is from 100 to 1000\nelements. So in case of (x+y)\\*(x-y) expression, we calculate it not for\nthe whole column but only for 100 values (assume that size of the chunk\nis 100). Splitting columns into chunks in successors of MonetDB x100 and\nHyPer allows to increase speed up to ten times.\n\n## \u003cspan id=\"vops\"\u003eVOPS\u003c/span\u003e\n\n\u003cspan id=\"overview\"\u003eOverview\u003c/span\u003e\n\nThere are several attempts to integrate columnar store in PostgreSQL.\nThe most known is [CStore FDW](https://github.com/citusdata/cstore_fdw)\nby CitusDB. It is implemented as foreign data wrapper (FDW) and is\nefficient for queries fetching relatively small fraction of columns. But\nit is using standard Postgres raw-based executor and so is not able to\ntake advantages of vector processing. There is interesting\n[project](https://github.com/citusdata/postgres_vectorization_test) done\nby CitusDB intern. He implements vector operations on top of CStore\nusing executors hooks for some nodes. IT reports 4-6 times speedup for\ngrand aggregates and 3 times speedup for aggregation with group by.\n\nAnother project is [IMCS](https://github.com/knizhnik/imcs): In-Memory\nColumnar Store. Here columnar store is implemented in memory and is\naccessed using special functions. So you can not use standard SQL query\nto work with this storage - you have to rewrite it using IMCS functions.\nIMCS provides vector operations (using tiles) and parallel execution.\n\nBoth CStore and IMCS are keeping data outside Postgres. But what if we\nwant to use vector operations for data kept in standard Postgres tables?\nDefinitely, the best approach is to impalement alternative heap format.\nOr even further: eliminate notion of heap at all - treat heap just as\nyet another access method, similar with other indexes.\n\nBut such radical changes requires deep redesign of all Postgres\narchitecture. It will be better to estimate first possible advantages we\ncan expect from usage of vector vector operations. Vector executor is\nwidely discussed in Postgres forums, but efficient vector executor is\nnot possible without underlying support at storage layer. Advantages of\nvector processing will be annihilated if vectors are formed from\nattributes of rows extracted from existed Postgres heap page.\n\nThe idea of VOPS extension is to implement vector operations for tiles\nrepresented as special Postgres types. Tiles should be used as table\ncolumn types instead of scalar types. For example instead of \"real\" we\nshould use \"vops\\_float4\" which is tile representing up to 64 values of\nthe correspondent column. Why 64? There are several reasons for choosing\nthis number:\n\n1.  We provide efficient access to tiles, we need that\n    size\\_of\\_tile\\*size\\_of\\_attribute\\*number\\_of\\_attributes is\n    smaller than page size. Typical record contains about 10 attributes,\n    default size of Postgres page is 8kb.\n2.  64 is number of bits in large word. We need to maintain bitmask to\n    mark null values. Certainly it is possible to store bitmask in array\n    with arbitrary size, but manipulation with single 64-bit integer is\n    more efficient.\n3.  Due to the arguments above, to efficiently utilize cache, size of\n    tile should be in range 100..1000.\n\nVOPS is implemented as Postgres extension. It doesn't change anything in\nPostgres executor and page format. It also doesn't setup any executors\nhooks or alter query execution plan. The main idea of this project was\nto measure speedup which can be reached by using vector operation with\nexisted executor and heap manager. VOPS provides set of standard\noperators for tile types, allowing to write SQL queries in the way\nsimilar with normal SQL queries. Right now vector operators can be used\ninside predicates and aggregate expressions. Joins are not currently\nsupported. Details of VOPS architecture are described below.\n\n### \u003cspan id=\"types\"\u003eTypes\u003c/span\u003e\n\nVOPS supports all basic Postgres numeric types: 1,2,4,8 byte integers\nand 4,8 bytes floats. Also it supports `date` and `timestamp` types but\nthem are using the same implementation as `int4` and `int8`\ncorrespondingly.\n\n| SQL type                | C type    | VOPS tile type  |\n| ----------------------- | --------- | --------------- |\n| bool                    | bool      | vops\\_bool      |\n| \"char\"                  | char      | vops\\_char      |\n| int2                    | int16     | vops\\_int2      |\n| int4                    | int32     | vops\\_int4      |\n| int8                    | int64     | vops\\_int8      |\n| float4                  | float4    | vops\\_float4    |\n| float8                  | float8    | vops\\_float8    |\n| date                    | DateADT   | vops\\_date      |\n| timestamp               | Timestamp | vops\\_timestamp |\n| char(N), varchar(N)     | text      | vops\\_text      |\n\nVOPS doesn't support work with strings (char or varchar types), except\ncase of single character. If strings are used as identifiers, in most\ncases it is preferable to place them in some dictionary and use integer\nidentifiers instead of original strings.\n\n### \u003cspan id=\"operators\"\u003eVector operators\u003c/span\u003e\n\nVOPS provides implementation of all built-in SQL arithmetic operations\nfor numeric types: **+ - / \\*** Certainly it also implements all\ncomparison operators: **= \\\u003c\\\u003e \\\u003e \\\u003e= \\\u003c \\\u003c=**. Operands of such\noperators can be either tiles, either scalar constants: `x=y` or `x=1`.\n\nBoolean operators `and`, `or`, `not` can not be overloaded. This is why\nVOPS provides instead of them operators **\u0026 | \\!**. Please notice that\nprecedence of this operators is different from `and`, `or`, `not`\noperators. So you can not write predicate as `x=1 | x=2` - it will cause\nsyntax error. To solve this problem please use parenthesis: `(x=1) |\n(x=2)`.\n\nAlso VOPS provides analog of between operator. In SQL expression `(x\nBETWEEN a AND b)` is equivalent to `(x \u003e= a AND x \u003c= b)`. But as far as\nAND operator can not be overloaded, such substitution will not work for\nVOPS tiles. This is why VOPS provides special function for range check.\nUnfortunately `BETWEEN` is reserved keyword, so no function with such\nname can be defined. This is why synonym `BETWIXT` is used.\n\n.\n\nPostgres requires predicate expression to have boolean type. But result\nof vector boolean operators is `vops_bool`, not `bool`. This is why\ncompiler doesn't allow to use it in predicate. The problem can be solved\nby introducing special `filter` function. This function is given\narbitrary vector boolean expression and returns normal boolean which ...\nis always true. So from Postgres executor point of view predicate value\nis always true. But `filter` function sets `filter_mask` which is\nactually used in subsequent operators to determine selected records. So\nquery in VOPS looks something like this:\n\n``` \n  select sum(price) from trades where filter(day \u003e= '2017-01-01'::date);\n```\n\nPlease notice one more difference from normal sequence: we have to use\nexplicit cast of string constant to appreciate data type (`date` type in\nthis example). For `betwixt` function it is not\nneeded:\n\n``` \n  select sum(price) from trades where filter(betwixt(day, '2017-01-01', '2017-02-01'));\n```\n\nFor `char`, `int2` and `int4` types VOPS provides concatenation operator\n**||** which produces doubled integer type: `(char || char) -\u003e int2`,\n`(int2 || int2) -\u003e int4`, `(int4 || int4) -\u003e int8`. Them can be used for\ngrouping by several columns (see below).\n\n| Operator              | Description                          |\n| --------------------- | ------------------------------------ |\n| `+`                   | Addition                             |\n| `-`                   | Binary subtraction or unary negation |\n| `*`                   | Multiplication                       |\n| `/`                   | Division                             |\n| `=`                   | Equals                               |\n| `\u003c\u003e`                  | Not equals                           |\n| `\u003c`                   | Less than                            |\n| `\u003c=`                  | Less than or Equals                  |\n| `\u003e`                   | Greater than                         |\n| `\u003e=`                  | Greater than or equals               |\n| `\u0026`                   | Boolean AND                          |\n| `\\|`                  | Boolean OR                           |\n| `!`                   | Boolean NOT                          |\n| `bitwixt(x,low,high)` | Analog of BETWEEN                    |\n| `is_null(x)`          | Analog of IS NULL                    |\n| `is_not_null(x)`      | Analog of IS NOT NULL                |\n| `ifnull(x,subst)`     | Analog of COALESCE                   |\n\n### \u003cspan id=\"aggregates\"\u003eVector aggregates\u003c/span\u003e\n\nOLAP queries usually perform some kind of aggregation of large volumes\nof data. These includes `grand` aggregates which are calculated for the\nwhole table or aggregates with `group by` which are calculated for each\ngroup. VOPS implements all standard SQL aggregates: `count, min, max,\nsum, avg, var_pop, var_sampl, variance, stddev_pop, stddev_samp,\nstddev`. Them can be used exactly in the same way as in normal SQL\nqueries:\n\n    select sum(l_extendedprice*l_discount) as revenue\n    from vops_lineitem\n    where filter(betwixt(l_shipdate, '1996-01-01', '1997-01-01')\n            \u0026 betwixt(l_discount, 0.08, 0.1)\n            \u0026 (l_quantity \u003c 24));\n\nAlso VOPS provides weighted average aggregate VWAP which can be used to\ncalculate volume-weighted average price:\n\n    select wavg(l_extendedprice,l_quantity) from vops_lineitem;\n\nUsing aggregation with group by is more complex. VOPS provides two\nfunctions for it: `map` and `reduce`. The work is actually done by\n**map**(*group\\_by\\_expression*, *aggregate\\_list*, *expr* {, *expr* })\nVOPS implements aggregation using hash table, which entries collect\naggregate states for all groups. And set returning function `reduce`\njust iterates through the hash table consrtucted by `map`. `reduce`\nfunction is needed because result of aggregate in Postgres can not be a\nset. So aggregate query with group by looks something like\n    this:\n\n    select reduce(map(l_returnflag||l_linestatus, 'sum,sum,sum,sum,avg,avg,avg',\n        l_quantity,\n        l_extendedprice,\n        l_extendedprice*(1-l_discount),\n        l_extendedprice*(1-l_discount)*(1+l_tax),\n        l_quantity,\n        l_extendedprice,\n        l_discount)) from vops_lineitem where filter(l_shipdate \u003c= '1998-12-01'::date);\n\nHere we use concatenation operator to perform grouping by two columns.\nRight now VOPS supports grouping only by integer type. Another serious\nrestriction is that all aggregated expressions should have the same\ntype, for example `vops_float4`. It is not possible to calculate\naggregates for `vops_float4` and `vopd_int8` columns in one call of\n`map` function, because it accepts aggregation arguments as variadic\narray, so all elements of this array should have the same type.\n\nAggregate string in `map` function should contain list of requested\naggregate functions, separated by colon. Standard lowercase names should\nbe used: `count, sum, agg, min, max`. Count is executed for the\nparticular column: `count(x)`. There is no need to explicitly specify\n`count(*)` because number of records in each group is returned by\n`reduce` function in any case.\n\n`reduce` function returns set of `vops_aggregate` type. It contains\nthree components: value of group by expression, number of records in the\ngroup and array of floats with aggregate values. Please notice that\nvalues of all aggregates, including `count` and `min/max`, are returned\nas\n    floats.\n\n    create type vops_aggregates as(group_by int8, count int8, aggs float8[]);\n    create function reduce(bigint) returns setof vops_aggregates;\n\nBut there is much simple and straightforward way of performing group\naggregates using VOPS. We need to partition table by *group by* fields.\nIn this case grouping keys will be stored in normal way and other fields\n- inside tiles. Now Postgres executor will execute VOPS aggregates for\neach group:\n\n    select\n        l_returnflag,\n        l_linestatus,\n        sum(l_quantity) as sum_qty,\n        sum(l_extendedprice) as sum_base_price,\n        sum(l_extendedprice*(1-l_discount)) as sum_disc_price,\n        sum(l_extendedprice*(1-l_discount)*(1+l_tax)) as sum_charge,\n        avg(l_quantity) as avg_qty,\n        avg(l_extendedprice) as avg_price,\n        avg(l_discount) as avg_disc,\n        countall(*) as count_order\n    from\n        vops_lineitem_projection\n    where\n        filter(l_shipdate \u003c= '1998-12-01'::date)\n    group by\n        l_returnflag,\n        l_linestatus\n    order by\n        l_returnflag,\n        l_linestatus;\n\nIn this example `l_returnflag` and `l_linestatus` fields of table\nvops\\_lineitem\\_projection have `\"char\"` type while all other used\nfields - tile types (`l_shipdate` has type `vops_date` and other fields\n- `vops_float4`). The query above is executed even faster than query\nwith `reduce(map(...))`. The main problem with this approach is that you\nhave to create projection for each combination of group by keys you want\nto use in queries.\n\n### \u003cspan id=\"window\"\u003eVector window functions\u003c/span\u003e\n\nVOPS provides limited support of Postgres window functions. It\nimplements `count, sum, min, max, avg` and `lag` functions. But\nunfortunately Postgres requires aggregates to have to similar final type\nfor moving (window) and plain implementations. This is why VOPS has to\nchoose define this aggregate under different names: `mcount, msum, mmin,\nmmax, mavg`.\n\nThere are also two important restrictions:\n\n1.  Filtering, grouping and sorting can be done only by scalar\n    (non-tile) attributes\n2.  Only `rows between unbounded preceding and current row` frame is\n    supported (but there is special version of `msum` which accepts\n    extra window size parameter)\n\nExample of using window functions with\n    VOPS:\n\n    select vops_unnest(t.*) from (select mcount(*) over w,mcount(x) over w,msum(x) over w,mavg(x) over w,mmin(x) over w,mmax(x) over w,x - lag(x) over w \n    from v window w as (rows between unbounded preceding and current row)) t;\n\n### \u003cspan id=\"indexes\"\u003eUsing indexes\u003c/span\u003e\n\nAnalytic queries are usually performed on the data for which no indexes\nare defined. And columnar store vector operations are most efficient in\nthis case. But it is still possible to use indexes with VOPS.\n\nAs far as each VOPS tile represents multiple values, index can be used\nonly for some preliminary, non-precise filtering of data. It is\nsomething similar with BRIN indexes. VOPS provides four functions:\n`first, last, high, low` which can be used to obtain high/low boundary\nof values stored in the tile. First two functions `first` and `last`\nshould be used for sorted data set. In this case first value is the\nsmallest value in the tile and last value is the largest value in the\ntile. If data is not sorted, then `low`high functions should be used,\nwhich are more expensive, because them need to inspect all tile values.\nUsing this four function it is possible to construct functional indexes\nfor VOPS table. BRIN index seems to be the best choice for VOPS\n    table:\n\n    create index low_boundary on trades using brin(first(day)); -- trades table is ordered by day\n    create index high_boundary on trades using brin(last(day)); -- trades table is ordered by day\n\nNow it is possible to use this indexes in query. Please notice that we\nhave to recheck precise condition because index gives only approximate\nresult:\n\n    select sum(price) from trades where first(day) \u003e= '2015-01-01' and last(day) \u003c= '2016-01-01'\n                                                   and filter(betwixt(day, '2015-01-01', '2016-01-01'));\n\n### \u003cspan id=\"populating\"\u003ePreparing data for VOPS\u003c/span\u003e\n\nNow the most interesting question (from which may be we should start) -\nhow we managed to prepare data for VOPS queries? Who and how will\ncombine attribute values of several rows inside one VOPS tile? It is\ndone by `populate` functions, provided by VOPS extension.\n\nFirst of all you need to create table with columns having VOPS tile\ntypes. It can map all columns of the original table or just some most\nfrequently used subset of them. This table can be treated as\n`projection` of original table (this concept of projections is taken\nfrom Vertica). Projection should include columns which are most\nfrequently used together in queries.\n\nOriginal table from TPC-H benchmark:\n\n    create table lineitem(\n       l_orderkey integer,\n       l_partkey integer,\n       l_suppkey integer,\n       l_linenumber integer,\n       l_quantity real,\n       l_extendedprice real,\n       l_discount real,\n       l_tax real,\n       l_returnflag \"char\",\n       l_linestatus \"char\",\n       l_shipdate date,\n       l_commitdate date,\n       l_receiptdate date,\n       l_shipinstruct char(25),\n       l_shipmode char(10),\n       l_comment char(44));\n\nVOPS projection of this table:\n\n    create table vops_lineitem(\n       l_shipdate vops_date not null,\n       l_quantity vops_float4 not null,\n       l_extendedprice vops_float4 not null,\n       l_discount vops_float4 not null,\n       l_tax vops_float4 not null,\n       l_returnflag vops_char not null,\n       l_linestatus vops_char not null\n    );\n\nOriginal table can be treated as write optimized storage (WOS). If it\nhas not indexes, then Postgres is able to provide very fast insertion\nspeed, comparable with raw disk write speed. Projection in VOPS format\ncan be treated as read-optimized storage (ROS), most efficient for\nexecution of OLAP queries.\n\nData can be transferred from original to projected table using VOPS\n`populate` function:\n\n    create function populate(destination regclass, \n                            source regclass, \n                            predicate cstring default null, \n                            sort cstring default null) returns bigint;\n\nTwo first mandatory arguments of this function specify target and source\ntables. Optional predicate and sort clauses allow to restrict amount of\nimported data and enforce requested order. By specifying predicate it is\npossible to update VOPS table using only most recently received records.\nThis functions returns number of loaded records. Example of populate\nfunction\n    invocation:\n\n    select populate(destination := 'vops_lineitem'::regclass, source := 'lineitem'::regclass);\n\nYou can use populated table in queries performing sequential scan. VOPS\noperators can speed up filtering of records and calculation of\naggregates. Aggregation with `group by` requires use of `reduce + map`\nfunctions. But as it was mentioned above in the section describing\naggregates, it is possible to populate table in such way, that standard\nPostgres grouping algorithm will be used.\n\nWe need to choose partitioning keys and sort original table by this\nkeys. Combination of partitioning keys expected to be NOT unique -\notherwise tiles can only increase used space and lead to performance\ndegradation. But if there are a lot of duplicates, then \"collapsing\"\nthem and storing other fields in tiles will help to reduce space and\nspeed up queries. Let's create the following projection of `lineitems`\ntable:\n\n    create table vops_lineitem_projection(                                                                                    \n       l_shipdate vops_date not null,\n       l_quantity vops_float4 not null,\n       l_extendedprice vops_float4 not null,\n       l_discount vops_float4 not null,\n       l_tax vops_float4 not null,\n       l_returnflag \"char\" not null,\n       l_linestatus \"char\" not null\n    );\n\nAs you can see, in this table `l_returnflag` and `l_linestatus` fields\nare scalars, and other fields - tiles. This projection can be populated\nusing the following\n    command:\n\n    select populate(destination := 'vops_lineitem_projection'::regclass, source := 'lineitem_projection'::regclass, sort := 'l_returnflag,l_linestatus');\n\nNow we can create normal index on partitioning keys, define standard\npredicates for them and use them in `group by` and `order by` clauses.\n\nSometimes it is not possible or not desirable to store two copies of the\nsame dataset. VOPS allows to load data directly from CSV file into VOPS\ntable with tiles, bypassing creation of normal (plain) table. It can be\ndone using `import`\n    function:\n\n    select import(destination := 'vops_lineitem'::regclass, csv_path := '/mnt/data/lineitem.csv', separator := '|');\n\n`import` function is defined in this way:\n\n    create function import(destination regclass, \n                           csv_path cstring, \n                           separator cstring default ',', \n                           skip integer default 0) returns bigint;\n\nIt accepts name of target VOPS table, path to CSV file, optional\nseparator (default is ',') and number of lines in CSV header (no header\nby default). The function returns number of imported rows.\n\n### \u003cspan id=\"vops_unnest\"\u003eBack to normal tuples\u003c/span\u003e\n\nA query from VOPS projection returns set of tiles. Output function of\ntile type is able to print content of the tile. But in some cases it is\npreferable to transfer result to normal (horizontal) format where each\ntuple represents one record. It can be done using `vops_unnest`\n    function:\n\n    postgres=# select vops_unnest(l.*) from vops_lineitem l where filter(l_shipdate \u003c= '1998-12-01'::date) limit 3;\n                  vops_unnest                 \n    ---------------------------------------\n     (1996-03-13,17,33078.9,0.04,0.02,N,O)\n     (1996-04-12,36,38306.2,0.09,0.06,N,O)\n     (1996-01-29,8,15479.7,0.1,0.02,N,O)\n    (3 rows)\n\n### \u003cspan id=\"fdw\"\u003eBack to normal tables\u003c/span\u003e\n\nAs it was mentioned in previous section, `vops_unnest` function can\nscatter records with VOPS types into normal records with scalar types.\nSo it is possible to use this records in arbitrary SQL queries. But\nthere are two problems with vops_unnest function:\n\n1.  It is not convenient to use. This function has no static knowledge\n    about the format of output record and this is why programmer has to\n    specify it manually, if here wants to decompose this record.\n2.  PostgreSQL optimizer has completely no knowledge on result of\n    transformation performed by vops_unnest() function. This is why it\n    is not able to choose optimal query execution plan for data\n    retrieved from VOPS table.\n\nFortunately Postgres provides solution for both of this problem: foreign\ndata wrappers (FDW). In our case data is not really \"foreign\": it is\nstored inside our own database. But in alternatives (VOPS) format. VOPS\nFDW allows to \"hide\" specific of VOPS format and run normal SQL queries\non VOPS tables. FDW allows the following:\n\n1.  Extract data from VOPS table in normal (horizontal) format so that\n    it can be proceeded by upper nodes in query execution plan.\n2.  Pushdown to VOPS operations that can be efficiently executed using\n    vectorized operations on VOPS types: filtering and aggregation.\n3.  Provide statistic for underlying table which can be used by query\n    optimizer.\n\nSo, by placing VOPS projection under FDW, we can efficiently perform\nsequential scan and aggregation queries as if them will be explicitly\nwritten for VOPS table and at the same time be able to execute any other\nqueries on this data, including joins, CTEs,... Query can be written in\nstandard SQL without usage of any VOPS specific functions.\n\nBelow is an example of creating VOPS FDW and running some queries on it:\n\n    create foreign table lineitem_fdw  (\n       l_suppkey int4 not null,\n       l_orderkey int4 not null,\n       l_partkey int4 not null,\n       l_shipdate date not null,\n       l_quantity float4 not null,\n       l_extendedprice float4 not null,\n       l_discount float4 not null,\n       l_tax      float4 not null,\n       l_returnflag \"char\" not null,\n       l_linestatus \"char\" not null\n    ) server vops_server options (table_name 'vops_lineitem');\n    \n    explain select\n       sum(l_extendedprice*l_discount) as revenue\n    from\n       lineitem_fdw\n    where\n       l_shipdate between '1996-01-01' and '1997-01-01'\n       and l_discount between 0.08 and 0.1\n       and l_quantity \u003c 24;\n                           QUERY PLAN                        \n    ---------------------------------------------------------\n     Foreign Scan  (cost=1903.26..1664020.23 rows=1 width=4)\n    (1 row)\n    \n    -- Filter was pushed down to FDW\n    \n    explain select\n        n_name,\n        count(*),\n        sum(l_extendedprice * (1-l_discount)) as revenue\n    from\n        customer_fdw join orders_fdw on c_custkey = o_custkey\n        join lineitem_fdw on l_orderkey = o_orderkey\n        join supplier_fdw on l_suppkey = s_suppkey\n        join nation on c_nationkey = n_nationkey\n        join region on n_regionkey = r_regionkey\n    where\n        c_nationkey = s_nationkey\n        and r_name = 'ASIA'\n        and o_orderdate \u003e= '1996-01-01'\n        and o_orderdate \u003c '1997-01-01'\n    group by\n        n_name\n    order by\n        revenue desc;\n                                                                  QUERY PLAN                                                              \n    --------------------------------------------------------------------------------------------------------------------------------------\n     Sort  (cost=2337312.28..2337312.78 rows=200 width=48)\n       Sort Key: (sum((lineitem_fdw.l_extendedprice * ('1'::double precision - lineitem_fdw.l_discount)))) DESC\n       -\u003e  GroupAggregate  (cost=2336881.54..2337304.64 rows=200 width=48)\n             Group Key: nation.n_name\n             -\u003e  Sort  (cost=2336881.54..2336951.73 rows=28073 width=40)\n                   Sort Key: nation.n_name\n                   -\u003e  Hash Join  (cost=396050.65..2334807.39 rows=28073 width=40)\n                         Hash Cond: ((orders_fdw.o_custkey = customer_fdw.c_custkey) AND (nation.n_nationkey = customer_fdw.c_nationkey))\n                         -\u003e  Hash Join  (cost=335084.53..2247223.46 rows=701672 width=52)\n                               Hash Cond: (lineitem_fdw.l_orderkey = orders_fdw.o_orderkey)\n                               -\u003e  Hash Join  (cost=2887.07..1786058.18 rows=4607421 width=52)\n                                     Hash Cond: (lineitem_fdw.l_suppkey = supplier_fdw.s_suppkey)\n                                     -\u003e  Foreign Scan on lineitem_fdw  (cost=0.00..1512151.52 rows=59986176 width=16)\n                                     -\u003e  Hash  (cost=2790.80..2790.80 rows=7702 width=44)\n                                           -\u003e  Hash Join  (cost=40.97..2790.80 rows=7702 width=44)\n                                                 Hash Cond: (supplier_fdw.s_nationkey = nation.n_nationkey)\n                                                 -\u003e  Foreign Scan on supplier_fdw  (cost=0.00..2174.64 rows=100032 width=8)\n                                                 -\u003e  Hash  (cost=40.79..40.79 rows=15 width=36)\n                                                       -\u003e  Hash Join  (cost=20.05..40.79 rows=15 width=36)\n                                                             Hash Cond: (nation.n_regionkey = region.r_regionkey)\n                                                             -\u003e  Seq Scan on nation  (cost=0.00..17.70 rows=770 width=40)\n                                                             -\u003e  Hash  (cost=20.00..20.00 rows=4 width=4)\n                                                                   -\u003e  Seq Scan on region  (cost=0.00..20.00 rows=4 width=4)\n                                                                         Filter: ((r_name)::text = 'ASIA'::text)\n                               -\u003e  Hash  (cost=294718.76..294718.76 rows=2284376 width=8)\n                                     -\u003e  Foreign Scan on orders_fdw  (cost=0.00..294718.76 rows=2284376 width=8)\n                         -\u003e  Hash  (cost=32605.64..32605.64 rows=1500032 width=8)\n                               -\u003e  Foreign Scan on customer_fdw  (cost=0.00..32605.64 rows=1500032 width=8)\n    \n    -- filter on orders range is pushed to FDW\n\n## \u003cspan id=\"transform\"\u003eStandard SQL query transformation\u003c/span\u003e\n\nPrevious section describes VOPS specific types, operators, functions,...\nGood news\\! You do not need to learn them. You can use normal SQL. Well,\nit is still responsibility of programmer or database administrator to\ncreate proper projections of original table. This projections need to\nuse tiles types for some attributes (vops\\_float4,...). Then you can\nquery this table using standard SQL. And this query will be executed\nusing vector operations\\!\n\nHow it works? There are absolutely no magic here. There are four main\ncomponents of the puzzle:\n\n1.  User defined types\n2.  User defined operator\n3.  User defined implicit type casts\n4.  Post parse analyze hook which performs query transformation\n\nSo VOPS defines tile types and standard SQL operators for this types.\nThen it defines implicit type cast from `vops_bool` (result of boolean\noperation with tiles) to boolean type. Now programmer do not have to\nwrap vectorized boolean operations in `filter()` function call. And the\nfinal transformation is done by post parse analyze hook, defined by VOPS\nextension. It replaces scalar boolean operations with vector boolean\noperations:\n\n| Original expression         | Result of transformation        |\n| --------------------------- | ------------------------------- |\n| `NOT filter(o1)`            | `filter(vops_bool_not(o1))`     |\n| `filter(o1) AND filter(o2)` | `filter(vops_bool_and(o1, o2))` |\n| `filter(o1) OR filter(o2)`  | `filter(vops_bool_or(o1, o2))`  |\n\nNow there is no need to use VOPS specific `BETIXT` operator: standard\nSQL `BETWEEN` operator will work (but still using `BETIXT` is slightly\nmore efficient, because it performs both comparions in one function).\nAlso there are no problems with operators precedence and extra\nparenthesis are not needed. If query includes vectorized aggregates,\nthen `count(*)` is transformed to `countall(*)`.\n\nThere is only one difference left between standard SQL and its\nvectorized extension. You still have to perform explicit type cast in\ncase of using string literal, for example `l_shipdate \u003c= '1998-12-01'`\nwill not work for `l_shipdate` column with tile type. Postgres have two\noverloaded versions of \\\u003c= operator which can be applied here:\n\n1.  `vops_date` **\\\u003c=** `vops_date`\n2.  `vops_date` **\\\u003c=** `date`\n\nAnd it decides that it is better to convert string to the tile type\n`vops_date`. In principle, it is possible to provide such conversion\noperator. But it is not good idea, because we have to generate dummy\ntile with all components equal to the specified constant and perform\n(*vector* **OP** *vector*) operation instead of more efficient (*vector*\n**OP** *scalar*).\n\nThere is one pitfall with post parse analyze hook: it is initialized in\nthe extension `_PG_init` function. But if extension was not registered\nin `shared_preload_libraries` list, then it will be loaded on demand\nwhen any function of this extension is requested. Unfortunately it\nhappens **after** parse analyze is done. So first time you execute VOPS\nquery, it will not be transformed. You can get wrong result in this\ncase. Either take it in account, either add `vops` to\n`shared_preload_libraries` configuration string. VOPS extension provides\nspecial function `vops_initialize()` which can be invoked to force\ninitialization of VOPS extension. After invocation of this function,\nextension will be loaded and all subsequent queries will be normally\ntransformed and produce expected results.\n\n## \u003cspan id=\"projections\"\u003eVOPS projections and automatic table sustitution\u003c/span\u003e\n\nVOPS provides some functions simplifying creation and usage of projections.\nIn future it may be added to SQL grammar, so that it is possible to write\n`CREATE PROJECTION xxx OF TABLE yyy(column1, column2,...) GROUP BY (column1, column2, ...)`.\nBut right now it can be done using `create_projection(projection_name text, source_table regclass, vector_columns text[], scalar_columns text[] default null, order_by text default null)` function.\nFirst argument of this function specifies name of the projection, second refers to existed Postgres table, `vector_columns` is array of\ncolumn names which should be stores as VOPS tiles, `scalar_columns`  is array of grouping columns which type is preserved and\noptional `order_by` parameter specifies name of ordering attribute (explained below).\nThe `create_projection(PNAME,...)` functions does the following:\n\n1. Creates projection table with specified name and attributes.\n2. Creates PNAME_refresh() functions which can be used to update projection.\n3. Creates functional BRIN indexes for `first()` and `last()` functions of ordering attribute (if any)\n4. Creates BRIN index on grouping attributes (if any)\n5. Insert information about created projection in `vops_projections` table. This table is used by optimizer to\n    automatically substitute table with partition.\n\nThe `order_by` attribute is one of the VOPS projection vector columns by which data is sorted. Usually it is some kind of timestamp\nused in *time series* (for example trade date). Presence of such column in projection allows to incrementally update projection.\nGenerated `PNAME_refresh()` method calls `populate` method with correspondent values of `predicate` and\n`sort` parameters, selecting from original table only rows with `order_by` column value greater than maximal\nvalue of this column in the projection. It assumes that `order_by` is unique or at least refresh is done at the moment when there is some gap\nin collected events. In addition to `order_by`, sort list for `populate` includes all scalar (grouping) columns.\nIt allows to efficiently group imported data by scalar columns and fill VOPS tiles (vector columns) with data.\n\nWhen `order_by` attribute is specified, VOPS creates two functional  BRIN indexes on `first()` and `last()`\nfunctions of this attribute. Presence of such indexes allows to efficiently select time slices. If original query contains\npredicate like `(trade_date between '01-01-2017' and '01-01-2018')` then VOPS projection substitution mechanism adds\n`(first(trade_date) \u003e= '01-01-2017' and last(trade_date) \u003e= '01-01-2018')` conjuncts which allow Postgres optimizer to use BRIN\nindex to locate affected pages.\n\nIn in addition to BRIN indexes for `order_by` attribute, VOPS also creates BRIN index for grouping (scalar) columns.\nSuch index allows to efficiently select groups and perform index join.\n\nLike materialized views, VOPS projections are not updated automatically. It is responsibility of programmer to periodically refresh them.\nCertainly it is possible to define trigger or rule which will automatically insert data in projection table when original table is updated.\nBut such approach will be extremely inefficient and slow. To take advantage of vector processing, VOPS has to group data in tiles.\nIt can be done only if there is some batch of data which can be grouped by scalar attributes. If you insert records in projection table on-by-one,\nthen most of VOPS tiles will contain just one element.\nThe most convenient way is to use generated `PNAME_refresh()` function.\nIf `order_by` attribute is specified, this function imports from original table only the new data (not present in projection).\n\nThe main advantage of VOPS projection mechanism is that it allows to automatically substitute queries on original tables with projections.\nThere is `vops.auto_substitute_projections` configuration parameter which allows to switch on such substitution.\nBy default it is switched off, because VOPS projects may be not synchronized with original table and query on projection may return different result.\nRight now projections can be automatically substituted only if:\n\n1. Query doesn't contain joins.\n2. Query performs aggregation of vector (tile) columns.\n3. All other expressions in target list, `ORDER BY` / `GROUP BY` clauses refer only to scalar attributes of projection.\n\nProjection can be removed using `drop_projection(projection_name text)` function.\nIt not only drops the correspondent table, but also removes information about it from `vops_partitions` table\nand drops generated refresh function.\n\nExample of using projections:\n```\ncreate extension vops;\n\ncreate table lineitem(\n   l_orderkey integer,\n   l_partkey integer,\n   l_suppkey integer,\n   l_linenumber integer,\n   l_quantity real,\n   l_extendedprice real,\n   l_discount real,\n   l_tax real,\n   l_returnflag \"char\",\n   l_linestatus \"char\",\n   l_shipdate date,\n   l_commitdate date,\n   l_receiptdate date,\n   l_shipinstruct char(25),\n   l_shipmode char(10),\n   l_comment char(44),\n   l_dummy char(1));\n\nselect create_projection('vops_lineitem','lineitem',array['l_shipdate','l_quantity','l_extendedprice','l_discount','l_tax'],array['l_returnflag','l_linestatus']);\n\n\\timing\n\ncopy lineitem from '/mnt/data/lineitem.tbl' delimiter '|' csv;\n\nselect vops_lineitem_refresh();\n\nselect\n    l_returnflag,\n    l_linestatus,\n    sum(l_quantity) as sum_qty,\n    sum(l_extendedprice) as sum_base_price,\n    sum(l_extendedprice*(1-l_discount)) as sum_disc_price,\n    sum(l_extendedprice*(1-l_discount)*(1+l_tax)) as sum_charge,\n    avg(l_quantity) as avg_qty,\n    avg(l_extendedprice) as avg_price,\n    avg(l_discount) as avg_disc,\n    count(*) as count_order\nfrom\n    lineitem\nwhere\n    l_shipdate \u003c= '1998-12-01'\ngroup by\n    l_returnflag,\n    l_linestatus\norder by\n    l_returnflag,\n    l_linestatus;\n\nset vops.auto_substitute_projections TO  on;\n\nselect\n    l_returnflag,\n    l_linestatus,\n    sum(l_quantity) as sum_qty,\n    sum(l_extendedprice) as sum_base_price,\n    sum(l_extendedprice*(1-l_discount)) as sum_disc_price,\n    sum(l_extendedprice*(1-l_discount)*(1+l_tax)) as sum_charge,\n    avg(l_quantity) as avg_qty,\n    avg(l_extendedprice) as avg_price,\n    avg(l_discount) as avg_disc,\n    count(*) as count_order\nfrom\n    lineitem\nwhere\n    l_shipdate \u003c= '1998-12-01'\ngroup by\n    l_returnflag,\n    l_linestatus\norder by\n    l_returnflag,\n    l_linestatus;\n```\n\n\n## \u003cspan id=\"example\"\u003eExample\u003c/span\u003e\n\nThe most popular benchmark for OLAP is [TPC-H](http://www.tpc.org/tpch).\nIt contains 21 different queries. We adopted for VOPS only two of them:\nQ1 and Q6 which are not using joins. Most of fragments of this code are\nalready mentioned above, but here we collect it together:\n\n``` \n-- Standard way of creating extension\ncreate extension vops; \n\n-- Original TPC-H table\ncreate table lineitem(\n   l_orderkey integer,\n   l_partkey integer,\n   l_suppkey integer,\n   l_linenumber integer,\n   l_quantity real,\n   l_extendedprice real,\n   l_discount real,\n   l_tax real,\n   l_returnflag \"char\",\n   l_linestatus \"char\",\n   l_shipdate date,\n   l_commitdate date,\n   l_receiptdate date,\n   l_shipinstruct char(25),\n   l_shipmode char(10),\n   l_comment char(44),\n   l_dummy char(1)); -- this table is needed because of terminator after last column in generated data \n\n-- Import data to it\ncopy lineitem from '/mnt/data/lineitem.tbl' delimiter '|' csv;\n\n-- Create VOPS projection\ncreate table vops_lineitem(\n   l_shipdate vops_date not null,\n   l_quantity vops_float4 not null,\n   l_extendedprice vops_float4 not null,\n   l_discount vops_float4 not null,\n   l_tax vops_float4 not null,\n   l_returnflag vops_char not null,\n   l_linestatus vops_char not null\n);\n\n-- Copy data to the projection table\nselect populate(destination := 'vops_lineitem'::regclass, source := 'lineitem'::regclass);\n\n-- For honest comparison creates the same projection without VOPS types\ncreate table lineitem_projection as (select l_shipdate,l_quantity,l_extendedprice,l_discount,l_tax,l_returnflag::\"char\",l_linestatus::\"char\" from lineitem);\n\n-- Now create mixed projection with partitioning keys:\ncreate table vops_lineitem_projection(                                                                                    \n   l_shipdate vops_date not null,\n   l_quantity vops_float4 not null,\n   l_extendedprice vops_float4 not null,\n   l_discount vops_float4 not null,\n   l_tax vops_float4 not null,\n   l_returnflag \"char\" not null,\n   l_linestatus \"char\" not null\n);\n\n-- And populate it with data sorted by partitioning key:\nselect populate(destination := 'vops_lineitem_projection'::regclass, source := 'lineitem_projection'::regclass, sort := 'l_returnflag,l_linestatus');\n\n\n-- Let's measure time\n\\timing\n\n-- Original Q6 query performing filtering with calculation of grand aggregate\nselect\n    sum(l_extendedprice*l_discount) as revenue\nfrom\n    lineitem\nwhere\n    l_shipdate between '1996-01-01' and '1997-01-01'\n    and l_discount between 0.08 and 0.1\n    and l_quantity \u003c 24;\n\n-- VOPS version of Q6 using VOPS specific operators\nselect sum(l_extendedprice*l_discount) as revenue\nfrom vops_lineitem\nwhere filter(betwixt(l_shipdate, '1996-01-01', '1997-01-01')\n        \u0026 betwixt(l_discount, 0.08, 0.1)\n        \u0026 (l_quantity \u003c 24));\n\n-- Yet another vectorized version of Q6, but now in stadnard SQL:\nselect sum(l_extendedprice*l_discount) as revenue\nfrom vops_lineitem\nwhere l_shipdate between '1996-01-01'::date AND '1997-01-01'::date\n   and l_discount between 0.08 and 0.1\n   and l_quantity \u003c 24;\n\n\n\n-- Original version of Q1: filter + group by + aggregation\nselect \n    l_returnflag,\n    l_linestatus,\n    sum(l_quantity) as sum_qty,\n    sum(l_extendedprice) as sum_base_price,\n    sum(l_extendedprice*(1-l_discount)) as sum_disc_price,\n    sum(l_extendedprice*(1-l_discount)*(1+l_tax)) as sum_charge,\n    avg(l_quantity) as avg_qty,\n    avg(l_extendedprice) as avg_price,\n    avg(l_discount) as avg_disc,\n    count(*) as count_order\nfrom\n    lineitem\nwhere\n    l_shipdate \u003c= '1998-12-01'\ngroup by\n    l_returnflag,\n    l_linestatus\norder by\n    l_returnflag,\n    l_linestatus;\n\n-- VOPS version of Q1, sorry - no final sorting\nselect reduce(map(l_returnflag||l_linestatus, 'sum,sum,sum,sum,avg,avg,avg',\n    l_quantity,\n    l_extendedprice,\n    l_extendedprice*(1-l_discount),\n    l_extendedprice*(1-l_discount)*(1+l_tax),\n    l_quantity,\n    l_extendedprice,\n    l_discount)) from vops_lineitem where filter(l_shipdate \u003c= '1998-12-01'::date);\n       \n-- Mixed mode: let's Postgres does group by and calculates VOPS aggregates for each group\nselect\n    l_returnflag,\n    l_linestatus,\n    sum(l_quantity) as sum_qty,\n    sum(l_extendedprice) as sum_base_price,\n    sum(l_extendedprice*(1-l_discount)) as sum_disc_price,\n    sum(l_extendedprice*(1-l_discount)*(1+l_tax)) as sum_charge,\n    avg(l_quantity) as avg_qty,\n    avg(l_extendedprice) as avg_price,\n    avg(l_discount) as avg_disc,\n    count(*) as count_order\nfrom\n    vops_lineitem_projection\nwhere\n    l_shipdate \u003c= '1998-12-01'::date\ngroup by\n    l_returnflag,\n    l_linestatus\norder by\n    l_returnflag,\n    l_linestatus;\n\n\n```\n\n## \u003cspan id=\"performance\"\u003ePerformance evaluation\u003c/span\u003e\n\nNow most interesting thing: compare performance results on original\ntable and using vector operations on VOPS projection. All measurements\nwere performed at desktop with 16Gb of RAM and quad-core i7-4770 CPU @\n3.40GHz processor with enabled hyper-threading. Data set for benchmark\nwas generated by dbgen utility included in TPC-H benchmark. Scale factor\nis 10 which corresponds to about 8Gb database. It can completely fit in\nmemory, so we are measuring best query execution time for *warm* data.\nPostgres was configured with shared buffer size equal to 8Gb. For each\nquery we measured time of sequential and parallel execution with 8\nparallel\nworkers.\n\n| Query                                   | Sequential execution (msec) | Parallel execution (msec) |\n| --------------------------------------- | --------------------------: | ------------------------: |\n| Original Q1 for lineitem                |                       38028 |                     10997 |\n| Original Q1 for lineitem\\_projection    |                       33872 |                      9656 |\n| Vectorized Q1 for vops\\_lineitem        |                        3372 |                       951 |\n| Mixed Q1 for vops\\_lineitem\\_projection |                        1490 |                       396 |\n| Original Q6 for lineitem                |                       16796 |                      4110 |\n| Original Q6 for lineitem\\_projection    |                        4279 |                      1171 |\n| Vectorized Q6 for vops\\_lineitem        |                         875 |                       284 |\n\n## \u003cspan id=\"conclusion\"\u003eConclusion\u003c/span\u003e\n\nAs you can see in performance results, VOPS can provide more than 10\ntimes improvement of query speed. And this result is achieved without\nchanging something in query planner and executor. It is better than any\nof existed attempt to speed up execution of OLAP queries using JIT (4\ntimes for Q1, 3 times for Q6): [Speeding up query execution in\nPostgreSQL using LLVM JIT\ncompiler](http://llvm.org/devmtg/2016-09/slides/Melnik-PostgreSQLLLVM.pdf).\n\nDefinitely VOPS extension is just a prototype which main role is to\ndemonstrate potential of vectorized executor. But I hope that it also\ncan be useful in practice to speedup execution of OLAP aggregation\nqueries for existed databases. And in future we should think about the\nbest approach of integrating vectorized executor in Postgres core.\n\nALL sources of VOPS project can be obtained from this [GIT\nrepository](https://github.com/postgrespro/vops). Chinese version of\ndocumentation is available\n[here](https://github.com/digoal/blog/blob/master/201702/20170225_01.md).\nPlease send any feedbacks, complaints, bug reports, change requests to\n[Konstantin Knizhnik](mailto:k.knizhnik@postgrespro.ru).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpostgrespro%2Fvops","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpostgrespro%2Fvops","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpostgrespro%2Fvops/lists"}