{"id":13423356,"url":"https://github.com/supabase/pg_net","last_synced_at":"2025-05-16T13:03:25.021Z","repository":{"id":44876015,"uuid":"385426741","full_name":"supabase/pg_net","owner":"supabase","description":"A PostgreSQL extension that enables asynchronous (non-blocking) HTTP/HTTPS requests with SQL","archived":false,"fork":false,"pushed_at":"2025-04-16T23:05:06.000Z","size":719,"stargazers_count":294,"open_issues_count":23,"forks_count":21,"subscribers_count":29,"default_branch":"master","last_synced_at":"2025-05-06T18:16:01.765Z","etag":null,"topics":["http-client","postgres","postgresql","postgresql-extension","sql","supabase"],"latest_commit_sha":null,"homepage":"https://supabase.github.io/pg_net","language":"C","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/supabase.png","metadata":{"funding":{"github":["supabase"],"patreon":null,"open_collective":null,"ko_fi":null,"tidelift":null,"community_bridge":null,"liberapay":null,"issuehunt":null,"otechie":null,"custom":null},"files":{"readme":"README.md","changelog":null,"contributing":"docs/contributing.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2021-07-13T00:52:44.000Z","updated_at":"2025-05-03T02:33:22.000Z","dependencies_parsed_at":"2024-01-29T23:51:05.870Z","dependency_job_id":"ae4ef58f-2d37-4f59-aaf3-e16caf9b9e2f","html_url":"https://github.com/supabase/pg_net","commit_stats":{"total_commits":194,"total_committers":8,"mean_commits":24.25,"dds":"0.34536082474226804","last_synced_commit":"d3125253f084ac605615afbdc69211570d1cd197"},"previous_names":[],"tags_count":21,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/supabase%2Fpg_net","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/supabase%2Fpg_net/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/supabase%2Fpg_net/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/supabase%2Fpg_net/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/supabase","download_url":"https://codeload.github.com/supabase/pg_net/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253795177,"owners_count":21965492,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["http-client","postgres","postgresql","postgresql-extension","sql","supabase"],"created_at":"2024-07-31T00:00:32.579Z","updated_at":"2025-05-16T13:03:25.001Z","avatar_url":"https://github.com/supabase.png","language":"C","readme":"# PG_NET\n*A PostgreSQL extension that enables asynchronous (non-blocking) HTTP/HTTPS requests with SQL*.\n\nRequires libcurl \u003e= 7.83. Compatible with PostgreSQL \u003e = 12.\n\n![PostgreSQL version](https://img.shields.io/badge/postgresql-12+-blue.svg)\n[![License](https://img.shields.io/pypi/l/markdown-subtemplate.svg)](https://github.com/supabase/pg_net/blob/master/LICENSE)\n[![Coverage Status](https://coveralls.io/repos/github/supabase/pg_net/badge.svg)](https://coveralls.io/github/supabase/pg_net)\n[![Tests](https://github.com/supabase/pg_net/actions/workflows/main.yml/badge.svg)](https://github.com/supabase/pg_net/actions)\n\n**Documentation**: [https://supabase.github.io/pg_net](https://supabase.github.io/pg_net)\n\n---\n\n# Contents\n- [Introduction](#introduction)\n- [Technical Explanation](#technical-explanation)\n- [Installation](#installation)\n- [Configuration](#extension-configuration)\n- [Requests API](#requests-api)\n    - Monitoring requests\n    - GET requests\n    - POST requests\n    - DELETE requests\n- [Practical Examples](#practical-examples)\n    - Syncing data with an external data source using triggers\n    - Calling a serverless function every minute with PG_CRON\n    - Retrying failed requests\n- [Contributing](#contributing)\n\n---\n\n# Introduction\n\nThe PG_NET extension enables PostgreSQL to make asynchronous HTTP/HTTPS requests in SQL. It eliminates the need for servers to continuously poll for database changes and instead allows the database to proactively notify external resources about significant events. It seamlessly integrates with triggers, cron jobs (e.g., [PG_CRON](https://github.com/citusdata/pg_cron)), and procedures, unlocking numerous possibilities. Notably, PG_NET powers Supabase's Webhook functionality, highlighting its robustness and reliability.\n\nCommon use cases for the PG_NET extension include:\n\n- Calling external APIs\n- Syncing data with outside resources\n- Calling a serverless function when an event, such as an insert, occurred\n\nHowever, it is important to note that the extension has a few limitations. Currently, it only supports three types of asynchronous requests:\n\n- async http GET requests\n- async http POST requests with a *JSON* payload\n- async http DELETE requests\n\nUltimately, though, PG_NET offers developers more flexibility in how they monitor and connect their database with external resources.\n\n---\n\n# Technical Explanation\n\nThe extension introduces a new `net` schema, which contains two unlogged tables, a type of table in PostgreSQL that offers performance improvements at the expense of durability. You can read more about unlogged tables [here](https://pgpedia.info/u/unlogged-table.html). The two tables are:\n\n1. **`http_request_queue`**: This table serves as a queue for requests waiting to be executed. Upon successful execution of a request, the corresponding data is removed from the queue.\n\n    The SQL statement to create this table is:\n\n    ```sql\n    CREATE UNLOGGED TABLE\n        net.http_request_queue (\n            id bigint NOT NULL DEFAULT nextval('net.http_request_queue_id_seq'::regclass),\n            method text NOT NULL,\n            url text NOT NULL,\n            headers jsonb NOT NULL,\n            body bytea NULL,\n            timeout_milliseconds integer NOT NULL\n        )\n    ```\n\n2. **`_http_response`**: This table holds the responses of each executed request.\n\n    The SQL statement to create this table is:\n\n    ```sql\n    CREATE UNLOGGED TABLE\n        net._http_response (\n            id bigint NULL,\n            status_code integer NULL,\n            content_type text NULL,\n            headers jsonb NULL,\n            content text NULL,\n            timed_out boolean NULL,\n            error_msg text NULL,\n            created timestamp with time zone NOT NULL DEFAULT now()\n        )\n    ```\n\nWhen any of the three request functions (`http_get`, `http_post`, `http_delete`) are invoked, they create an entry in the `net.http_request_queue` table.\n\nThe extension employs C's [libCurl](https://curl.se/libcurl/c/) library within a PostgreSQL [background worker](https://www.postgresql.org/docs/current/bgworker.html) to manage HTTP requests. This background worker regularly checks the `http_request_queue` table and executes the requests it finds there.\n\nOnce a response is received, it gets stored in the `_http_response` table. By monitoring this table, you can keep track of response statuses and messages.\n\n---\n\n# Installation\n\n## Enabling the Extension with Supabase\n\nYou can activate the `pg_net` extension via Supabase's dashboard by following these steps:\n\n1. Navigate to the 'Database' page.\n2. Select 'Extensions' from the sidebar.\n3. Search for \"pg_net\" and enable the extension.\n\n## Local Setup\n\n### Configuring Your Device/Server\n\nClone this repo and run\n\n```bash\nmake \u0026\u0026 make install\n```\n\nTo make the extension available to the database add on `postgresql.conf`:\n\n```\nshared_preload_libraries = 'pg_net'\n```\n\nBy default, pg_net is available on the `postgres` database. To use pg_net on a different database, you can add the following on `postgresql.conf`:\n\n```\npg_net.database_name = '\u003cdbname\u003e';\n```\n\nUsing pg_net on multiple databases in a cluster is not supported.\n\n### Installing in PostgreSQL\n\nTo activate the extension in PostgreSQL, run the create extension command. The extension creates its own schema named net to avoid naming conflicts.\n\n```psql\ncreate extension pg_net;\n```\n\n---\n\n# Extension Configuration\n\nthe extension creates 3 configurable variables:\n\n1. **pg_net.batch_size** _(default: 200)_: An integer that limits the max number of rows that the extension will process from _`net.http_request_queue`_ during each read\n2. **pg_net.ttl** _(default: 6 hours)_: An interval that defines the max time a row in the _`net.http_response`_ will live before being deleted\n3. **pg_net.database_name** _(default: 'postgres')_: A string that defines which database the extension is applied to\n4. **pg_net.username** _(default: NULL)_: A string that defines which user will the background worker be connected with. If not set (`NULL`), it will assume the bootstrap user.\n\nAll these variables can be viewed with the following commands:\n```sql\nshow pg_net.batch_size;\nshow pg_net.ttl;\nshow pg_net.database_name;\nshow pg_net.username;\n```\n\nYou can change these by editing the `postgresql.conf` file (find it with `SHOW config_file;`) or with `ALTER SYSTEM`:\n\n```\nalter system set pg_net.ttl to '1 hour'\nalter system set pg_net.batch_size to 500;\n```\n\nThen, reload the settings and restart the `pg_net` background worker with:\n\n```\nselect net.worker_restart();\n```\n\nNote that doing `ALTER SYSTEM` requires SUPERUSER but on PostgreSQL \u003e= 15, you can do:\n\n```\ngrant alter system on parameter pg_net.ttl to \u003crole\u003e;\ngrant alter system on parameter pg_net.batch_size to \u003crole\u003e;\n```\n\nTo allow regular users to update `pg_net` settings.\n\n# Requests API\n\n## GET requests\n\n### net.http_get function signature\n\n```sql\nnet.http_get(\n    -- url for the request\n    url text,\n    -- key/value pairs to be url encoded and appended to the `url`\n    params jsonb default '{}'::jsonb,\n    -- key/values to be included in request headers\n    headers jsonb default '{}'::jsonb,\n    -- the maximum number of milliseconds the request may take before being cancelled\n    timeout_milliseconds int default 1000\n)\n    -- request_id reference\n    returns bigint\n\n    strict\n    volatile\n    parallel safe\n    language plpgsql\n```\n\n### Examples:\nThe following examples use the [Postman Echo API](https://learning.postman.com/docs/developer/echo-api/).\n\n#### Calling an API\n\n```sql\nSELECT net.http_get (\n    'https://postman-echo.com/get?foo1=bar1\u0026foo2=bar2'\n) AS request_id;\n```\n\n\u003e NOTE: You can view the response with the following query:\n\u003e\n\u003e ```sql\n\u003e SELECT *\n\u003e FROM net._http_response;\n\u003e ```\n\n#### Calling an API with URL encoded params\n\n```sql\nSELECT net.http_get(\n  'https://postman-echo.com/get',\n  -- Equivalent to calling https://postman-echo.com/get?foo1=bar1\u0026foo2=bar2\u0026encoded=%21\n  -- The \"!\" is url-encoded as %21\n  '{\"foo1\": \"bar1\", \"foo2\": \"bar2\", \"encoded\": \"!\"}'::JSONB\n) AS request_id;\n```\n\n#### Calling an API with an API-KEY\n\n```sql\nSELECT net.http_get(\n  'https://postman-echo.com/get?foo1=bar1\u0026foo2=bar2',\n   headers := '{\"API-KEY-HEADER\": \"\u003cAPI KEY\u003e\"}'::JSONB\n) AS request_id;\n```\n\n## POST requests\n### net.http_post function signature\n\n```sql\nnet.http_post(\n    -- url for the request\n    url text,\n    -- body of the POST request\n    body jsonb default '{}'::jsonb,\n    -- key/value pairs to be url encoded and appended to the `url`\n    params jsonb default '{}'::jsonb,\n    -- key/values to be included in request headers\n    headers jsonb default '{\"Content-Type\": \"application/json\"}'::jsonb,\n    -- the maximum number of milliseconds the request may take before being cancelled\n    timeout_milliseconds int default 1000\n)\n    -- request_id reference\n    returns bigint\n\n    volatile\n    parallel safe\n    language plpgsql\n```\n### Examples:\nThe following examples post to the [Postman Echo API](https://learning.postman.com/docs/developer/echo-api/).\n\n#### Sending data to an API\n\n```sql\nSELECT net.http_post(\n    'https://postman-echo.com/post',\n    '{\"key\": \"value\", \"key\": 5}'::JSONB,\n    headers := '{\"API-KEY-HEADER\": \"\u003cAPI KEY\u003e\"}'::JSONB\n) AS request_id;\n```\n\n#### Sending single table row as a payload\n\n\u003e NOTE: If multiple rows are sent using this method, each row will be sent as a separate request.\n\n```sql\nWITH selected_row AS (\n    SELECT\n        *\n    FROM target_table\n    LIMIT 1\n)\nSELECT\n    net.http_post(\n        'https://postman-echo.com/post',\n        to_jsonb(selected_row.*),\n        headers := '{\"API-KEY-HEADER\": \"\u003cAPI KEY\u003e\"}'::JSONB\n    ) AS request_id\nFROM selected_row;\n```\n\n#### Sending multiple table rows as a payload\n\n\u003e WARNING: when sending multiple rows, be careful to limit your payload size.\n\n```sql\nWITH selected_rows AS (\n    SELECT\n        -- Converts all the rows into a JSONB array\n        jsonb_agg(to_jsonb(target_table)) AS JSON_payload\n    FROM target_table\n    -- Generally good practice to LIMIT the max amount of rows\n)\nSELECT\n    net.http_post(\n        'https://postman-echo.com/post'::TEXT,\n        JSON_payload,\n        headers := '{\"API-KEY-HEADER\": \"\u003cAPI KEY\u003e\"}'::JSONB\n    ) AS request_id\nFROM selected_rows;\n```\n\n## DELETE requests\n### net.http_delete function signature\n\n```sql\nnet.http_delete(\n    -- url for the request\n    url text,\n    -- key/value pairs to be url encoded and appended to the `url`\n    params jsonb default '{}'::jsonb,\n    -- key/values to be included in request headers\n    headers jsonb default '{}'::jsonb,\n    -- the maximum number of milliseconds the request may take before being cancelled\n    timeout_milliseconds int default 2000\n)\n    -- request_id reference\n    returns bigint\n\n    strict\n    volatile\n    parallel safe\n    language plpgsql\n    security definer\n```\n\n### Examples:\nThe following examples use the [Dummy Rest API](https://dummy.restapiexample.com/employees).\n\n#### Sending a delete request to an API\n\n```sql\nSELECT net.http_delete(\n    'https://dummy.restapiexample.com/api/v1/delete/2'\n) AS request_id;\n```\n\n#### Sending a delete request with a row id as a query param\n\n```sql\nWITH selected_id AS (\n    SELECT\n        id\n    FROM target_table\n    LIMIT 1 -- if not limited, it will make a delete request for each returned row\n)\nSELECT\n    net.http_delete(\n        'https://dummy.restapiexample.com/api/v1/delete/'::TEXT,\n        format('{\"id\": \"%s\"}', id)::JSONB\n    ) AS request_id\nFROM selected_id;\n```\n\n#### Sending a delete request with a row id as a path param\n\n```sql\nWITH selected_id AS (\n    SELECT\n        id\n    FROM target_table\n    LIMIT 1 -- if not limited, it will make a delete request for each returned row\n)\nSELECT\n    net.http_delete(\n        'https://dummy.restapiexample.com/api/v1/delete/' || id\n    ) AS request_id\nFROM selected_row\n```\n\n---\n\n# Practical Examples\n\n## Syncing data with an external data source using triggers\n\nThe following example comes from [Typesense's Supabase Sync guide](https://typesense.org/docs/guide/supabase-full-text-search.html#syncing-individual-deletes)\n\n```sql\n-- Create the function to delete the record from Typesense\nCREATE OR REPLACE FUNCTION delete_record()\n    RETURNS TRIGGER\n    LANGUAGE plpgSQL\nAS $$\nBEGIN\n    SELECT net.http_delete(\n        url := format('\u003cTYPESENSE URL\u003e/collections/products/documents/%s', OLD.id),\n        headers := '{\"X-Typesense-API-KEY\": \"\u003cTypesense_API_KEY\u003e\"}'\n    )\n    RETURN OLD;\nEND $$;\n\n-- Create the trigger that calls the function when a record is deleted from the products table\nCREATE TRIGGER delete_products_trigger\n    AFTER DELETE ON public.products\n    FOR EACH ROW\n    EXECUTE FUNCTION delete_products();\n```\n\n## Calling a serverless function every minute with PG_CRON\n\nThe [PG_CRON](https://github.com/citusdata/pg_cron) extension enables PostgreSQL to become its own cron server. With it you can schedule regular calls to activate serverless functions.\n\n\u003e Useful links:\n\u003e\n\u003e * [Supabase PG_CRON Installation Guide](https://supabase.com/docs/guides/database/extensions/pgcron)\n\u003e * [Cron Syntax Helper](https://crontab.guru/)\n\n### Example Cron job to call serverless function\n\n```sql\nSELECT cron.schedule(\n\t'cron-job-name',\n\t'* * * * *', -- Executes every minute (cron syntax)\n\t$$\n\t    -- SQL query\n\t    SELECT net.http_get(\n\t\t-- URL of Edge function\n\t\turl:='https://\u003creference id\u003e.functions.Supabase.co/example',\n\t\theaders:='{\n\t\t    \"Content-Type\": \"application/json\",\n\t\t    \"Authorization\": \"Bearer \u003cTOKEN\u003e\"\n\t\t}'::JSONB\n\t    ) as request_id;\n\t$$\n);\n```\n\n## Retrying failed requests\n\nEvery request made is logged within the net._http_response table. To identify failed requests, you can execute a query on the table, filtering for requests where the status code is 500 or higher.\n\n### Finding failed requests\n\n```sql\nSELECT\n    *\nFROM net._http_response\nWHERE status_code \u003e= 500;\n```\n\nWhile the net.\\_http_response table logs each request, it doesn't store all the necessary information to retry failed requests. To facilitate this, we need to create a request tracking table and a wrapper function around the PG_NET request functions. This will help us store the required details for each request.\n\n### Creating a Request Tracker Table\n\n```sql\nCREATE TABLE request_tracker(\n    method TEXT,\n    url TEXT,\n    params JSONB,\n    body JSONB,\n    headers JSONB,\n    request_id BIGINT\n)\n```\n\nBelow is a function called request_wrapper, which wraps around the PG_NET request functions. This function records every request's details in the request_tracker table, facilitating future retries if needed.\n\n### Creating a Request Wrapper Function\n\n```sql\nCREATE OR REPLACE FUNCTION request_wrapper(\n    method TEXT,\n    url TEXT,\n    params JSONB DEFAULT '{}'::JSONB,\n    body JSONB DEFAULT '{}'::JSONB,\n    headers JSONB DEFAULT '{}'::JSONB\n)\nRETURNS BIGINT\nAS $$\nDECLARE\n    request_id BIGINT;\nBEGIN\n\n    IF method = 'DELETE' THEN\n        SELECT net.http_delete(\n            url:=url,\n            params:=params,\n            headers:=headers\n        ) INTO request_id;\n    ELSIF method = 'POST' THEN\n        SELECT net.http_post(\n            url:=url,\n            body:=body,\n            params:=params,\n            headers:=headers\n        ) INTO request_id;\n    ELSIF method = 'GET' THEN\n        SELECT net.http_get(\n            url:=url,\n            params:=params,\n            headers:=headers\n        ) INTO request_id;\n    ELSE\n        RAISE EXCEPTION 'Method must be DELETE, POST, or GET';\n    END IF;\n\n    INSERT INTO request_tracker (method, url, params, body, headers, request_id)\n    VALUES (method, url, params, body, headers, request_id);\n\n    RETURN request_id;\nEND;\n$$\nLANGUAGE plpgsql;\n```\n\nTo retry a failed request recorded via the wrapper function, use the following query. This will select failed requests, retry them, and then remove the original request data from both the net.\\_http_response and request_tracker tables.\n\n### Retrying failed requests\n\n```sql\nWITH retry_request AS (\n    SELECT\n        request_tracker.method,\n        request_tracker.url,\n        request_tracker.params,\n        request_tracker.body,\n        request_tracker.headers,\n        request_tracker.request_id\n    FROM request_tracker\n    INNER JOIN net._http_response ON net._http_response.id = request_tracker.request_id\n    WHERE net._http_response.status_code \u003e= 500\n    LIMIT 3\n),\nretry AS (\n    SELECT\n        request_wrapper(retry_request.method, retry_request.url, retry_request.params, retry_request.body, retry_request.headers)\n    FROM retry_request\n),\ndelete_http_response AS (\n    DELETE FROM net._http_response\n    WHERE id IN (SELECT request_id FROM retry_request)\n    RETURNING *\n)\nDELETE FROM request_tracker\nWHERE request_id IN (SELECT request_id FROM retry_request)\nRETURNING *;\n```\n\nThe above function can be called using cron jobs or manually to retry failed requests. It may also be beneficial to clean the request_tracker table in the process.\n\n# Contributing\n\nCheckout the [Contributing](docs/contributing.md) page to learn more about adding to the project.\n","funding_links":["https://github.com/sponsors/supabase"],"categories":["C","Python"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsupabase%2Fpg_net","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsupabase%2Fpg_net","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsupabase%2Fpg_net/lists"}