{"id":25635273,"url":"https://github.com/hatamiarash7/duckdb-netquack","last_synced_at":"2026-02-20T10:03:06.555Z","repository":{"id":276664634,"uuid":"922426316","full_name":"hatamiarash7/duckdb-netquack","owner":"hatamiarash7","description":"DuckDB extension for parsing, extracting, and analyzing domains, URIs, and paths with ease.","archived":false,"fork":false,"pushed_at":"2026-02-15T17:02:37.000Z","size":662,"stargazers_count":31,"open_issues_count":3,"forks_count":2,"subscribers_count":1,"default_branch":"main","last_synced_at":"2026-02-15T23:39:47.024Z","etag":null,"topics":["database","database-extension","duckdb","duckdb-community","duckdb-database","duckdb-extension","duckdb-udf","extension","network-utilities","sql"],"latest_commit_sha":null,"homepage":"https://duckdb.org/community_extensions/extensions/netquack.html","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/hatamiarash7.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":"CITATION.cff","codeowners":".github/CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-01-26T06:59:06.000Z","updated_at":"2026-02-15T17:02:41.000Z","dependencies_parsed_at":"2025-03-02T17:25:54.167Z","dependency_job_id":"8f7218d4-2ef1-4293-903a-8713883ffddc","html_url":"https://github.com/hatamiarash7/duckdb-netquack","commit_stats":null,"previous_names":["hatamiarash7/duckdb-netquack"],"tags_count":9,"template":false,"template_full_name":"duckdb/extension-template","purl":"pkg:github/hatamiarash7/duckdb-netquack","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hatamiarash7%2Fduckdb-netquack","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hatamiarash7%2Fduckdb-netquack/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hatamiarash7%2Fduckdb-netquack/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hatamiarash7%2Fduckdb-netquack/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/hatamiarash7","download_url":"https://codeload.github.com/hatamiarash7/duckdb-netquack/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hatamiarash7%2Fduckdb-netquack/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29647768,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-20T09:27:29.698Z","status":"ssl_error","status_checked_at":"2026-02-20T09:26:12.373Z","response_time":59,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["database","database-extension","duckdb","duckdb-community","duckdb-database","duckdb-extension","duckdb-udf","extension","network-utilities","sql"],"created_at":"2025-02-22T23:27:48.781Z","updated_at":"2026-02-20T10:03:06.549Z","avatar_url":"https://github.com/hatamiarash7.png","language":"C++","funding_links":[],"categories":["Extensions"],"sub_categories":["[Community Extensions](https://duckdb.org/community_extensions/)"],"readme":"# DuckDB Netquack Extension\n\n[![DuckDB Badge](https://img.shields.io/badge/Built_With-DuckDB-fff100)](https://duckdb.org/community_extensions/extensions/netquack.html) [![GitHub License](https://img.shields.io/github/license/hatamiarash7/duckdb-netquack)](https://github.com/hatamiarash7/duckdb-netquack/blob/main/LICENSE) [![GitHub Release](https://img.shields.io/github/v/release/hatamiarash7/duckdb-netquack)](https://github.com/hatamiarash7/duckdb-netquack/releases/latest)\n\n![logo](./.github/logo.jpg)\n\nThis extension is designed to simplify working with domains, URIs, and web paths directly within your database queries. Whether you're extracting top-level domains (TLDs), parsing URI components, or analyzing web paths, Netquack provides a suite of intuitive functions to handle all your network tasks efficiently. Built for data engineers, analysts, and developers.\n\nWith Netquack, you can unlock deeper insights from your web-related datasets without the need for external tools or complex workflows.\n\nNetQuack uses ClickHouse-inspired character-by-character parsing and gperf-generated perfect hash functions for optimal performance.\n\nTable of Contents\n\n- [DuckDB Netquack Extension](#duckdb-netquack-extension)\n  - [Installation 🚀](#installation-)\n  - [Usage Examples 📚](#usage-examples-)\n    - [Extracting The Main Domain](#extracting-the-main-domain)\n    - [Extracting The Path](#extracting-the-path)\n    - [Extracting The Host](#extracting-the-host)\n    - [Extracting The Schema](#extracting-the-schema)\n    - [Extracting The Query](#extracting-the-query)\n      - [Query String](#query-string)\n      - [Query Parameters](#query-parameters)\n    - [Extracting The Port](#extracting-the-port)\n    - [Extracting The File Extension](#extracting-the-file-extension)\n    - [Extracting The TLD (Top-Level Domain)](#extracting-the-tld-top-level-domain)\n    - [Extracting The Sub Domain](#extracting-the-sub-domain)\n    - [Get Tranco Rank](#get-tranco-rank)\n      - [Update Tranco List](#update-tranco-list)\n      - [Get Tranco Ranking](#get-tranco-ranking)\n    - [IP Address Functions](#ip-address-functions)\n      - [IP Calculator](#ip-calculator)\n    - [Get Extension Version](#get-extension-version)\n  - [Build Requirements](#build-requirements)\n  - [Debugging](#debugging)\n  - [Roadmap 🗺️](#roadmap-️)\n  - [Contributing 🤝](#contributing-)\n  - [Issues 🐛](#issues-)\n\n## Installation 🚀\n\n**netquack** is distributed as a [DuckDB Community Extension](https://duckdb.org/community_extensions/) and can be installed using SQL:\n\n```sql\nSET allow_community_extensions = true;\nINSTALL netquack FROM community;\nLOAD netquack;\n```\n\nIf you previously installed the `netquack` extension, upgrade using the FORCE command\n\n```sql\nFORCE INSTALL netquack FROM community;\nLOAD netquack;\n```\n\nAlso, you can check for any available updates for the extension using this command:\n\n```sql\nUPDATE EXTENSIONS (netquack);\n```\n\n## Usage Examples 📚\n\nOnce installed, the [macro functions](https://duckdb.org/community_extensions/extensions/netquack.html#added-functions) provided by the extension can be used just like built-in functions.\n\n### Extracting The Main Domain\n\nThis function extracts the main domain from a URL using an optimized static TLD lookup system. The extension uses Mozilla's Public Suffix List compiled into a gperf-generated perfect hash function for O(1) TLD lookups with zero collisions.\n\n```sql\nD SELECT extract_domain('a.example.com') AS domain;\n┌─────────────┐\n│   domain    │\n│   varchar   │\n├─────────────┤\n│ example.com │\n└─────────────┘\n\nD SELECT extract_domain('https://b.a.example.com/path') AS domain;\n┌─────────────┐\n│   domain    │\n│   varchar   │\n├─────────────┤\n│ example.com │\n└─────────────┘\n```\n\nThe TLD lookup is built into the extension at compile time using the latest Mozilla Public Suffix List. No runtime downloads or database operations are required.\n\n### Extracting The Path\n\nThis function extracts the path from a URL.\n\n```sql\nD SELECT extract_path('https://b.a.example.com/path/path') AS path;\n┌────────────┐\n│    path    │\n│  varchar   │\n├────────────┤\n│ /path/path │\n└────────────┘\n\nD SELECT extract_path('example.com/path/path/image.png') AS path;\n┌──────────────────────┐\n│         path         │\n│       varchar        │\n├──────────────────────┤\n│ /path/path/image.png │\n└──────────────────────┘\n```\n\n### Extracting The Host\n\nThis function extracts the host from a URL.\n\n```sql\nD SELECT extract_host('https://b.a.example.com/path/path') AS host;\n┌─────────────────┐\n│      host       │\n│     varchar     │\n├─────────────────┤\n│ b.a.example.com │\n└─────────────────┘\n\nD SELECT extract_host('example.com:443/path/image.png') AS host;\n┌─────────────┐\n│    host     │\n│   varchar   │\n├─────────────┤\n│ example.com │\n└─────────────┘\n```\n\n### Extracting The Schema\n\nThis function extracts the schema from a URL. Supported schemas for now:\n\n- `http` | `https`\n- `ftp`\n- `mailto`\n- `tel` | `sms`\n\n```sql\nD SELECT extract_schema('https://b.a.example.com/path/path') AS schema;\n┌─────────┐\n│ schema  │\n│ varchar │\n├─────────┤\n│ https   │\n└─────────┘\n\nD SELECT extract_schema('mailto:someone@example.com') AS schema;\n┌─────────┐\n│ schema  │\n│ varchar │\n├─────────┤\n│ mailto  │\n└─────────┘\n\nD SELECT extract_schema('tel:+123456789') AS schema;\n┌─────────┐\n│ schema  │\n│ varchar │\n├─────────┤\n│ tel     │\n└─────────┘\n```\n\n### Extracting The Query\n\n#### Query String\n\nThe `extract_query_string` function extracts the query string from a URL as a single string.\n\n```sql\nD SELECT extract_query_string('example.com?key=value') AS query;\n┌───────────┐\n│   query   │\n│  varchar  │\n├───────────┤\n│ key=value │\n└───────────┘\n\nD SELECT extract_query_string('http://example.com.ac/path/?a=1\u0026b=2') AS query;\n┌─────────┐\n│  query  │\n│ varchar │\n├─────────┤\n│ a=1\u0026b=2 │\n└─────────┘\n```\n\n#### Query Parameters\n\nThe `extract_query_parameters` table function parses the query string and returns each key-value pair as a separate row. This is useful for analyzing URL parameters in a structured way.\n\n```sql\nD SELECT * FROM extract_query_parameters('http://example.com/path/?a=1\u0026b=2');\n┌─────────┬─────────┐\n│   key   │  value  │\n│ varchar │ varchar │\n├─────────┼─────────┤\n│ a       │ 1       │\n│ b       │ 2       │\n└─────────┴─────────┘\n\nD SELECT * FROM extract_query_parameters('https://example.com/search?q=duckdb\u0026hl=en\u0026num=10');\n┌─────────┬─────────┐\n│   key   │  value  │\n│ varchar │ varchar │\n├─────────┼─────────┤\n│ q       │ duckdb  │\n│ hl      │ en      │\n│ num     │ 10      │\n└─────────┴─────────┘\n\nD SELECT m.media_url,\n  e.key,\n  e.value\nFROM instagram_posts m,\n  LATERAL extract_query_parameters(m.media_url) e\nORDER BY m.id;\n\n┌───────────────────────────────────────────────────────────────────────────────────────────┬────────────┬───────────┐\n│                                         media_url                                         │    key     │   value   │\n│                                          varchar                                          │  varchar   │  varchar  │\n├───────────────────────────────────────────────────────────────────────────────────────────┼────────────┼───────────┤\n│ https://cdn.instagram.com/media/abc123.jpg?utm_source=instagram\u0026utm_medium=social\u0026id=1001 │ id         │ 1001      │\n│ https://cdn.instagram.com/media/abc123.jpg?utm_source=instagram\u0026utm_medium=social\u0026id=1001 │ utm_medium │ social    │\n│ https://cdn.instagram.com/media/abc123.jpg?utm_source=instagram\u0026utm_medium=social\u0026id=1001 │ utm_source │ instagram │\n│ https://cdn.instagram.com/media/def456.jpg?quality=hd\u0026format=webp\u0026user=arash              │ user       │ arash     │\n│ https://cdn.instagram.com/media/def456.jpg?quality=hd\u0026format=webp\u0026user=arash              │ format     │ webp      │\n│ https://cdn.instagram.com/media/def456.jpg?quality=hd\u0026format=webp\u0026user=arash              │ quality    │ hd        │\n│ https://cdn.instagram.com/media/ghi789.mp4?autoplay=true\u0026loop=false\u0026session_id=xyz987     │ session_id │ xyz987    │\n│ https://cdn.instagram.com/media/ghi789.mp4?autoplay=true\u0026loop=false\u0026session_id=xyz987     │ loop       │ false     │\n│ https://cdn.instagram.com/media/ghi789.mp4?autoplay=true\u0026loop=false\u0026session_id=xyz987     │ autoplay   │ true      │\n└───────────────────────────────────────────────────────────────────────────────────────────┴────────────┴───────────┘\n```\n\n### Extracting The Port\n\nThis function extracts the port from a URL.\n\n```sql\nD SELECT extract_port('https://example.com:8443/') AS port;\n┌─────────┐\n│  port   │\n│ varchar │\n├─────────┤\n│ 8443    │\n└─────────┘\n\nD SELECT extract_port('[::1]:6379') AS port;\n┌─────────┐\n│  port   │\n│ varchar │\n├─────────┤\n│ 6379    │\n└─────────┘\n```\n\n### Extracting The File Extension\n\nThis function extracts the file extension from a URL. It will return the file extension without the dot.\n\n```sql\nD SELECT extract_extension('http://example.com/image.jpg') AS ext;\n┌─────────┐\n│   ext   │\n│ varchar │\n├─────────┤\n│ jpg     │\n└─────────┘\n```\n\n### Extracting The TLD (Top-Level Domain)\n\nThis function extracts the top-level domain from a URL using the optimized gperf-based public suffix lookup system. The function correctly handles multi-part TLDs (like `com.au`) using the longest-match algorithm from Mozilla's Public Suffix List.\n\n```sql\nD SELECT extract_tld('https://example.com.ac/path/path') AS tld;\n┌─────────┐\n│   tld   │\n│ varchar │\n├─────────┤\n│ com.ac  │\n└─────────┘\n\nD SELECT extract_tld('a.example.com') AS tld;\n┌─────────┐\n│   tld   │\n│ varchar │\n├─────────┤\n│ com     │\n└─────────┘\n```\n\n### Extracting The Sub Domain\n\nThis function extracts the sub-domain from a URL using the optimized public suffix lookup system to correctly identify the domain boundary and extract everything before it.\n\n```sql\nD SELECT extract_subdomain('http://a.b.example.com/path') AS dns_record;\n┌────────────┐\n│ dns_record │\n│  varchar   │\n├────────────┤\n│ a.b        │\n└────────────┘\n\nD SELECT extract_subdomain('test.example.com.ac') AS dns_record;\n┌────────────┐\n│ dns_record │\n│  varchar   │\n├────────────┤\n│ test       │\n└────────────┘\n```\n\n### Get Tranco Rank\n\n#### Update Tranco List\n\nThis function returns the [Tranco](https://tranco-list.eu/) rank of a domain. You have an `update_tranco` function to update the Tranco list manually.\n\n```sql\nD SELECT update_tranco(true);\n┌─────────────────────────────────────┐\n│ update_tranco(CAST('f' AS BOOLEAN)) │\n│               varchar               │\n├─────────────────────────────────────┤\n│ Tranco list updated                 │\n└─────────────────────────────────────┘\n```\n\nThis function will get the latest Tranco list and save it into the `tranco_list` table. There will be a `tranco_list_%Y-%m-%d.csv` file in the current directory after the function is called. The extension will use this file to prevent downloading the list again.\n\nYou can ignore the file and force the extension to download the list again by calling the function with `true` as a parameter. If you don't want to download the list again, you can call the function with `false` as a parameter.\n\n```sql\nD SELECT update_tranco(false);\n```\n\nAs the latest Tranco list is for the last day, you can download your list manually and rename it to `tranco_list_%Y-%m-%d.csv` to use it with the extension too.\n\n#### Get Tranco Ranking\n\nYou can use this function to get the ranking of a domain:\n\n```sql\nD SELECT get_tranco_rank('microsoft.com') AS rank;\n┌─────────┐\n│  rank   │\n│ varchar │\n├─────────┤\n│ 2       │\n└─────────┘\n\nD SELECT get_tranco_rank('cloudflare.com') AS rank;\n┌─────────┐\n│  rank   │\n│ varchar │\n├─────────┤\n│ 13      │\n└─────────┘\n```\n\nYou can use the `get_tranco_rank_category` function to retrieve the category utility column that gives you the domain's rank category. The `category` value is on a log10 scale with half steps (e.g., top 1k, top 5k, top 10k, top 50k, top 100k, top 500k, top 1M, top 5m, etc.), with each rank excluding the previous (e.g., top 5k is actually 4k domains, excluding top 1k).\n\n```sql\nD SELECT get_tranco_rank_category('microsoft.com') AS category;\n┌──────────┐\n│ category │\n│ varchar  │\n├──────────┤\n│ top1k    │\n└──────────┘\n```\n\n### IP Address Functions\n\nThis extension provides various functions for manipulating and analyzing IP addresses, including calculating networks, hosts, and subnet masks.\n\n#### IP Calculator\n\n\u003e [!WARNING]\n\u003e It's an experimental function.\n\nThe `ipcalc` function takes an IP address and netmask and calculates the resulting broadcast, network, wildcard mask, and host range.\n\n![ipcalc-sc](./.github/ipcalc-sc.png)\n\n```sql\nSELECT * FROM ipcalc('192.168.1.0/24');\n```\n\nIt's a table function that provides various details about IP addresses, including:\n\n- Address\n- Netmask\n- Wildcard\n- Network / Hostroute\n- HostMin\n- HostMax\n- Broadcast\n- Hosts count\n\nYou can use this table function with your data easily:\n\n```sql\nD CREATE OR REPLACE TABLE ips AS SELECT '127.0.0.1' AS ip UNION ALL SELECT '192.168.1.0/22';\n\nD SELECT i.IP,\n      (\n          SELECT hostsPerNet\n          FROM ipcalc(i.IP)\n      ) AS hosts\n  FROM ips AS i;\n┌────────────────┬───────┐\n│       ip       │ hosts │\n│    varchar     │ int64 │\n├────────────────┼───────┤\n│ 127.0.0.1      │   254 │\n│ 192.168.1.0/22 │  1022 │\n└────────────────┴───────┘\n```\n\n### Get Extension Version\n\nYou can use the `netquack_version` function to get the extension version.\n\n```sql\nD SELECT * FROM netquack_version();\n┌─────────┐\n│ version │\n│ varchar │\n├─────────┤\n│ v1.8.1  │\n└─────────┘\n```\n\n## Build Requirements\n\n- **C++ compiler**: Needs C++17 or later (e.g., `g++`, `clang++`).\n- **gperf**: Perfect hash generation requires `gperf`.\n- **CMake**\n- **GNU Make**\n\n```bash\n# On Debian-based systems\nsudo apt-get install gperf cmake make\n\n# On MacOS using Homebrew\nbrew install gperf cmake make\n```\n\n## Debugging\n\nThe debugging process for DuckDB extensions is not an easy job. For Netquack, we have created a log file in the current directory. The log file is named `netquack.log` and contains all the logs for the extension. You can use this file to debug your code.\n\nAlso, there will be stdout errors for background tasks like CURL.\n\n## Roadmap 🗺️\n\n- [ ] Implement `extract_custom_format` function\n- [ ] Implement `parse_uri` function - Return a STRUCT with all components (scheme, host, port, path, query, fragment) in a single call\n- [ ] Save Tranco data as Parquet\n- [ ] Implement GeoIP functionality\n- [ ] Return default value for `get_tranco_rank`\n- [ ] Implement `extract_fragment` function - Extract the fragment (`#section`) from a URL\n- [ ] Implement `normalize_url` function - Canonicalize URLs (lowercase scheme/host, remove default ports, sort query params, remove trailing slashes)\n- [ ] Implement `is_valid_url` function - Return whether a string is a well-formed URL\n- [ ] Implement `url_encode` / `url_decode` functions - Standalone percent-encoding and decoding\n- [ ] Implement `is_valid_ip` function - Return whether a string is a valid IPv4 or IPv6 address\n- [ ] Implement `is_private_ip` function - Check if an IP is in a private/reserved range (RFC 1918, loopback, link-local)\n- [ ] Implement `ip_to_int` / `int_to_ip` functions - Convert between dotted-quad notation and integer representation\n- [ ] Implement `ip_in_range` function - Check if an IP falls within a given CIDR block\n- [ ] Implement `ip_version` function - Return `4` or `6` for the IP version of a given address\n- [ ] Support internationalized domain names (IDNs)\n- [ ] Implement `punycode_encode` / `punycode_decode` functions - Convert internationalized domain names to/from ASCII-compatible encoding\n- [ ] Implement `is_valid_domain` function - Validate a domain name against RFC rules\n- [ ] Implement `domain_depth` function - Return the number of levels in a domain\n- [ ] Implement `base64_encode` / `base64_decode` functions - Encode and decode Base64 strings\n- [ ] Implement `extract_path_segments` table function - Split a URL path into individual segment rows\n\n## Contributing 🤝\n\nDon't be shy and reach out to us if you want to contribute 😉\n\n1. Fork it!\n2. Create your feature branch: `git checkout -b my-new-feature`\n3. Commit your changes: `git commit -am 'Add some feature'`\n4. Push to the branch: `git push origin my-new-feature`\n5. Submit a pull request\n\n## Issues 🐛\n\nEach project may have many problems. Contributing to the better development of this project by [reporting them](https://github.com/hatamiarash7/duckdb-netquack/issues). 👍\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhatamiarash7%2Fduckdb-netquack","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhatamiarash7%2Fduckdb-netquack","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhatamiarash7%2Fduckdb-netquack/lists"}