{"id":51992940,"url":"https://github.com/KoliStat/the-stats-duck","last_synced_at":"2026-08-02T01:00:18.215Z","repository":{"id":350192886,"uuid":"1199842332","full_name":"KoliStat/the-stats-duck","owner":"KoliStat","description":"A statistical computing toolkit for DuckDB.","archived":false,"fork":false,"pushed_at":"2026-07-01T11:32:33.000Z","size":1374,"stargazers_count":47,"open_issues_count":9,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-07-01T13:15:54.955Z","etag":null,"topics":["data-analysis","data-science","duckdb","hypothesis-testing","statistical-tests","statistics"],"latest_commit_sha":null,"homepage":"https://kolistat.com/products/the-stats-duck","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/KoliStat.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":"CITATION.cff","codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-04-02T19:08:43.000Z","updated_at":"2026-07-01T11:25:27.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/KoliStat/the-stats-duck","commit_stats":null,"previous_names":["caerbannogwhite/the-stats-duck","kolistat/the-stats-duck"],"tags_count":9,"template":false,"template_full_name":null,"purl":"pkg:github/KoliStat/the-stats-duck","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/KoliStat%2Fthe-stats-duck","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/KoliStat%2Fthe-stats-duck/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/KoliStat%2Fthe-stats-duck/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/KoliStat%2Fthe-stats-duck/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/KoliStat","download_url":"https://codeload.github.com/KoliStat/the-stats-duck/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/KoliStat%2Fthe-stats-duck/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":36175203,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-07-20T02:08:10.276Z","status":"online","status_checked_at":"2026-08-01T02:00:05.789Z","response_time":100,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data-analysis","data-science","duckdb","hypothesis-testing","statistical-tests","statistics"],"created_at":"2026-07-31T03:00:14.380Z","updated_at":"2026-08-02T01:00:18.124Z","avatar_url":"https://github.com/KoliStat.png","language":"C++","funding_links":[],"categories":["Extensions"],"sub_categories":["[Community Extensions](https://duckdb.org/community_extensions/)"],"readme":"# The Stats Duck\n\nA statistical computing toolkit for DuckDB.\n\nThe Stats Duck brings statistical workflows — descriptive statistics, hypothesis\ntests, grammar-of-graphics visualization, and direct readers/writers for SAS,\nSPSS, and Stata files — into SQL. Functions are implemented as streaming\naggregates and scalar primitives, so they scale from local notebooks to\nbillion-row warehouses and also run inside DuckDB-WASM in the browser.\n\n\u003e The extension installs and loads in DuckDB under the technical name\n\u003e `stats_duck` (matching the binary, the SQL function namespace, and the\n\u003e `INSTALL` keyword). \"The Stats Duck\" is the project / brand name; `stats_duck`\n\u003e is what you type at the SQL prompt.\n\n## Scope\n\nThe Stats Duck is meant to cover the everyday work of a general-purpose\nstatistician without leaving SQL. The current release covers four areas:\n\n- **Hypothesis tests** — parametric and non-parametric, with effect sizes and\n  confidence intervals returned alongside the test statistic.\n- **Visualizations (VISUALIZE)** — `VISUALIZE … FROM \u003ctable\u003e DRAW \u003cmark\u003e`, a\n  Posit-published Grammar-of-Graphics SQL dialect compiled to Vega-Lite v5.\n  No server-side rendering: the extension emits a spec + per-layer SQL, and\n  the client (browser, notebook, …) runs the SQL and feeds the rows to vega.\n- **Statistical file I/O** — first-class readers AND writers for SAS, SPSS, and\n  Stata files, integrated with DuckDB's virtual file system so they work\n  transparently with local paths, `httpfs://`, `s3://`, and registered WASM\n  file buffers.\n- **Streaming aggregates** — every test is a single-pass aggregate over the\n  data, so it composes naturally with `GROUP BY`, window frames, and DuckDB's\n  parallel execution.\n\nFuture releases will add Spearman/Kendall correlations, regression with full\ndiagnostics, multiple-testing corrections, and more distribution families.\n\n## Functions\n\n### Hypothesis tests (aggregate)\n\n| Function                                                              | Description                                  |\n| --------------------------------------------------------------------- | -------------------------------------------- |\n| `ttest_1samp(column, [mu], [alpha], [alternative])`                   | One-sample t-test                            |\n| `ttest_2samp(column1, column2, [equal_var], [alpha], [alternative])`  | Two-sample t-test (Welch's or Student's)     |\n| `ttest_paired(column1, column2, [alpha], [alternative])`              | Paired t-test                                |\n| `mann_whitney_u(column1, column2, [alternative], [continuity])`       | Mann-Whitney U test (Wilcoxon rank-sum)      |\n| `wilcoxon_signed_rank(column1, column2, [alternative], [continuity])` | Wilcoxon signed-rank test                    |\n| `pearson_test(x, y, [alpha], [alternative])`                          | Pearson correlation with significance        |\n| `spearman_test(x, y, [alpha], [alternative])`                         | Spearman rank correlation                    |\n| `kendall_test(x, y, [alternative])`                                   | Kendall's tau-b rank correlation             |\n| `anova_oneway(value, group)`                                          | One-way ANOVA                                |\n| `chisq_independence(row, col, [continuity])`                          | Chi-square test of independence              |\n| `chisq_goodness_of_fit(category)`                                     | Chi-square goodness-of-fit (uniform)         |\n| `jarque_bera(column)`                                                 | Jarque-Bera normality test                   |\n| `shapiro_wilk(column)`                                                | Shapiro-Wilk normality test (Royston AS R94) |\n| `anderson_darling(column)`                                            | Anderson-Darling normality test              |\n| `ks_test_1samp(column)`                                               | Kolmogorov-Smirnov one-sample (vs fitted normal) |\n| `ks_test_2samp(column1, column2)`                                     | Kolmogorov-Smirnov two-sample                |\n| `sign_test_1samp(column, [mu], [alternative])`                        | Sign test on the median                      |\n| `sign_test_paired(column1, column2, [alternative])`                   | Paired sign test                             |\n\nAll tests return a `STRUCT` with the test statistic, degrees of freedom,\np-value, and relevant effect sizes / confidence intervals.\n\n#### Common parameters\n\n| Parameter     | Type      | Default       | Description                                                      |\n| ------------- | --------- | ------------- | ---------------------------------------------------------------- |\n| `mu`          | `DOUBLE`  | `0.0`         | Hypothesized population mean (one-sample t-test only)            |\n| `equal_var`   | `BOOLEAN` | `false`       | Assume equal variances — Student's pooled test (two-sample only) |\n| `alpha`       | `DOUBLE`  | `0.05`        | Significance level for confidence intervals (t-tests only)       |\n| `alternative` | `VARCHAR` | `'two-sided'` | `'two-sided'`, `'less'`, or `'greater'`                          |\n| `continuity`  | `BOOLEAN` | `false`       | Apply continuity correction (mann_whitney_u, chisq_independence) |\n\n#### Result struct fields\n\n**t-test:** `test_type`, `t_statistic`, `df`, `p_value`, `alternative`, `mean_diff`, `ci_lower`, `ci_upper`, `cohens_d`\n\n**Mann-Whitney:** `test_type`, `u_statistic`, `z_statistic`, `p_value`, `alternative`, `rank_biserial`\n\n**Wilcoxon:** `test_type`, `w_statistic`, `z_statistic`, `p_value`, `alternative`, `effect_size_r`\n\n**Pearson:** `test_type`, `r`, `t_statistic`, `df`, `p_value`, `alternative`, `ci_lower`, `ci_upper`, `n`\n\n**Spearman:** `test_type`, `rho`, `t_statistic`, `df`, `p_value`, `alternative`, `ci_lower`, `ci_upper`, `n`\n\n**Kendall:** `test_type`, `tau`, `z_statistic`, `p_value`, `alternative`, `n`\n\n**ANOVA:** `test_type`, `f_statistic`, `df_between`, `df_within`, `p_value`, `ss_between`, `ss_within`, `eta_squared`, `n_groups`, `n`\n\n**Chi-square:** `test_type`, `chi_square`, `df`, `p_value`, `n`, `n_rows`/`n_cols` or `n_categories`\n\n**Jarque-Bera:** `test_type`, `jb_statistic`, `skewness`, `excess_kurtosis`, `df`, `p_value`, `n`\n\n**Anderson-Darling:** `test_type`, `a_squared`, `a_squared_adjusted`, `p_value`, `n`\n\n**Shapiro-Wilk:** `test_type`, `w_statistic`, `p_value`, `n`\n\n**KS one-sample:** `test_type`, `d_statistic`, `p_value`, `n`\n\n**KS two-sample:** `test_type`, `d_statistic`, `p_value`, `n_x`, `n_y`\n\n**Sign test:** `test_type`, `m_statistic`, `n_pos`, `n_neg`, `n_zero`, `p_value`, `alternative`, `n`\n\n### Descriptive statistics (aggregate)\n\n| Function                                                    | Description                                                                                                              |\n| ----------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------ |\n| `summary_stats(column, [bias_correction], [quantile_type])` | n, n_missing, mean, sd, variance, min, q1, median, q3, max, iqr, skewness, kurtosis, mode, mode_frequency, is_multimodal |\n\n`bias_correction` (BOOLEAN, default `true`) toggles the skewness/kurtosis\nformulas. With `true` the output matches SAS PROC MEANS, scipy with\n`bias=False`, and Excel SKEW/KURT. With `false` the population formulas\n`m3/m2^1.5` and `m4/m2² - 3` are used, matching R's default. Mean, SD,\nvariance, quantiles, and IQR are unaffected.\n\n`mode` is the smallest modal value (when more than one value shares the\nmaximum frequency, the smallest of them is returned and `is_multimodal`\nis set to `true`). For all-distinct input — every value appears exactly\nonce — `mode` is `NaN` and `mode_frequency` is `0`, matching SAS PROC\nUNIVARIATE's \"Mode .\" output.\n\n`quantile_type` (INTEGER, default `7`) picks the Hyndman \u0026 Fan (1996)\nquantile algorithm used for `q1`, `median`, and `q3`. Supported values:\n\n- `7` — R / Excel INC default. `position = 1 + q * (n - 1)`.\n- `5` — SAS PROC UNIVARIATE default. `position = q * n + 0.5`.\n\nFor `x = [1, 2, 3, 4]`: type 7 gives Q1=1.75 / Q3=3.25 (matching R); type 5\ngives Q1=1.5 / Q3=3.5 (matching SAS PROC MEANS).\n\n### Distribution functions (scalar)\n\n| Function                 | Description             |\n| ------------------------ | ----------------------- |\n| `dnorm(x, [mean], [sd])` | Normal PDF              |\n| `pnorm(x, [mean], [sd])` | Normal CDF              |\n| `qnorm(p, [mean], [sd])` | Normal quantile         |\n| `dt(x, df)`              | Student's t PDF         |\n| `pt(x, df)`              | Student's t CDF         |\n| `qt(p, df)`              | Student's t quantile    |\n| `dchisq(x, df)`          | Chi-square PDF          |\n| `pchisq(x, df)`          | Chi-square CDF          |\n| `qchisq(p, df)`          | Chi-square quantile     |\n| `df(x, df1, df2)`        | F distribution PDF      |\n| `pf(x, df1, df2)`        | F distribution CDF      |\n| `qf(p, df1, df2)`        | F distribution quantile |\n| `dgamma(x, shape, [rate])` | Gamma PDF (rate=1 default) |\n| `pgamma(x, shape, [rate])` | Gamma CDF             |\n| `qgamma(p, shape, [rate])` | Gamma quantile        |\n| `dbeta(x, alpha, beta)`  | Beta PDF on [0, 1]      |\n| `pbeta(x, alpha, beta)`  | Beta CDF                |\n| `qbeta(p, alpha, beta)`  | Beta quantile           |\n| `dexp(x, [rate])`        | Exponential PDF (rate=1 default) |\n| `pexp(x, [rate])`        | Exponential CDF         |\n| `qexp(p, [rate])`        | Exponential quantile (closed form) |\n| `dweibull(x, shape, [scale])` | Weibull PDF (scale=1 default) |\n| `pweibull(x, shape, [scale])` | Weibull CDF |\n| `qweibull(p, shape, [scale])` | Weibull quantile (closed form) |\n| `dlnorm(x, [meanlog], [sdlog])` | Log-normal PDF (meanlog=0, sdlog=1 defaults) |\n| `plnorm(x, [meanlog], [sdlog])` | Log-normal CDF |\n| `qlnorm(p, [meanlog], [sdlog])` | Log-normal quantile |\n| `dpois(k, lambda)`       | Poisson PMF (discrete) |\n| `ppois(q, lambda)`       | Poisson CDF |\n| `qpois(p, lambda)`       | Poisson quantile (integer search) |\n| `dnbinom(k, size, prob)` | Negative binomial PMF (count of failures before `size` successes; `prob` per-trial) |\n| `pnbinom(q, size, prob)` | Negative binomial CDF — closed form via regularized incomplete beta |\n| `qnbinom(p, size, prob)` | Negative binomial quantile (integer search) |\n| `dhyper(x, m, n, k)`     | Hypergeometric PMF (`m` successes / `n` failures / `k` draws without replacement) |\n| `phyper(q, m, n, k)`     | Hypergeometric CDF (direct PMF sum; tractable for typical population sizes) |\n| `qhyper(p, m, n, k)`     | Hypergeometric quantile |\n| `rnorm([mean], [sd])`    | Random sample from a normal — **volatile, per-row** |\n| `rt(df)` / `rchisq(df)` / `rf(df1, df2)` | Student-t / χ² / F samples |\n| `rgamma(shape, [rate])` / `rbeta(alpha, beta)` / `rexp([rate])` | Gamma / Beta / Exponential samples |\n| `rweibull(shape, [scale])` / `rlnorm([meanlog], [sdlog])` / `rpois(lambda)` | Weibull / Log-normal / Poisson samples |\n| `rnbinom(size, prob)` / `rhyper(m, n, k)` | Negative binomial / Hypergeometric samples |\n| `poibin_cdf(probs LIST\u003cDOUBLE\u003e, k BIGINT)` | Poisson Binomial CDF — `P(X ≤ k)` for `X = Σᵢ Bᵢ`, `Bᵢ ∼ Bernoulli(pᵢ)` |\n| `bin_edges(x [, method])` *(aggregate)* | Auto bin-edge vector for `x` — `sturges` (default), `fd`, `scott`, `sqrt`, `rice`, `auto` |\n| `bin_label(x, edges)` | Label for the bin containing `x` given an edge vector (typically from `bin_edges`) |\n| `bootstrap(x, statistic, n_iters [, seed])` *(aggregate)* | With-replacement resampling — emits `LIST\u003cDOUBLE\u003e` of length `n_iters`. `statistic` ∈ `{mean, median, sum, stddev, variance, min, max}` |\n\n### Dataset profile (table function)\n\n| Function       | Description                                              |\n| -------------- | -------------------------------------------------------- |\n| `meta(data)`   | One row per column with kind classification + light stats |\n\n```sql\nSELECT * FROM meta('penguins');\n```\n\nOutput columns (fixed schema):\n`column_name`, `column_type`, `kind`, `n_rows`, `n_missing`, `n_distinct`,\n`min`, `p25`, `median`, `p75`, `max`, `mean`, `stddev`, `top`, `top_freq`.\n\n- `kind` is a semantic classification —\n  `numeric` / `categorical` / `temporal` / `boolean` / `other` — derived\n  from the catalog type. It controls which distribution columns are\n  populated: numeric kinds fill `min` / `p25` / `median` / `p75` / `max` /\n  `mean` / `stddev` (cast to DOUBLE; `quantile_cont` type 7, sample\n  stddev), and `categorical` / `boolean` fill `top` (the mode, ties broken\n  by smaller value) and `top_freq` (its count).\n- `n_rows` is the source table's row count, repeated on every row so any\n  single row is self-contained.\n- Dataset-level summaries fall out via aggregation:\n\n  ```sql\n  SELECT count(*) FILTER (WHERE kind = 'numeric')     AS n_numeric,\n         count(*) FILTER (WHERE kind = 'categorical') AS n_categorical,\n         sum(n_missing)                               AS total_missing,\n         count(*) FILTER (WHERE n_missing \u003e 0)        AS cols_with_nulls\n  FROM meta('penguins');\n  ```\n\n- Overlaps DuckDB's built-in `SUMMARIZE` but is a table function\n  (joinable, filterable, composable in CTEs), adds the kind classifier,\n  and reports a mode for categorical / boolean columns. Per-column\n  detail (skewness, kurtosis, custom quantile types, bias-corrected\n  variants) lives in `summary_stats(column)`.\n\n### Table 1 summary (table function)\n\n| Function                                          | Description                                                |\n| ------------------------------------------------- | ---------------------------------------------------------- |\n| `table_one(data, variables [, by])`               | Long-format descriptives table for mixed variable types    |\n| `corr_matrix(data, variables [, method])`         | Long-format pairwise correlation matrix (`pearson` / `spearman` / `kendall`) |\n| `lm(data, y, x)` / `lm_summary(data, y, x)`       | OLS regression — `lm` returns per-term coefficients, `lm_summary` returns model R² / F / σ |\n| `lm_fit(y, x [, vcov [, cluster] [, add_intercept]])` | OLS regression **aggregate** (one model per `GROUP BY`) with classical, HC0–HC3, or CR0/CR1 cluster-robust standard errors |\n\n```sql\nSELECT * FROM table_one(\n    'patients',\n    variables := ['age', 'sex', 'bmi'],\n    by := ['arm']         -- optional; pass multiple columns for cross-stratification\n);\n```\n\nOutput columns (long format, fixed schema):\n`variable`, `level`, `statistic`, `stratum`, `display`, `p_value`, `effect_size`\n\n- Each numeric variable yields rows for `n`, `missing`, `mean (sd)`,\n  `median [q1, q3]`, `min, max` — `level` is NULL.\n- Each categorical variable yields one row per level with `n (%)` plus a\n  trailing `Missing` level row that is always emitted (filter with\n  `WHERE level \u003c\u003e 'Missing'` if you don't want it). All level percentages\n  share the stratum-total denominator so they sum to 100%.\n- `stratum` is `'Overall'` when `by` is unset / empty; otherwise the Cartesian\n  product of distinct value tuples across the listed by-columns, labelled\n  by joining values with `' / '` in declared order (e.g. `'Adelie / female'`\n  for `by := ['species', 'sex']`). Rows where any by-column is NULL are\n  excluded from the stratum breakdown.\n- `p_value` is the between-group test result, repeated on every row of the\n  same variable so a PIVOT can grab it with `FIRST(p_value)`. NULL when\n  `by` is unset or has only one stratum. Numeric variables use one-way\n  ANOVA (`anova_oneway`); categorical variables use chi-square independence\n  (`chisq_independence`). NULL when the underlying test is infeasible (zero\n  variance, too few samples).\n- `effect_size` is the matching magnitude — **η² (eta-squared)** for\n  numeric variables (from ANOVA's `ss_between / ss_total`), **Cramér's V**\n  for categorical (`√(χ² / (n · (min(rows, cols) - 1)))`). Both are in\n  [0, 1] and larger means stronger association, so a single uniform\n  column name works across kinds. Same repetition and NULL handling as\n  `p_value`.\n- Variable types are auto-classified from the catalog: integer / floating-\n  point types are numeric, everything else (VARCHAR, BOOLEAN, ENUM,\n  date/time) is categorical. Override per-variable with\n  `force_categorical := ['stage']` (integer column that's really a\n  category) or `force_numerical := ['height']` (VARCHAR column holding\n  numeric strings). Entries must appear in `variables`, and the two lists\n  must not overlap.\n\nPivot to wide for display:\n\n```sql\nPIVOT table_one('patients', variables := ['age', 'sex'], by := ['arm'])\n    ON stratum USING first(display)\n    GROUP BY variable, level, statistic;\n```\n\n### Linear regression (table function)\n\n```sql\nSELECT * FROM lm('mtcars', y := 'mpg', x := ['wt', 'hp']);\n--   term        estimate  std_error  t_statistic  p_value\n--   (Intercept) 37.2273   1.5988     23.285       2.57e-20\n--   wt          -3.8778   0.6327     -6.129       1.12e-06\n--   hp          -0.0318   0.0090     -3.519       1.45e-03\n\nSELECT * FROM lm_summary('mtcars', y := 'mpg', x := ['wt', 'hp']);\n--   r_squared  adj_r_squared  f_statistic  f_p_value  df_model  df_residual  sigma  n\n--   0.8268     0.8148         69.21        9.11e-12   2         29           2.593  32\n```\n\nA `formula` named parameter accepts an R-style spec as an alternative to the\nexplicit `y` / `x` form:\n\n```sql\nSELECT * FROM lm('mtcars', formula := 'mpg ~ wt + hp');\nSELECT * FROM lm('mtcars', formula := 'mpg ~ wt + hp - 1');  -- no intercept\nSELECT * FROM lm('weird_names', formula := '\"My Y\" ~ \"x.with.dots\"');\n```\n\nThe formula grammar supports additive predictors, `- 1` or `+ 0` to drop the\nintercept, bare and `\"...\"`-quoted identifiers, and free whitespace.\nInteractions (`x1:x2`), wildcards (`*`, `^`, `.`) and inline expressions\n(`I(x^2)`, `log(x)`) are not supported in v0.6 — wrap into a CTE if you need\ntransformed columns. `formula` and `y` / `x` are mutually exclusive.\n\nOLS via Cholesky decomposition of `X'X`. Rows with NULL in `y` or any `x` are\ndropped (complete-case). Term order follows the user-supplied predictor order,\nafter the intercept. Calling `lm` and `lm_summary` with the same arguments\nfits the model twice — use a CTE if you need both shapes from a single fit.\nErrors on singular `X'X` (perfectly collinear predictors) or insufficient rows\n(`n ≤ k` parameters). When the intercept is removed, R²/adj-R² use the\nuncentered TSS = Σ y² to match R's `summary.lm` — interpret with care.\n\n### Linear regression aggregate with robust SEs (`lm_fit`)\n\n`lm_fit` is the **aggregate** companion to `lm`: it fits OLS over the rows of a\ngroup, so a single `GROUP BY` returns one regression per key. Where `lm` takes\ncolumn *names* (and labels its terms), the aggregate takes the design-matrix row\nas a `LIST(DOUBLE)` of predictor *values* — so coefficients come back **by\nposition**. The intercept is prepended automatically.\n\n```sql\n-- example data: y ~ x1 over 8 rows in a table `points(y, x1)`\nSELECT (u).term, (u).estimate, (u).std_error\nFROM (SELECT unnest((lm_fit(y, [x1])).coefficients) AS u FROM points);\n--   term         estimate  std_error\n--   (Intercept)  0.0357    0.1404\n--   x1           1.9976    0.0278\n```\n\nThe headline is **heteroskedasticity-consistent standard errors**. A third,\nconstant argument selects the covariance estimator — `'const'` (default,\nclassical), `'HC0'` (Eicker–Huber–White), `'HC1'` (the Stata `,robust`\ndefault, `× n/(n−k)`), `'HC2'`, or `'HC3'` (the recommended small-sample\ndefault). Only the `std_error` / `t_statistic` / `p_value` change; the point\nestimates do not:\n\n```sql\nSELECT (u).term, (u).std_error\nFROM (SELECT unnest((lm_fit(y, [x1], 'HC3')).coefficients) AS u FROM points);\n--   term         std_error\n--   (Intercept)  0.1313\n--   x1           0.0282\n```\n\nThe killer query is a regression *per group*, computed where the data lives:\n\n```sql\n-- a separate fit for every cylinder count, robust SEs, in one pass\nSELECT cyl, lm_fit(mpg, [wt, hp], 'HC1') AS model\nFROM mtcars GROUP BY cyl;\n```\n\n**Cluster-robust standard errors** (`'CR0'`, or `'CR1'` — the Stata\n`vce(cluster)` / statsmodels `cov_type='cluster'` default) account for\nwithin-cluster correlation. Unlike `vcov`, the cluster key is a real **per-row\ncolumn** (not a constant); pass it as `VARCHAR` and cast a non-text key with\n`::VARCHAR`:\n\n```sql\n-- SEs clustered by firm; 'CR1' applies the [G/(G−1)]·[(N−1)/(N−k)] correction\nSELECT (u).term, (u).estimate, (u).std_error\nFROM (SELECT unnest((lm_fit(ret, [mktrf, smb], 'CR1', firm_id::VARCHAR)).coefficients) AS u\n      FROM panel);\n```\n\nCluster-robust inference uses a `t(G − 1)` reference (G = number of clusters,\nsurfaced as `n_clusters`), so it differs from the `t(n − k)` used by classical /\nHC. `'cluster'` is accepted as an alias for `'CR1'`.\n\n`lm_fit` returns a single `STRUCT`:\n\n| Field                                                       | Type                | Notes |\n| ----------------------------------------------------------- | ------------------- | ----- |\n| `coefficients`                                              | `LIST\u003cSTRUCT\u003e`      | one element per term: `term`, `estimate`, `std_error`, `t_statistic`, `p_value` |\n| `n`, `k`, `df_residual`                                     | `BIGINT`            | rows used, parameters (incl. intercept), `n − k` |\n| `r_squared`, `adj_r_squared`, `sigma`                       | `DOUBLE`            | classical model fit |\n| `f_statistic`, `f_p_value`                                  | `DOUBLE`            | classical overall-significance F (not robustified) |\n| `has_intercept`                                             | `BOOLEAN`           | |\n| `vcov_type`                                                 | `VARCHAR`           | the estimator actually used |\n| `n_clusters`                                                | `BIGINT`            | number of clusters G (CR0/CR1 only; NULL otherwise) |\n\nA trailing constant `add_intercept := false` (positionally\n`lm_fit(y, x, 'const', false)`, or `lm_fit(y, x, 'CR1', cluster, false)` when\nclustered) drops the constant term — note aggregates take **positional**\nconstants, not the `name := value` form `lm` uses. Rows with a NULL `y`, any NULL\nlist element, or (when clustered) a NULL cluster key are dropped (complete-case).\nA group with too few rows (`n ≤ k`), a singular/collinear design, or — for CR0/CR1\n— fewer than two clusters yields a **NULL** result for that group rather than\naborting the query. `t`/`p` use the t(n−k) distribution for classical/HC and\nt(G−1) for CR0/CR1 (matching Stata / statsmodels `use_t`). All numerics run on the\nshared header-only linear-algebra kernel (`(X'X)⁻¹`, the robust sandwich),\nvalidated against statsmodels — see `test/cpp/test_lm_fit.cpp`. The bias-reduced\nCR2/CR3 cluster estimators are a planned follow-up.\n\n### Multiple-testing correction (scalar)\n\n| Function                                | Description                                                                |\n| --------------------------------------- | -------------------------------------------------------------------------- |\n| `adjust_p(pvals, method)`               | Apply a multiple-testing correction to a list of p-values                  |\n\n`adjust_p` takes a `LIST\u003cDOUBLE\u003e` of raw p-values and a method name, and\nreturns adjusted p-values in input order. Methods (case-sensitive, matching\nR's `p.adjust`):\n\n- `'bonferroni'` — `min(1, n · p_i)`.\n- `'holm'` — Holm step-down (1979).\n- `'hochberg'` — Hochberg step-up (1988).\n- `'BH'` (alias `'fdr'`) — Benjamini-Hochberg FDR (1995).\n- `'BY'` — Benjamini-Yekutieli FDR (2001) for arbitrary dependence.\n- `'none'` — pass-through, returns the input unchanged.\n\nNULLs in the input list are passed through to the output at the same\nposition and are excluded from `n`.\n\n### Data import (table function)\n\n| Function                                | Description                   |\n| --------------------------------------- | ----------------------------- |\n| `read_stat(path, [format], [encoding])` | Read SAS / SPSS / Stata files |\n\n### Data export (COPY function)\n\n| Statement                         | Description                       |\n| --------------------------------- | --------------------------------- |\n| `COPY \u003ctable\u003e TO 'file.xpt'`      | Write SAS Transport (XPT v5)      |\n| `COPY \u003ctable\u003e TO 'file.sas7bdat'` | Write SAS7BDAT (see caveat below) |\n| `COPY \u003ctable\u003e TO 'file.sav'`      | Write SPSS SAV                    |\n\n\u003e **SAS7BDAT caveat.** ReadStat's SAS7BDAT writer is reverse-engineered: files\n\u003e round-trip through ReadStat-family readers (this extension's `read_stat()`,\n\u003e pyreadstat, haven, R) but are **not opened by real SAS / SAS Universal\n\u003e Viewer / SAS OnDemand**. Use XPT for SAS-native readability.\n\n### Visualizations (VISUALIZE parser extension)\n\nA Grammar-of-Graphics SQL dialect: `VISUALIZE` returns a single row with two\ncolumns — `spec` (a complete Vega-Lite v5 JSON spec) and `layer_sqls` (a\n`MAP(VARCHAR, VARCHAR)` of named SQL strings, one per layer). The client\nruns each layer's SQL and feeds the rows to vega-embed via the `datasets` API.\n\n**Tutorial:** [`docs/visualize.md`](docs/visualize.md) walks through a worked\nexample of every mark and clause.\n\n**See also:** [posit-dev/ggsql-duckdb](https://github.com/posit-dev/ggsql-duckdb)\n— the dedicated grammar-of-graphics DuckDB extension from the ggplot2 team that\ninspired this syntax. `stats_duck`'s\n`VISUALIZE` is **not** a reimplementation of ggsql and doesn't track its syntax — it's\na deliberately minimal, WebAssembly-friendly built-in supporting only a fixed set of\nmarks and clauses, for plotting stats output inline. For the full grammar of graphics,\nuse ggsql.\n\n```\n[WITH [RECURSIVE] \u003ccte\u003e AS (...) [, \u003ccte\u003e AS (...)]*]\nVISUALIZE \u003cexpr\u003e AS \u003caesthetic\u003e [: \u003ctype\u003e] (, \u003cexpr\u003e AS \u003caesthetic\u003e ...)\nFROM \u003ctable\u003e\nDRAW \u003cmark\u003e [STAT \u003cidentity|smooth|summary\u003e] (DRAW \u003cmark\u003e [STAT ...])*\n[FACET BY \u003cexpr\u003e [ROWS | COLS] | FACET BY \u003crow_expr\u003e, \u003ccol_expr\u003e]\n[SCALE \u003cchannel\u003e {TO \u003cscheme\u003e | ZERO true|false | DOMAIN \u003clo\u003e \u003chi\u003e | LABEL '\u003ctext\u003e'}+]*\n[TITLE '\u003ctext\u003e' [SUBTITLE '\u003ctext\u003e']]\n```\n\nMultiple `SCALE` options may be **stacked** on one channel\n(`SCALE x LABEL 'Bill Depth' ZERO false`) or split across repeated `SCALE x`\nclauses — they merge into one scale/axis block either way. SQL comments\n(`-- …` to end of line, and `/* … */`) may appear anywhere in a `VISUALIZE`\nstatement; they're skipped like whitespace (but never treated as comments\ninside a string literal).\n\nA leading `WITH` clause is supported; CTEs are scoped to each layer's\nprojected SQL so they compose with wrapping marks (`line`, `bar`, `area`,\n`errorband`, `regression`) without extra work. `WITH … SELECT …` statements\nwithout a top-level `VISUALIZE` keyword fall through to DuckDB's normal SQL\nparser unchanged.\n\n**Marks:** `point`, `line`, `bar`, `histogram`, `text`, `area`, `rule`, `tick`,\n`errorbar`, `errorband`, `boxplot`, `violin`, `heatmap`, `density`, `regression`. Custom\nmarks register as `visualize_mark_v1_\u003cname\u003e` scalar functions and are discovered\nvia DuckDB's catalog, so other extensions can ship their own marks without\nmodifying stats_duck.\n\n`heatmap` is a `rect` mark with ordinal x/y and quantitative color (correlation\nmatrices, contingency tables). `density` runs Vega-Lite's KDE on the `x`\naesthetic, grouped by `color` if mapped (one curve per category). `violin`\nrenders one horizontal density per category of `x`, laid out via vega-lite's\n`column` facet (composes with `FACET BY ... ROWS` but conflicts with\n`FACET BY ... COLS`). `regression` fits a linear `y ~ x` model server-side\nvia Vega-Lite's regression transform, also grouped by `color`. Use\n`DRAW point DRAW regression` for a scatter-with-fit overlay.\n\n**Aesthetic channels:** `x`, `y`, `color`, `fill`, `stroke`, `shape`, `size`,\n`opacity`, `tooltip`, `text`, `x2`, `y2`. Unknown channels are silently dropped.\n\n**Type overrides:** append `:quantitative`, `:ordinal`, `:nominal`, or\n`:temporal` to an aesthetic to force its Vega-Lite type\n(e.g. `year AS color:ordinal`).\n\n**Axis labels:** `SCALE x LABEL 'Bill length (mm)'` injects an `axis.title` into\nthe channel; pairs with `TO` / `ZERO` / `DOMAIN` on the same channel.\n\n**Titles:** `TITLE 'Plot title' [SUBTITLE 'Plot subtitle']` appears once per\nspec, after `SCALE` clauses. Always emitted as a Vega-Lite `TitleParams` object\nso a subtitle can be added without reshaping consumer code.\n\n## SAS compatibility\n\nstats_duck defaults follow modern statistical conventions (scipy /\nR with `correct=FALSE` / `var.equal=FALSE`). The one exception is\n`summary_stats` — its skewness/kurtosis formulas are the Fisher-Pearson\nbias-corrected ones, which is what SAS PROC MEANS, pandas, and Excel\nreport. To reproduce SAS PROC output exactly, use the toggles below.\n\n| SAS procedure / statistic                      | stats_duck call                                                     |\n| ---------------------------------------------- | ------------------------------------------------------------------- |\n| PROC MEANS — mean, SD, skewness, kurtosis      | `summary_stats(x)` _(default already matches SAS)_                  |\n| PROC TTEST — Pooled (equal variances)          | `ttest_2samp(x, y, true)`                                           |\n| PROC TTEST — Satterthwaite (default)           | `ttest_2samp(x, y)` _(default Welch's matches Satterthwaite)_       |\n| PROC NPAR1WAY — Wilcoxon two-sample Z          | `mann_whitney_u(x, y, 'two-sided', true)` _(continuity correction)_ |\n| PROC FREQ — Continuity Adj. χ² (2x2 only)      | `chisq_independence(row, col, true)` _(Yates' correction)_          |\n| PROC FREQ — Chi-Square (no adjustment)         | `chisq_independence(row, col)` _(default)_                          |\n| PROC CORR — Pearson / Spearman / Kendall       | `pearson_test(x, y)` / `spearman_test(x, y)` / `kendall_test(x, y)` |\n| PROC GLM — one-way ANOVA F-test                | `anova_oneway(value, group)`                                        |\n| PROC UNIVARIATE — Signed Rank (sign-rank test) | `wilcoxon_signed_rank(x, 0)` _(against a 0 column or constant)_     |\n| PROC UNIVARIATE — Sign test (M statistic)      | `sign_test_1samp(x, [mu_0])`                                        |\n| PROC UNIVARIATE — quantiles (Type 5)           | `summary_stats(x, true, 5)`                                         |\n\nDefaults preserve modern conventions so users on the modern side of the\nfence get sensible numbers without touching the API; SAS users add the\nappropriate flag during migration and validation.\n\n## Examples\n\n### Hypothesis tests\n\n#### One-sample t-test\n\n```sql\n-- Test whether the mean of v3 differs from zero\nSELECT ttest_1samp(v3) FROM measurements;\n\n-- Test against a specific mean\nSELECT ttest_1samp(v3, 5.0) FROM measurements;\n\n-- One-sided test with 99% confidence interval\nSELECT ttest_1samp(v3, 0.0, 0.01, 'greater') FROM measurements;\n```\n\n#### Two-sample t-test\n\n```sql\n-- Welch's t-test (default, does not assume equal variances)\nSELECT ttest_2samp(group_a, group_b) FROM experiment;\n\n-- Student's t-test (assumes equal variances)\nSELECT ttest_2samp(group_a, group_b, true) FROM experiment;\n```\n\n#### Paired t-test\n\n```sql\n-- Compare before/after measurements\nSELECT ttest_paired(before, after) FROM patients;\n```\n\n#### Mann-Whitney U test\n\nNon-parametric alternative to the two-sample t-test:\n\n```sql\n-- Compare two independent samples\nSELECT mann_whitney_u(group_a, group_b) FROM experiment;\n\n-- One-sided test\nSELECT mann_whitney_u(group_a, group_b, 'less') FROM experiment;\n```\n\n#### Wilcoxon signed-rank test\n\nNon-parametric alternative to the paired t-test:\n\n```sql\n-- Compare paired measurements\nSELECT wilcoxon_signed_rank(before, after) FROM patients;\n```\n\n### Working with results\n\nThe result of any hypothesis test is a `STRUCT` — access individual fields with\ndot notation:\n\n```sql\nSELECT (ttest_1samp(v3)).t_statistic,\n       (ttest_1samp(v3)).p_value\nFROM measurements;\n```\n\nOr unpack all fields:\n\n```sql\nSELECT (r).*\nFROM (SELECT ttest_1samp(v3) AS r FROM measurements);\n```\n\n#### Group-by\n\nRun a test per group with no extra plumbing:\n\n```sql\nSELECT id3,\n       (ttest_1samp(v3)).t_statistic,\n       (ttest_1samp(v3)).p_value\nFROM measurements\nGROUP BY id3;\n```\n\n#### Inline data\n\nUse `VALUES` for quick experiments with literal data:\n\n```sql\nSELECT (r).*\nFROM (SELECT ttest_1samp(v) AS r FROM (VALUES (2.0), (4.0), (6.0), (8.0), (10.0)) AS t(v));\n```\n\n### Data import\n\n#### Reading statistical file formats\n\nRead SAS (`.sas7bdat`, `.xpt`), SPSS (`.sav`, `.zsav`, `.por`), and Stata\n(`.dta`) files:\n\n```sql\n-- Auto-detect format from file extension\nSELECT * FROM read_stat('data.sas7bdat');\n\n-- Explicit format\nSELECT * FROM read_stat('data.dat', format := 'dta');\n\n-- Replacement scan: query files directly\nSELECT * FROM 'survey.sav';\n```\n\nDate, datetime, and time columns are automatically detected from format\nmetadata and converted to DuckDB's native temporal types.\n\n`read_stat` is wired through DuckDB's virtual file system, so the same call\nworks against:\n\n- local paths: `read_stat('/data/survey.sav')`\n- remote files (with the `httpfs` extension loaded): `read_stat('https://…/survey.sav')`\n- object stores: `read_stat('s3://bucket/survey.sav')`\n- in-browser DuckDB-WASM file buffers registered via `registerFileBuffer()`\n\n#### Writing statistical file formats\n\nThe same VFS-backed writers, exposed as DuckDB `COPY` functions:\n\n```sql\n-- SAS Transport (XPT v5) — universally readable, FDA / pharma standard\nCOPY survey TO 'survey.xpt';\n\n-- SAS7BDAT — round-trips through stats_duck and pyreadstat (NOT real SAS, see caveat)\nCOPY survey TO 'survey.sas7bdat' (COMPRESSION 'rows');\n\n-- SPSS SAV\nCOPY survey TO 'survey.sav';\n\n-- COPY \u003csubquery\u003e form works too\nCOPY (SELECT * FROM huge_table WHERE region = 'EU') TO 'eu.xpt';\n```\n\nNULL handling differs by storage format: numeric NULLs round-trip as\nSAS/SPSS system-missing, but VARCHAR NULLs collapse to empty strings (these\nformats have no NULL/empty distinction for character columns).\n\n### Visualizations (VISUALIZE)\n\n```sql\n-- Set up — penguin morphology, classic ggplot2 demo data\nCREATE TABLE penguins AS SELECT * FROM (VALUES\n    (39.1, 18.7, 'Adelie',    2018), (39.5, 17.4, 'Adelie',    2019),\n    (44.0, 18.0, 'Gentoo',    2020), (45.2, 14.5, 'Chinstrap', 2021)\n) AS t(bill_len, bill_dep, species, year);\n```\n\n#### Scatter\n\n```sql\nVISUALIZE bill_len AS x, bill_dep AS y FROM penguins DRAW point;\n```\n\n#### Scatter colored by species, with a viridis scale\n\n```sql\nVISUALIZE bill_len AS x, bill_dep AS y, species AS color\nFROM penguins\nDRAW point\nSCALE color TO viridis;\n```\n\n#### Multi-layer (scatter + line overlay)\n\n```sql\nVISUALIZE bill_len AS x, bill_dep AS y FROM penguins DRAW point DRAW line;\n```\n\n#### Faceted plot — one panel per species, vertically stacked\n\n```sql\nVISUALIZE bill_len AS x, bill_dep AS y FROM penguins\nDRAW point FACET BY species ROWS;\n```\n\n#### 2D facet grid — species rows × sex columns\n\n```sql\nVISUALIZE bill_len AS x, bill_dep AS y FROM penguins\nDRAW point FACET BY species, sex;\n```\n\n#### Scatter with LOESS overlay via STAT smooth\n\n```sql\nVISUALIZE bill_len AS x, bill_dep AS y FROM penguins\nDRAW point\nDRAW line STAT smooth;\n```\n\n#### SQL expressions in mappings\n\n```sql\nVISUALIZE bill_len * 2 AS x, log(bill_dep) AS y FROM penguins DRAW point;\n```\n\n#### Type-annotated aesthetic (year is INTEGER but should be ordinal here)\n\n```sql\nVISUALIZE bill_len AS x, bill_dep AS y, year AS color:ordinal\nFROM penguins\nDRAW point\nSCALE color TO viridis;\n```\n\nThe output of any `VISUALIZE` query is a single row with `(spec, layer_sqls)`.\nThe client side (a browser app, a notebook renderer, …) is responsible for\nrunning each layer's SQL and feeding the rows into vega-embed via its\n`datasets` API. See [Bedevere Wise](https://github.com/caerbannogwhite/bedevere-wise)\nfor a reference DuckDB-WASM consumer.\n\n## Building\n\n```bash\ngit submodule update --init --recursive\nmake release\n```\n\n### Building with MinGW\n\nDuckDB extensions are tagged at build time with a _platform string_ that the\nhost process uses to validate ABI compatibility before loading. The default\n`make release` on Windows uses MSVC and stamps `windows_amd64`. A consumer\nDuckDB built with mingw-w64 (e.g. via zig's bundled clang + mingw-w64\nsysroot, or the rtools/MSYS2 toolchains) will refuse to load that binary\nwith:\n\n```\nFailed to load 'stats_duck.duckdb_extension'. The file was built for the\nplatform 'windows_amd64', but we can only load extensions built for\nplatform 'windows_amd64_mingw'.\n```\n\nFor those hosts, build with:\n\n```bash\nmake mingw_release\n```\n\nThis drives a CMake build with `-G \"MinGW Makefiles\"`, sets `CC=gcc`,\n`CXX=g++`, and stamps `DUCKDB_EXPLICIT_PLATFORM=windows_amd64_mingw`. The\nbinary lands at:\n\n```\nbuild/mingw_release/extension/stats_duck/stats_duck.duckdb_extension\nbuild/mingw_release/repository/\u003cduckdb_version\u003e/windows_amd64_mingw/stats_duck.duckdb_extension\n```\n\nThe MSVC `windows_amd64` binary at `build/release/...` is left alone — both\nflavors can coexist on disk and ship side by side. CI already produces both\non every release; this target is for local-only verification or for\ndistributing to consumers that bundle their own DuckDB and need an exact\nABI match.\n\n**Toolchain requirements.** Anything ABI-compatible with `x86_64-w64-mingw32`\nworks. Tested with TDM-GCC 10.3 and MSYS2's `mingw-w64-x86_64-toolchain`. On\nTDM-GCC the Makefile passes `-D_WIN32_WINNT=0x0A00` because TDM-GCC's default\npredates the Vista-era APIs DuckDB uses; MSYS2 already defaults to that\nvalue, where the flag is harmless.\n\n**vcpkg is intentionally not used** for the MinGW build — its Windows\ndefault triplet is MSVC and would not match the toolchain. zlib (the only\nexternal dep, used for SPSS `.zsav` read/write) becomes optional and\ngracefully degrades if absent (see `CMakeLists.txt:97`); install zlib via\nyour mingw package manager if you need `.zsav` support.\n\n### Building with zig as the C++ toolchain (matches sassy)\n\nDuckDB's platform string `windows_amd64_mingw` does not distinguish which C++\nruntime the binary is linked against. Two `windows_amd64_mingw` extensions can\nboth pass the platform check on load and then segfault during function\nregistration if their `std::map` / `std::shared_ptr` / etc. layouts differ —\nwhich they do across **libstdc++** (GNU's STL, what mingw-w64 GCC uses) and\n**libc++** (LLVM's STL, what zig's bundled clang uses with `link_libcpp`).\n\nThe downstream [sassy](https://github.com/caerbannogwhite/sassy) SAS\ninterpreter builds DuckDB with zig + `link_libcpp = true`, so its DuckDB\nlinks against libc++. The `make mingw_release` target above produces a\nlibstdc++-linked binary and is _not_ compatible — it'll segfault inside\nsassy. Use the zig variant instead:\n\n```bash\nmake zig_mingw_release\n```\n\nThis drives the same CMake configuration as `mingw_release` but routes\n`CC` / `CXX` through `scripts/zig-shims/zig-cc.cmd` / `zig-cxx.cmd`, which\nforward to `zig cc -target x86_64-windows-gnu` and\n`zig c++ -target x86_64-windows-gnu`. Output:\n\n```\nbuild/zig_mingw_release/extension/stats_duck/stats_duck.duckdb_extension\nbuild/zig_mingw_release/repository/\u003cduckdb_version\u003e/windows_amd64_mingw/stats_duck.duckdb_extension\n```\n\nSame platform stamp (`windows_amd64_mingw`), different C++ runtime. On a\ncorrectly-built artifact, `objdump -p stats_duck.duckdb_extension | grep DLL`\nshows only Windows system DLLs (`KERNEL32.dll`, `api-ms-win-crt-*`) — no\n`libstdc++-6.dll` or `libgcc_s_seh-1.dll`. If any GNU runtime DLL appears,\nthe build slipped back to GCC.\n\n**Requirements.** zig 0.16+ on `PATH`. Tested with the `zig.zig`\nWinGet-installed copy.\n\n## Testing\n\n```bash\nmake release \u0026\u0026 make test\n```\n\nOr on Windows where the Makefile test runner may not work:\n\n```bash\n./build/release/test/Release/unittest.exe \"test/sql/*\"\n```\n\n## License\n\nThe Stats Duck is released under the [Apache License 2.0](LICENSE). If you use\nit in academic work, see [`CITATION.cff`](CITATION.cff) for citation metadata.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FKoliStat%2Fthe-stats-duck","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FKoliStat%2Fthe-stats-duck","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FKoliStat%2Fthe-stats-duck/lists"}