{"id":24915735,"url":"https://github.com/fd/zettalm","last_synced_at":"2025-03-28T06:29:27.642Z","repository":{"id":26857530,"uuid":"30317521","full_name":"fd/zettalm","owner":"fd","description":"Go code to build linear regression models on zettabytes of data","archived":false,"fork":false,"pushed_at":"2015-02-04T19:44:32.000Z","size":130,"stargazers_count":3,"open_issues_count":0,"forks_count":2,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-02-02T07:19:23.774Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":false,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/fd.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2015-02-04T19:44:07.000Z","updated_at":"2022-10-03T21:20:47.000Z","dependencies_parsed_at":"2022-07-24T13:30:04.620Z","dependency_job_id":null,"html_url":"https://github.com/fd/zettalm","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fd%2Fzettalm","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fd%2Fzettalm/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fd%2Fzettalm/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fd%2Fzettalm/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/fd","download_url":"https://codeload.github.com/fd/zettalm/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245983694,"owners_count":20704786,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-02-02T07:19:09.648Z","updated_at":"2025-03-28T06:29:27.623Z","avatar_url":"https://github.com/fd.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"Project Zettalm\n===============\n\nGo code to build linear regression models on zettabytes of data - using Alan Miller's AS 274 online QR decomposition\n\n* fits linear models to data\n\nThis golang code does basic linear regression, otherwise know as least-squares fitting.\n\n* but: handles really big data\n\nThis particular solution to fitting linear models can handle infinite data,\nand that's the point.\n\nThere are lots of model fitting packages that can handle finite data; they handle data that fits in RAM, or fits on disk, or in the cloud. The point of interest here\nis that this online QR decomposition algorithm can handle Zetta-Bytes of observations,\n*actually unlimited* rows of data. It needs space only proportional to\nO(p^2) for p variables. It only ever has to look at each row once. The original\nFortran90 did not handle more than one dependent variable, but the current\ncode does. Again, observations of x and y variables are not stored in memory.\nWeighted observations are supported, including negative weights to delete\ncases.\n\n* is still quite fast\n\nYou are certainly most likely to be I/O-bound. For CPU time, I benchmarked\nthe current code at 10x the performance of R's linear model solver.\n\nAsymptodically, for p variables (both x and y variables), each additional\nrow of observations incurs O(p^2) time at the time of addition (see the Includ() routine).\nWhen you want to get the regression coefficients for the current model, \nit costs one O(p^2) time operation, a call to Regcf(), to extract and\nreturn the betas. If you need standard errors (to compute p-values for the\nbetas), the Cov() routine will supply them, along with the parameter covariance matrix,\nand needs O(p^3) time (as does its subroutine, Inv()). SingularCheck() \nand SS() are both O(p) time routines.\n\nIn summary, for a typical fit on N rows for p variables, where N \u003e p,\nthe time complexity is O(N*p^2). If p \u003c N, then cost is O(p^3) time due to the \nCov() call. Overall, time complexity is O(N*p^2 + p^3), as is typical for\nlinear regression.  Note that the structure of the code means that you\ncan fit multiple dependent y-variables at once, and still only ever make one\npass through the data. Data need not fit in main memory, nor even ever be\nstored all in one place. For big problems, the savings on space can make\na huge difference in what is viable to handle.\n\n\n* enhancements over the fortran code\n\nI've added the ability to handle multiple y-variables at once, as well\nas online computation of means and standard deviations for all x and y variables.\nAlso I've fixed minor bugs that were present when using the downdating functionality\nto remove rows from the set. \n\nMost importantly, I've added an extensive acceptance test-suite to\nallow easy refactoring. Execute the tests, as usual in go, with \"go test -v\".\n\n* online, but can be a moving window too\n\nEven though it only ever looks at any row once, rows are weighted, and\nhence rows can be deleted, if you wish, from the data set considered\nby adding the same row again but using a weight of -1 (instead of the default +1 weight)\non the second call to Includ().\n\n\nOrigins:\n\nThe structure and algorithm at the heart of this code is derived from\nAlan Miller's AS 274 Algorithm and his Fortran90 source code for that publication.\nAS 274 in turn builds upon W. Morven Gentleman's AS 75 algorithm. Refer to:\n\nhttp://www.jstor.org/stable/2347147\n\nhttp://lib.stat.cmu.edu/apstat/274\n\nhttp://lib.stat.cmu.edu/apstat/75\n\nAdditional information and the Fortran90 source is available at\nhttp://jblevins.org/mirror/amiller and http://jblevins.org/mirror/amiller/lsq.f90\nThe web page http://jblevins.org/mirror/amiller/ indicates that Alan Miller's \nversion of the code has been placed in the public domain. Extensive credit is due to\nDr. Miller for the implementation and Dr. Gentleman for the theory (using Fast Given's rotations and avoiding square root operations during computation of the Cholesky factorization) in this numerically exacting algorithm.\n\nNon-public enhancement: I've written an extension that allows scatter-gather fitting of models in parallel. This drastically reduces the fitting time for many models. Contact me directly if you are interested in licensing the parallelization code.\n\nLicense: MIT\n\nCopyright (c) 2014, Jason E. Aten, Ph.D. \u003cj.e.aten@gmail.com\u003e\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffd%2Fzettalm","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffd%2Fzettalm","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffd%2Fzettalm/lists"}