{"id":17640634,"url":"https://github.com/microprediction/microprediction","last_synced_at":"2025-05-15T01:09:28.518Z","repository":{"id":42449601,"uuid":"241911060","full_name":"microprediction/microprediction","owner":"microprediction","description":"If you can measure it, consider it predicted","archived":false,"fork":false,"pushed_at":"2025-03-14T18:18:53.000Z","size":124443,"stargazers_count":348,"open_issues_count":23,"forks_count":61,"subscribers_count":15,"default_branch":"master","last_synced_at":"2025-05-12T03:13:29.500Z","etag":null,"topics":["fbprophet","filterpy","hmmlearn","kalman-filter","keras","nowcasting","online-algorithms","pmdarima","time-series","timeseries","timeseries-analysis","timeseries-clustering","timeseries-data","timeseries-database","timeseries-forecasting","timeseries-prediction","tsfresh","tslearn"],"latest_commit_sha":null,"homepage":"http://www.microprediction.org","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/microprediction.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2020-02-20T14:54:03.000Z","updated_at":"2025-05-11T19:49:23.000Z","dependencies_parsed_at":"2023-02-15T00:30:46.428Z","dependency_job_id":"65781cc7-02b5-4cac-a98b-19332db97dbd","html_url":"https://github.com/microprediction/microprediction","commit_stats":{"total_commits":2180,"total_committers":15,"mean_commits":"145.33333333333334","dds":0.1417431192660551,"last_synced_commit":"e4678304773e3760ad54eb6396a69af6c512dad6"},"previous_names":[],"tags_count":138,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/microprediction%2Fmicroprediction","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/microprediction%2Fmicroprediction/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/microprediction%2Fmicroprediction/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/microprediction%2Fmicroprediction/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/microprediction","download_url":"https://codeload.github.com/microprediction/microprediction/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253666151,"owners_count":21944624,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["fbprophet","filterpy","hmmlearn","kalman-filter","keras","nowcasting","online-algorithms","pmdarima","time-series","timeseries","timeseries-analysis","timeseries-clustering","timeseries-data","timeseries-database","timeseries-forecasting","timeseries-prediction","tsfresh","tslearn"],"created_at":"2024-10-23T06:04:26.338Z","updated_at":"2025-05-15T01:09:23.497Z","avatar_url":"https://github.com/microprediction.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"\n# microprediction (Peter's Repos)\nIf you were redirected from: *Is Facebook's Prophet the Time-Series Messiah or Just a Very Naughty Boy?*, here's the [article](https://medium.com/geekculture/is-facebooks-prophet-the-time-series-messiah-or-just-a-very-naughty-boy-8b71b136bc8c).\n\nOtherwise hi\n\n - My home page page contains [papers, working papers, articles](https://github.com/microprediction/home).\n - My [thanks for reaching out](https://github.com/microprediction/monteprediction/blob/main/TFRO.md) page contains contact suggestions. \n - Medium [blog](https://microprediction.medium.com/)\n - [Linked-In](https://www.linkedin.com/in/petercotton/) content. \n\nI'm a career quant, applied mathematician, open-source developer, entrepreneur and dad.  \n\n### Interests\n\n - Portfolio and ensemble construction (e.g. [paper](https://github.com/microprediction/home/blob/main/workingpapers/Hierarchical_Minimum_Variance_Portfolios.pdf) and [blog](https://medium.com/geekculture/schur-complementary-portfolios-fix-hierarchical-risk-parity-28b0efa1f35f) where I unified the two sides of portfolio theory).  \n - OTC microstructure (my day job so not public .. but for older work see [this](https://github.com/microprediction/home/blob/main/presentations/trading_illiquid.pdf) or [that](https://github.com/microprediction/home/blob/main/presentations/who_ya_gonna_call.pdf) or [the other](https://github.com/microprediction/home/blob/main/presentations/Benchmark___as_presented_at_NYU_Tandon_2016%20(1).pdf)).\n - Thurstone models (contests, LLM samplers etc)\n - Derivative-free optimization\n - Time-series\n - Sports analytics\n - Collective Intelligence, Decentralized AI\n - LLM uses for alpha, code gen, prediction and instrumentation (e.g. [pi](https://pi.crunchdao.com/)).\n\n### Microprediction\n\nI wrote a book on what we might call the Strong [Indispensable Markets Hypothesis](https://github.com/microprediction/home/blob/main/workingpapers/The_Indispensible_Markets_Hypothesis.pdf), and to help it along in a small way I run contests like [mid-one](https://mid-one.crunchdao.com/) which you might want to check out too as there is a lot of prize-money. I like to provoke people into using market-inspired collective mechanisms for prediction. I used the options market to effortlessly beat 97% of participants in the year-long M6 contest - see the [post](https://www.linkedin.com/posts/petercotton_the-options-market-beat-94-of-participants-activity-7020917422085795840-Pox0?utm_source=share\u0026utm_medium=member_desktop) or [article](https://medium.com/geekculture/the-options-market-beat-94-of-participants-in-the-m6-financial-forecasting-contest-fa4f47f57d33). My [book](https://mitpress.mit.edu/books/microprediction) is a meditation on the power of mini-markets and algorithmic micro-managers in a very specific yet ubiquitous domain: frequently repeated prediction. It predates and extends phrases like \"DeAI\" and \"Info Finance\" (Buterin). Read the [awards and reviews](https://microprediction.github.io/building_an_open_ai_network/feedback.html). \n\n![](https://github.com/microprediction/microprediction/blob/master/docs/assets/images/cotton_microprediction_3d_down.png)\n\n### Monteprediction \nAllow me to mention a long-running game where you hurl a million 11-dimensional Monte Carlo samples at my server. \n\n1. Open this [colab notebook](https://github.com/microprediction/monteprediction_colab_examples/blob/main/monteprediction_entry.ipynb) or [script](https://github.com/microprediction/monteprediction_colab_examples/blob/main/monteprediction_entry.py)  (yes there's an [R version](https://github.com/microprediction/monteprediction_colab_examples/blob/main/monteprediction_entry_rlang.ipynb)), \n2. Change the email, at minimum,\n3. Run it. Every weekend.\n4. Check your scores at [www.monteprediction.com](https://www.monteprediction.com)\n\nThe notebook also describes the scoring mechanism. Ask questions in the slack (see bottom of [leaderboard](https://www.monteprediction.com) for slack invite).\n\nNow, here's some attempt to introduce you to my open source work ...\n\n### Derivative-free optimizer comparisons\n\nThe [humpDay](https://github.com/microprediction/humpday) package is intended to help you choose a derivative-free optimizer for your use case. \n\n![](https://i.imgur.com/FCiSrMQ.png)\n\n### Schur Complementary Portfolios\n\nMy work on unifying Hierarchical Risk Parity with minimum variance portfolio optimization sits in the [precise](https://github.com/microprediction/precise) package. See [slides](https://github.com/microprediction/home/blob/main/presentations/Schur_Complementary_Portfolios_2025__slides_.pdf) from a recent talk. \n\n\u003ca href=\"https://medium.com/geekculture/schur-complementary-portfolios-fix-hierarchical-risk-parity-28b0efa1f35f\"\u003e\n\u003cimg src=\"https://github.com/microprediction/precise/blob/main/docs/assets/images/schur_reaction.png\" width=\"600\"\u003e\u003c/a\u003e\n\n### Incremental time-series and benchmarking\n\nThe [timemachines](https://github.com/microprediction/timemachines) package enumerates online methods and makes some effort to evaluate univariate methods against the corpus of time-series drawn from the microprediction platform. It is an attempt to reduce everything to relatively pure functions:\n\n$$\n    f : (y_t, state; k) \\mapsto ( [\\hat{y}(t+1),\\hat{y}(t+2),\\dots,\\hat{y}(t+k) ], [\\sigma(t+1),\\dots,\\sigma(t+k)], posterior\\ state))\n$$\n\nwhere $\\sigma(t+l)$ estimates the standard error of the prediction $\\hat{y}(t+l)$. \n\n\n![](https://i.imgur.com/elu5muO.png)\n\n\n### Benchmarking overview\n\n| Topic                  | Package           | Elo ratings | Methods                                                                                                                                                                                  | Data sources | \n|------------------------|-------------------|-------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------| \n| Univariate time-series | [timemachines](https://github.com/microprediction/timemachines)  | [Timeseries Elo ratings](https://microprediction.github.io/timeseries-elo-ratings/html_leaderboards/univariate-k_003.html) | Most popular packages ([list](https://github.com/microprediction/timemachines/tree/main/timemachines/skaters))                                                                           | [microprediction streams](https://www.microprediction.org/browse_streams.html)                                      |\n| Global derivative-free optimization | [humpday](https://github.com/microprediction/humpday) |  [Optimizer Elo ratings](https://microprediction.github.io/optimizer-elo-ratings/html_leaderboards/overall.html) | Most popular packages ([list](https://github.com/microprediction/humpday/tree/main/humpday/optimizers))                                                                                  | A mix of classic and new [objectives](https://github.com/microprediction/humpday/tree/main/humpday/objectives)      |\n| Covariance, precision, correlation | [precise](https://github.com/microprediction/precise) | See [notebooks](https://github.com/microprediction/precise/tree/main/examples_colab_notebooks) | [cov](https://github.com/microprediction/precise/blob/main/LISTING_OF_COV_SKATERS.md) and [portfolio](https://github.com/microprediction/precise/blob/main/LISTING_OF_MANAGERS.md) lists |Stocks, electricity etc                                                                                              | \n\nThese packages aspire to advance online autonomous prediction in a small way, but also help me notice if anyone else does.  \n\n## Winning\n\nThe [winning](https://github.com/microprediction/winning) package includes my recently published fast algorithm for inferring relative ability from win probabilities, at any scale. As explained in the [paper](https://github.com/microprediction/winning/blob/main/docs/Horse_Race_Problem__SIAM_updated.pdf) the uses extend well beyond the pricing of quinellas! \n\n![](https://i.imgur.com/83iFzel.png) \n\n## First Down\n\nMy [firstdown](https://github.com/microprediction/firstdown) repo contains analysis aspiring to ruin great game of football. See Wilmott [paper](https://github.com/microprediction/firstdown/blob/main/wilmott_paper/44-49_Cotton_PDF5_Jan22%20(2).pdf) and for heaven's sake, don't stretch out for the first down. That's obviously nuts.  \n\n  ![](https://github.com/microprediction/firstdown/blob/main/images/firstdownpaper.png)\n\n## Some other repos\n\n- [randomcov](https://github.com/microprediction/randomcov) - A set of quirky correlation and covariance matrix generators (I'd love your ideas). \n- [embarrassingly](https://github.com/microprediction/embarrassingly) - A speculative approach to robust optimization that sends impure objective functions to optimizers.\n- [pandemic](https://github.com/microprediction/pandemic) - Ornstein-Uhlenbeck epidemic simulation (related [paper](https://arxiv.org/abs/2005.10311))\n- [momentum](https://github.com/microprediction/momentum) - My most personally re-used mini package ... for incremental mean, var, skew, kurtosis.\n- [muid](https://github.com/microprediction/muid) - Memorable Unique Identifiers ... try to figure out how that can't be an oxymoron.\n- [timeseries-notebooks](https://github.com/microprediction/timeseries-notebooks) - Lots of examples of using open source timeseries packages.\n- [correlationbounds](https://github.com/microprediction/correlationbounds) - Mini package for conf bounds\n- [building_an_open_ai_network](https://github.com/microprediction/building_an_open_ai_network) - Book related.\n- [recalibrate](https://github.com/microprediction/recalibrate) - Utils related to Platt scaling etc. \n  \n## The currently defunct real-time time-series platform\n\nThe real-time system previously maintained by yours truly has entered a trisoloran dehydrated state but will hopefully be revived at a future date, after one of my three hundred ChatGPT generated scientific grant proposals is successful. Here's how some of the open-source stuff used to propagate down into the \"algo fight club\". \n\n1. The \"[/skaters](https://github.com/microprediction/timemachines/tree/main/timemachines/skaters)\" provide canonical, single-line of code access to functionality drawn from packages like [river](https://github.com/online-ml/river), [pydlm](https://github.com/wwrechard/pydlm), [tbats](https://github.com/intive-DataScience/tbats), [pmdarima](http://alkaline-ml.com/pmdarima/), [statsmodels.tsa](https://www.statsmodels.org/stable/tsa.html), [neuralprophet](https://neuralprophet.com/), Facebook [Prophet](https://facebook.github.io/prophet/), \n   Uber's [orbit](https://eng.uber.com/orbit/), Facebook's [greykite](https://engineering.linkedin.com/blog/2021/greykite--a-flexible--intuitive--and-fast-forecasting-library) and more. \n   \n2. The [StreamSkater](https://microprediction.github.io/microprediction/predict-using-python-streamskater.html) makes it easy to use any \"skater\". \n\n3. Choices are sometimes advised by [Elo ratings](https://microprediction.github.io/timeseries-elo-ratings/html_leaderboards/special-k_003.html), but anyone can do what they want. \n\n4. It's not too hard to use my [HumpDay](https://github.com/microprediction/humpday) package for offline meta-param tweaking, et cetera. \n\n5. It's not too hard to use my [precise](https://github.com/microprediction/precise) package for online ensembling. \n\nA few repos that drove this:\n\n- The [muid](https://github.com/microprediction/muid) identifier package is explained in this [video](https://vimeo.com/397352413). \n- [microconventions](https://github.com/microprediction/microconventions) captures things common to client and server, and may answer many of your more specific questions about prediction horizons, et cetera.  \n- [rediz](https://github.com/microprediction/rediz) contains server side code. For the brave. \n- There are other rats and mice like [getjson](https://github.com/microprediction/getjson), [runthis](https://github.com/microprediction/runthis) and [momentum](https://github.com/microprediction/momentum).  \n\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmicroprediction%2Fmicroprediction","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmicroprediction%2Fmicroprediction","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmicroprediction%2Fmicroprediction/lists"}