{"id":16508743,"url":"https://github.com/mobeets/psychometric-fits","last_synced_at":"2026-05-13T09:31:47.343Z","repository":{"id":17973219,"uuid":"20972899","full_name":"mobeets/psychometric-fits","owner":"mobeets","description":"Exploring various methods of fitting psychometric functions","archived":false,"fork":false,"pushed_at":"2014-06-24T23:02:58.000Z","size":604,"stargazers_count":2,"open_issues_count":0,"forks_count":1,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-01-12T17:11:49.705Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mobeets.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2014-06-18T18:18:27.000Z","updated_at":"2023-11-20T15:09:19.000Z","dependencies_parsed_at":"2022-08-26T10:40:58.401Z","dependency_job_id":null,"html_url":"https://github.com/mobeets/psychometric-fits","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mobeets%2Fpsychometric-fits","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mobeets%2Fpsychometric-fits/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mobeets%2Fpsychometric-fits/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mobeets%2Fpsychometric-fits/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mobeets","download_url":"https://codeload.github.com/mobeets/psychometric-fits/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":241460086,"owners_count":19966511,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-11T15:47:19.587Z","updated_at":"2026-05-13T09:31:47.304Z","avatar_url":"https://github.com/mobeets.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"A good overview of various Markov-chain Monte Carlo methods is available [here](http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.13.7133\u0026rep=rep1\u0026type=pdf).\n\n### Metropolis-Hastings sampling\n\n_Metropolis-Hastings_ sampling (M-H) aims to draw samples from a posterior using only a _posterior-ish function_ (e.g. the unnormalized posterior given some measured data) and a _proposal function_. M-H is provided with an initial draw from the posterior and aims to generate a series of samples by moving from that initial draw to the next, following a certain rule. (More details can be found [here](http://www.journalofvision.org/content/5/5/8.short).)\n\nThe posterior-ish function is used to determine whether or not the next potential sample is in a region of higher posterior probability compared with the previous sample.\n\nThe proposal function describes how to generate the next potential sample. (If the proposal function is symmetric this sampling procedure is sometimes called _Metropolis samping_. If it is independent of the location of the most recent sample it is called an _independent sampler_.) The result is a sort of random walk of samples through the posterior-ish function; often the proposal function is a Gaussian centered on the most recent sample with a given standard deviation. The choice of standard deviation greatly determines how well M-H samples the entire posterior, which you can see in the image below.\n\n![Example of proposal function of M-H](/img/proposal-fcn.png?raw=true \"Example of proposal function of M-H\")\n\n#### Example 1: Generating samples from weibull pdf\n\nThe function `example_1()` function in `examples.py` currently uses a gaussian proposal function to draw samples from a weibull pdf with given shape and scale parameters. As you can see, the samples generated by `metropolis_hastings()` (after pruning) are a close match to the actual weibull pdf:\n\n![Example of posterior samples from M-H](/img/example-1.png?raw=true \"Example of posterior samples from M-H\")\n\n#### Example 2: Fitting weibull scale parameter of data generated by a weibull pdf\n\nBut `example_1()` is more like a sanity-check, isn't it? If we hand M-H the pdf, it can generate samples from that pdf--not that impressive.\n\nOn the other hand, `example_2()` is a little closer to what we'd want M-H to do. It simulates data from a weibull pdf given shape and scale parameters, and M-H tries to generate samples of (i.e. fit) the shape parameter, by calculating the log-likelihood of the simulated data.\n\nSo now, the generated samples are all estimates of the shape parameter. The shape parameter used to simulate the data was 3, with a fairly small sample size of only 1000. The estimates cluster near but not quite at 3, which is fine, though repeated simulations would probably show more encouraging results (since here I'm simulating such a low sample size).\n\n![Example of posterior samples from M-H](/img/example-2.png?raw=true \"Example of posterior samples from M-H\")\n\n### Simulated annealing\n\nMetropolis-Hastings aims to approximate the entire posterior distribution. However, in the case of fitting, often all you want is the maximum a posteriori (MAP) estimate of the posterior: in other words, you don't need to approximate the entire posterior--you just want to know the mode!\n\nSimulated annealing (good overview [here](http://stuff.mit.edu/~dbertsim/papers/Optimization/Simulated%20annealing.pdf)) is a generalization of Metropolis-Hastings, with an added parameter function called the \"cooling schedule\" that is non-increasing with each iteration of your sampler. \n\nUsing M-H to approximate the mode of the posterior is inefficient since it tries to spread itself along the entire posterior. Simulated annealing, however, uses its cooling schedule function to slowly hone in on the mode of the posterior.\n\nOne common choice of a cooling schedule function `T(i)` is, for a given `d`:\n\n    T = lambda i: d/np.log(i+2)\n\n(The only rules for `T` is that it must be non-increasing, and as i -\u003e ∞, T(i) -\u003e 0.)\n\nNow, at each iteration, the `p_pdf_fcn` of `metropolis_hastings()` is instead calculated as `p_pdf_fcn(x)^(1/T_i)`, where i is the current iteration of the sampler.\n\nThe very last sample generated is your MAP estimate of the posterior.\n\n#### Example 3: Simulated annealing MAP estimate\n\nJust as the spread of the proposal function is crucial to fully exploring the posterior in M-H, here the choice and parameters of your cooling function are crucial to appropriately estimating the mode of the posterior.\n\nJust as in Example 2, the sample size of my simulated data set is only 1000, so though the theta that generated the data was 3, it's not too surprising that our MAP estimate for theta is not exactly 3. Also, I somewhat hastily set the parameter `d = 1` for my cooling function. Larger sample size and better parameter fitting would definitely improve your simulated annealing experience.\n\n![Example of MAP from simulated annealing](/img/example-3.png?raw=true \"Example of MAP from simulated annealing\")\n\nSo what are the relative tradeoffs apparent so far between M-H and simulated annealing? Well, M-H aims to simulate the entire posterior, which is inefficient if you're just looking for an estimate of the mode. Simulated annealing, on the other hand, should converge on this mode in fewer iterations, but you now have an additional parameter to adjust. (And adjusting this parameter involves, ironically enough, assessing the shape of your posterior.)\n\n### Scipy's minimization methods\n\n`scipy.optimize` actually has a pretty broad selection of [minimization methods](http://docs.scipy.org/doc/scipy/reference/tutorial/optimize.html), including simulated annealing. So if you're not looking to view the posterior but are instead looking for its mode, this is probably your best bet (in terms of development time, testing, _and_ solver speed).\n\nI simulated data in the same way as I did in Examples 2 and 3 and called `scipy.optimize.minimize` with all of its relevant method arguments: `[\"nelder-mead\", \"powell\", \"Anneal\", \"BFGS\", \"TNC\", \"L-BFGS-B\", \"SLSQP\"]`. Every single one of these solvers found solutions in no time at all, and all of their solutions were basically identical. Hooray!\n\nOnly problem is that I'm still fitting only one parameter, and I'm not including a prior. When you fit a real-life psychometric function, you need up to four parameters _and_ a prior. Do all of these methods scale with more parameters and messier function evaluations?\n\n#### Example 4: Fitting four parameters of data generated by a weibull pdf\n\nA general form of the psychometric curve is as follows:\n\n`Ψ(x; α, β, γ, λ) = γ + (λ - γ)F(x; α, β)`.\n\n`α` is the scale parameter, `β` the shape parameter, `λ` the upper-bound of performance, and `γ` the lower-bound. `F` is typically a sigmoid function--in this case I'm using the cdf of a weibull distribution.\n\nI wanted to compare all the fitting methods of `scipy.optimize` that allow bounds on the parameters. These are `[\"TNC\", \"L-BFGS-B\", \"SLSQP\"]`, but `TNC` ended up being too slow so I dropped it from consideration.\n\nI simulated a series of datasets from a model psychometric curve with fixed parameters. Each dataset contained trials collected at `x_i = i/20` for `i = 1..20`. For each `x`, I simulated 50 trials. Each trial was either 1 or 0, for success/failure, where the probability of success on each trial was `Ψ(x; α, β, γ, λ)`. In other words, each trial was a draw from a bernoulli distribution with probability `Ψ(x; α, β, γ, λ)`.\n\nFor each data set, I estimated the model parameters using the `L-BFGS-B` and `SLSQP` arguments to `scipy.optimize`, representing the [limited-memory BFGS](https://en.wikipedia.org/wiki/Limited-memory_BFGS) and [sequential least squares programming](http://www.pyopt.org/reference/optimizers.slsqp.html) algorithms, respectively. (Both of these are [quasi-Newton](https://en.wikipedia.org/wiki/Quasi-Newton_method) methods, meaning they use approximations of the Hessian to find the minima.)\n\nThe solutions for `L-BFGS-B` and `SLSQP` are essentially identical, so I'll plot the results of only one of them.\n\nSimulation 1: Results from fitting 100 datasets drawn from model with parameters: α = 0.1, β=0.5, γ=0.03,$ and  λ=0.98.\n\n![four-parameter psychometric fitting, part 1](/img/pmf_4-1.png?raw=true \"four-parameter psychometric fitting, part 1\")\n\nSimulation 2: Results from fitting 1000 datasets drawn from model with parameters: α = 0.3, β=0.9, γ=0.045, and λ=0.96.\n\n![four-parameter psychometric fitting, part 2](/img/pmf_4-2.png?raw=true \"four-parameter psychometric fitting, part 2\")\n\nWhen interpreting these plots it should be noted that each model fit _kinda_ got to cheat because I gave the fitting function the actual generating set of parameters as its initial guess. The resulting spread in the parameter estimates, then, is very much a best-case scenario.\n\nOverall, the scale and shape parameter estimates appear unbiased in the long-run. But look how poorly it fits the upper-bound, `λ`! And only in the second simulation did it fit `γ` well on average. Why is this? Not really sure yet, but both of these parameters represent performance at the edge-cases of `x`, which means their estimates will be very dependent on the presence of rare events in the datasets.\n\n### Future examples\n\n* Example 5: Fitting with a prior over parameters\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmobeets%2Fpsychometric-fits","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmobeets%2Fpsychometric-fits","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmobeets%2Fpsychometric-fits/lists"}