{"id":17222889,"url":"https://github.com/cheind/py-globalflow","last_synced_at":"2025-04-14T00:17:43.881Z","repository":{"id":45712973,"uuid":"388178894","full_name":"cheind/py-globalflow","owner":"cheind","description":"Python implementation of \"Global Data Association for MOT Tracking using Network Flows\"","archived":false,"fork":false,"pushed_at":"2021-08-16T17:45:45.000Z","size":3703,"stargazers_count":14,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"develop","last_synced_at":"2025-03-27T14:21:17.414Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/cheind.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-07-21T16:24:50.000Z","updated_at":"2024-09-08T15:03:15.000Z","dependencies_parsed_at":"2022-09-23T08:30:18.702Z","dependency_job_id":null,"html_url":"https://github.com/cheind/py-globalflow","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cheind%2Fpy-globalflow","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cheind%2Fpy-globalflow/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cheind%2Fpy-globalflow/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cheind%2Fpy-globalflow/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/cheind","download_url":"https://codeload.github.com/cheind/py-globalflow/tar.gz/refs/heads/develop","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248799961,"owners_count":21163404,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-15T04:06:36.908Z","updated_at":"2025-04-14T00:17:43.854Z","avatar_url":"https://github.com/cheind.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\n\n\n![](https://www.travis-ci.com/cheind/py-globalflow.svg?branch=main)\n\n# **py-globalflow**\nPython implementation of _Global Data Association for MOT Tracking using Network Flows_ (zhang2008global) with minor tweaks and applications to human pose tracking.\n\nFeatures\n- **Problem agnostic**: No restrictions are set on the type of observations. Related costs can be computed in a flexible manner by providing cost function object. Standard costs based on negative log-probabilities, as presented in the the paper (zhang2008global), are implemented as a specialized cost function objects.\n- **Occlusions**: Short-term occlusions can be handled by enabling short-cut connections between flow nodes (see remarks below).\n- **Plotting**: Helpers for plotting the flow graph and any solution.\n- **Application**: An application for tracking human poses by means of geometric and appearance information is included. Appearance information is taken from compressed Re-ID features predicted by deep neural architectures ([torchreid](https://github.com/KaiyangZhou/deep-person-reid)).\n\n## Example\n```python\nimport matplotlib.pyplot as plt\nimport numpy as np\nimport scipy.stats\n\nimport globalflow as gflow\n\ntimeseries = [\n    [0.0, 1.0],  # obs. at t=0\n    [-0.5, 0.1, 0.5, 1.1],  # obs. at t=1\n    [0.2, 0.6, 1.2],  # obs. at t=2\n]\n\n# Define the class that provides costs. Here we inherit from\n# gflow.StandardGraphCosts which already predefines some costs\n# based on equations found in the paper.\nclass GraphCosts(gflow.StandardGraphCosts):\n    def __init__(self) -\u003e None:\n        super().__init__(\n            penter=1e-3, pexit=1e-3, beta=0.05, max_obs_time=len(timeseries) - 1\n        )\n\n    def transition_cost(self, x: gflow.FlowNode, y: gflow.FlowNode) -\u003e float:\n        tdiff = y.time_index - x.time_index\n        logprob = scipy.stats.norm.logpdf(\n            y.obs, loc=x.obs + 0.1 * tdiff, scale=0.5\n        ) + np.log(0.1)\n        return -logprob\n\n# Setup the graph\nflowgraph = gflow.build_flow_graph(timeseries, GraphCosts())\n\n# Solve the problem globally\nflowdict, ll, num_traj = gflow.solve(flowgraph)\n\nprint(\n    \"optimum: log-likelihood\", ll, \"number of trajectories\", num_traj\n)  # optimum: log-likelihood 6.72 number of trajectories 2\n\n# Note, while the flowdict structure is the most general representation\n# for the solution of the problem, it might be hard to work with in\n# practice. See gflow.find_trajectories and gflow.label_observations\n# to convert between representations.\n\n# Plot the graph and the result.\n\nplt.figure(figsize=(12, 8))\ngflow.draw.draw_graph(flowgraph)\nplt.figure(figsize=(12, 8))\ngflow.draw.draw_flowdict(flowgraph, flowdict)\nplt.show()\n```\n\nThe following graph shows the optimal trajectories\n\n\u003cdiv align=\"center\"\u003e\n\u003cimg src=\"etc/flow.svg\" width=\"80%\"\u003e\n\u003c/div\u003e\n\nand problem setup\n\n\u003cdiv align=\"center\"\u003e\n\u003cimg src=\"etc/graph.svg\" width=\"80%\"\u003e\n\u003c/div\u003e\n\n## Install\n```bash\npip install git+https://github.com/cheind/py-globalflow\n```\n\nIn case you intend to run the examples as well, better clone this repo and install locally.\n```bash\ngit clone https://github.com/cheind/py-globalflow.git\ncd py-globalflow\npip install -e .[dev]\n```\n\n## Remarks\n\nThe paper (zhang2008global) considers the problem of finding the global optimal trajectories _T_ from a given set of observations _X_. Optimality is defined in terms maximizing the posterior probability p(_T_|_X_). Given some independence assumptions (section 3.1) the paper decomposes the distribution into two main factors: a) the likelihoods of observations p(xi|_T_) and b) the probability of a single trajectory Ti p(Ti):\n- p(xi|_T_) ~ Bernoulli(1-beta)\n- p(Ti) ~ Markov chain consisting of appearance, linking and disappearing probabilities between involved observations\n\nGiven probabilistic formulation, the task of finding optimal trajectories can be mapped to a min-cost-flow problem. The interpretation of this mapping is quite intuitive\n\u003e Each flow path can be interpreted as an object trajectory, the amount of the flow\nsent from s to t is equal to the number of object trajectories, and the total cost of the flow on G corresponds to the loglikelihood of the association hypothesis (zhang2008global).\n\n### Observation probabilities p(xi|_T_)\n\np(xi|_T_) is modeled as a Bernoulli variable with parameter (1-b), where b(eta) is probability of being a false-positive. The derived cost term (eq. 11) Ci = log(b/(1-b)), is derived as follows ()\n```\nlog p(xi|_T)        = log((1-bi)^fi*bi^(1-fi))\n                    = fi*log(1-bi) + (1-fi)*log(bi)\n                    = fi*log(1-bi) - fi*log(bi) + log(bi)\n-log p(xi|_T)       = -fi*log(1-bi) + fi*log(bi) - log(bi)\n                    = fi*log(bi/(1-bi)) - log(bi)\namin -log p(xi|_T)  = amin fi*log(bi/(1-bi))\n                    = amin fi*ci\n```\nwith fi being the indicator variable of whether xi is part of the solution or not. The term -log(bi) vanishes as it can be regarded constant wrt to argmin. The plot below graphs bi vs ci.\n\n\u003cdiv align=\"center\"\u003e\n\u003cimg src=\"etc/fpcost.svg\" width=\"60%\"\u003e\n\u003c/div\u003e\n\nAs the probability of false-positive drops below 0.5, the auxiliary edge cost between ui/vi edge cost gets negative. This allows the optimization to introduce new trajectories that increase the total flow likelihood. All other costs (pairing, appearance, disappearance) are negative log probabilities and hence positive.\n\n## Short-term occlusions\nThis library supports short-term occlusions via transition edges that connect different observation times. We make use of the time ordered sequence of observations and limit transition edges up to the previous `l+1` timesteps. Hence, transition edges allow observations at time `t` to pair previous observations up to `t-l-1`, where `l` is the number of skip layers (defaults to zero).\n\nGiven a similar set of observations as above\n```python\ntimeseries = [\n    [0.0, 1.0],  # obs. at t=0\n    [-0.5, 0.1, 0.5, 1.1],  # obs. at t=1\n    [0.2, 0.6],  # obs. at t=2\n    [0.3, 0.6, 1.3],  # obs. at t=3\n]\n```\n\nwe see that a potential track `(1.0, 1.1, -, 1.3)` is occluded at time 2. Setting skip-layers `l=1` we get the following solution that successfully connects this track.\n\n\u003cdiv align=\"center\"\u003e\n\u003cimg src=\"etc/occlusions.svg\" width=\"80%\"\u003e\n\u003c/div\u003e\n\nNote, that the transition probability p(xi|xj) will need to incorporate the time-difference (i.e via a motion model that is application dependent). See `examples/minimal_occlusions.py` for full details.\n\n## Human Pose Tracking\nThis repository contains an example to use **py-globalflow** for tracking 2D human pose outputs. The application, by default, uses on geometric joint properties and hence only 2D pose results are required. Optionally, Re-ID features can be incorporated to improve long-term occlusion handling. See\n\n```\npython -m examples.track_poses --help \n```\n\n### 2D Pose Results\n\nBelow are two videos (click on the images to play) that compare input to found trajectories without short-cut layers. The pose 2D human pose prediction is done by (metha2018single, wang2020deep) on samples from the MuPoTS-3D (mehta2018single) and MPI-INF-3DHP (mono-3dhp2017).\n\nTS1 Sequence\n\u003cdiv align=\"center\"\u003e\n    \u003ca href=\"https://www.youtube.com/watch?v=bO0R1tq_wcI\"\u003e\u003cimg src=\"etc/ts1.png\" width=\"60%\"\u003e\u003c/a\u003e\n\u003c/div\u003e\n\nTS18 Sequence\n\u003cdiv align=\"center\"\u003e\n    \u003ca href=\"https://www.youtube.com/watch?v=wY8X0AO-MTo\"\u003e\u003cimg src=\"etc/ts18.png\" width=\"80%\"\u003e\u003c/a\u003e\n\u003c/div\u003e\n\n### Re-ID Features\n\nWhen using Re-ID features, tracking incorporates appearance information in tracking. Re-ID features are computed using `examples/reid_features.py`, which relies on a deep neural network for feature prediction. The application then, by default, compresses these features onto 2D space. These compressed features are then subject to a multivariate normal distribution during transition probability computation. Experimentally, we've found that compressing Re-ID features for segmented videos generally maintains cluster information well, as shown below for TS1 sequence.\n\n\u003cdiv align=\"center\"\u003e\n\u003cimg src=\"etc/reid-compression.png\" width=\"80%\"\u003e\n\u003c/div\u003e\n\nThe following video shows improved occlusion handling of **py-globalflow** on TS18 sequence.\n\n[![](etc/reid-cover.PNG)](https://www.youtube.com/watch?v=6LUWGpCFOm4)\n\n\n\n### 3D Pose Results\n\nThe following video shows the beneficial effect of **py-globalflow** on 3D human pose estimation. This is based on the work of (veges2020temporal) that notes the following drawback of their method:\n\u003e Also, one drawback of our approach is that it does not include tracking, the combination with a tracking algorithm\nremains future work.\n\nWhen applied to multi-person scenarios and  the person IDs get mixed up, the algorithm tends towards their middle poses. That is, the person on one side is attracted to the other side and vice versa. This leads to hallucinations that look like artificial dances of the involved persons. \n\nIn this case we use **py-globalflow** to generate correct pose tracks based on geometric joint and Re-ID information. When we then re-run the temporal smoothing module we get much more realistic results as shown in the video below. Left is the input video, middle is with **py-globalflow** and right is without pose tracking.\n\n#### TS1 Sequence\n\n[![](etc/posesmooth-cover.PNG)](https://www.youtube.com/watch?v=aU3whnxvXFc)\n\n#### TS18 Sequence\nThis sequence benefits in particular from Re-ID appearance cost terms in tracking to recover from the mid-term occlusions. The flying pose towards the end of the video is due to the 3D pose estimator.\n\n[![](etc/posesmooth-ts18-cover.PNG)](https://www.youtube.com/watch?v=3pb1-teTw44)\n\n\n## References\n```bibtex\n@inproceedings{zhang2008global,\n  title={Global data association for multi-object tracking using network flows},\n  author={Zhang, Li and Li, Yuan and Nevatia, Ramakant},\n  booktitle={2008 IEEE Conference on Computer Vision and Pattern Recognition},\n  pages={1--8},\n  year={2008},\n  organization={IEEE}\n}\n\n@inproceedings{mehta2018single,\n  title={Single-shot multi-person 3d pose estimation from monocular rgb},\n  author={Mehta, Dushyant and Sotnychenko, Oleksandr and Mueller, Franziska and Xu, Weipeng and Sridhar, Srinath and Pons-Moll, Gerard and Theobalt, Christian},\n  booktitle={2018 International Conference on 3D Vision (3DV)},\n  pages={120--130},\n  year={2018},\n  organization={IEEE}\n}\n\n@article{wang2020deep,\n  title={Deep high-resolution representation learning for visual recognition},\n  author={Wang, Jingdong and Sun, Ke and Cheng, Tianheng and Jiang, Borui and Deng, Chaorui and Zhao, Yang and Liu, Dong and Mu, Yadong and Tan, Mingkui and Wang, Xinggang and others},\n  journal={IEEE transactions on pattern analysis and machine intelligence},\n  year={2020},\n  publisher={IEEE}\n}\n\n@inproceedings{mono-3dhp2017,\n author = {Mehta, Dushyant and Rhodin, Helge and Casas, Dan and Fua, Pascal and Sotnychenko, Oleksandr and Xu, Weipeng and Theobalt, Christian},\n title = {Monocular 3D Human Pose Estimation In The Wild Using Improved CNN Supervision},\n booktitle = {3D Vision (3DV), 2017 Fifth International Conference on},\n url = {http://gvv.mpi-inf.mpg.de/3dhp_dataset},\n year = {2017},\n organization={IEEE},\n doi={10.1109/3dv.2017.00064},\n} \n\n@inproceedings{veges2020temporal,\n  author=\"V{\\'e}ges, M. and L{\\H{o}}rincz, A.\",\n  title=\"Temporal Smoothing for 3D Human Pose Estimation and Localization for Occluded  People\",\n  booktitle=\"Neural Information Processing\",\n  year=\"2020\",\n  pages=\"557--568\",\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcheind%2Fpy-globalflow","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcheind%2Fpy-globalflow","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcheind%2Fpy-globalflow/lists"}