{"id":32995409,"url":"https://oxuva.github.io/long-term-tracking-benchmark/","last_synced_at":"2025-11-22T23:01:14.913Z","repository":{"id":75638454,"uuid":"119158448","full_name":"oxuva/long-term-tracking-benchmark","owner":"oxuva","description":"[ECCV'18] Long-term Tracking in the Wild: A Benchmark","archived":false,"fork":false,"pushed_at":"2019-12-26T12:27:30.000Z","size":28892,"stargazers_count":178,"open_issues_count":8,"forks_count":37,"subscribers_count":11,"default_branch":"master","last_synced_at":"2024-04-15T09:11:07.823Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://oxuva.github.io/long-term-tracking-benchmark","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/oxuva.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2018-01-27T11:30:17.000Z","updated_at":"2024-01-04T16:20:19.000Z","dependencies_parsed_at":null,"dependency_job_id":"f0474ea1-a90e-47a7-9fad-7bff7ac86d70","html_url":"https://github.com/oxuva/long-term-tracking-benchmark","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/oxuva/long-term-tracking-benchmark","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oxuva%2Flong-term-tracking-benchmark","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oxuva%2Flong-term-tracking-benchmark/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oxuva%2Flong-term-tracking-benchmark/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oxuva%2Flong-term-tracking-benchmark/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/oxuva","download_url":"https://codeload.github.com/oxuva/long-term-tracking-benchmark/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oxuva%2Flong-term-tracking-benchmark/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":285873538,"owners_count":27246054,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-11-22T02:00:05.934Z","response_time":64,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-11-13T12:00:21.881Z","updated_at":"2025-11-22T23:01:14.895Z","avatar_url":"https://github.com/oxuva.png","language":"Python","funding_links":[],"categories":["Dataset"],"sub_categories":["Object Tracking  Dataset"],"readme":"# OxUvA long-term tracking benchmark [ECCV'18]\n**Note:** *If, while reading this tutorial, you are stuck somewhere or you are unsure you are interpreting the instructions correctly, do not hesitate to open an issue here on GitHub.*\n\nThis repository provides Python code to measure the quality of a tracker's predictions and generate all figures of the paper.\nThe following sections provide instructions for each stage.\n\n1. [Obtain the data](#1-obtain-the-data)\n2. [Set up the environment](#2-set-up-the-environment)\n3. [Run your tracker](#3-run-your-tracker)\n4. [Submit to the evaluation server](#4-submit-to-the-evaluation-server)\n5. [Generate the plots for a paper](#5-generate-the-plots-for-a-paper)\n6. [Add your tracker to the results page](#6-add-your-tracker-to-the-results-page)\n\nThe challenge is split into two tracks: \"constrained\" and \"open\".\nTo be eligible for the \"constrained\" challenge, a tracker must use *only* the data in `annotations/dev_constrained_ytbb_train.csv` and `annotations/dev.csv` for development.\nAll other trackers must enter the \"open\" challenge. With *development* we intend, in addition to training, also pre-training, validation and hyper-parameter search.\nFor example, SINT uses pre-trained weights and SiamFC is trained from scratch on ImageNet VID.\nHence they are both in the \"open\" challenge.\n\nThe results of all *citeable* trackers are maintained in a [results repository](https://github.com/oxuva/long-term-tracking-results/).\nThis repo should be used for comparison against state-of-the-art.\nIt is updated periodically according to a [schedule](https://docs.google.com/document/d/1BtoMzxMGfKMM7DtYOm44dXNr18HrG5CqN9cyxDAem-M/edit).\n\n\n## 1. Obtain the data\n\nThe ground-truth labels for the dev set can be found *in this repository* in [`dataset/annotations`](dataset/annotations).\nThe tracker initialization for the dev *and* test sets can be found in [`dataset/tasks`](dataset/tasks).\n**Note:** Only the annotations for the *dev* set are public.\nThese can be useful for diagnosing failures and hyperparameter search.\nFor the *test* set, the annotations are secret and trackers can only be assessed via the evaluation server (explained later).\n\nTo obtain the images, fill in [this form](https://docs.google.com/forms/d/e/1FAIpQLSepA_sLCMrqnZXBPnZFNmggf-MdEGa2Um-Q7pRGQt4SxvGNeg/viewform) and then download `images_dev.tar` and `images_test.tar`.\nExtract these archives in `dataset/`.\n\nThe structure of `dataset/` should be:\n```\ndataset/images/{subset}/{video}/{frame:06d}.jpeg\ndataset/tasks/{subset}.csv\ndataset/annotations/{subset}.csv\n```\nwhere `{subset}` is either `dev` or `test`, `{video}` is the video ID e.g. `vid0042`, and `{frame:06d}` is the frame number e.g. `002934`.\n\n\n### Task format\n\nA tracking \"task\" consists of the initial and final frame numbers and the rectangle that defines the target in the initial frame.\nA collection of tracking tasks are specified in a single CSV file (e.g. [`dataset/tasks/dev.csv`](dataset/tasks/dev.csv)) with the following fields.\n\n* `video_id`: (string) Name of video.\n* `object_id`: (string) Name of object within the video.\n* `init_frame`: (integer) Frame in which initial rectangle is specified.\n* `last_frame`: (integer) Last frame in which tracker is required to make a prediction (inclusive).\n* `xmin`, `xmax`, `ymin`, `ymax`: (float) Rectangle in the initial frame. Co-ordinates are relative to the image: zero means top and left, one means bottom and right.\n\nA tracker should output predictions for frames `init_frame` + 1, `init_frame` + 2, ..., `last_frame`.\n\nThe function `oxuva.load_dataset_tasks_csv` will return a `VideoObjectDict` of `Task`s from such a file.\n\n### Annotation format\n\nA track \"annotation\" gives the ground-truth path of an object.\nThis can be used for training and evaluating trackers.\nThe annotation includes the class, but this information is not provided for a \"task\", and thus will not be available at testing time.\n\nA collection of track annotations are specified in a single CSV file (e.g. [`dataset/annotations/dev.csv`](dataset/annotations/dev.csv)) with the following fields.\n\n* `video_id`: (string) Name of video.\n* `object_id`: (string) Name of object within the video.\n* `class_id`: (integer) Index of object class. Matches YTBB.\n* `class_name`: (string) Name of object class. Matches YTBB.\n* `contains_cuts`: (string) Either `true`, `false` or `unknown`.\n* `always_visible`: (string) Either `true`, `false` or `unknown`.\n* `frame_num`: (integer) Frame of current annotation.\n* `object_presence`: (string) Either `present` or `absent`.\n* `xmin`, `xmax`, `ymin`, `ymax`: (float) Rectangle in the current frame if present, otherwise it should be ignored.\n\nThe function `oxuva.load_dataset_annotations_csv` will return a `VideoObjectDict` of track annotation dict from such a file.\nThe functions `oxuva.make_track_label` and `oxuva.make_frame_label` are used to construct track annotation dicts.\nThe function `oxuva.make_task_from_track` converts a track annotation into a tracking task with ground-truth labels.\n\n\n## 2. Set up the environment\n\nTo run the code in this repository, it is necessary to install the Python libraries listed in [`requirements.txt`](requirements.txt).\nTo install these dependencies using `pip` (perhaps in a virtual environment):\n```bash\npip install -r requirements.txt\n```\n\nYou must also add the parent directory of `oxuva/` to `PYTHONPATH` to be able to import the `oxuva` package.\n```bash\nexport PYTHONPATH=\"path/to/long-term-tracking-benchmark/python:$PYTHONPATH\"\n```\nAlternatively, for convenience, you can `source` the script `pythonpath.sh` in `bash`:\n```bash\nsource path/to/long-term-tracking-benchmark/pythonpath.sh\n```\n\n\n## 3. Run your tracker\n\n**Note:** Unlike the VOT or OTB toolkits, ours does not execute your tracker.\nYour tracker should output all predictions in the format described below.\nFor Python trackers, we provide the utility functions `oxuva.load_dataset_tasks_csv` and `oxuva.dump_predictions_csv` to make this easy.\nSee [`examples/opencv/track.py`](examples/opencv/track.py) for an example.\n\nAll rectangle co-ordinates are relative to the image: zero means top and left, one means bottom and right.\nIf the object extends beyond the image boundary, ground-truth rectangles are clipped to \\[0, 1\\].\n\n### Prediction format\n\nThe predictions for a tracker are specified with one CSV file per track.\nThe names of these files must be `{video}_{object}.csv`.\nThe fields of these CSV files are:\n\n* `video_id`: (string) Name of video.\n* `object_id`: (string) Name of object within the video.\n* `frame_num`: (integer) Frame of current annotation.\n* `present`: (string) Either `present` or `absent` (can use `true`/`false` or `0`/`1` too)\n* `score`: (float) Number that represents confidence of object presence.\n* `xmin`, `xmax`, `ymin`, `ymax`: (float) Rectangle in the current frame if present, otherwise it is ignored.\n\nThe score is only used for diagnostics, it does not affect the main evaluation of the tracker.\nIf the object is predicted `absent`, then the score and the rectangle will not be used.\nSince the ground-truth annotations do not extend beyond the edge of the image, the evaluation toolkit will truncate the predicted rectangles to the image frame before computing the IOU.\n\n\n## 4. Submit to the evaluation server\n\nSince the annotations for the test set are secret, in order to evaluate your tracker and produce plots similar to the one in our paper you need to submit the raw prediction csv files to the [evaluation server](https://competitions.codalab.org/competitions/19529#participate), hosted on CodaLab.\n\nFirst, create a CodaLab account (if you do not already have one) and request to join the OxUvA competition.\nNote that the CodaLab account is per human, not per tracker.\nDo *not* create a username for your tracker.\nThe name of your tracker will appear when you add it to the results repository (point 6 of this tutorial).\nPlease choose a username that enables us to identify you, such as your real name or your GitHub account.\n\nTo submit the results, create a zip archive containing all predictions in CSV format (as described above).\nThere should be one file per object with the filename `{video}_{object}.csv`.\nIt doesn't matter whether the CSV files are contained at the root level of the zip archive or below a single subdirectory of any name.\nIf a submission encounters an error (for example, a missing prediction file), you will be able to view the verbose error log, and the submission will not count towards your quota.\n(If you want, you can first upload your predictions for the dev set to confirm that your predictions are in the correct format.)\nPlease consider that for the dev set your quota is of 500 submissions in total (max 50 per day), while for the test set the limit is of *10 submissions in total (max 1 per day)*.\n\nOnce the submission has been successful, you can download the generated output files.\nThese will be used to generate the plots and submit to the results repo.\n\n**Note:** You will notice that the CodaLab challenge shows a leaderboard with usernames and scores.\nFor the purpose of writing a paper, you do not need to compare against the most recent methods: what matters are the state-of-the-art results for citeable trackers in the results repository (point 6).\n\n\n## 5. Generate the plots for a paper\n\nFirst, clone the results repo.\n```bash\ngit clone https://github.com/oxuva/long-term-tracking-results.git\ncd long-term-tracking-results\n```\nThis repo contains several snapshots of the past state-of-the-art as git tags (*TODO* generate tags).\nThe tag `eccv18` indicates the set of methods in our original paper, and successive tags are of the form `{year}-{month:02d}`, for example:\n```bash\ngit checkout 2018-07\n```\n\nYou can state in your paper which tag you are comparing against.\nWhen writing a paper, you are not required to compare against the most recent state-of-the-art... but clearly the most recent the better, as your results will be more convincing.\n\nAdd an entry for your tracker to `trackers.json`.\nYou must specify a human-readable name for your tracker, and whether your tracker is eligible for the constrained-data challenge.\n```json\n    \"tracker_id\": {\"name\": \"Tracker Name\", \"constrained\": false},\n```\nUse `python -m json.tool --sort-keys` to standardize the formatting and order of this file.\n\nFor the test set, copy the `iou_0dx.json` files returned by the evaluation server to the directory\n```results/assess/test/{tracker_id}/```\nin the results repo.\nThe script `oxuva.tools.analyze` will try to load this summary of the tracker assessment from these files before attempting to read the complete predictions of the tracker and the ground-truth annotations.\n\nFor the dev set, you may follow the same procedure as above.\nHowever, it is possible to evaluate your tracker's predictions locally, without using the evaluation server.\nTo do this, put the CSV files of your tracker's predictions (that is, the input to the evaluation server) in the directory\n```predictions/dev/{tracker_id}/```\nin the results repo.\nThe script will generate the corresponding files in the `assess/` directory.\nNote that if you update the predictions, you should erase the corresponding files in the `assess/` directory, or specify `--no_use_summary`.\nIf desired, the predictions of other trackers on the *dev* set are available from Google Drive (TODO).\nPlease do not publish your predictions on the *test* set, as it may enable someone to construct an approximate ground-truth using a consensus method.\n\nTo generate all plots and tables, use `analyze_paper.sh` or `analyze_web.sh`:\n```bash\nbash analyze_paper.sh --data=test --challenge=open --loglevel=warning\n```\n\nTo just generate the table of results:\n```bash\npython -m oxuva.tools.analyze table --data=dev --challenge=open\n```\nThe results table will be written to `analysis/dev/open/table.csv`.\nUse `--help` to discover the optional flags.\nFor example, you can use `--iou_thresholds 0.1 0.3 0.5 0.7 0.9` to generate the results with a wider range of thresholds.\n\nSimilarly, to just generate the main figure, use:\n```bash\npython -m oxuva.tools.analyze plot_tpr_tnr --data=dev --challenge=open\n```\n**Note:** Please do *not* put the dev set plots in the paper without the test set.\nIn general, comparison statements of the type *A is better than B* should be done using the test set.\n\n\n## 6. Add your tracker to the results page\n\nSeparately from the evaluation server, we are maintaining a [results page/repository](https://github.com/oxuva/long-term-tracking-results) that reflects the state-of-the-art on our dataset.\n\nIn order to have your tracker added to the plots, you need to:\n\n1) Have completed all the previous points and produced the test set plots of your tracker.\n2) Have a document that describes your tracker. It does not need to be a peer-reviewed conference - arXiv is fine - we just need a *citeable* method.\n3) Do a pull request to the results repository, containing everything we need to update the plots (i.e. `assess/test/{tracker_name}/iou_0d{3,5,7}.json`. Remember to specify the name of your tracker and whether it qualifies for the constrained challenge in `trackers.json`. In the comment section, please include a) your CodaLab username, b) the paper that describes your method and optionally d) a short description of your method. Do not include the generated plots, we will update these after merging the pull request.\n4) The organizers will manually review your request according to [this schedule](https://docs.google.com/document/d/1BtoMzxMGfKMM7DtYOm44dXNr18HrG5CqN9cyxDAem-M/edit).\n\nRemember that even if your method is not in first place, submitting your tracker to the results repository is valuable to the community and it increases the chance of having your paper read and cited.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/oxuva.github.io%2Flong-term-tracking-benchmark%2F","html_url":"https://awesome.ecosyste.ms/projects/oxuva.github.io%2Flong-term-tracking-benchmark%2F","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/oxuva.github.io%2Flong-term-tracking-benchmark%2F/lists"}