{"id":13590982,"url":"https://github.com/cvondrick/vatic","last_synced_at":"2025-12-17T04:02:47.976Z","repository":{"id":141375951,"uuid":"2019922","full_name":"cvondrick/vatic","owner":"cvondrick","description":"Efficiently Scaling Up Video Annotation with Crowdsourced Marketplaces. IJCV 2012","archived":false,"fork":false,"pushed_at":"2020-07-15T20:01:58.000Z","size":2382,"stargazers_count":607,"open_issues_count":91,"forks_count":255,"subscribers_count":36,"default_branch":"master","last_synced_at":"2024-11-06T11:45:11.898Z","etag":null,"topics":["computer-vision","video","video-annotation"],"latest_commit_sha":null,"homepage":"http://mit.edu/vondrick/vatic/","language":"HTML","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/cvondrick.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2011-07-08T20:36:53.000Z","updated_at":"2024-09-28T01:26:03.000Z","dependencies_parsed_at":null,"dependency_job_id":"b783566c-eeef-4e33-9680-30c6e32ae009","html_url":"https://github.com/cvondrick/vatic","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cvondrick%2Fvatic","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cvondrick%2Fvatic/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cvondrick%2Fvatic/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cvondrick%2Fvatic/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/cvondrick","download_url":"https://codeload.github.com/cvondrick/vatic/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247860881,"owners_count":21008374,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["computer-vision","video","video-annotation"],"created_at":"2024-08-01T16:00:52.417Z","updated_at":"2025-12-17T04:02:42.916Z","avatar_url":"https://github.com/cvondrick.png","language":"HTML","funding_links":[],"categories":["HTML"],"sub_categories":[],"readme":"# Please Note\n\nIntel has created an excellent annotation tool with the latest technologies. https://github.com/opencv/cvat \n\nThe project below is archived, and no further updates are expected.\n\n2009 to 2020\n\n# VATIC - Video Annotation Tool from Irvine, California\n\n\n\nVATIC is an online video annotation tool for computer vision research that\ncrowdsources work to Amazon's Mechanical Turk. Our tool makes it easy to build\nmassive, affordable video data sets. \n\n\u003cimg src='http://i.imgur.com/z6jl5Bs.jpg'\u003e\n\n# INSTALLATION \n\nNote: VATIC has only been tested on Ubuntu with Apache 2.2 HTTP server and a\nMySQL server. This document will describe installation on this platform,\nhowever it should work any operating system and with any server.\n\n## Download\n\nYou can download and extract VATIC from our website. Note: do NOT run the \ninstaller as root. \n\n    $ wget http://mit.edu/vondrick/vatic/vatic-install.sh\n    $ chmod +x vatic-install.sh\n    $ ./vatic-install.sh\n    $ cd vatic\n\n## HTTP Server Configuration \n\nOpen the Apache configuration file. On Ubuntu, this file is located at:\n\n    /etc/apache2/sites-enabled/000-default\n\nIf you do not use Apache on this computer for any other purpose, replace the\ncontents of the file with:\n\n    WSGIDaemonProcess www-data\n    WSGIProcessGroup www-data\n\n    \u003cVirtualHost *:80\u003e\n        ServerName vatic.domain.edu\n        DocumentRoot /path/to/vatic/public\n\n        WSGIScriptAlias /server /path/to/vatic/server.py\n        CustomLog /var/log/apache2/access.log combined\n    \u003c/VirtualHost\u003e\n\nupdating ServerName with your domain name, DocumentRoot with the path to\nthe public directory in VATIC, and WSGIScriptAlias to VATIC's server.py file.\n\nIf you do use Apache for other purposes, you will have to setup a new virtual\nhost with the correct document root and script alias, as shown above.\n\nMake sure you have the mod_headers module enabled:\n\n    $ sudo cp /etc/apache2/mods-available/headers.load /etc/apache2/mods-enabled\n\nAfter making these changes, restart Apache:\n\n    $ sudo apache2ctl graceful\n\n## SQL Server Configuration \n\nWe recommend creating a separate database specifically for VATIC:\n\n    $ mysql -u root\n    mysql\u003e create database vatic;\n\nThe next section will automatically create the necessary tables.\n\n## Setup\n\nInside the vatic directory, copy config.py-example to config.py:\n\n    $ cp config.py-example config.py\n\nThen open config.py and make changes to the following variables in order to\nconfigure VATIC:\n\n    signature       Amazon Mechanical Turk AWS signature (secret access key)\n    accesskey       Amazon Mechanical Turk AWS access key (access key ID)\n    sandbox         If true, put into Mturk sandbox mode. For debugging.\n    localhost       The local HTTP address: http://vatic.domain.edu/ so it\n                    matches the ServerName in Apache.\n    database        Database connection string: for example,\n                    mysql://user:pass@localhost/vatic\n    geolocation     API key from ipinfodb.com for geolocation services\n\nIf you do not plan on using VATIC on Mechcanical Turk (offlien mode only), you\ncan leave the signature and accesskey empty.\n\nAfter saving results, you can then initialize the database:\n\n    $ turkic setup --database\n\nNote: if you want to reset the database, you can do this with:\n\n    $ turkic setup --database --reset\n\nwhich will require confirmation to reset in order to prevent data loss.\n\nFinally, you must also allow VATIC to access turkic, a major dependency:\n\n    $ turkic setup --public-symlink\n\n# ANNOTATION \n\nBefore you continue, you should verify that the installation was correct. You\ncan verify this with:\n\n    $ turkic status --verify\n\nIf you receive any error messages, it means the installation was not complete\nand you should review the previous section. Note: If you do not plan on\nusing Mechanical Turk, you can safely ignore any errors caused by Mechanical\nTurk.\n\n## Frame Extraction \n\nOur system requires that videos are extracted into JPEG frames. Our tool can \ndo this automatically for you:\n\n    $ mkdir /path/to/output/directory\n    $ turkic extract /path/to/video.mp4 /path/to/output/directory\n\nBy default, our tool will resize the frames to fit within a 720x480 rectangle.\nWe believe this resolution is ideal for online video viewing. You can change \nresolution with options:\n\n    $ turkic extract /path/to/video.mp4 /path/to/output/directory\n      --width 1000 --height 1000\n\nor\n\n    $ turkic extract /path/to/video.mp4 /path/to/output/directory\n      --no-resize\n\nThe tool will maintain aspect ratio in all cases.\n\nAlternatively, if you have already extracted frames, you can use the\nformatframes command to format the video into a format that VATIC understands:\n\n    $ turkic formatframes /path/to/frames/ /path/to/output/directory\n\nThe above command will read all the images in /path/to/frames and create\nhard links (soft copy) in /path/to/output/directory.\n\n## Importing a Video\n\nAfter extracting frames, the video can be imported into our tool for \nannotation. The general syntax for this operation is:\n\n    $ turkic load identifier /path/to/output/directory Label1 Label2 LabelN\n\nwhere identifier is a unique string that you will use to refer to this video,\n/path/to/output/directory is the directory of frames, and LabelX are class\nlabels that you want annotated (e.g., Person, Car, Bicycle). You can have as\nmany class labels as you wish, but you must have at least one.\n\nWhen a video is imported, it is broken into small segments typically of only a\nfew seconds. When all the segments are annotated, the annotations are merged\nacross segments because each segment overlaps another by a small margin.\n\nThe above command specifies all of the required options, but there are many\noptions available as well. We recommend using these options.\n\n    MTurk Options\n        --title         The title that MTurk workers see\n        --description   The description that MTurk workers see\n        --duration      Time in seconds that a worker has to complete the task\n        --lifetime      Time in seconds that the task is online\n        --keywords      Keywords that MTurk workers can search on\n        --offline       Disable MTurk and use for self annotation only\n\n    Compensation Options\n        --cost                  The price advertised to MTurk workers\n        --per-object-bonus      A bonus in dollars paid for each object\n        --completion-bonus      A bonus in dollars paid for completing the task\n\n    Qualification Options\n        --min-approved-percent  Minimum percent of tasks the worker must have\n                                approved before they can work for you\n        --min-approved-amount   Minimum number of tasks that the worker must \n                                have completed before they can work for you\n\n    Video Options\n        --length        The length of each segment for this video in frames\n        --overlap       The overlap between segments in frames\n        --use-frames    When splitting into segments, only the frame intervals\n                        specified in this file. Each line should contain a\n                        start frame, followed by a space, then the stop frame.\n                        Frames outside the intervals in this file will be\n                        ignored.\n        --skip          If specified, request annotations only every N frames.\n        --blow-radius   When a user marks an annotation, blow away all other\n                        annotations within this many frames. If you want to\n                        allow the user to make fine-grained annotations, set\n                        this number to a small integer, or 0 to disable. By\n                        default, this is 5, which we recommend.\n\nYou can also specify temporal attributes that each object label can take on.\nFor example, you may have a person object with attributes \"walking\", \"running\",\nor \"sitting\". You can specify attributes the same way as labels, except you\nprepend an ~ before the text, which bind the attribute to the previous label:\n\n    $ turkic load identifier /path/to/output/directory Label1 ~Attr1A ~Attr1B\n      Label2 ~Attr2A ~Attr2B ~Attr2C Label3 \n\nIn the above example, Label1 will have attributes Attr1A and Attr1B, Label2\nwill have attributes Attr2B, Attr2B, and Attr2C and Label3 will have no \nattributes. Specifying attributes is optional.\n\n## Gold Standard Training \n\nIt turns out that video annotation is extremely challenging and most MTurk\nworkers lack the necessary patience. For this reason, we recommend requiring\nworkers to pass a \"gold standard\" video. When a new worker visits the task,\nthey will be redirected to a video for which the annotations are already known.\nIn order to move on to the true annotations, the worker must correctly annotate\nthe gold standard video first. We have found that this approach significantly\nimproves the quality of the annotations.\n\nTo use this feature, import a video to be used as the gold standard:\n\n    $ turkic load identifier-train /path/to/frames Label1 Label2 LabelN\n      --for-training --for-training-start 0 --for-training-stop 500\n      --for-training-overlap 0.5 --for-training-tolerance 0.1\n      --for-training-mistakes 1\n\nYou can also use any of the options described above. Explanations for the new\noptions are as follows:\n\n    --for-training              Specifies that this video is gold standard\n    --for-training-start        Specifies the first frame to use\n    --for-training-stop         Specifies the last frame to use\n    --for-training-overlap      Percent overlap that worker's boxes must match \n    --for-training-tolerance    Percent that annotations must agree temporally\n    --for-training-mistakes     The number of completely wrong annotations \n                                allowed. We recommend setting this to a small,\n                                nonzero integer.\n\nAfter running the above command, it will provide you with an URL for you to\ninput the ground truth annotation. You must make this ground truth annotation\nas careful as possible, as it will be used to evaluate future workers.\n\nYou can now specify that a video should use a gold standard video:\n\n    $ turkic load identifier /path/to/output/directory Label1 Label2 LabelN\n      --train-with identifier-train\n\nWhen a not-yet-seen worker visits this video, they will now be redirected to\nto the training video and be required to pass the evaluation test first.\n\n## Publishing Tasks \n\nWhen you are ready for the MTurk workers to annotate, you must publish the \ntasks, which will allow workers to start annotating:\n    \n    $ turkic publish\n\nYou can limit the number of tasks that are published:\n\n    $ turkic publish --limit 100\n\nRunning above command repeatedly will launch tasks in batches of 100. You can\nalso disable all pending tasks:\n    \n    $ turkic publish --disable\n\nwhich will \"unpublish\" tasks that have not yet been completed.\n\nIf you have videos that are offline only, you can see their access URLs with\nthe command:\n\n    $ turkic publish --offline\n\nNote: for the above command to work, you must have loaded the video with the\n--offline parameter as well: \n\n    $ turkic load identifier /path/to/frames Person --offline\n\n## Checking the Status \n\nYou can check the status of the video annotation server with the command:\n\n    $ turkic status\n\nThis will list various statistics about the server, such as number of jobs\npublished and how many are completed. You can get even more statistics by\nrequesting additional information from Amazon:\n\n    $ turkic status --turk\n\nwhich will output how much money is left in your account, among other\nstatistics.\n\nWhen all the videos are annotated, the last line will read:\n\n    Server is offline.\n\n## Retrieving Annotations\n\nYou can get all the annotations for a video with the command:\n\n    $ turkic dump identifier -o output.txt\n\nwhich will write the file \"output.txt\" where each line contains one\nannotation. Each line contains 10+ columns, separated by spaces. The\ndefinition of these columns are:\n\n    1   Track ID. All rows with the same ID belong to the same path.\n    2   xmin. The top left x-coordinate of the bounding box.\n    3   ymin. The top left y-coordinate of the bounding box.\n    4   xmax. The bottom right x-coordinate of the bounding box.\n    5   ymax. The bottom right y-coordinate of the bounding box.\n    6   frame. The frame that this annotation represents.\n    7   lost. If 1, the annotation is outside of the view screen.\n    8   occluded. If 1, the annotation is occluded.\n    9   generated. If 1, the annotation was automatically interpolated.\n    10  label. The label for this annotation, enclosed in quotation marks.\n    11+ attributes. Each column after this is an attribute.\n\nBy default, the above command will not attempt to merge annotations across\nshot segments. You can request merging with the command:\n\n    $ turkic dump identifier -o output.txt --merge --merge-threshold 0.5\n\nThe --merge-threshold option is optional, but it is a number between 0 and 1\nthat represents much the paths must agree in order to merge. 1 specifies a\nperfect match and 0 specifies no match. In practice, 0.5 is sufficient. Merging\nis done using the Hungarian algorithm.\n\nYou can also scale annotations by a factor, which is useful for when the\nvideos have been downsampled:\n\n    $ turkic dump identifier -o output.txt -s 2.8\n\nor force it to fit within a max dimension:\n\n    $ turkic dump identifier -o output.txt --dimensions 400x200\n\nor force it to fit within the dimensions of the original video:\n\n    $ turkic dump identifier -o output.txt --original-video /path/to/video.mp4\n\nThe command can also output to many different formats. Available formats are:\n\n    --xml       Use XML\n    --json      Use JSON\n    --matlab    Use MATLAB\n    --pickle    Use Python's Pickle\n    --labelme   Use LabelMe video's XML format\n    --pascal    Use PASCAL VOC format, treating each frame as an image\n\nThe specifications for these formats should be self explanatory.\n\n## Visualizing Videos \n\nYou can preview the annotations by visualizing the results:\n\n    $ turkic visualize identifier /tmp --merge\n\nwhich will output frames to /tmp with the bounding boxes with the file name\nas the frame number. The visualization will contain some meta information\nthat can help you identify bad workers. You can remove this meta information\nwith the option:\n\n    $ turkic visualize identifer /tmp --merge --no-augment\n\nIf you want to make a video of the visualization (e.g., with ffmpeg), it is\nuseful to renumber the frames so that they start counting at 0 and do not\nhave any gaps:\n\n    $ turkic visualize identifier /tmp --merge --renumber\n\nIf you wish to display the class label and their attributes next to the box,\nspecify the --labels option:\n\n    $ turkic visualize identifier /tmp --labels\n\n## Compensating Workers \n\nWhen you are ready, you can compensate workers:\n\n    $ turkic compensate --default accept\n\nwhich will pay all workers for all outstanding tasks. We strongly recommend\npaying all workers regardless of their quality. You should attempt to pay\nworkers at least once per day.\n\n## Finding Jobs \n\nIf you have found a small mistake in a video and want to make\nthe correction yourself, you can start an annotation session initialized with\nthe MTurk workers annotations:\n\n    $ turkic find --id identifier\n    $ turkic find --id identifier --frame frame\n\nwhere identifier is the identifier for the video and frame is the frame number\nthat the error occurs. In most cases, this command will return one URL for you\nto make the corrections. If it outputs two URLs, it means the frame number\noccurs in two overlapping segments, and so you may have to make changes to both\nof the segments. You can also omit the frame argument, in which case it will\noutput all URLs for that video.\n\nIf you want to find the HIT id, assignment ID, or worker ID for a particular\nvideo, specify the --ids parameter to the vet command:\n\n    $ turkic find --id identifer --ids\n    $ turkic find --id identifer --frame frame --ids\n\nwill print a list of all the IDs for the video. If the corresponding segment\nhas been published and completed, it will list three strings: the HIT ID,\nassignment ID, and the worker ID. If the job has been published but not\nfinished, it will just list the HIT ID. If the job has not yet been published,\nit prints \"(not published)\".\n\nAdditionally, if you want to find the job that corresponds to a particular\nHIT ID, you can use the find command:\n\n    $ turkic find --hitid HITID\n\n## Quality Control \n\nThe gold standard does a \"pretty good\" job of weeding out bad workers.\nNonetheless, there will always be bad workers that we must identify and\ninvalidate. Our tool provides a method to sample the annotations provided by\nworkers, which you can then manually verify for correctness:\n\n    $ turkic sample /tmp\n\nwhich by default will pick 3 random videos that the worker has completed, and\npick 4 random frames from each of those videos, and write visualiations to a\nfile in /tmp. You can tweak the number of videos and the number of frames with\nthe options:\n\n    $ turkic sample /tmp --number 3 --frames 4\n\nMoreover, you can only look at work from a certain date:\n\n    $ turkic sample /tmp --since \"yesterday\"\n\nThe filename will follow the format of WORKERID-JOBID.jpg. Once you have\nidentified a mallicious worker, you can block them, invalidate ALL of their\nwork, and respawn their jobs with the command:\n\n    $ turkic invalidate workerid\n\nThe options are also available:\n\n    --no-block      invalidate and respawn, but don't block\n    --no-publish    block and invalidate, but don't respawn\n\nYou can also invalidate and respawn individual jobs with the command:\n\n    $ turkic invalidate --hit hitid\n\n## Listing all Videos \n\nYou can retrieve a list of all videos in the system with:\n\n    $ turkic list\n\nIf you want just the videos that have been published:\n\n    $ turkic list --published\n\nIf you want just the videos that have been worked on:\n\n    $ turkic list --completed\n\nIf you instead want the videos that are used for gold standard:\n\n    $ turkic list --training\n\nFinally, if you just want to count how many videos are in the system, use the\n--count option, in combination with any of the above:\n\n    $ turkic list --count\n    $ turkic list --published --count\n\nIf you want statistics about each video, then give the --stats option:\n\n    $ turkic list --stats\n\n## Managing Workers \n\nYou can list all known workers with the command:\n\n    $ turkic workers\n\nwhich will dump every worker with the number of jobs they have completed. You\ncan also use this command to block and unblock workers:\n\n    $ turkic workers --block workerid\n    $ turkic workers --unblock workerid\n\nYou can also search for workers by the first few letters of their ID:\n\n    $ turkic workers --search A3M\n\n## Deleting a Video \n\nYou can delete a video at any time with:\n\n    $ turkic delete identifier\n\nIf the video has already been annotated (even partially), this command will \nwarn you and abort. You can force deletion with:\n\n    $ turkic delete identifier --force\n\nwhich will REMOVE ALL DATA AND CANNOT BE UNDONE.\n\n# REFERENCES \n\nWhen using our system, please cite:\n\n    Carl Vondrick, Donald Patterson, Deva Ramanan. \"Efficiently Scaling Up\n    Crowdsourced Video Annotation\" International Journal of Computer Vision\n    (IJCV). June 2012.\n\n# FEEDBACK AND BUGS \n\nPlease direct all comments and report all bugs to:\n\n    Carl Vondrick\n    vondrick@mit.edu\n\nThanks for using our system!\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcvondrick%2Fvatic","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcvondrick%2Fvatic","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcvondrick%2Fvatic/lists"}