{"id":50358188,"url":"https://github.com/kingsdigitallab/framesense","last_synced_at":"2026-05-29T23:30:58.962Z","repository":{"id":299635397,"uuid":"1003666200","full_name":"kingsdigitallab/framesense","owner":"kingsdigitallab","description":"A modular command-line tool to process video collections.","archived":false,"fork":false,"pushed_at":"2026-05-11T17:03:26.000Z","size":93171,"stargazers_count":10,"open_issues_count":19,"forks_count":2,"subscribers_count":2,"default_branch":"main","last_synced_at":"2026-05-11T19:08:09.579Z","etag":null,"topics":["command-line-tool","reusable","video-processing"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/kingsdigitallab.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":"codemeta.json","zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-06-17T13:39:08.000Z","updated_at":"2026-05-11T17:05:39.000Z","dependencies_parsed_at":"2025-06-17T14:43:15.241Z","dependency_job_id":"45735abd-8255-4efa-aae3-58a165f04fda","html_url":"https://github.com/kingsdigitallab/framesense","commit_stats":null,"previous_names":["kingsdigitallab/framesense"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/kingsdigitallab/framesense","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kingsdigitallab%2Fframesense","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kingsdigitallab%2Fframesense/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kingsdigitallab%2Fframesense/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kingsdigitallab%2Fframesense/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/kingsdigitallab","download_url":"https://codeload.github.com/kingsdigitallab/framesense/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kingsdigitallab%2Fframesense/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33675019,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-05-29T02:00:06.066Z","response_time":107,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["command-line-tool","reusable","video-processing"],"created_at":"2026-05-29T23:30:57.692Z","updated_at":"2026-05-29T23:30:58.953Z","avatar_url":"https://github.com/kingsdigitallab.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# 🎞️FrameSense\n\nFrameSense is a highly modular command line tool designed to process your video collections.\n\n**Current focus**: consolidation phase aiming to improve stability, performance and architecture.\n\n## Requirements\n\n* python 3.10+\n* Docker or Singularity\n\n## Initial set Up\n\n1. Clone this repository anywhere on your system;\n2. Copy [`docs/collections.json`](docs/collections.json) to the folder containing your video collections and adapt its contents to your needs. All paths `collections.json` are relative to that file.\n3. Copy [`docs/.env`](docs/.env) in the folder that contains `framesense.py` and adapt the values to your environment. `FRAMESENSE_COLLECTIONS` should be the absolute path to your copy of `collections.json`.\n4. Create the python virtual environment: `python3 -m venv venv`\n5. Activate the virtual environment: `source venv/bin/activate`\n6. Install the required packages: `pip install -r requirements.txt`\n\n## Concepts\n\nIn FrameSense your collection is broken down into a hierarchy of smaller units. A `Collection` contains `videos`, which are made of `Clips`. Each Clip is a sequence of `Shots`. And each Shot is composed of a series of still `Frames`.\n\nEach operation provided by FrameSense works on a particular unit in that hierarchy.\n\nA video can be manually annotated in an `annotation file` that contains a list of annotations. An `annotation` describes and locates a clip within the video.\n\n## Expected folder structure of your collections\n\n* COLLECTIONS\n    * collections.json*\n    * COLLECTION1*\n        * VIDEO1*\n            * VIDEO1.mp4\n            * CLIP1\n                * CLIP1.mp4\n                * shots\n                    * SHOT_INDEX # three digits, zero-padded\n                        * shot.mp4\n                        * 01-XXX.jpg # first frame\n                        * 02-XXX.jpg # middle frame\n                        * 03-XXX.jpg # last frame\n                        * frames.json\n    * ANNOTATIONS1\n        * VIDEO1.json\n\nIn the above tree, a file or folder name in capital can be name whichever way you like. \nFile or folder with an asterisk are mandatory. Names in lowercase are predefined, you can't change them.\nInitially each video folder must have either a video file (e.g. godfather/godfather.mp4) or at least one clip file (e.g. godfather/godfather/baptism/baptism.mp4).\n\n![image](https://github.com/user-attachments/assets/c3d8ae45-505c-4ade-b0b0-58b0ccd4759e)\n\n\n## Usage\n\nThe tool offers a command-line interface to a series of modular operators acting on your video collections.\n\nTo see the list of available operators:\n\n`python framesense.py operators`\n\nFor instance, to see the collections and the videos they contain:\n\n`python framesense.py collections -v`\n\n### arguments\n\n* `-f FILTER` : only input files which path contain the FILTER string (case insensitive) will be processed by the operator\n* `-r` : forces the operator to **redo** the operation, even if the output already exists. **USE WITH CAUTION** as it can destroy outputs from previous operations\n* `-v` : **verbose** mode, print more stuff\n* `--dry-run` : does not make any change on disk. A way of testing an operation's scope before running it\n\nThe operator will return an error if you try to use an argument that it doesn't support (yet).\n\n## Architectural principles\n\n* **Modular**: open architecture so each operator has its own module and containerised dependencies; easy to extend or swap operations;\n* **Incremental**: an operation builds on top of outputs from other operations, enabling caching, reuse of intermediate results and custom pipelines;\n* **HPC-friendly**: non-interactive command line tool which is easy to install and run on SLURM;\n* **Portable**: should be easy to run on different machines, including lower-end personal computers\n\n## Operators\n\nFrameSense comes with a battery of built-in operators. \n\nFor a complete and up to date list, please use `python framesense.py operators`.\n\nCheck the README.md under each operator folder for a specification card. (Work in progress)\n\n### Built-in operators\n\n#### Information\n\n* **operators**:\n    List all available operators\n* **annotations**:\n    List all annotation files\n* **collections**:\n    List all collections\n\n#### Segmentation\n\n* **[make_clips_ffmpeg](operators/make_clips_ffmpeg/)**:\n    Extract clips from videos based on timecodes in annotation files\n* **[make_shots_scenedetect](operators/make_shots_scenedetect)**:\n    Extract shots from clips using PySceneDetect\n* **[make_frames_ffmpeg](operators/make_frames_ffmpeg)**:\n    Extract frames from shots using ffmpeg\n\n#### Detection\n\n* **[scale_frames_sssabet](operators/scale_frames_sssabet)**:\n    Shot scale classification from frames based on https://github.com/sssabet/Shot_Type_Classification\n\n#### Sound\n\n* **[extract_sound_ffmpeg](operators/extract_sound_ffmpeg/)**:\n    Extract clip audio channel into a sound file\n\n#### Transcoding\n\n* **[transcode_clips_ffmpeg](operators/transcode_clips_ffmpeg/)**:\n    Convert a clip from one format to another\n\n#### Question answering\n\n* **[answer_videos_vlm](operators/answer_videos_vlm/)**:\n    Answer questions about a video file using a video/vision language model\n* **[answer_transcription_ollama](operators/answer_transcription_ollama/)**:\n    Answer questions about a clip transcription using a large language model\n* **[answer_frames_vlm](operators/answer_frames_vlm/)**:\n    Answer questions about a frame using a vision language model\n\n### Design principles\n\nIt is expected that each operator:\n* is functionally minimal (\"does one thing and does it well\");\n* is atomic (only valid and complete output are persisted);\n* implements a single method or strategy;\n* should only process the input if its output doesn't already exist;\n* uses containers to isolate its software dependencies;\n* works on all files at one specific level in the hierarchy (e.g. make_shots splits all your clips into shots);\n* has a name which reflects what it does, on what unit, with which method (e.g. make_clips_ffmpeg);\n* is written as a Python class within a module `operator.py` under a package which name matches the name of the operator (e.g. `operators/make_clips_ffmpeg/operator.py`);\n* inherits from the [base operator](operators/base/operator.py);\n\n### Containers\n\nAny software dependency needed for an operator to run \nshould be packaged into a container image. \nFrameSense supports Docker and Singularity container engines.\n\n**The first time an operator is executed the container image \nwill be built** from its Dockerfile.\nAfter that, the container image will only be rebuilt \nif the Dockerfile has changed.\n\nIf the operator folder contains an `./app` directory,\nit will be mounted to `/app` within the container.\n\n#### Singularity\n\nWhen using Singularity (instead of Docker), \nthe Dockerfile is converted to a .def file \nbefore the conainer image is built. \nAll singularity definition and image files \nare generated and stored under the `./singularity` folder.\n\nIt is possible to build the images on one machine \nand copy the `singularity` folder across to another\nso they are ready to be used.\n\nThe build process will be offloaded to \nthe remote Singularity endpoint \nyou are logged into.\nIf you are not logged into a remote endpoint\nthe build process uses the `--fakeroot` method.\nBut this method requires privileges \nonly available on a personal computer.\n\nOn a HPC environment with Singularity\nyou can either log into a remote endpoint\nor copy the `singularity` folder from another machine.\n\nIf you are not logged and `--fakeroot` is not allowed,\nyou may encounter this error:\n\n`FATAL:   fakeroot requires to set 'allow setuid = yes' in /etc/singularity/singularity.conf`\n\n## Features and Bugs\n\nPlease use [github issue tracker](https://github.com/kingsdigitallab/framesense/issues) to report bugs or request new features.\n\nThis tool is currently being developed primarily to serve the needs of the [ISSA research project](https://github.com/kingsdigitallab/issa). \nTickets related to ISSA will therefore take priority. \nUnrelated tickets are welcome but we can't guarantee that they will be addressed promptly or at all\nuntil FrameSense receives more dedicated support (external contributors or additional funding).\n\n## Performance and HPC\n\nWe are aiming to support low end (laptop) and high end (HPCs) compute environment. \nHowever at the moment support is limited and the operators are very slow.\n\n## Testing\n\nSee [tests/README.md](tests/README.md) for details.\n\n## Environment variables\n\nVariables affecting how FrameSense works can be set \nin your session environment\nor the `.env` file located in the same folder as framesense.py.\n\nYou can point to a different `.env` file \nby setting the absolute or relative (to `framesense.py`) path \nin the `FRAMESENSE_DOTENV_PATH` environment variable.\n\n[`./docs/.env`](./docs/.env) is a template \nthat contains a list of all available variables and their default values.\n\nIf CUDA_VISIBLE_DEVICES='' \nthe containers will not be able to access the host's GPU.\n\nAny operator parameter can be passed as an environment variable.\nThe name is OPERATOR_PARAM, \nwhere OPERATOR is the uppercase name of the operator,\nand PARAM the uppercase name of the parameter.\nFor instance ANSWER_FRAMES_VLM_MODEL=\"qwen3-vl:2b\" \nto set the `model` param of `answer_frames_vlm` to \"qwen3-vl:2b\".\n\n## Operator parameters\n\nThe value of any operator parameter originates from these sources, \nin the following order of precedence:\n* environment variable (see above)\n* variable in `.env` file\n* value under meta.params.OPERATOR.PARAM in the collection json file (note that OPERATOR and PARAM values should be in lower case, e.g. `answer_frames_vlm` and `model`)\n* the default value set in the operator's `params.json` file\n\n## GPU access\n\nDocker containers can access the host GPU \nif the [nvidia container toolkit](https://docs.nvidia.com/ai-enterprise/deployment/vmware/latest/docker.html) is installed.\n\nTo check support run this command:\n\n`docker run --rm --gpus all alpine`\n\nIf it returns an error related to gpu in the message \nthe toolkit might be missing.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkingsdigitallab%2Fframesense","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkingsdigitallab%2Fframesense","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkingsdigitallab%2Fframesense/lists"}