{"id":28022854,"url":"https://github.com/trailofbits/datasig","last_synced_at":"2026-02-18T13:02:50.447Z","repository":{"id":286143752,"uuid":"884147076","full_name":"trailofbits/datasig","owner":"trailofbits","description":"Dataset fingerprinting for AIBOM","archived":false,"fork":false,"pushed_at":"2026-02-11T13:53:56.000Z","size":1096,"stargazers_count":15,"open_issues_count":4,"forks_count":0,"subscribers_count":4,"default_branch":"main","last_synced_at":"2026-02-11T21:49:02.302Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"agpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/trailofbits.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2024-11-06T08:26:11.000Z","updated_at":"2026-02-11T13:53:24.000Z","dependencies_parsed_at":"2025-04-04T16:27:51.523Z","dependency_job_id":"4bd3bac5-4d02-495d-954b-c43a39570bcf","html_url":"https://github.com/trailofbits/datasig","commit_stats":null,"previous_names":["trailofbits/datasig"],"tags_count":6,"template":false,"template_full_name":null,"purl":"pkg:github/trailofbits/datasig","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/trailofbits%2Fdatasig","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/trailofbits%2Fdatasig/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/trailofbits%2Fdatasig/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/trailofbits%2Fdatasig/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/trailofbits","download_url":"https://codeload.github.com/trailofbits/datasig/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/trailofbits%2Fdatasig/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29580643,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-18T08:38:15.585Z","status":"ssl_error","status_checked_at":"2026-02-18T08:38:14.917Z","response_time":162,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-05-10T23:01:10.017Z","updated_at":"2026-02-18T13:02:50.431Z","avatar_url":"https://github.com/trailofbits.png","language":"Python","funding_links":[],"categories":["Defense \u0026 Security Controls"],"sub_categories":["Data \u0026 Supply Chain Security"],"readme":"# Dataset fingerprinting\nThis repository contains our proof-of-concept for fingerprinting a dataset.\n\n## Local installation\n```python\ngit clone https://github.com/trailofbits/datasig \u0026\u0026 cd datasig\nuv sync\n```\n\n## Usage\n### Fingerprinting\nThe code below shows experimental usage of the library.\nThis will be subject to frequent changes in early development stages. \n\n```python\nfrom torchvision.datasets import MNIST\nfrom datasig.dataset import TorchVisionDataset\nfrom datasig.algo import KeyedShaMinHash, UID\n\ntorch_dataset = MNIST(root=\"/tmp/data\", train=True, download=True)\n\n# Wrap the dataset with one of the classes in `datasig.dataset`.\n# These classes provide a uniform interface to access serialized data points.\ndataset = TorchVisionDataset(torch_dataset)\n\n# Pass the dataset to the fingerprinting algorithm.\nprint(\"Dataset UID: \", UID(dataset).digest())\nprint(\"Dataset fingerprint: \", KeyedShaMinHash(dataset).digest())\n```\n\n### Manual serialization/deserialization\nThe dataset classes defined in `datasig.dataset` provide static serialization\nand deserialization to convert datapoints between their usual representation\nand bytes.\n\n```python\nfrom torchvision.datasets import MNIST\nfrom datasig.dataset import TorchVisionDataset\n\ntorch_dataset = MNIST(root=\"/tmp/data\", train=True, download=True)\n\n# Serializing data points to bytes\nserialized = TorchVisionDataset.serialize_data_point(torch_dataset[0])\n\n# Deserializing data points from bytes\ndeserialized = TorchVisionDataset.deserialize_data_point(serialized)\n```\n\n## Development\n### Unit tests\nTests are in the `datasig/test` directory. You can run the tests with:\n\n```bash\nuv run python -m pytest # Run all tests\nuv run python -m pytest -s datasig/test/test_csv.py # Run only one test file\nuv run python -m pytest -s datasig/test/test_csv.py -k test_similarity # Run only one specific test function\n```\n\n### Profiling\nThe profiling script generates a profile for dataset processing and fingerprint generation using cProfile. To profile the MNIST dataset from the torch framework,\nyou can run:\n\n```bash\nuv run python profiling.py torch_mnist --full\n```\n\nThe `--full` argument tells the script to include dataset canonization, UID generation, and fingerprint generation in the profile. If you want to profile only some of these steps you can cherry pick by using or omitting the following arguments instead:\n\n```bash\nuv run python profiling.py torch_mnist --canonical --uid --fingerprint\n```\n\nYou can optionally specify the datasig config version to use (at the time of writing we have only v0) with:  \n\n```bash\nuv run python profiling.py torch_mnist -v 0 --all\n```\n\nCurrently we support only one target dataset: `torch_mnist`. To add another dataset, add a class in `profiling.py` similar to `TorchMNISTV0`, that implements the `_setup()` method which is responsible for loading the dataset.\n\n### Benchmarking\n\n!!! This is currently broken !!!\n\nDatasig has a built-in `benchmark` module that allows to run experiments to benchmark speed and accuracy of various fingerprinting methods with varying configurations and on several datasets.\n\nBenchmarks are configured programmatically using the `datasig` library directly.\nThe `benchmarking.py` script gives a comprehensive overview of how to configure and run a benchmark, export results, as well as plot them on graph.\n\nYou can run the example benchmark with\n\n```bash\nuv run python benchmarking.py\n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftrailofbits%2Fdatasig","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftrailofbits%2Fdatasig","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftrailofbits%2Fdatasig/lists"}