{"id":18906779,"url":"https://github.com/oduwsdl/off-topic-memento-toolkit","last_synced_at":"2025-10-09T13:32:39.752Z","repository":{"id":57449833,"uuid":"121306624","full_name":"oduwsdl/off-topic-memento-toolkit","owner":"oduwsdl","description":"This system evaluates a collection of mementos (archived web pages) to determine which are off topic. The collection can be part of an Archive-It collection, a single TimeMap, or stored in a WARC file.","archived":false,"fork":false,"pushed_at":"2021-12-19T18:04:15.000Z","size":98216,"stargazers_count":9,"open_issues_count":2,"forks_count":3,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-10-09T13:32:23.177Z","etag":null,"topics":["cosine","measure","memento","simhash","timemap","topic","warc"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/oduwsdl.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":"CITATION.cff","codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2018-02-12T21:51:42.000Z","updated_at":"2024-06-04T08:21:04.000Z","dependencies_parsed_at":"2022-09-26T17:31:17.210Z","dependency_job_id":null,"html_url":"https://github.com/oduwsdl/off-topic-memento-toolkit","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"purl":"pkg:github/oduwsdl/off-topic-memento-toolkit","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oduwsdl%2Foff-topic-memento-toolkit","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oduwsdl%2Foff-topic-memento-toolkit/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oduwsdl%2Foff-topic-memento-toolkit/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oduwsdl%2Foff-topic-memento-toolkit/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/oduwsdl","download_url":"https://codeload.github.com/oduwsdl/off-topic-memento-toolkit/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oduwsdl%2Foff-topic-memento-toolkit/sbom","scorecard":{"id":702738,"data":{"date":"2025-08-11","repo":{"name":"github.com/oduwsdl/off-topic-memento-toolkit","commit":"77cba436a6219fab8b8b7c773c3fc273388a8a2c"},"scorecard":{"version":"v5.2.1-40-gf6ed084d","commit":"f6ed084d17c9236477efd66e5b258b9d4cc7b389"},"score":2.8,"checks":[{"name":"Packaging","score":-1,"reason":"packaging workflow not detected","details":["Warn: no GitHub/GitLab publishing workflow detected."],"documentation":{"short":"Determines if the project is published as a package that others can easily download, install, easily update, and uninstall.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#packaging"}},{"name":"Dangerous-Workflow","score":-1,"reason":"no workflows found","details":null,"documentation":{"short":"Determines if the project's GitHub Action workflows avoid dangerous patterns.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#dangerous-workflow"}},{"name":"Maintained","score":0,"reason":"0 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project is \"actively maintained\".","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#maintained"}},{"name":"Token-Permissions","score":-1,"reason":"No tokens found","details":null,"documentation":{"short":"Determines if the project's workflows follow the principle of least privilege.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#token-permissions"}},{"name":"Code-Review","score":0,"reason":"Found 0/12 approved changesets -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project requires human code review before pull requests (aka merge requests) are merged.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#code-review"}},{"name":"Binary-Artifacts","score":10,"reason":"no binaries found in the repo","details":null,"documentation":{"short":"Determines if the project has generated executable (binary) artifacts in the source repository.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#binary-artifacts"}},{"name":"CII-Best-Practices","score":0,"reason":"no effort to earn an OpenSSF best practices badge detected","details":null,"documentation":{"short":"Determines if the project has an OpenSSF (formerly CII) Best Practices Badge.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#cii-best-practices"}},{"name":"Vulnerabilities","score":10,"reason":"0 existing vulnerabilities detected","details":null,"documentation":{"short":"Determines if the project has open, known unfixed vulnerabilities.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#vulnerabilities"}},{"name":"Fuzzing","score":0,"reason":"project is not fuzzed","details":["Warn: no fuzzer integrations found"],"documentation":{"short":"Determines if the project uses fuzzing.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#fuzzing"}},{"name":"Security-Policy","score":0,"reason":"security policy file not detected","details":["Warn: no security policy file detected","Warn: no security file to analyze","Warn: no security file to analyze","Warn: no security file to analyze"],"documentation":{"short":"Determines if the project has published a security policy.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#security-policy"}},{"name":"Pinned-Dependencies","score":0,"reason":"dependency not pinned by hash detected -- score normalized to 0","details":["Warn: containerImage not pinned by hash: Dockerfile:1: pin your Docker image by updating python:3.7.3-stretch to python:3.7.3-stretch@sha256:205cafadbb7f0c48f2a6d655eb100a9675aa85e6e73958860f488406cce09779","Warn: pipCommand not pinned by hash: Dockerfile:7","Info:   0 out of   1 containerImage dependencies pinned","Info:   0 out of   1 pipCommand dependencies pinned"],"documentation":{"short":"Determines if the project has declared and pinned the dependencies of its build process.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#pinned-dependencies"}},{"name":"License","score":10,"reason":"license file detected","details":["Info: project has a license file: LICENSE.txt:0","Info: FSF or OSI recognized license: MIT License: LICENSE.txt:0"],"documentation":{"short":"Determines if the project has defined a license.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#license"}},{"name":"Signed-Releases","score":-1,"reason":"no releases found","details":null,"documentation":{"short":"Determines if the project cryptographically signs release artifacts.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#signed-releases"}},{"name":"Branch-Protection","score":0,"reason":"branch protection not enabled on development/release branches","details":["Warn: branch protection not enabled for branch 'master'"],"documentation":{"short":"Determines if the default and release branches are protected with GitHub's branch protection settings.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#branch-protection"}},{"name":"SAST","score":0,"reason":"SAST tool is not run on all commits -- score normalized to 0","details":["Warn: 0 commits out of 19 are checked with a SAST tool"],"documentation":{"short":"Determines if the project uses static code analysis.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#sast"}}]},"last_synced_at":"2025-08-22T05:35:04.382Z","repository_id":57449833,"created_at":"2025-08-22T05:35:04.383Z","updated_at":"2025-08-22T05:35:04.383Z"},"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":279001480,"owners_count":26083102,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-09T02:00:07.460Z","response_time":59,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cosine","measure","memento","simhash","timemap","topic","warc"],"created_at":"2024-11-08T09:18:41.977Z","updated_at":"2025-10-09T13:32:39.711Z","avatar_url":"https://github.com/oduwsdl.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![Build Status](https://travis-ci.org/oduwsdl/off-topic-memento-toolkit.svg?branch=master)](https://travis-ci.org/oduwsdl/off-topic-memento-toolkit)\n\nGiven a collection of archived web pages, known as mementos, the Off Topic Memento Toolkit (OTMT) allows one to determine which mementos are off-topic. Mementos are produced by crawling live web pages, resulting in collections that often contain different versions of the same web page. In time, due to hacking, loss of ownership of the domain, or even website restructuring, a web page can go off-topic, resulting in the collection containing off-topic mementos. The OTMT helps users detect these off-topic mementos before conducting research on their collections of archived web pages.\n\nThis code is based on work by:\n\nAlNoamany, Y., Weigle, M.C. \u0026 Nelson, M.L. Detecting off-topic pages within TimeMaps in Web archives. *International Journal on Digital Libraries* (2016) 17: 203. https://doi.org/10.1007/s00799-016-0183-5\n\nand an early version of it was evaluated in:\n\nJones, S.M., Weigle, M.C \u0026 Nelson, M.L. The Off-Topic Memento Toolkit. *Proceedings of the 15th International Conference on Digital Preservation* (2018) https://arxiv.org/abs/1806.06870\n\n# Quick start\n\nThe software is now available on PyPI **and requires a minimum of Python 3.6**:\n\n`pip install otmt`\n\nThis installs the `detect_off_topic` command on your system, along with the `offtopic` Python library. To determine if the content in Archive-It collection is off-topic:\n\n`detect_off_topic -i archiveit=7877 -o outputfile.json`\n\nThis stores the information about each memento and TimeMap of Archive-It collection 7877 in a JSON-formatted file named `output.json`.\n\n# More details\n\n## Input types\n\nTo accomplish this, OTMT accepts the following inputs:\n* an Archive-It collection ID\n* URIs for one or more Memento TimeMaps (see RFC 7089)\n* one or more files in Web ARChive (WARC) format (see ISO 28500)\n\nThese inputs are converted internally into a series of files and folders used for the rest of the evaluation.\n\nTo specify an Archive-It collection use the `archiveit` keyword followed by an = and the collection ID, like so:\n`detect_off_topic -i archiveit=7877 -o outputfile.json`\n\nFor one or more TimeMaps, specify them with the `timemap` keyword followed by an = and the URI-T of the TimeMap:\n`detect_off_topic -i timemap=http://archive.example.org/timemap1,http://archive.example.org/timemap2 -o outputfile.json`\n\nLikewise, for one or more WARC files:\n`detect_off_topic -i warc=example1.warc.gz,example2.warc.gz -o outputfile.json`\n\n### Notes For WARCs:\nWhen analyzing WARCs, OTMT:\n1. generates a TimeMap for each seed it encounters\n2. generates an entry in this TimeMap for the seed with an internal URI-M and a memento-datetime derived from record's WARC-Date header\n3. once done with all WARCs, presents the TimeMaps as if they had been downloaded to the measure part of the architecture\n4. The measure part of the architecture (currently) takes the first memento and compares it to each other memento in the TimeMap\n\nIn summary, OTMT pulls in all of the data from all WARCs and analyzes it together. For TimeMap Measures, it uses the first memento in the TimeMap as the basis for comparison with other mementos in the TimeMap.\n\nThere is no URI canonicalization like the Internet Archive does with URIs like edition.cnn.com and www.cnn.com. OTMT does not compare example.org and example2.org because they would have separate URI-Rs and hence separate TimeMaps.\n\n### Notes For TimeMaps:\n\nIn this case, OTMT does not aggregate data from all TimeMaps, instead treating them separately. If two TimeMaps serve the same URI-R, then OTMT does not reconcile them into one. Such functionality would indeed be useful. If added, one could compare TimeMaps for the same URI-R across multiple collections or even multiple archives. If this functionality is desired, [please request it as a feature](https://github.com/oduwsdl/off-topic-memento-toolkit/issues).\n\n## TimeMap Measures\nWith TimeMap measures, each memento in a TimeMap is compared to the first memento of that TimeMap. The comparison is performed using one or more of the following measures:\n* Cosine Similarity (keyword: `cosine`) - this is the default, combined with wordcount\n* Word Count (keyword: `wordcount`) - this is the default, combined with cosine\n* Byte Count (keyword: `bytecount`)\n* Simhash on the raw memento content (keyword: `raw_simhash`)\n* Simhash on the term frequencies of the raw memento content (keyword: `tf_simhash`)\n* Jaccard Distance (keyword: `jaccard`)\n* Sørensen-Dice Distance (keyword: `sorensen`)\n* Latent Semantic Indexing with Gensim (keyword: `gensim_lsi`)\n\nTimeMap measures are specified by the `-tm` argument followed by the keyword of the desired measure. Optionally, one can specify a threshold value followed by a =, like so:\n\n`detect_off_topic -i archiveit=7877 -o outputfile.json -tm jaccard=0.80`\n\nMultiple measures can be specified, separated by commas:\n\n`detect_off_topic -i archiveit=7877 -o outputfile.json -tm jaccard=0.80,bytecount=-0.50`\n\nIf a threshold value is not specified the hard-coded default values are used.\n\n## Output file formats\n\nThe output JSON file has the following format:\n```\n{\n  \"URI of a TimeMap\": {\n    \"URI of a Memento\": {\n      \"timemap measures\": {\n        \"[name of measure]\": {\n          \"comparison score\": [score],\n          \"topic status\": [\"on-topic\"|\"off-topic\"]\n          }\n        }\n      }\n    }\n    ...\n```\n\nCSV output is also supported via the `-ot` option:\n`detect_off_topic -i archiveit=7877 -o outputfile.csv -ot csv`\n\n## Installing for development\n\nTo run the tests associated with the OTMT, execute:\n`python ./setup.py test`\n\nTo install to run locally, run (within the base of the source directory):\n`pip install .`\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Foduwsdl%2Foff-topic-memento-toolkit","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Foduwsdl%2Foff-topic-memento-toolkit","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Foduwsdl%2Foff-topic-memento-toolkit/lists"}