{"id":22039113,"url":"https://github.com/timm/tested","last_synced_at":"2025-09-12T08:40:10.468Z","repository":{"id":64532421,"uuid":"569981645","full_name":"timm/tested","owner":"timm","description":"The best way to test \"it\" is to watch someone else try to break \"it\".","archived":false,"fork":false,"pushed_at":"2023-04-04T05:15:30.000Z","size":101307,"stargazers_count":1,"open_issues_count":1,"forks_count":24,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-05-07T21:35:34.635Z","etag":null,"topics":["ai","data-mining","lua"],"latest_commit_sha":null,"homepage":"https://timm.github.io/tested","language":"Lua","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-2-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/timm.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":"CITATION.cff","codeowners":null,"security":null,"support":null,"governance":null,"roadmap":"ROADMAP.md","authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2022-11-24T04:05:46.000Z","updated_at":"2023-04-30T17:25:41.000Z","dependencies_parsed_at":"2025-04-13T03:49:27.384Z","dependency_job_id":null,"html_url":"https://github.com/timm/tested","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/timm/tested","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/timm%2Ftested","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/timm%2Ftested/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/timm%2Ftested/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/timm%2Ftested/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/timm","download_url":"https://codeload.github.com/timm/tested/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/timm%2Ftested/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":265080256,"owners_count":23708121,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai","data-mining","lua"],"created_at":"2024-11-30T11:09:27.152Z","updated_at":"2025-07-13T02:10:54.022Z","avatar_url":"https://github.com/timm.png","language":"Lua","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003csmall\u003e\u003cp\u003e\u0026nbsp;\n\u003ca name=top\u003e\u003c/a\u003e\n\u003ctable\u003e\u003ctr\u003e \n\u003ctd\u003e\u003ca href=\"/README.md#top\"\u003ehome\u003c/a\u003e\n\u003ctd\u003e\u003ca href=\"/ROADMAP.md\"\u003eroadmap\u003c/a\u003e\n\u003ctd\u003e\u003ca href=\"http:github.com/timm/tested/issues\"\u003eissues\u003c/a\u003e\n\u003ctd\u003e \u003ca href=\"/LICENSE.md\"\u003e\u0026copy;2022,2023\u003c/a\u003e by \u003ca href=\"http://menzies.us\"\u003etim menzies\u003c/a\u003e\n\u003c/tr\u003e\u003c/table\u003e\u003c/small\u003e\n\u003cimg  align=center width=600 src=\"/docs/img/banner.png\"\u003e\u003c/p\u003e\n\u003cp\u003e \u003cimg src=\"https://img.shields.io/badge/task-ai-blueviolet\"\u003e\u003ca\nhref=\"https://github.com/timm/tested/actions/workflows/tests.yml\"\u003e \u003cimg \n src=\"https://github.com/timm/tested/actions/workflows/tests.yml/badge.svg\"\u003e\u003c/a\u003e \u003cimg \n src=\"https://img.shields.io/badge/language-lua-orange\"\u003e \u003cimg \n src=\"https://img.shields.io/badge/purpose-teaching-yellow\"\u003e \u003ca \n href=\"https://zenodo.org/badge/latestdoi/569981645\"\u003e \u003cimg \n src=\"https://zenodo.org/badge/569981645.svg\" alt=\"DOI\"\u003e\u003c/a\u003e\u003c/p\u003e\n\n\u003c!-- table\u003e\n\u003ctr\u003e\u003ctd\u003eStudy guide \u003c/td\u003e\u003ctd\u003eSub-term\n\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd\u003edata \u003c/td\u003e\u003ctd\u003e\ndata: row (a.k.a. example, instance);\ncolumn (a.k.a. attribute, feature)\n\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd\u003elearning type \u003c/td\u003e\u003ctd\u003e\nunsupervised or semi-supervised or supervised;\u003cbr\u003e\ninstance or model-based\u003cbr\u003e\n\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd\u003ealgorothm \u003c/td\u003e\u003ctd\u003e\ndecision tree learning;\nclustering;\nFASTMAP (and cosine rule)\u003cbr\u003e\n\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd\u003eaccountability \u003c/td\u003e\u003ctd\u003e\nlogging, watched, presence\u003cbr\u003e\n\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd\u003emulti-objective reasoning \u003c/td\u003e\u003ctd\u003e\ndomination, Pareto frontier, Zitzler predicate\u003cbr\u003e\n\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd\u003emisc \u003c/td\u003e\u003ctd\u003e\nstakeholder,\nmanifold, labelling cost, curse of dimensionality, explanation\n\u003c/td\u003e\u003c/tr\u003e\n\u003c/table --\u003e\n\n\u003chr\u003e\n\nTESTED is  a semi-supervised, multi-objective, model-based\nexplanation system.\nThe code is a refactoring of decades of work by dozens of Ph.D. students.\n\nTESTED assumes that the best way to test \"it\" is to watch someone else \n(specifically, stakeholders [^stake])\ntry to break \"it\".  TESTED lets  people  explore more and fix\nmore (but  sampling less around a system).\n\nTESTED is not a pretty GUI. Rather, it is a programming toolkit that shows that\nthese kinds of tools are (very) simple to build. For example, everything\nhere is just a few hundred lines of LUA (or less). And all of that code\nshares the same similar structures (so once you can code on tool, you can\ncode much of the others).\n\n[^stake]: Definition: \"Stakeholders\"  are individuals or organizations having\n  a right, share, claim, or interest in a system or in its possession\n  of characteristics that meet their needs and expectations \n  [(ISO/IEC/IEEE\n  2015)](https://www.iso.org/standard/63711.html).\n\n## Frequently Asked Questions\n\nSo what is TESTED really about?\n- is it about how to reconfigure broken things to make them better? \n- is it about requirements engineering?\n- is it about software engineering?\n- is it about configuration?\n- is it about data mining?\n- is it about testing?\n\nTo which the answer is \"yes\". All these things share the same underlying\nmethods and challenges. Which means tools built for one of these tasks\ncan help the other [^duo][^abduction]. \n\nTESTED aims to support \"stakeholder testing\". This  is  a kind of black box tested aimed at offering a big picture summary of some code, not\njust for the developers, but also for those that have to use the software. More than just acceptance testing (which is usually\nsome contractual thing), stakeholder testing aims to measure then mitigate problems with the system. That is, normal testing\njust finds bugs while stakeholder testing finds gradients along which the system can slip up or down to better or worse behavior\n\n# Install\n\nInstall Lua:\n\n    brew install lua       # on max os/x\n    lua -v                 # TESTED needs at least LUA version 5.3\n\nCheck-out this repo\n\n    git clone https://github.com/timm/tested\n\nCheck your installation\n\n    cd tested/src\n    lua 101.lua -g all # should show one crash and several passes\n\n# What is TESTED??\n\n\nWhen people like me\n(i.e. a developer) write software that \nis used by other people \n(i.e. the stakeholders), those other people should be able to\nverify that the built software is right,\nand validate that the right software is being built.\n\nSuch \"stakeholder testing\" is challenging since,\noften, stakeholders may\nnot understand everything  about what goes on inside the code.\nHence stakeholder testing  needs special kinds of tools\nthat helps helps\n   humans find the   best things or fix the worst things;\nwithout\n  having to offer too much information on each thing. \n\nThe central claim of TESTED is that these:\n**tools are surprisingly easy to build**. To say that another way:\n- people can (and should) understand AI systems;\n- then use those systems to build a better world.\n\nEvery tool is less than a few hundred lines of LUA code and all those tools share most of the same internal structure.\nStudents can learn this simpler approach to AI as a set of weekly homeworks where they recode the tools\nin any language at all (except LUA).\nThen, for graduate students, they can also do a final four week project\nwhere they try  to improve on a stakeholder testing tool called \"fishing\", provided in this kit.\n\n(Just an aside, the way I do homeworks is that every week, everyone\nhas to submit something, even if it is broken. Homeworks can get submitted multiple times\nso I grade them \"2\" (for \"good\"); \"1\" (for \"invited to resubmit\"); \"0\" (for \"bad\" or \"no submission\".)\n\n## What is \"STAKEHOLDER TESTING\"\n\nBetter methods for better searching for better solutions is important.\nThere are too many examples  of terrible software solutions.\nFor example:\n- Amazon had to scrap an automated recruiting tool as it \n  was found to be [biased against women](https://reut.rs/2Od9fPr).\n- A widely used face recognition software was found to be biased against \n  [dark-skinned women](https://news.mit.edu/2018/study-finds-gender-skin-type-bias-artificial-intelligence-systems-0212) and\n  [dark-skinned men](https://www.nytimes.com/2020/06/24/technology/facial-recognition-arrest.html).\n- Google Translate, the most popular translation engine in the world, \n  [shows gender bias](https://science.sciencemag.org/content/356/6334/183). \n  “She is an engineer, \n  He is a nurse” is translated into Turkish and then again into English becomes “He is an engineer, She is a nurse” [5].  \n- Chapter six of Safiya Noble’s book Algorithms of Oppression [^noble] \n  tells the sad tale of  how a design quirk of  Yelp ruined a small business:\n  As one of Noble’s interviewees put it \"Black people don’t ‘check in’ and let people know where they’re at when they sit in my (hair dressing salon). \n  They already feel like they are being hunted;  they aren't going to tell the Man where they are\". \n  Hence, that salon fell in the Yelp ratings (losing customers) since its patrons rarely  pressed the   “checked-in”  button. \n\nFor our purposes, the  important point of the Noble example\nis this:\n- if software designers had been more intentional about\nsoliciting feedback from the Black community...\n- then they could have\nchanged how check-ins are weighted in the overall Yelp rating system.\n\nAs to the other examples, in each case there was some discriminatory\neffect which was easy to detect and repair [^joy]], but developers\njust failed to test for those biases.\n\nThere is a solution to all these problems\n- if a small group of\npeople build software for the larger community\n- they need to listen\nmore to the  concerns of the larger community. \n\nFor that to work,\nthe smaller group of developers have to admit the larger group into\ntheir design processes– either via\n- changing the reward structures\n  such that there are inducements for the few to listen to the many\n  (e.g. by better government legislation or  professional standards);\n  or \n- inclusion practices that admits the broader community into\n  the developer community, or by \n- review practices where the\n  developers can take better and faster feedback from the community.\n\nThe first two of these points requires major organization\nchanges. This repository is more about the third point which can we said another way:\nfrom an ethical perspective, it is good\npractice to give software to stakeholders and let them try to break\nit. \n\n[^noble]: Noble, Safiya Umoja. \"Algorithms of oppression.\" Algorithms of Oppression. New York University Press, 2018.\n[^joy]: Chakraborty, Joymallya, Suvodeep Majumder, and Tim Menzies. \"Bias in machine learning software: why? how? what to do?.\" Foundations of Software Engineering, 2021\n\n\n\n\n[^abduction]: For more on the mysterious machine that runs deep\n  within testing, SE, requirements engineering, configuration, etc,\n  see [my Ph.D. thesis](https://menzies.us/pdf/96abkl.pdf). In summary,\n  by the time you can test \"it\" then you can also exercise \"it\";\n  i.e. properly designed, a good test engine is also a good\n  execution engine. For years I tried coding all this up in a logical\n  framework. Then I found ways to use data mining for very faster,\n  scalable, approximate reasoning. So now I offer my private\n  theory-of-everything in a procedural framework, embedded with some\n  data mining tools. Specifically, data miners divide a space and\n  optimizers tell you how to jump around that space.\n\n[^duo]: [Better Software Analytics via \"DUO\": Data Mining Algorithms Using/Used-by Optimizers](https://arxiv.org/pdf/1812.01550.pdf)\n  Amritanshu Agrawal, Tim Menzies, Leandro L. Minku, Markus Wagner, and Zhe Yu. 2020. \n  Empirical Softw. Engg. 25, 3 (May 2020), 2099–2136. https://doi.org/10.1007/s10664-020-09808-9\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftimm%2Ftested","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftimm%2Ftested","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftimm%2Ftested/lists"}