{"id":31382264,"url":"https://github.com/empiricalci/emp","last_synced_at":"2025-09-28T11:32:25.011Z","repository":{"id":140802423,"uuid":"47239760","full_name":"empiricalci/emp","owner":"empiricalci","description":":microscope: Empirical CLI","archived":false,"fork":false,"pushed_at":"2020-09-10T17:57:55.000Z","size":239,"stargazers_count":42,"open_issues_count":15,"forks_count":7,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-08-23T19:32:58.433Z","etag":null,"topics":["ai","artificial-intelligence","benchmark-framework","benchmarking","docker","reproducible-research","reproducible-science","science"],"latest_commit_sha":null,"homepage":"https://empiricalci.com","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/empiricalci.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2015-12-02T05:44:31.000Z","updated_at":"2024-01-04T16:02:14.000Z","dependencies_parsed_at":null,"dependency_job_id":"a8d70312-f121-4dc0-9f7d-d3870c5eb1d5","html_url":"https://github.com/empiricalci/emp","commit_stats":{"total_commits":109,"total_committers":2,"mean_commits":54.5,"dds":0.00917431192660545,"last_synced_commit":"93cadafa273a747531a23854e91842fc47ee6c8c"},"previous_names":[],"tags_count":17,"template":false,"template_full_name":null,"purl":"pkg:github/empiricalci/emp","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/empiricalci%2Femp","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/empiricalci%2Femp/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/empiricalci%2Femp/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/empiricalci%2Femp/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/empiricalci","download_url":"https://codeload.github.com/empiricalci/emp/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/empiricalci%2Femp/sbom","scorecard":{"id":376241,"data":{"date":"2025-08-11","repo":{"name":"github.com/empiricalci/emp","commit":"93cadafa273a747531a23854e91842fc47ee6c8c"},"scorecard":{"version":"v5.2.1-40-gf6ed084d","commit":"f6ed084d17c9236477efd66e5b258b9d4cc7b389"},"score":3.2,"checks":[{"name":"Packaging","score":-1,"reason":"packaging workflow not detected","details":["Warn: no GitHub/GitLab publishing workflow detected."],"documentation":{"short":"Determines if the project is published as a package that others can easily download, install, easily update, and uninstall.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#packaging"}},{"name":"Dangerous-Workflow","score":-1,"reason":"no workflows found","details":null,"documentation":{"short":"Determines if the project's GitHub Action workflows avoid dangerous patterns.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#dangerous-workflow"}},{"name":"Token-Permissions","score":-1,"reason":"No tokens found","details":null,"documentation":{"short":"Determines if the project's workflows follow the principle of least privilege.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#token-permissions"}},{"name":"Maintained","score":0,"reason":"0 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project is \"actively maintained\".","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#maintained"}},{"name":"Code-Review","score":0,"reason":"Found 0/11 approved changesets -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project requires human code review before pull requests (aka merge requests) are merged.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#code-review"}},{"name":"Binary-Artifacts","score":10,"reason":"no binaries found in the repo","details":null,"documentation":{"short":"Determines if the project has generated executable (binary) artifacts in the source repository.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#binary-artifacts"}},{"name":"CII-Best-Practices","score":0,"reason":"no effort to earn an OpenSSF best practices badge detected","details":null,"documentation":{"short":"Determines if the project has an OpenSSF (formerly CII) Best Practices Badge.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#cii-best-practices"}},{"name":"Security-Policy","score":0,"reason":"security policy file not detected","details":["Warn: no security policy file detected","Warn: no security file to analyze","Warn: no security file to analyze","Warn: no security file to analyze"],"documentation":{"short":"Determines if the project has published a security policy.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#security-policy"}},{"name":"Fuzzing","score":0,"reason":"project is not fuzzed","details":["Warn: no fuzzer integrations found"],"documentation":{"short":"Determines if the project uses fuzzing.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#fuzzing"}},{"name":"Vulnerabilities","score":10,"reason":"0 existing vulnerabilities detected","details":null,"documentation":{"short":"Determines if the project has open, known unfixed vulnerabilities.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#vulnerabilities"}},{"name":"License","score":10,"reason":"license file detected","details":["Info: project has a license file: LICENSE:0","Info: FSF or OSI recognized license: MIT License: LICENSE:0"],"documentation":{"short":"Determines if the project has defined a license.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#license"}},{"name":"Signed-Releases","score":-1,"reason":"no releases found","details":null,"documentation":{"short":"Determines if the project cryptographically signs release artifacts.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#signed-releases"}},{"name":"Branch-Protection","score":-1,"reason":"internal error: error during branchesHandler.setup: internal error: githubv4.Query: Resource not accessible by integration","details":null,"documentation":{"short":"Determines if the default and release branches are protected with GitHub's branch protection settings.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#branch-protection"}},{"name":"Pinned-Dependencies","score":0,"reason":"dependency not pinned by hash detected -- score normalized to 0","details":["Warn: containerImage not pinned by hash: Dockerfile:6: pin your Docker image by updating node:6-slim to node:6-slim@sha256:2cdb3951bc2cd934a056b1b1ff2224da2c092f28a6ac73133d76ce72dc54252e","Warn: containerImage not pinned by hash: test/bad_dockerfile:1: pin your Docker image by updating python:2.7-alpine to python:2.7-alpine@sha256:724d0540eb56ffaa6dd770aa13c3bc7dfc829dec561d87cb36b2f5b9ff8a760a","Warn: containerImage not pinned by hash: test/fixtures/gpu-experiment/Dockerfile:1","Warn: containerImage not pinned by hash: test/fixtures/standalone_project/Dockerfile:2: pin your Docker image by updating python:2.7-alpine to python:2.7-alpine@sha256:724d0540eb56ffaa6dd770aa13c3bc7dfc829dec561d87cb36b2f5b9ff8a760a","Warn: containerImage not pinned by hash: test/fixtures/test-report/data/Dockerfile:2: pin your Docker image by updating python:2.7-alpine to python:2.7-alpine@sha256:724d0540eb56ffaa6dd770aa13c3bc7dfc829dec561d87cb36b2f5b9ff8a760a","Warn: npmCommand not pinned by hash: Dockerfile:12","Warn: pipCommand not pinned by hash: test/bad_dockerfile:2","Info:   0 out of   5 containerImage dependencies pinned","Info:   0 out of   1 npmCommand dependencies pinned","Info:   0 out of   1 pipCommand dependencies pinned"],"documentation":{"short":"Determines if the project has declared and pinned the dependencies of its build process.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#pinned-dependencies"}},{"name":"SAST","score":0,"reason":"SAST tool is not run on all commits -- score normalized to 0","details":["Warn: 0 commits out of 29 are checked with a SAST tool"],"documentation":{"short":"Determines if the project uses static code analysis.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#sast"}}]},"last_synced_at":"2025-08-18T14:19:03.658Z","repository_id":140802423,"created_at":"2025-08-18T14:19:03.658Z","updated_at":"2025-08-18T14:19:03.658Z"},"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":277315120,"owners_count":25797567,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-27T02:00:08.978Z","response_time":73,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai","artificial-intelligence","benchmark-framework","benchmarking","docker","reproducible-research","reproducible-science","science"],"created_at":"2025-09-28T11:32:19.429Z","updated_at":"2025-09-28T11:32:20.092Z","avatar_url":"https://github.com/empiricalci.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"## This project is no longer being maintained. Feel free to fork \u0026 take over.\n\n\u003ch1 align='center'\u003e\n  \u003ca href='https://empiricalci.com'\u003e\n    \u003cimg src='https://cloud.githubusercontent.com/assets/689720/22778232/1723bff0-ee6b-11e6-9703-07b9d7d5a58d.png'/\u003e\n  \u003c/a\u003e\n\u003c/h1\u003e\n\u003cp align='center'\u003e\n  \u003ci\u003eA Power Tool for Empirical Science\u003c/i\u003e\u003cbr/\u003e\u003cbr/\u003e\n  \u003ca href='https://travis-ci.org/empiricalci/emp'\u003e\n    \u003cimg src='https://travis-ci.org/empiricalci/emp.svg?branch=master' alt='build status'/\u003e\n  \u003c/a\u003e\n  \u003ca href='https://gitter.im/empiricalci/emp?utm_source=badge\u0026utm_medium=badge\u0026utm_campaign=pr-badge\u0026utm_content=badge'\u003e\n    \u003cimg src='https://badges.gitter.im/empiricalci/emp.svg' alt='gitter'/\u003e\n  \u003c/a\u003e\n\u003c/p\u003e\n\n**emp** is a command line tool that helps you run and replicate experiments\nusing the [Empirical Framework](https://empiricalci.com/docs/framework)\n\n\u003cp align='center'\u003e\n  \u003cimg src='https://cloud.githubusercontent.com/assets/689720/22812270/97c874b0-eef8-11e6-819b-cf17485cfe7e.gif'/\u003e\n\u003c/p\u003e\n\n## Dependencies\nThere are only 2 requirements:\n\n- **Node.js** is a JavaScript runtime required to run  this program on your computer. Download and install Node.js [from here](https://nodejs.org/en/).\n- **Docker** allows us to create self-contained portable environments that work accross different platforms. Follow [these instructions](https://docs.docker.com/engine/installation/) to install Docker.\n\n## Install\n\n**emp** is supported on Windows, Mac and Linux.\nOnce you have node.js installed on your computer, you can install **emp** using ``npm``. \n```\nnpm install -g empirical-cli\n```\n\n## Get started\nTest your installation by replicating an experiment.\n```\nemp run empirical/mnist-sample/x/Hke43sz5e\n```\n\nThis will clone the source code from GitHub, checkout the appropriate version, \nbuild the experiment image, download the required datasets, run the experiment, \nand save the results on your computer. All in **one single command**.\n\n## Run your experiments\nOnce you have defined your experiment using the [Empirical Framework](http://empiricalci.com/docs).\n\n### From GitHub\nRun an experiment directly from a GitHub repository.\n```\nemp run mnist https://github.com/empiricalci/mnist-sample\n```\n\nOr provide a specific commit.\n```\nemp run mnist https://github.com/empiricalci/mnist-sample#0e6b04363c374992eb94a80dd2db0895711fb60f\n```\n\n### From empiricalci.com\nYou can easily replicate any experiment pushed to [empiricalci](https://empiricalci.com) by running:\n```\nemp run \u003cexperimentId\u003e\n```\n\n### From a local directory\nOr you can run an experiment from a local directory.\n```\nemp run my-protocol /path/to/code\n```\n\n## Keep track of your experiments\n\n### Authenticate with the server\nAuthenticating with [empiricalci.com](https://empiricalci.com) allows you to save the results of your experiments\nand share them with your peers.  \n1. If you haven't done so, sign up for an account on [empiricalci.com](http://empiricalci.com)    \n2. Login using the CLI: ``emp login`` will ask for your credentials and store them on your computer  \n\n### Push your results\n\nPush your results to the online dashboard by providing a message a path to the output directory created by ``emp run`` and the project id.\n\n```\nemp push /path/to/report \u003cprojectId\u003e\n```\n\n## Configure\n\nEmpirical uses a directory to cache all the datasets downloaded and to save any files generated during the\ntests or experiments. This defaults to ``~/empirical``. You can change this by doing:\n```\nemp configure\n```\n\n## Notes\n\n### GPU Support\n\nGPU support is provided for Linux via [nvidia-docker](https://github.com/NVIDIA/nvidia-docker), you can follow the instructions on the [\"Quick start\"](https://github.com/NVIDIA/nvidia-docker#quick-start) to install it and learn how to run GPU enabled experiments [here](http://empiricalci.com/docs/gpu-support).\n\nCurrently Windows and MacOS are not supported since they rely on VMs to run Docker. See [this issue](https://github.com/NVIDIA/nvidia-docker/issues/101) for more information.\n\n\n### About Docker on Windows\n\nIf you're using the [Docker for Windows](https://docs.docker.com/docker-for-windows) version, as opposed to [Docker toolbox](https://docs.docker.com/toolbox/overview/) make sure of the following:\n\n- **emp** requires to mount certain directories on your containers, so make sure to [share your local drives](https://docs.docker.com/docker-for-windows/#/shared-drives).\n- **emp** can usually communicate with Docker using the default settings. If it fails to connect, make sure the ``DOCKER_HOST`` environment variable is set to the correct value. On [Docker for Windows](https://docs.docker.com/docker-for-windows) this defaults to ``127.0.0.1:2375``. You can set it using CMD as ``setx DOCKER_HOST \"127.0.0.1:2375\"``.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fempiricalci%2Femp","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fempiricalci%2Femp","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fempiricalci%2Femp/lists"}