https://github.com/empiricalci/emp
:microscope: Empirical CLI
https://github.com/empiricalci/emp
ai artificial-intelligence benchmark-framework benchmarking docker reproducible-research reproducible-science science
Last synced: 9 months ago
JSON representation
:microscope: Empirical CLI
- Host: GitHub
- URL: https://github.com/empiricalci/emp
- Owner: empiricalci
- License: mit
- Created: 2015-12-02T05:44:31.000Z (over 10 years ago)
- Default Branch: master
- Last Pushed: 2020-09-10T17:57:55.000Z (almost 6 years ago)
- Last Synced: 2025-08-23T19:32:58.433Z (10 months ago)
- Topics: ai, artificial-intelligence, benchmark-framework, benchmarking, docker, reproducible-research, reproducible-science, science
- Language: JavaScript
- Homepage: https://empiricalci.com
- Size: 233 KB
- Stars: 42
- Watchers: 2
- Forks: 7
- Open Issues: 15
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- License: LICENSE
Awesome Lists containing this project
README
## This project is no longer being maintained. Feel free to fork & take over.
A Power Tool for Empirical Science
**emp** is a command line tool that helps you run and replicate experiments
using the [Empirical Framework](https://empiricalci.com/docs/framework)
## Dependencies
There are only 2 requirements:
- **Node.js** is a JavaScript runtime required to run this program on your computer. Download and install Node.js [from here](https://nodejs.org/en/).
- **Docker** allows us to create self-contained portable environments that work accross different platforms. Follow [these instructions](https://docs.docker.com/engine/installation/) to install Docker.
## Install
**emp** is supported on Windows, Mac and Linux.
Once you have node.js installed on your computer, you can install **emp** using ``npm``.
```
npm install -g empirical-cli
```
## Get started
Test your installation by replicating an experiment.
```
emp run empirical/mnist-sample/x/Hke43sz5e
```
This will clone the source code from GitHub, checkout the appropriate version,
build the experiment image, download the required datasets, run the experiment,
and save the results on your computer. All in **one single command**.
## Run your experiments
Once you have defined your experiment using the [Empirical Framework](http://empiricalci.com/docs).
### From GitHub
Run an experiment directly from a GitHub repository.
```
emp run mnist https://github.com/empiricalci/mnist-sample
```
Or provide a specific commit.
```
emp run mnist https://github.com/empiricalci/mnist-sample#0e6b04363c374992eb94a80dd2db0895711fb60f
```
### From empiricalci.com
You can easily replicate any experiment pushed to [empiricalci](https://empiricalci.com) by running:
```
emp run
```
### From a local directory
Or you can run an experiment from a local directory.
```
emp run my-protocol /path/to/code
```
## Keep track of your experiments
### Authenticate with the server
Authenticating with [empiricalci.com](https://empiricalci.com) allows you to save the results of your experiments
and share them with your peers.
1. If you haven't done so, sign up for an account on [empiricalci.com](http://empiricalci.com)
2. Login using the CLI: ``emp login`` will ask for your credentials and store them on your computer
### Push your results
Push your results to the online dashboard by providing a message a path to the output directory created by ``emp run`` and the project id.
```
emp push /path/to/report
```
## Configure
Empirical uses a directory to cache all the datasets downloaded and to save any files generated during the
tests or experiments. This defaults to ``~/empirical``. You can change this by doing:
```
emp configure
```
## Notes
### GPU Support
GPU support is provided for Linux via [nvidia-docker](https://github.com/NVIDIA/nvidia-docker), you can follow the instructions on the ["Quick start"](https://github.com/NVIDIA/nvidia-docker#quick-start) to install it and learn how to run GPU enabled experiments [here](http://empiricalci.com/docs/gpu-support).
Currently Windows and MacOS are not supported since they rely on VMs to run Docker. See [this issue](https://github.com/NVIDIA/nvidia-docker/issues/101) for more information.
### About Docker on Windows
If you're using the [Docker for Windows](https://docs.docker.com/docker-for-windows) version, as opposed to [Docker toolbox](https://docs.docker.com/toolbox/overview/) make sure of the following:
- **emp** requires to mount certain directories on your containers, so make sure to [share your local drives](https://docs.docker.com/docker-for-windows/#/shared-drives).
- **emp** can usually communicate with Docker using the default settings. If it fails to connect, make sure the ``DOCKER_HOST`` environment variable is set to the correct value. On [Docker for Windows](https://docs.docker.com/docker-for-windows) this defaults to ``127.0.0.1:2375``. You can set it using CMD as ``setx DOCKER_HOST "127.0.0.1:2375"``.