{"id":18811301,"url":"https://github.com/activeloopai/snark-doc","last_synced_at":"2026-01-11T06:30:14.277Z","repository":{"id":103329117,"uuid":"138695864","full_name":"activeloopai/snark-doc","owner":"activeloopai","description":"Serverless Deep Learning and Hyperparameter Search","archived":false,"fork":false,"pushed_at":"2018-11-14T18:29:15.000Z","size":5512,"stargazers_count":50,"open_issues_count":3,"forks_count":4,"subscribers_count":7,"default_branch":"master","last_synced_at":"2025-02-16T14:15:37.613Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://snark.ai","language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/activeloopai.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-06-26T06:44:38.000Z","updated_at":"2024-01-04T16:24:09.000Z","dependencies_parsed_at":"2023-06-02T12:15:57.357Z","dependency_job_id":null,"html_url":"https://github.com/activeloopai/snark-doc","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/activeloopai%2Fsnark-doc","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/activeloopai%2Fsnark-doc/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/activeloopai%2Fsnark-doc/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/activeloopai%2Fsnark-doc/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/activeloopai","download_url":"https://codeload.github.com/activeloopai/snark-doc/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":239748254,"owners_count":19690232,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-07T23:25:34.623Z","updated_at":"2026-01-11T06:30:14.216Z","avatar_url":"https://github.com/activeloopai.png","language":null,"funding_links":[],"categories":[],"sub_categories":[],"readme":"\n\n# \u003cimg src=\"https://avatars3.githubusercontent.com/u/34816118?s=200\u0026v=4\" data-canonical-src=\"hhttps://avatars3.githubusercontent.com/u/34816118?s=200\u0026v=4\" width=\"40\" height=\"40\" /\u003e Snark AI\n\n## Basic Usage\n**Step 1**. Install snark with pip3\n```bash\n  pip3 install snark\n```\nIn case of difficulties with installation, take a look at the [troubleshooting](#troubleshooting) section. \n\n**Step 2**. Go to [lab.snark.ai](https://lab.snark.ai) to sign up. Sign in through the CLI\n```bash\n  snark login\n```\n\n**Step 3**. Create Yaml description\n\nSnark uses yaml training workflow descriptions. Below is a basic MNIST example. \n```yaml\n  version: 1\n  experiments:\n    mnist:\n      image: pytorch/pytorch:latest\n      hardware:\n        gpu: K80\n      command:\n        - git clone https://github.com/pytorch/examples\n        - cd examples/mnist\n        - python main.py\n```\nThis yaml file describes an `mnist` experiment which uses _pytorch_ on a single _K80 GPU_.\nThe workflow runs as a combination of three commands described by the file.\nSave as `mnist.yml`.\n\n**Step 4**. Start the workflow\n\nUse `snark up` to start the workflow.\n```bash\n  snark up -f mnist.yml\n```\n\nSnark up starts the experiment described by the yaml file. It spins up a cluster with 1 K80 GPU.\nIn general workflows executed by snark can be of different nature:\n- single-GPU ML training \n- distributed ML training \n- hyperparameter search\n\n**Step 5** List Experiments\n\n`snark ps` command will list the experiments along with their ids and states.\n```bash\n  snark ps\n```\nExperiment Ids are used to tear down the workflows.\n\n**Step 6** Tear Down Experiments\nSnark workflows can be torn down using `snark down` command\n```bash\n  snark down {experiment_id}\n```\nThe `snark down` command shuts down all cloud resources utilized by the experiment workflow.\n\n\n## Monitoring\nLogin to [lab.snark.ai](https://lab.snark.ai) to check the GPU usage and credits left.\n\n### Experiments\nOn the cli, use `snark ps` to keep track of all experiments.\n```bash\n  snark ps\n```\n\n### Tasks\nTo check the state of each task per experiment run.\n```\nsnark ps {experiment_id}\n```\n\n### Logs\nTo get logs of running experiments \n```bash\n  snark logs {experiment_id}\n```\n\n## Properties \n\nSnark executes serverless ML workflows described in Yaml files. \nThis section walks through properties that can be used in the Yaml file.\n\n### Multiple Experiments\nSnark executes experiments described in Yaml file. Those experiments are ML workflows such as model training, hyper parameter search etc.\n\nHere is how to declare them in the Yaml file:\n```yaml\n  version: 1\n  experiments:\n    yolo: # the experiment name\n      # here go the details of the workflow\n      ...\n    my_second_experiment:\n      # description of another experiment\n      ...\n    third_experiment:\n      ...\n\n```\n`snark ps` will show the experiment ids and states after they are run.\n\n\n### Docker Image\n\nSnark executes the workflow commands against the given docker image.\n**_Note:_** _Currently only public images are accepted_\n```yaml\n  version: 1\n  experiments:\n    yolo:\n      image: pytorch/pytorch # if no tag is provided, snark will pull :latest image\n```\n\n### Multiple GPUs\nThis property describes the hardware to run the experiment on. \n```yaml\n  version: 1\n  experiments:\n    yolo:\n      image: pytorch/pytorch \n      hardware:\n        gpu: K80 # use K80 GPU\n        gpu_count: 2 # use 2 K80 GPUs\n```\n`gpu` is the name of the gpu to use\n`gpu_count` describes the number of gpus for the given experiment to run on. By default it's value is **1**.\n\nSupported GPUs are K80 and V100 and the supported `gpu_count`s for them are **(1, 8, 16)** and **(1, 4, 8)** respectively\n\n\n### Commands\nSnark Workflows comprise of commands. The commands are executed against the docker image provided.\n```yaml\n  version: 1\n  experiments:\n    yolo:\n      image: pytorch/pytorch \n      hardware:\n        gpu: K80\n        gpu_count: 2\n      commands: # Commands is an array of commands to execute against the image declared above\n        - git clone https://github.com/pytorch/examples\n        - cd examples/mnist\n        - python main.py\n```\nThe above example demonstrates how snark workflows are executed: using provided image and running the given commands on that image.\n\n\n### Hyperparameter search\nIt is possible to sample from discrete or continous range of parameters. You would need to provide the sampling method, number of samples, number of parallel workers and variables in the command execution list. Params are templated in commands using double handlebars `{{param}}`.\n\n```yaml\nversion: 1\nexperiments:\n  mnist_hyperparam_search:\n    image: pytorch/pytorch:latest\n    parameters:\n      github: https://github.com/pytorch/examples\n      batch_size: [32,64,128,256]\n      lr: \"0.01-0.09\"\n    hardware:\n      gpu: k80\n      gpu_count: 1\n    sampling: 'random'\n    samples: 8\n    workers: 4\n    command:\n      - git clone {{github}} \u0026\u0026 cd examples/mnist\n      - python main.py --batch-size {{batch_size}} --lr {{lr}}\n```\n\n\n\u003ca name=\"troubleshooting\"\u003e\u003c/a\u003e\n### Troubleshooting\n\n#### If you get a `Permission denied` message:\n```bash\n  sudo pip3 install snark\n```\n#### If you don't have `sudo` access:\n```bash\n  pip3 install snark --user\n```\n\n**AND** add the following to your `~/.bashrc` file:\n```bash\n  export PY_USER_BIN=$(python3 -c 'import site; print(site.USER_BASE + \"/bin\")')\n  export PATH=$PY_USER_BIN:$PATH\n```\n**AND** reload your `~/.bashrc`:\n```bash\n  source ~/.bashrc\n```\n#### Snark Not Found\nIf you tried all above and still get `snark not found` error message, try:\n1) Updating your pip3 through `pip3 install --upgrade pip3`\n2) Try installing specific version through tarball:\n```bash\n  pip3 uninstall snark\n  pip3 install https://files.pythonhosted.org/packages/6b/c4/1112f032a3d90686d757e5b0b325564a047488fc74fa43a138148dc2b8a5/snark-0.3.2.0.tar.gz\n```\n\nIn case of questions or issues please contact us at support@snark.ai\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Factiveloopai%2Fsnark-doc","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Factiveloopai%2Fsnark-doc","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Factiveloopai%2Fsnark-doc/lists"}