{"id":27431018,"url":"https://github.com/bigmlcom/bigml-bash","last_synced_at":"2025-04-14T15:28:20.099Z","repository":{"id":3165467,"uuid":"4196294","full_name":"bigmlcom/bigml-bash","owner":"bigmlcom","description":"Simple bash/curl bindings for BigML.io","archived":false,"fork":false,"pushed_at":"2012-05-01T20:49:12.000Z","size":88,"stargazers_count":6,"open_issues_count":0,"forks_count":5,"subscribers_count":16,"default_branch":"master","last_synced_at":"2024-04-16T10:58:59.685Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Shell","has_issues":false,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":"npm/policies","license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/bigmlcom.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2012-05-01T20:36:51.000Z","updated_at":"2024-04-16T10:58:59.686Z","dependencies_parsed_at":"2022-08-20T10:10:22.492Z","dependency_job_id":null,"html_url":"https://github.com/bigmlcom/bigml-bash","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bigmlcom%2Fbigml-bash","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bigmlcom%2Fbigml-bash/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bigmlcom%2Fbigml-bash/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bigmlcom%2Fbigml-bash/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/bigmlcom","download_url":"https://codeload.github.com/bigmlcom/bigml-bash/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248905886,"owners_count":21181069,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-04-14T15:28:18.974Z","updated_at":"2025-04-14T15:28:20.082Z","avatar_url":"https://github.com/bigmlcom.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Accessing BigML.io from shell scripts\n\nIn this repository you'll find a set of very simple shell scripts\ninteracting with https://bigml.io/.\n\n## Requirements\n\nThese shell scripts use bash, curl and python to interact with\nhttps://bigml.io and create datasources, datasets, models and,\neventually, predictions.  You'll be also able to retrieve them in JSON\nformat.\n\nThe scripts are written in **bash**, and use **curl** to send HTTPS\nrequests to our servers.  Responses, in JSON format, are\npretty-printed in the console via python's **json.tool** module.\n\n## Common setup\n\nTo access bigml.io you need a username and an API key, which are\nassigned to you one you register in [our site](https://bigml.com).\n\nThe scripts look for them in the environment variables `BIGML_USERNAME`\nand `BIGML_API_KEY`, which are combined in the authentication token\n`BIGML_AUTH`.  The [env/api_key.sh](env/api_key.sh) file provides a\ntemplate you can use in your `.bashrc` or `.bash_profile` to set those\nvariables automatically when you log in:\n\n```bash\nBIGML_USERNAME=\u003cusername\u003e\nBIGML_API_KEY=\u003capikey\u003e\nexport BIGML_URL=https://bigml.io/andromeda/\nexport BIGML_AUTH=\"username=$BIGML_USERNAME;api_key=$BIGML_API_KEY;\"\n```\n\nAll scripts in the repo live in the `bin` directory.\n\n## Creating sources\n\nTo create a datasource from a local data file, use `create_source.sh`.\nThis simple script takes a single parameter, namely, the path to the\ndata file, and uses *curl* to upload the file to BigML's servers,\nregister an associated datasource and prints its JSON descriptor.\n\nHere's a sample invocation:\n\n\n```bash\n~/bigml/io/bash $ ./create_source.sh ../csv/iris.csv\n  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n                                 Dload  Upload   Total   Spent    Left  Speed\n100  5461  100   653  100  4808    114    843  0:00:05  0:00:05 --:--:--  8094\n{\n    \"code\": 201,\n    \"content_type\": \"application/octet-stream\",\n    \"created\": \"2012-03-23T01:44:25.687600\",\n    \"credits\": 0.0,\n    \"file_name\": \"iris.csv\",\n    \"md5\": \"d1175c032e1042bec7f974c91e4a65ae\",\n    \"name\": \"iris.csv\",\n    \"number_of_datasets\": 0,\n    \"number_of_models\": 0,\n    \"number_of_predictions\": 0,\n    \"private\": true,\n    \"resource\": \"source/4f6bd5791552687fb5000003\",\n    \"size\": 4608,\n    \"source_parser\": {\n        \"header\": true,\n        \"locale\": \"en-US\",\n        \"missing_tokens\": [\n            \"N/A\",\n            \"n/a\",\n            \"NA\",\n            \"na\",\n            \"-\",\n            \"?\"\n        ],\n        \"quote\": \"\\\"\",\n        \"separator\": \",\",\n        \"trim\": true\n    },\n    \"status\": {\n        \"code\": 2,\n        \"elapsed\": 0,\n        \"message\": \"The source creation has been started\"\n    },\n    \"type\": 0,\n    \"updated\": \"2012-03-23T01:44:25.687628\"\n}\n```\n\n\nResource creation is asynchronous: the created resource status code\nwill be, as you see in the sample above, 2 (i.e., *in-progress*).  You\ncan retrieve the descriptor (with its updated status) at any time\nusing the `get.sh` script with the **resource** identifier as its only\nargument.\n\n```bash\n~/bigml/io/bash $ ./get.sh source/4f6bd5791552687fb5000003\n  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n                                 Dload  Upload   Total   Spent    Left  Speed\n100  1050  100  1050    0     0    179      0  0:00:05  0:00:05 --:--:--  2908\n{\n    \"code\": 200,\n    \"content_type\": \"application/octet-stream\",\n    \"created\": \"2012-03-23T01:44:25.687000\",\n    \"credits\": 0.0087890625,\n    \"fields\": {\n        \"000000\": {\n            \"column_number\": 0,\n            \"name\": \"sepal length\",\n            \"optype\": \"numeric\"\n        },\n        \"000001\": {\n            \"column_number\": 1,\n            \"name\": \"sepal width\",\n            \"optype\": \"numeric\"\n        },\n        \"000002\": {\n            \"column_number\": 2,\n            \"name\": \"petal length\",\n\n[...]\n\n    \"status\": {\n        \"code\": 5,\n        \"elapsed\": 2969,\n        \"message\": \"The source has been created\"\n    },\n    \"type\": 0,\n    \"updated\": \"2012-03-23T01:44:28.755000\"\n}\n```\n\n## Creating datasets\n\nOnce you have created a datasource, you can create a *dataset*, which\nis a processed version of the data accompanied by metadata describing\nits contents (data types, histograms, etc.).\n\nCreating a dataset containing all columns in the source and using the\ndefault parsing options is accomplished via `create_dataset.sh`, which\ntakes the datasource resource identifier as its only argument.\n\n\n```bash\n~/bigml/io/bash $ ./create_dataset.sh source/4f6bd5791552687fb5000003\n  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n                                 Dload  Upload   Total   Spent    Left  Speed\n100   919  100   874  100    45    156      8  0:00:05  0:00:05 --:--:--  1758\n{\n    \"code\": 201,\n    \"columns\": 5,\n    \"created\": \"2012-03-23T02:07:49.831948\",\n    \"credits\": 0.0087890625,\n    \"fields\": {\n        \"000000\": {\n            \"column_number\": 0,\n            \"name\": \"sepal length\",\n            \"optype\": \"numeric\"\n        },\n        \"000001\": {\n            \"column_number\": 1,\n            \"name\": \"sepal width\",\n            \"optype\": \"numeric\"\n        },\n        \"000002\": {\n            \"column_number\": 2,\n            \"name\": \"petal length\",\n            \"optype\": \"numeric\"\n        },\n        \"000003\": {\n            \"column_number\": 3,\n            \"name\": \"petal width\",\n            \"optype\": \"numeric\"\n        },\n        \"000004\": {\n            \"column_number\": 4,\n            \"name\": \"species\",\n            \"optype\": \"categorical\"\n        }\n    },\n    \"locale\": \"en-US\",\n    \"name\": \"iris' dataset\",\n    \"number_of_models\": 0,\n    \"number_of_predictions\": 0,\n    \"private\": true,\n    \"resource\": \"dataset/4f6bdaf51552687fb3000006\",\n    \"rows\": 0,\n    \"size\": 4608,\n    \"source\": \"source/4f6bd5791552687fb5000003\",\n    \"source_status\": true,\n    \"status\": {\n        \"code\": 1,\n        \"message\": \"The dataset is being processed and will be created soon\"\n    },\n    \"updated\": \"2012-03-23T02:07:49.831969\"\n}\n```\n\nAgain, the process is asynchronous: your task has been scheduled\n(status 1, i.e., *queued*), and the new resource has been assigned an\nidentifier (*dataset/4f6bdaf51552687fb3000006*) which you can use in\nconjunction with the `get.sh` script to recover the dataset's JSON\nmetadata at any other time:\n\n```bash\n./get.sh dataset/4f6bdaf51552687fb3000006\n```\n\n## Creating models\n\nWith a dataset resource in your hand, you can proceed to the creation\nof a predictive model, using `create_model.sh`.\n\n```bash\n~/bigml/io/bash (master) $ ./create_model.sh dataset/4f6bdaf51552687fb3000006\n  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n                                 Dload  Upload   Total   Spent    Left  Speed\n100   691  100   644  100    47    112      8  0:00:05  0:00:05 --:--:--  1018\n{\n    \"code\": 201,\n    \"columns\": 5,\n    \"created\": \"2012-03-23T02:28:57.741150\",\n    \"credits\": 0.03515625,\n    \"dataset\": \"dataset/4f6bdaf51552687fb3000006\",\n    \"dataset_status\": true,\n    \"holdout\": 0.0,\n    \"input_fields\": [],\n    \"locale\": \"en-US\",\n    \"max_columns\": 5,\n    \"max_rows\": 150,\n    \"name\": \"iris' dataset model\",\n    \"number_of_predictions\": 0,\n    \"objective_fields\": [],\n    \"private\": true,\n    \"range\": [\n        1,\n        150\n    ],\n    \"resource\": \"model/4f6bdfe9035d075177000005\",\n    \"rows\": 150,\n    \"size\": 4608,\n    \"source\": \"source/4f6bd5791552687fb5000003\",\n    \"source_status\": true,\n    \"status\": {\n        \"code\": 1,\n        \"message\": \"The model is being processed and will be created soon\"\n    },\n    \"updated\": \"2012-03-23T02:28:57.741177\"\n}\n```\n\nAgain, the model is scheduled for creation, and you can retrieve its\nstatus at any time by means of `get.sh` and its resource identifier.\n\n## Creating predictions\n\nYou can now use the model resource identifier together with some\ninput parameters to ask for predictions, using `create_prediction.sh`:\n\n```bash\n~/bigml/io/bash $ ./create_prediction.sh model/4f6bdfe9035d075177000005 '{\"000002\":1.2}'\n  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n                                 Dload  Upload   Total   Spent    Left  Speed\n100  1040    0   967  100    73    170     12  0:00:06  0:00:05  0:00:01  1729\n{\n    \"code\": 201,\n    \"created\": \"2012-03-23T02:52:23.651644\",\n    \"credits\": 0.01,\n    \"dataset\": \"dataset/4f6bdaf51552687fb3000006\",\n    \"dataset_status\": true,\n    \"fields\": {\n        \"000002\": {\n            \"column_number\": 2,\n            \"datatype\": \"double\",\n            \"name\": \"petal length\",\n            \"optype\": \"numeric\"\n        },\n        \"000004\": {\n            \"column_number\": 4,\n            \"datatype\": \"string\",\n            \"name\": \"species\",\n            \"optype\": \"categorical\"\n        }\n    },\n    \"input_data\": {\n        \"000002\": 1.2\n    },\n    \"locale\": \"en-US\",\n    \"model\": \"model/4f6bdfe9035d075177000005\",\n    \"model_status\": true,\n    \"name\": \"Prediction for species\",\n    \"objective_fields\": [\n        \"000004\"\n    ],\n    \"prediction\": {\n        \"000004\": \"Iris-setosa\"\n    },\n    \"prediction_path\": {\n        \"bad_fields\": [],\n        \"next_predicates\": [],\n        \"path\": [\n            {\n                \"field\": \"000002\",\n                \"operator\": \"\u003c=\",\n                \"value\": 2.45\n            }\n        ],\n        \"unknown_fields\": []\n    },\n    \"private\": true,\n    \"resource\": \"prediction/4f6be5671552687fb5000005\",\n    \"source\": \"source/4f6bd5791552687fb5000003\",\n    \"source_status\": true,\n    \"status\": {\n        \"code\": 5,\n        \"message\": \"The prediction has been created\"\n    },\n    \"updated\": \"2012-03-23T02:52:23.651667\"\n}\n```\n## Deleting resources\n\nGiven an identifier, the corresponding resource can be deleted with\n`delete.sh`:\n\n```bash\n~/bigml/io/bash $ ./delete.sh prediction/4f6be5671552687fb5000005\n~/bigml/io/bash $ ./get.sh prediction/4f6be5671552687fb5000005\n  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n                                 Dload  Upload   Total   Spent    Left  Speed\n100    65  100    65    0     0     12      0  0:00:05  0:00:05 --:--:--   211\n{\"code\": 404, \"status\": {\"code\": -1104, \"message\": \"Not found\"}}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbigmlcom%2Fbigml-bash","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbigmlcom%2Fbigml-bash","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbigmlcom%2Fbigml-bash/lists"}