{"id":13631563,"url":"https://github.com/evilsocket/ergo","last_synced_at":"2025-04-04T20:13:28.050Z","repository":{"id":39886232,"uuid":"158589080","full_name":"evilsocket/ergo","owner":"evilsocket","description":"🧠 A tool that makes AI easier. ","archived":false,"fork":false,"pushed_at":"2024-06-17T22:49:03.000Z","size":829,"stargazers_count":296,"open_issues_count":1,"forks_count":35,"subscribers_count":12,"default_branch":"master","last_synced_at":"2025-03-28T19:11:55.774Z","etag":null,"topics":["dataset","deep-learning","ergo","gpu","keras","machine-learning","neural-networks","tensorflow","training-algorithm"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/evilsocket.png","metadata":{"files":{"readme":"README.md","changelog":"changelog.sh","contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"github":null,"patreon":"evilsocket","open_collective":null,"ko_fi":null,"tidelift":null,"community_bridge":null,"liberapay":null,"issuehunt":null,"otechie":null,"custom":null}},"created_at":"2018-11-21T18:15:31.000Z","updated_at":"2025-03-24T00:17:41.000Z","dependencies_parsed_at":"2024-09-03T09:41:48.611Z","dependency_job_id":"9993c3a4-7050-4fa7-9c67-8d174a431b0d","html_url":"https://github.com/evilsocket/ergo","commit_stats":{"total_commits":255,"total_committers":3,"mean_commits":85.0,"dds":0.3686274509803922,"last_synced_commit":"12633fc13e807fa9329e95b4318e725b8c614a4d"},"previous_names":[],"tags_count":25,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/evilsocket%2Fergo","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/evilsocket%2Fergo/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/evilsocket%2Fergo/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/evilsocket%2Fergo/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/evilsocket","download_url":"https://codeload.github.com/evilsocket/ergo/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247242680,"owners_count":20907134,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["dataset","deep-learning","ergo","gpu","keras","machine-learning","neural-networks","tensorflow","training-algorithm"],"created_at":"2024-08-01T22:02:30.126Z","updated_at":"2025-04-04T20:13:28.026Z","avatar_url":"https://github.com/evilsocket.png","language":"Python","readme":"\u003cp align=\"center\"\u003e\n  \u003cimg alt=\"ergo\" src=\"https://i.imgur.com/EO9PdNp.jpg\"/\u003e\n  \u003cp align=\"center\"\u003e\n    \u003ca href=\"https://github.com/evilsocket/ergo/releases/latest\"\u003e\u003cimg alt=\"Release\" src=\"https://img.shields.io/github/release/evilsocket/ergo.svg?style=flat-square\"\u003e\u003c/a\u003e\n    \u003ca href=\"https://github.com/evilsocket/ergo/blob/master/LICENSE.md\"\u003e\u003cimg alt=\"Software License\" src=\"https://img.shields.io/badge/license-GPL3-brightgreen.svg?style=flat-square\"\u003e\u003c/a\u003e\n  \u003c/p\u003e\n\u003c/p\u003e\n\n`ergo` (from the Latin sentence *[\"Cogito ergo sum\"](https://en.wikipedia.org/wiki/Cogito,_ergo_sum)*) is a command line tool that makes machine learning with [Keras](https://keras.io/) easier. \n\n**It can be used to**: \n\n* scaffold new projects in seconds and customize only a minimum amount of code.\n* encode samples, import and optimize CSV datasets and train the model with them.\n* visualize the model structure, loss and accuracy functions during training.\n* determine how each of the input features affects the accuracy by differential inference.\n* export a simple REST API to use your models from a server.\n\n### Installing\n\n    sudo pip3 install ergo-ai\n\n### Installing from Sources\n\n    git clone https://github.com/evilsocket/ergo.git\n    cd ergo\n    sudo pip3 install -r requirements.txt\n    python3 setup.py build\n    sudo python3 setup.py install\n\n### Enable GPU support (optional)\n\nMake sure you have [CUDA 11 and cuDNN 8.0 installed](https://medium.com/@zhanwenchen/install-cuda-and-cudnn-for-tensorflow-gpu-on-ubuntu-79306e4ac04e) and then:\n\n    sudo pip3 uninstall tensorflow\n    sudo pip3 install tensorflow-gpu\n\n### Example Projects\n\n- A [planes detector](https://github.com/evilsocket/ergo-planes-detector) from satellite imagery.\n- An [anti malware API](https://github.com/evilsocket/ergo-pe-av) for Windows.\n\n### Usage\n\nTo print the general help menu:\n\n    ergo help\n\nTo print action specific help:\n\n    ergo \u003caction\u003e -h\n\nStart by printing the available actions by running `ergo help`, you can also print the software version (ergo, keras and tensorflow versions) and some hardware info with `ergo info` to verify your installation. \n\n#### Creating a Project\n\nOnce ready, create a new project named `example` (`ergo create -h` to see how to customize the initial model):\n\n    ergo create example\n\nInside the newly created `example` folder, there will be three files: \n\n1. `prepare.py`, used to preprocess your dataset and inputs (if, for instance, you're using pictures instead of a csv file).\n2. `model.py`, that you can change to customize the model.\n3. `train.py`, for the training algorithm.\n\nBy default, ergo will simply read the dataset as a CSV file, build a small neural network with 10 inputs, two hidden layers of 30 neurons each and 2 outputs and use a pretty standard training algorithm.\n\n#### Exploration (optional) \n\nExplore properties of the dataset. Ergo can generate graphs and tables that can be useful for the feature engineering of the problem. \n\nExplore can show:\n\n1. Metrics of each feature (min, max, standard deviation) - Which can be used to discard constant features in the dataset. \n2. Feature correlation of each feature with the target - Which can give an idea of how good is feature is as a linear predictor.\n3. Feature correlation matrix.\n4. [PCA decomposition](https://en.wikipedia.org/wiki/Principal_component_analysis): \n    * 2D projection of the data based on classes.\n    * Explained variance of each principal component with 90, 95 and 99 % explanation values.    \n5. [Kmeans clustering](https://en.wikipedia.org/wiki/K-means_clustering) or [DBSCAN clustering](https://en.wikipedia.org/wiki/DBSCAN) of the data.\n6. [Elbow method](https://en.wikipedia.org/wiki/Elbow_method_(clustering)) to determine the optimal number of clusters for kmeans.\n\nExample with a dataset `some/path/data.csv`: \n\n    ergo explore example --dataset some/path/data.csv -p\n\nThis will show the PCA decomposition of the dataset, saving (and optionally showing) the explained variance vs the number of principal component vectors used and the 2D projection of the dataset (colored by labels).\n\nA full exploratory analysis can be performed using the `--all` flag:\n    \n    ergo explore example --dataset some/path/data.csv --all \n\n#### Encoding (optional)\n\nIn case you implemented the `prepare_input` function in the `prepare.py` script, ergo can be used to encode raw samples, being them executables, images, strings or whatever, into vectors of scalars that are then saved into a `dataset.csv` file suitable for training\n\nExample with a folder `/path/to/data` which contains a `pos` and `neg` subfolders, in auto labeling mode each group of sample is labeled with its parent directory name:\n\n    ergo encode example /path/to/data\n\nExample with a single folder and manual labeling:\n\n    ergo encode example /path/to/data --label 'some-label'\n\nExample with a single text file containing multiple inputs, one per line:\n\n    ergo encode example /path/to/data --label 'some-label' -m\n\n#### Training\n\nAfter defining the model structure and the training process, you can import a CSV dataset (first column must be the label) and start training using 2 GPUs:\n\n    ergo train example --dataset /some/path/data.csv --gpus 2\n\nThis will split the dataset into a train, validation and test sets (partitioned with the `--test` and `--validation` arguments), start the training and once finished show the model statistics.\n\nIf you want to update a model and/or train it on already imported data, you can simply:\n\n    ergo train example --gpus 2\n\n#### Testing\n\nNow it's time to visualize the model structure and how the the `accuracy` and `loss` metrics changed during training (requires `sudo apt-get install graphviz python3-tk`):\n    \n    ergo view example\n\nIf the `data-test.csv` file is still present in the project folder (`ergo clean` has not been called yet), `ergo view` will also show the ROC curve.\n\nYou can use the `relevance` command to evaluate the model on a given set (or a subset of it, see `--ratio 0.1`) by nulling one attribute at a time and measuring how that influenced the accuracy (`feature.names` is an optional file with the names of the attributes, one per line):\n\n    ergo relevance example --dataset /some/path/data.csv --attributes /some/path/feature.names --ratio 0.1\n\nOnce you're done, you can remove the train, test and validation temporary datasets with:\n\n    ergo clean example\n\n#### Inference\n\nTo load the model and start a REST API for evaluation (can be customized with `--address`, `--port`, `--classes` and `--debug` options): \n\n    ergo serve example\n\nTo run an inference on a vector of scalars:\n\n    curl \"http://localhost:8080/?x=0.345,1.0,0.9,...\"\n\nIf you customized the `prepare_input` function in `prepare.py` (see the `Encoding` section), you can run an inference on a raw sample:\n\n    curl \"http://localhost:8080/?x=/path/to/sample\"\n\nThe input `x` can also be passed as a POST request:\n\n    curl --data 'x=...' \"http://localhost:8080/\"\n\nOr as a file upload:\n\n    curl -F 'x=@/path/to/file' \"http://localhost:8080/\"\n\nThe API can also be used to perform encoding only:\n\n    curl -F 'x=@/path/to/file' \"http://localhost:8080/encode\"\n\nThis will return the raw features vector that can be used for inference later.\n\n#### Other commands\n\nTo reset the state of a project (**WARNING**: this will remove the datasets, the model files and all training statistics):\n\n    ergo clean example --all\n\nEvaluate and compare the performances of two trained models on a given dataset and (optionally) output the differences to a json file:\n\n    ergo cmp example_a example_b --dataset /path/to/data.csv --to-json diffs.json\n\nFreeze the graph and convert the model to the [TensorFlow](https://www.tensorflow.org/) protobuf format:\n\n    ergo to-tf example\n\nConvert the Keras model to [frugally-deep](https://github.com/Dobiasd/frugally-deep) format:\n\n    ergo to-fdeep example\n\nOptimize a dataset (get unique rows and reuse 15% of the total samples, customize ratio with the `--reuse-ratio` argument, customize output with `--output`):\n\n    ergo optimize-dataset /some/path/data.csv\n\n### License\n\n`ergo` was made with ♥  by [the dev team](https://github.com/evilsocket/ergo/graphs/contributors) and it is released under the GPL 3 license.\n\n","funding_links":["https://patreon.com/evilsocket"],"categories":["Python"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fevilsocket%2Fergo","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fevilsocket%2Fergo","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fevilsocket%2Fergo/lists"}