{"id":18624127,"url":"https://github.com/wildonion/unixerr","last_synced_at":"2025-04-11T03:32:04.446Z","repository":{"id":41506265,"uuid":"236285956","full_name":"wildonion/uniXerr","owner":"wildonion","description":"♾️ universe〰️X〰️error ❌","archived":true,"fork":false,"pushed_at":"2023-07-02T07:47:14.000Z","size":24580,"stargazers_count":2,"open_issues_count":4,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-02-18T06:14:43.368Z","etag":null,"topics":["ai","artificial-intelligence","artificial-neural-networks","authentication","chat-application","classification","clustering-algorithm","credit-scoring","deep-learning","digital-companion","event-forecasting","jwt-authentication","machine-learning","position-clustering-data","python","pytorch-implementation","recognizer","recommendation-engine","rust","university-project"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/wildonion.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-01-26T08:31:06.000Z","updated_at":"2023-08-29T07:34:01.000Z","dependencies_parsed_at":"2024-11-07T04:39:47.780Z","dependency_job_id":null,"html_url":"https://github.com/wildonion/uniXerr","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wildonion%2FuniXerr","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wildonion%2FuniXerr/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wildonion%2FuniXerr/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wildonion%2FuniXerr/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/wildonion","download_url":"https://codeload.github.com/wildonion/uniXerr/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248335517,"owners_count":21086606,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai","artificial-intelligence","artificial-neural-networks","authentication","chat-application","classification","clustering-algorithm","credit-scoring","deep-learning","digital-companion","event-forecasting","jwt-authentication","machine-learning","position-clustering-data","python","pytorch-implementation","recognizer","recommendation-engine","rust","university-project"],"created_at":"2024-11-07T04:27:32.879Z","updated_at":"2025-04-11T03:31:59.435Z","avatar_url":"https://github.com/wildonion.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cp align=\"center\"\u003e\n    \u003cimg src=\"https://github.com/wildonion/uniXerr/blob/master/assets/uniXerr_R50.png\"\n\u003c/p\u003e\n\n\u003e 🌐 See the [Wiki](https://github.com/wildonion/uniXerr/wiki) to understand how stuff works!\n\n* 🪙 [coiniXerr Blockchain](https://github.com/wildonion/uniXerr/tree/master/infra/valhalla/coiniXerr)\n\n* 💳 [walleXerr](https://github.com/wildonion/uniXerr/tree/master/infra/valhalla/walleXerr)\n\n# AI Core Development Guide\n\n###### ⚠️ If you are working on development part, remember to change the local host(_127.0.0.1_) inside `/etc/hosts/` to `api.unixerr.com` and `tensorboard.api.unixerr.com` for API and TensorBoard server respectively.\n###### ⚠️ Remember to call `/users/add/info` and `/users/add/positions` routes of API server after the classification is done on csv file of input data. \n###### ⚠️ You can't create an environment if the environment was exported on a different platform than the target machine.\n###### ❗️ Both `piper` and `infra` folders can only be controlled using `controller.py` middleware.\n\n### Setup\n\n* Start an `Apache Cassandra` server and fill out `.env` file with necessary variables\n* Create an environment with a specific python version: ```conda create -n uniXerr python=3.8```\n* Create the environment using the `uniXerr.yml` file: ```conda env create -f uniXerr.yml```\n* Activate `uniXerr` environment: ```conda activate uniXerr```\n* Update the environment using `uniXerr.yml` file: ```conda env update -f uniXerr.yml --prune```\n* Export your active environment to `uniXerr.yml` file: ```conda env export | grep -v \"^prefix: \" \u003e uniXerr.yml```\n* Install `pm2`: ```wget -qO- https://getpm2.com/install.sh | bash```\n* Install completion for `typer-cli`: ```typer --install-completion```\n* Create a docs file from `uniXerr` **CLI**: ```typer app.py utils docs --name uniXerr --output uniXerr-cli.md```\n\n### Usage\n\n```console\n$ typer app.py run\nUsage: typer run [OPTIONS] COMMAND [ARGS]...\n\n  【  uniXerr CLI controller  】\n\nOptions:\n  --help  Show this message and exit.\n\nCommands:\n  classify-positions\n  cluster-positions\n  develop\n\n$ typer app.py run cluster-positions --help\nUsage: typer run cluster-positions [OPTIONS]\n\nOptions:\n  --generate-fake-samples      Generating fake samples for training.\n  --epoch INTEGER RANGE        Number of epoch for training VAE.\n  --batch-size INTEGER RANGE   Number of batch size for training VAE.\n  --device TEXT                Training device. cpu or cuda\n  --num-workers INTEGER RANGE  Number of workers for pytroch dataloader\n                               object.\n\n  --latent-dim INTEGER RANGE   Dimension of VAE latent space.\n  --ddo                        Force deletion with confirmation for dataloader\n                               object.\n\n  --dpm                        Force deletion with confirmation for pre-\n                               trained VAE model.\n\n  --cluster-on-raw-data        Clustering on pc_features dataset, default is\n                               set to VAE latent space\n\n  --cluster-method TEXT        Clustering method. kmeans or hdbscan; hdbscan\n                               is not suitable for latent space of VAE and has\n                               some drawbacks for new dataset.\n\n  --plot-method TEXT           Plotting method for data. pca or tsne; if you\n                               want plot data before clustering on different\n                               methods just remove the pc_dataloader.pth with\n                               --ddo option.\n\n  --help                       Show this message and exit.\n\n$ typer app.py run classify-positions --help\nUsage: typer run classify-positions [OPTIONS]\n\nOptions:\n  --csv-path FILE              Path to labeled pc_features csv dataset.\n  --input-data-csv-path FILE   Path to input data csv for classification.\n  --ddo                        Force deletion with confirmation for dataloader\n                               objects.\n\n  --dpm                        Force deletion with confirmation for pre-\n                               trained classifier model.\n\n  --epoch INTEGER RANGE        Number of epoch for training classifier.\n  --batch-size INTEGER RANGE   Number of batch size for training classifier.\n  --device TEXT                Training device. cpu or cuda\n  --num-workers INTEGER RANGE  Number of workers for pytroch dataloader\n                               object.\n\n  --help                       Show this message and exit.\n\n$ typer app.py run develop --help\nUsage: app.py run develop [OPTIONS]\n\nOptions:\n  --workers INTEGER RANGE  Number of workers\n  --help                   Show this message and exit.\n```\n\n\u003e Running in development mode: __ [API docs](http://api.unixerr.com:8000/docs) \n\n```console\n$ typer app.py run develop --workers 10\n```\n\n\u003e Export cassandra table into csv file:\n\n```console\n$ cqlsh api.unixerr.com -u username -p password -e \"copy unixerr.table_name to '/path/to/table_name.csv' with HEADER = true\"\n```\n\n\u003e Import exported csv file into cassandra table:\n\n```console\n$ cqlsh api.unixerr.com -u username -p password -e \"copy unixerr.table_name from '/path/to/table_name.csv' with HEADER = true\"\n```\n\n\u003e Running TensorBoard for visualization of training and testing DL models:\n\n```console\n$ tensorboard --host=tensorboard.unixerr.com --logdir=runs\n```\n\n\u003e [uniXerr CLI usage](https://github.com/wildonion/uniXerr/blob/master/uniXerr-cli.md)\n\n---\n\n# Results\n\n### 📌 Position Clustering Process\n\n[Dataloader Object - MinMax Scaler](https://github.com/wildonion/uniXerr/blob/master/piper/dataset/pc_dataloader-DATALOADER.pth)\n\n[Fake Dataset for Offline Training](https://github.com/wildonion/uniXerr/blob/master/piper/dataset/pc_features.csv)\n\n\u003e 📊 Plotted Dataset before Clustering using PCA - Standard Scaler\n\u003cp align=\"center\"\u003e\n    \u003cimg src=\"https://github.com/wildonion/uniXerr/blob/master/piper/dataset/pca_pc_beforeClustering.png\"\u003e\n\u003c/p\u003e\n\n\u003e 📊 Plotted Dataset before Clustering using TSNE - Standard Scaler\n\u003cp align=\"center\"\u003e\n    \u003cimg src=\"https://github.com/wildonion/uniXerr/blob/master/piper/dataset/tsne_pc_beforeClustering.png\"\u003e\n\u003c/p\u003e\n    \n[Clustered Dataset Based on Latent Space of Pre-trained VAE model](https://github.com/wildonion/uniXerr/blob/master/piper/dataset/pc_features_labeled-latent.csv)\n\n[Clustered Dataset Based on Position Clustering data](https://github.com/wildonion/uniXerr/blob/master/piper/dataset/pc_features_labeled-raw.csv)\n\n[VAE Pre-trained Model - Normal PDF](https://github.com/wildonion/uniXerr/blob/master/infra/position_clustering/utils/pc_model_vae.pth)\n\n\u003e 📊 Clusters Found by KMeans on Latent Space of Pre-trained VAE model\n\u003cp align=\"center\"\u003e\n    \u003cimg src=\"https://github.com/wildonion/uniXerr/blob/master/infra/position_clustering/utils/clusters-kmeans-latent.png\"\u003e\n\u003c/p\u003e\n\n\u003e 📊 Clusters Found by KMeans on Position Clustering Dataset - Plotted using PCA | Standard Scaler\n\u003cp align=\"center\"\u003e\n    \u003cimg src=\"https://github.com/wildonion/uniXerr/blob/master/infra/position_clustering/utils/clusters-kmeans-pca-raw.png\"\u003e\n\u003c/p\u003e\n\n\u003e 📊 Clusters Found by KMeans on Position Clustering Dataset - Plotted using TSNE | Standard Scaler\n\u003cp align=\"center\"\u003e\n    \u003cimg src=\"https://github.com/wildonion/uniXerr/blob/master/infra/position_clustering/utils/clusters-kmeans-tsne-raw.png\"\u003e\n\u003c/p\u003e\n\n\u003e 📊 VAE Model Training Loss \n\u003cp align=\"center\"\u003e\n    \u003cimg src=\"https://github.com/wildonion/uniXerr/blob/master/infra/position_clustering/utils/pc_model_loss.png\"\u003e\n\u003c/p\u003e\n\n### 📌 Position Classification Process\n\n[Training Dataloader Object of Clustered Dataset Based on Latent Space of Pre-trained VAE model](https://github.com/wildonion/uniXerr/blob/master/piper/dataset/pc_features_labeled_training_tensors-latent-DATALOADER.pth)\n\n[Testing Dataloader Object of Clustered Dataset Based on Latent Space of Pre-trained VAE model](https://github.com/wildonion/uniXerr/blob/master/piper/dataset/pc_features_labeled_testing_tensors-latent-DATALOADER.pth)\n\n[Training Dataloader Object of Clustered Dataset Based on Position Clustering data](https://github.com/wildonion/uniXerr/blob/master/piper/dataset/pc_features_labeled_training_tensors-raw-DATALOADER.pth)\n\n[Testing Dataloader Object of Clustered Dataset Based on Position Clustering data](https://github.com/wildonion/uniXerr/blob/master/piper/dataset/pc_features_labeled_testing_tensors-raw-DATALOADER.pth)\n\n\u003e 📊 Percentage of Positions before Classification on Clustered Dataset Based on Latent Space of Pre-trained VAE model\n\u003cp align=\"center\"\u003e\n    \u003cimg src=\"https://github.com/wildonion/uniXerr/blob/master/piper/dataset/pp_pc_beforeClassification-latent.png\"\u003e\n\u003c/p\u003e\n\n\u003e 📊 Percentage of Positions before Classification on Clustered Dataset Based on Position Clustering data\n\u003cp align=\"center\"\u003e\n    \u003cimg src=\"https://github.com/wildonion/uniXerr/blob/master/piper/dataset/pp_pc_beforeClassification-raw.png\"\u003e\n\u003c/p\u003e\n\n[Classifier Pre-trained Model - Trained and Tested on Clustered Dataset Based on Latent Space of Pre-trained VAE model](https://github.com/wildonion/uniXerr/blob/master/infra/position_classification/utils/pc_model_classifier-latent.pth)\n\n[Classifier Pre-trained Model - Trained and Tested on Clustered Dataset Based on Position Clustering data](https://github.com/wildonion/uniXerr/blob/master/infra/position_classification/utils/pc_model_classifier-raw.pth)\n\n\u003e 📊 Classifier Model Training Accuracy - Clustered Dataset Based on Latent Space of Pre-trained VAE model\n\u003cp align=\"center\"\u003e\n    \u003cimg src=\"https://github.com/wildonion/uniXerr/blob/master/infra/position_classification/utils/pc_model_training_acc-latent.png\"\u003e\n\u003c/p\u003e\n\n\u003e 📊 Classifier Model Testing Accuracy - Clustered Dataset Based on Latent Space of Pre-trained VAE model\n\u003cp align=\"center\"\u003e\n    \u003cimg src=\"https://github.com/wildonion/uniXerr/blob/master/infra/position_classification/utils/pc_model_testing_acc-latent.png\"\u003e\n\u003c/p\u003e\n\n\u003e 📊 Classifier Model Training Loss - Clustered Dataset Based on Latent Space of Pre-trained VAE model\n\u003cp align=\"center\"\u003e\n    \u003cimg src=\"https://github.com/wildonion/uniXerr/blob/master/infra/position_classification/utils/pc_model_training_loss-latent.png\"\u003e\n\u003c/p\u003e\n\n\u003e 📊 Classifier Model Training Accuracy - Clustered Dataset Based on Position Clustering data\n\u003cp align=\"center\"\u003e\n    \u003cimg src=\"https://github.com/wildonion/uniXerr/blob/master/infra/position_classification/utils/pc_model_training_acc-raw.png\"\u003e\n\u003c/p\u003e\n\n\u003e 📊 Classifier Model Testing Accuracy - Clustered Dataset Based on Position Clustering data\n\u003cp align=\"center\"\u003e\n    \u003cimg src=\"https://github.com/wildonion/uniXerr/blob/master/infra/position_classification/utils/pc_model_testing_acc-raw.png\"\u003e\n\u003c/p\u003e\n\n\u003e 📊 Classifier Model Training Loss - Clustered Dataset Based on Position Clustering data\n\u003cp align=\"center\"\u003e\n    \u003cimg src=\"https://github.com/wildonion/uniXerr/blob/master/infra/position_classification/utils/pc_model_training_loss-raw.png\"\u003e\n\u003c/p\u003e\n\n[Classification Results on Arbitrary Inputs - Classified using Pre-trained Model of Clustered Dataset Based on Latent Space of Pre-trained VAE model and Clustered Dataset Based on Position Clustering data](https://github.com/wildonion/uniXerr/blob/master/piper/db/_imported/)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwildonion%2Funixerr","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fwildonion%2Funixerr","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwildonion%2Funixerr/lists"}