{"id":20935776,"url":"https://github.com/rodrigopivi/aida","last_synced_at":"2025-05-13T20:33:14.749Z","repository":{"id":34923209,"uuid":"143543220","full_name":"rodrigopivi/aida","owner":"rodrigopivi","description":"🤖💬 Tiny experimental NLP deep learning library for text classification and NER. Built with Tensorflow.js, Keras and Chatito. Implemented in JS and Python.","archived":false,"fork":false,"pushed_at":"2023-01-11T19:40:07.000Z","size":180736,"stargazers_count":77,"open_issues_count":16,"forks_count":23,"subscribers_count":7,"default_branch":"master","last_synced_at":"2025-04-22T12:35:14.535Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://aida.dor.ai","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/rodrigopivi.png","metadata":{"files":{"readme":"readme.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-08-04T16:03:50.000Z","updated_at":"2024-01-04T16:25:11.000Z","dependencies_parsed_at":"2023-01-15T10:30:51.118Z","dependency_job_id":null,"html_url":"https://github.com/rodrigopivi/aida","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rodrigopivi%2Faida","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rodrigopivi%2Faida/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rodrigopivi%2Faida/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rodrigopivi%2Faida/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/rodrigopivi","download_url":"https://codeload.github.com/rodrigopivi/aida/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254021440,"owners_count":22000919,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-18T22:16:21.412Z","updated_at":"2025-05-13T20:33:09.740Z","avatar_url":"https://github.com/rodrigopivi.png","language":"TypeScript","readme":"\u003cp align=\"center\"\u003e\n  \u003ca href=\"https://aida.dor.ai\"\u003e\n    \u003cimg alt=\"Aida\" src=\"icon.png\" width=\"60\" /\u003e\n  \u003c/a\u003e\n\u003c/p\u003e\n\u003ch1 align=\"center\"\u003e\n  Aida\n\u003c/h1\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003cstrong\u003eBuild simple conversational assistants, chatbots, and more.\u003c/strong\u003e\u003cbr\u003e\n  Simple deep learning language models that can train and predict from the browser, nodejs and python.\n\u003c/p\u003e\n\n\u003ch3 align=\"center\"\u003e\n  \u003ca href=\"https://aida.dor.ai/demo\"\u003e\n    Demo\n  \u003c/a\u003e | \n  \u003ca href=\"https://aida.dor.ai/train\"\u003e\n    Train an assistant\n  \u003c/a\u003e | \n  \u003ca href=\"https://aida.dor.ai/overview\"\u003e\n    Technical overview\n  \u003c/a\u003e\n\u003c/h3\u003e\n\n\u003chr /\u003e\n\u003cbr/\u003e\n\n\u003cdiv\u003e\n  \u003ch2\u003eGetting started\u003c/h2\u003e\n\n  \u003ch3\u003eAida helps you prototype chatbots, fast.\u003c/h3\u003e\n  \u003cp\u003e\n  This is an experimental library for building natural language processing models. It can help you build a simple chatbot, and simple assistants. It's like building a smart regular expression for detecting sentence intentions and extracting key entities, but its much better because it's using neural networks and pre-trained characters bigram embeddings, so it has some general language knowledge.\n  \u003c/p\u003e\n  \u003cul\u003e\n      \u003cli\u003e\n          \u003cstrong\u003eIt's easy to use:\u003c/strong\u003e Getting started by creating a dataset and training couldn't be easier thanks to\u0026nbsp;\n          \u003ca href=\"https://rodrigopivi.github.io/Chatito\" target=\"_blank\"\u003e\n              Chatito\n          \u003c/a\u003e\n          , you can create a large dataset in minutes, and start training without any setup, just from the browser.\n      \u003c/li\u003e\n        \u003cli\u003e\n          \u003cstrong\u003eLow memory consumption:\u003c/strong\u003e Having small file size and memory consumption is very important to be able to predict from the browser and mobile devices. The language embeddings give up some information and performance by not using a full word dictionary, instead the model uses pre-trained word character bigrams provided by\n          \u003ca href=\"https://fasttext.cc/\" target=\"_blank\"\u003e\n              fastText\n          \u003c/a\u003e, to overcome this problem and get good predictive performance, it is required additional training examples, that is why a generative scripting language is provided to overcome this problem (Chatito DSL).\n      \u003c/li\u003e\n      \u003cli\u003e\n          \u003cstrong\u003eAccurate:\u003c/strong\u003e Although the model throws away some information by using bigrams to compose words instead of using full words, the models are able to get to good prediction rates given more data to learn from.\n      \u003c/li\u003e\n      \u003cli\u003e\n          \u003cstrong\u003eUniversal:\u003c/strong\u003e The trained models should be able to run from multiple environments, that is why the\n          models have two mirror implementations: in\u0026nbsp;\n          \u003ca href=\"https://js.tensorflow.com/\" target=\"_blank\"\u003e\n              TensorflowJS\n          \u003c/a\u003e\n          \u0026nbsp;to be able to train and run from the browser or Nodejs, and \u0026nbsp;\n          \u003ca href=\"https://keras.io/\" target=\"_blank\"\u003e\n              Keras\n          \u003c/a\u003e with Tensorflow backend for Python.\n      \u003c/li\u003e\n      \u003cli\u003e\n          \u003cstrong\u003eOffline support:\u003c/strong\u003e It should be able to train and make predictions without connectivty, no need to have a server-side api, although the trained models can also run server-side behind an api if desired. (TODO: add example running as AWS Lambda function)\n      \u003c/li\u003e\n  \u003c/ul\u003e\n\u003c/div\u003e\n\n## [Check the demo](https://aida.dor.ai/demo)\n\nIt's a chatbott running from the browser using Tensorflow.js and using the Web Speech API for speach to text and text to speach.\n\n## Train online\n\nYou can train from the browser [using Javascript and Tensorflow.js](https://aida.dor.ai/train) (using your local GPU resources) or from the browser [using Python and Tensorflow with Keras](https://colab.research.google.com/drive/1nzjxR7w2X99qlxjSD4pGOWksMLqK0eqZ) thanks to Google Colaboratory's free TPU's. There is no need to setup a local environment, the trained models can be saved for later use.\n\n\n## Local NPM package setup\n\n- Install the npm package:\n\n```\nyarn add aida-nlp\n```\n\n  - Create your chatito definition files, here you define your intents and your possible sentence models in mutiple `.chatito` files, and save them to a directory. e.g.: ´./chatito´\n\n  - Create a config file like `aida_config.json` where you define the path to your chatito definition files, the chatito dataset output path and the output path for the trained NLP models:\n\n```\n{\n  \"chatito\": {\n    \"inputPath\": \"./chatito\",\n    \"outputPath\": \"./dataset\"\n  },\n  \"aida\": {\n    \"outputPath\": \"./model\",\n    \"language\": \"en\"\n  }\n}\n```\n\n  - Generate and encode the dataset for training: `npx aida-nlp aida_config.json --action dataset`. The dataset will be available at the configured output path.\n\n  - Start training: `npx aida-nlp aida_config.json --action train`. The models will be saved at the configured output path.\n\n  - Run `npx aida-nlp aida_config.json --action test` for trying the generated testing dataset.\n\n## Local setup cloning the project\n\nAlternatively to training online and using npm package, you can setup the project locally. Clone the GH proejct and install dependencies for node and python (given NodeJS with yarn and Python3 are installed):\n\n  - Run `yarn install` from the `./typescript` directory\n  - Run `pip3 install -r requirements.txt` from the `./python` directory\n\n## Create a dataset\n\nEdit or create the chatito files inside `./typescript/examples/en/intents` to customize the dataset generation as you need. You can read more about [Chatito](https://rodrigopivi.github.io/Chatito).\n\nThen, from `./typescript` directory, run `npm run dataset:en:process`. This will generate many files at the `./typescript/public/models` directory. The dataset, the dataset parameters, the testing parameters and the embeddings dictionary. (Note: Aida also supports spanish language, if you need other language you can add if you first download the fastText embeddings for that language).\n\n## Training\n\nTtrain from 3 local environments:\n      - For python: open `./python/main.ipynb` with jupyter notebook or jupyter lab. Python will load your custom settings generated at step 3. And save the models in a TensorflowJS compatible format at the `output` directory.\n\n      - For web browsers: from `./typescript` run `npm run web:start`. Then navigate to `http://localhost:8000/train` for the training web UI. After training, downloading the model to the `./typescript/public/pretrained/web` directory (NOTE: this will also generate and download a new dataset).\n\n      - For Node.js: from `./typescript` run `npm run node:start`. This will load the previously dataset generated files from `./typescript/public/models`.\n\n## Technical Overview\n\nRead the [technical overview documentation](https://aida.dor.ai/overview).\n\n## Future ideas\n\n- Add tests\n\n- Add example that predicts from AWS Lambda\n\n- Experiment with multi layer language models based on character features like bigrams or trigrams for transfer learning, probably using a custom BiLSTM or LSTM architecture similar but simplier to [Universal Language Model Fine-tuning for Text Classification](https://arxiv.org/abs/1801.06146) [(blog post)](http://nlp.fast.ai/classification/2018/05/15/introducting-ulmfit.html).\n\n# Author\nRodrigo Pimentel\n","funding_links":[],"categories":["📦 Legacy \u0026 Inactive Projects"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frodrigopivi%2Faida","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frodrigopivi%2Faida","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frodrigopivi%2Faida/lists"}