{"id":29906747,"url":"https://github.com/josephrp/datatonic","last_synced_at":"2025-10-11T15:39:19.069Z","repository":{"id":212701070,"uuid":"730648002","full_name":"Josephrp/DataTonic","owner":"Josephrp","description":"🌟DataTonic : A Data-Capable AGI-style Agent Builder of Agents , that creates swarms , runs commands and securely processes and creates datasets, databases, visualisations, and analyses.","archived":false,"fork":false,"pushed_at":"2025-07-19T07:39:22.000Z","size":256,"stargazers_count":89,"open_issues_count":3,"forks_count":31,"subscribers_count":5,"default_branch":"main","last_synced_at":"2025-08-01T21:37:40.347Z","etag":null,"topics":["agent-builder","agi","autogen","azure","chroma","data","data-science","data-visualization","database","memgpt","semantic-kernel","semantic-memory","taskweaver"],"latest_commit_sha":null,"homepage":"https://www.tonic-ai.com","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Josephrp.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2023-12-12T11:28:24.000Z","updated_at":"2025-07-27T22:51:36.000Z","dependencies_parsed_at":"2023-12-15T18:25:13.441Z","dependency_job_id":"d4ae7652-0692-4603-b1e5-08c63f516990","html_url":"https://github.com/Josephrp/DataTonic","commit_stats":null,"previous_names":["tonic-ai/datatonic","josephrp/datatonic"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/Josephrp/DataTonic","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Josephrp%2FDataTonic","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Josephrp%2FDataTonic/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Josephrp%2FDataTonic/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Josephrp%2FDataTonic/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Josephrp","download_url":"https://codeload.github.com/Josephrp/DataTonic/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Josephrp%2FDataTonic/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":279007600,"owners_count":26084334,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-11T02:00:06.511Z","response_time":55,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["agent-builder","agi","autogen","azure","chroma","data","data-science","data-visualization","database","memgpt","semantic-kernel","semantic-memory","taskweaver"],"created_at":"2025-08-01T21:12:45.465Z","updated_at":"2025-10-11T15:39:19.042Z","avatar_url":"https://github.com/Josephrp.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# DataTonic\n\n- turn your laptop computer into a webserver of nested data and code capable agents !\n\nA Data-Capable AGI-style Agent Builder of Agents , that creates swarms , runs commands and securely processes and creates datasets, databases, visualizations, and analyses.\n\n- DataTonic solves simple tasks that require complex data processing\n- it's perfect for data analytics and business intelligence\n\nyou can use image or audio input in your native language : **DataTonic is AGI for all**\n\n#### What it does :\n\n- your request is first processed according to a statement of work\n- additional data is retrieved and stored to enrich it\n- multiple agents are created based on your specific use case\n- Multiple Multi-Agent Environments produce files and folders according to your use case.\n\n## Use Case\n\nDataTonic produces fixed business intelligence assets based on autonomous multimedia data processing. \n\n- Sales Profiles\n- Adaptive Summaries\n- Dataset Analytics\n- Research Reports\n- Business Automation Applications\n\nBased on those it can produce :\n- Strategies\n- Applications\n- Analyses\n- rich business intelligence\n\n### Business Case\n\nDataTonic provides junior executives with an extremely effective solution for basic and time-consuming data processing, document creation or business intelligence tasks.\n\nNow anyone can :\n- get rapid client profile and sales strategy including design assets for executions with a single request\n- create a functional web application\n- create entire databases of business intelligence that can be used by enterprise systems. \n\n### Enterprise Autonomation Agent\n\nDo not wait for accounting, legal or business intelligence reporting with uncertain quality and long review cycles. DataTonic accelerates the slowest part of analysis : data processing and project planning execution.\n\n## Main Benefits\n\nDataTonic is unique for many reasons :\n\n- local and secure application threads.\n- compatible with microsoft  enterprise environments.\n- based on a rigorous and reproducible evaluation method.\n- developper friendly : easily plug in new functionality and integrations.\n\n## Do more with DataTonic \n\n- is DataTonic Accessible ?\n\n**yes** DataTonic is accessible both audio and image input.\n\n- can i use it to make beautiful graphs and statistical analyses with little or no starting data ?\n \n**yes.** DataTonic will look for the data it needs but you can add your .db files or any other types of files with DataTonic.\n  \n- can i write a book or a long report ?\n\n**yes.** DataTonic produces rich , full-length content.\n\n- can i make an app ?\n\n**yes.** DataTonic is more tailored to business intelligence but it is able to produce functioning applications inside generated repositories.\n\n- can it do my job ?\n\n**yes** DataTonic is able to automate many junior positions and it will include more enterprise connectors, soon !\n\n### How we use it : Multi-Consult Technology\n\nYou can use datatonic however you want, here's how we're using it :\n- add case books to your folder for embedding : now DataTonic always presents its results in a case study!\n- add medical textbooks to your folder for embedding : now DataTonic helps you through med-school !\n- add entire company business information : Data tonic is now your strategic advisor !\n- ask data tonic to create targetted sales strategies : now DataTonic is your sales assistant !\n\n**Data Tonic is the first multi-nested agent-builder-of-agents!**\n\n## How Data Tonic Was Created\n\nDataTonic Team started by evaluating multiple models against the new google/gemini models , testing all functions. Based on our evaluation results we optimized default prompts and created new prompts and prompt pipeline configurations. \n\nLearn more about using TruLens and our scientific method in the [evaluation folder](https://github.com/Tonic-AI/DataTonic/tree/main/evaluation). We share our results in the [evaluation/results](https://github.com/Tonic-AI/DataTonic/tree/main/evaluation/results) folder.\n\n**you can also replicate our evaluation by following the instructions in #Easy Deploy**\n\n## How it works :\n\nDataTonic is the first application to use a **doubly nested multi-environment multi-agent builder-of-agents configuration** . Here's how it works ! \n\n### Orchestration\n\nData Tonic uses a novel combination of three orchestration libraries. \n- Each library creates it's own multi-agent environment. \n- Each of these environments includes a code execution and code generation capability. \n- Each of these stores data and embeddings on it's own datalake.\n- Autogen is at the interface with the user and orchestrates the semantic kernel hub as well as using Taskweaver for data processing tasks.\n- Semantic-kernel is a hub that includes internet browsing capabilities and is specifically designed to use taskweaver for data storage and retrieval and produce fixed intelligence assets also specifically designed for Autogen.\n- Taskweaver is used as a plugin in semantic kernel for data storage and retrieval and also in autogen, but remains an autonomous task that can execute complex tasks in its multi-environment execution system.\n\n### Technology : \n\n- Gemini is used in various configurations both for text using the autogen connector and for multimodal/image information processing. \n- Autogen uses a semantic-kernel function calling agent to access the internet using the google api semantic-kernel then processes the new information and stores it inside a SQL database orchestrated by Taskweaver.\n\n# Easy Deploy\n\nPlease try the methods below to use and deploy DataTonic.\n\n### Easy Deploy DataTonic evaluation/results\n\nThe easiest way to use DataTonic is to deploy on github spaces and use the **notebooks in the evaluation/results folder** . \n\nClick here for easy_deploy [COMING SOON!]\n\n**in the mean time please follow the instructions below:**\n\n1. Star then Fork this repository\n2. [use these instructions to deploy a code space for DataTonic](https://docs.github.com/en/codespaces/developing-in-a-codespace/creating-a-codespace-for-a-repository)\n3. Configure DataTonic according to the instructions below or \n4. navigate to [evaluation/results](https://github.com/Tonic-AI/DataTonic/tree/main/evaluation/results) to use our evaluation methods.\n\n# How To Use\n\nPlease follow the instructions in this readme **exactly**. \n\n### Follow Tonic-AI\n\n### Star \u0026 Fork this repository\n\n![image](https://github.com/Tonic-AI/DataTonic/assets/18212928/54e2f12a-0379-49d5-866c-985ea36f0e2b)\n1. Step 1 : Star this repository\n2. Step 2 : Fork this repository\n\n## Set Up\n\nplease use command line with administrator priviledges for the below.\n\n### Set Up Gemini\n\n- run the following command to install googlecloud/vertex cli :\n```bash\npip install google-cloud-aiplatform\n```\n\n- navigate to this url :\n```\nhttps://console.cloud.google.com/vertex-ai\n```\n- and click create new project.\n![image](https://github.com/Tonic-AI/DataTonic/assets/18212928/582ee276-e8b7-4d5d-b251-c1f730eaa84c)\n\n- Create a new project and add a payment method.\n\n- click 'enable all recommended APIs' \n![image](https://github.com/Tonic-AI/DataTonic/assets/18212928/2193f80b-fd46-444c-9173-81044166b3a6)\n\n- click on 'multimodal' on the left then 'my prompts' on the top:\n![image](https://github.com/Tonic-AI/DataTonic/assets/18212928/3ca60a1b-9012-432a-b431-b13eccbb57ff)\n\n- click on 'create prompt' and 'GET CODE' on the top right in the next screen:\n![image](https://github.com/Tonic-AI/DataTonic/assets/18212928/f563af64-cb9f-40a0-8158-564cd42bbc45)\n\n- then click on 'curl' on the top right to find your 'endpoint' and projectid , and other information\ne.g.\n```curl\ncat \u003c\u003c EOF \u003e request.json\n{\n    \"contents\": [\n        {\n            \"role\": \"user\",\n            \"parts\": []\n        }\n    ],\n    \"generation_config\": {\n        \"maxOutputTokens\": 2048,\n        \"temperature\": 0.4,\n        \"topP\": 1,\n        \"topK\": 32\n    }\n}\nEOF\n\nAPI_ENDPOINT=\"us-central1-aiplatform.googleapis.com\"\nPROJECT_ID=\"focused-album-408018\"\nMODEL_ID=\"gemini-pro-vision\"\nLOCATION_ID=\"us-central1\"\n\ncurl \\\n-X POST \\\n-H \"Authorization: Bearer $(gcloud auth print-access-token)\" \\\n-H \"Content-Type: application/json\" \\\n\"https://${API_ENDPOINT}/v1/projects/${PROJECT_ID}/locations/${LOCATION_ID}/publishers/google/models/${MODEL_ID}:streamGenerateContent\" -d '@request.json'\n```\n\n### Get Google API key\n\n run the following command to find your API key after following the instructions above:\n```bash\ngcloud auth print-access-token\n```\n**IMPORTANT : this key will expire every less than 30 minutes, so please refresh it regularly and accordingly**\n\n### Get Open AI Key(s)\n\n- navigate to openai and create a new key :\n```\nhttps://platform.openai.com/api-keys\n```\n\n### Set Up Azure\n\n- use the Azure OAI portal by navigating to this page :\n```\nhttps://oai.azure.com/portal\n```\n- deploy your models\n![image](https://github.com/Tonic-AI/DataTonic/assets/18212928/db19a2f3-5996-462c-9273-ef4af5710d0e)\n\n- go to playground\n- click on view code :\n![image](https://github.com/Tonic-AI/DataTonic/assets/18212928/192f55fb-c174-45d6-a15b-12f7bce5e0b2)\n- make a note of your `endpoint` , `API Key` , and `model name` to use it later.\n\n### Set Up Sqlite\n\n- visit this url and download and install the required packages :\n```https://www.sqlite.org/download.html```\n**For Windows** : \nYou need the SQLite source files, including the sqlite3.h header file, for the pysqlite3 installation.\n\n- Go to the SQLite Download Page.\n- Download the sqlite-amalgamation-*.zip file under the \"Source Code\" section.\n- Extract the contents of this zip file to a known directory (e.g., C:\\sqlite).\n\nSet Environment Variables:\n\nYou need to ensure that the directory where you extracted the SQLite source files is included in your system's PATH environment variable.\n\n- Right-click on 'This PC' or 'My Computer' and select 'Properties'.\n- Click on 'Advanced system settings' and then 'Environment Variables'.\n- Under 'System Variables', find and select the 'Path' variable, then click 'Edit'.\n- Add the path to the directory where you extracted the SQLite source files (e.g., C:\\sqlite).\n- Click 'OK' to close all dialog boxes.\n\nadd your path :\n```bash\nsetx SQLITE_INC \"C:\\sqlite\"\n```\nproceed with the rest of the setup below.\n\n### Install Taskweaver\n\nTaskWeaver requires Python \u003e= 3.10. It can be installed by running the following command:\n\n```\n# [optional to create conda environment]\n# conda create -n taskweaver python=3.10\n# conda activate taskweaver\n\n# clone the repository\ngit clone https://github.com/microsoft/TaskWeaver.git\ncd TaskWeaver\n# install the requirements\npip install -r requirements.txt\n```\n\n**Command Prompt**:\ndownload and install wsl:\n```bash\npip install wsl\n```\nthen run\n```bash\nwsl\n```\nthen install sqlite\n```bash\nsudo apt-get update\nsudo apt-get install libsqlite3-dev #or : sqlite-devel\nsudo pip install pysqlite3\n```\n\n## DataTonic Setup Instructions\n\nThis section provides instructions on setting up the project. Please turn off your firewall and use administrator priviledges on the command line.\n\n### Step 1 : Clone the repository\n\nclone this repository using the command line :\n\n```bash\ngit clone https://github.com/Tonic-AI/DataTonic.git\n```\n\n## Step 2: Configure DataTonic\n\n### Add Your Files to DataTonic \n\n1. add relevant files one by one with no folder to the folder called 'src/autogen/add_your_files_here' \n    - supported file types : \".pdf\" ,  \".html\" ,  \".eml\" \u0026 \".xlsx\":\n2. \n\n### Configuration\n1. you'll need the keys you made above for the following.\n2. use a text editor , and IDE  or command line to edit the following documents.\n3. Edit then save the files \n\n#### OAI_CONFIG_LIST\n\nedit 'OAI_CONFIG_LIST'\n\n```json\n        \"api_key\": \"your OpenAI Key goes here\",\n```\nand \n```json\n        \"api_key\": \"your Google's GenAI Key goes here\",\n```\n\n#### Configure OpenAI Key(s)\n    1. modify Line 135 in autogen_module.py\n        ```python\n        os.environ['OPENAI_API_KEY'] = 'Your key here'\n        ```\n    2. modify .env.example\n        ```os\n        OPENAI_API_KEY = \"your_key_here\"\n        ```\n        save as '.env' - this should create a new file.\n        **or** \n        rename to '.env' - this will rename the existing file.\n\n    3. modify src\\tonicweaver\\taskweaver_config.json\n        ```json\n        {\n            \"llm.api_base\": \"https://api.openai.com/v1\",\n            \"llm.api_key\": \"\",\n            \"llm.model\": \"gpt-4-1106-preview\"\n        }\n        ```\n\n    4. \n#### Google API\n\nedit ./src/semantic_kernel/semantic_kernel_module.py\n```python\nline 64:    semantic_kernel_data_module = SemanticKernelDataModule('\u003cgoogle_api_key\u003e', '\u003cgoogle_search_engine_id\u003e')\n```\nand \n```python\nline 158:    semantic_kernel_data_module = SemanticKernelDataModule('\u003cgoogle_api_key\u003e', '\u003cgoogle_search_engine_id\u003e')\n```\nwith your google API key and Search Engine ID , made above.\n\nsrc/semantic_kernel/googleconnector.py\n\n\n## Step 3: Install DataTonic\n\n### Install Taskweaver\n\nfrom the project directory :\n```bash\ncd ./src/tonicweaver\ngit clone https://github.com/microsoft/TaskWeaver.git\ncd ./src/tonicweaver/TaskWeaver\n# install the requirements\npip install -r requirements.txt\n```\n### Install DataTonic\nfrom the project directory :\n```bash\npip install -r requirements.txt\n```\n\n## Step 4: Run the application\n\n```bash\npython run app.py\n```\n\n# Developpers\n\nWe welcome contributions from the community! Whether you're opening a bug report, suggesting a new feature, or submitting a pull request, every contribution is valuable to us. Please follow these guidelines to contribute to DataTonic.\n\n## Getting Started\n\nBefore you begin, ensure you have the latest version of the main branch:\n\n```bash\ngit checkout main\ngit pull origin main\n```\nThen, create a new branch for your contribution:\n\n```bash\nCopy code\ngit checkout -b \u003cyour-branch-name\u003e\n```\n## Bug Reports\n\nIf you encounter any bugs, please file an issue on our GitHub repository. Include as much detail as possible:\n\n- A clear and concise description of the bug\n- Steps to reproduce the behavior\n- Expected behavior vs actual behavior\n- Screenshots if applicable\n- Any additional context or logs\n\n## Suggesting Enhancements\n\nWe are always looking for suggestions to improve DataTonic. If you have an idea, please open an issue with the tag 'enhancement'. Provide:\n\n- A clear and concise description of the proposed feature\n- Any relevant examples or mockups\n- A description of the benefits to DataTonic users\n\n## Code Contributions\nIf you'd like to contribute code, please follow these steps:\n\n### Step 1: Set Up Your Environment\nFollow the setup instructions in the README to get DataTonic running on your local machine.\n\n### Step 2: Make Your Changes\nEnsure that your changes adhere to the existing code structure and standards. Add or update tests as necessary.\n\n### Step 3: Commit Your Changes\nWrite clear and meaningful commit messages. This helps to understand the purpose of your changes and speed up the review process.\n\n```bash\ngit commit -m \"A brief description of the commit\"\n```\n### Step 4: Push to the Branch\nPush your changes to your remote branch:\n\n ```bash\ngit push origin \u003cyour-branch-name\u003e\n```\n### Step 5: Open a Pull Request\nGo to the repository on GitHub and open a new pull request against the main branch. Provide a clear description of the problem you're solving. Link any relevant issues.\n\n### Step 6: Code Review\nMaintainers will review your pull request. Be responsive to feedback to ensure a smooth process.\n\n# Code of Conduct\nPlease note that this project is released with a Contributor Code of Conduct. By participating in this project, you agree to abide by its terms.\n\n# License\nBy contributing to DataTonic, you agree that your contributions will be licensed under its LICENSE.\n\n**Thank you for contributing to DataTonic!🚀**\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjosephrp%2Fdatatonic","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjosephrp%2Fdatatonic","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjosephrp%2Fdatatonic/lists"}