{"id":22557441,"url":"https://github.com/austin-starks/Promptimizer","last_synced_at":"2025-08-04T14:31:50.488Z","repository":{"id":250235119,"uuid":"833879790","full_name":"austin-starks/Promptimizer","owner":"austin-starks","description":"An Automated AI-Powered Prompt Optimization Framework","archived":false,"fork":false,"pushed_at":"2024-07-31T11:54:58.000Z","size":273,"stargazers_count":158,"open_issues_count":1,"forks_count":13,"subscribers_count":3,"default_branch":"master","last_synced_at":"2024-12-03T17:04:46.258Z","etag":null,"topics":["ai","anthropic","aritificial-intelligence","genetic-algorithm","large-language-model","llama3","llama3-1","llamacpp","llm","machine-learning","mongodb","ollama","open-source","openai","optimization","prompt-engineering","prompt-management"],"latest_commit_sha":null,"homepage":"https://nexustrade.io/","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/austin-starks.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-07-26T00:42:40.000Z","updated_at":"2024-12-02T15:33:09.000Z","dependencies_parsed_at":"2024-09-13T22:42:07.877Z","dependency_job_id":"09c53f53-0b30-4e03-bbdf-9b086129902f","html_url":"https://github.com/austin-starks/Promptimizer","commit_stats":{"total_commits":10,"total_committers":2,"mean_commits":5.0,"dds":0.4,"last_synced_commit":"3acdcbf01e29defefb0a524c648b9f67855693f7"},"previous_names":["austin-starks/automated-prompt-optimization","austin-starks/promptimizer"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/austin-starks%2FPromptimizer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/austin-starks%2FPromptimizer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/austin-starks%2FPromptimizer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/austin-starks%2FPromptimizer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/austin-starks","download_url":"https://codeload.github.com/austin-starks/Promptimizer/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":228651889,"owners_count":17951895,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai","anthropic","aritificial-intelligence","genetic-algorithm","large-language-model","llama3","llama3-1","llamacpp","llm","machine-learning","mongodb","ollama","open-source","openai","optimization","prompt-engineering","prompt-management"],"created_at":"2024-12-07T20:02:11.449Z","updated_at":"2024-12-07T20:02:40.984Z","avatar_url":"https://github.com/austin-starks.png","language":"TypeScript","funding_links":[],"categories":["*Ops for AI","TypeScript"],"sub_categories":["LLMOps"],"readme":"# Promptimizer – Automated AI-Powered Prompt Optimization Framework\n\nThis project implements an automated system for optimizing AI prompts using genetic algorithms and machine learning techniques. It's designed to evolve and improve any LLM prompt. The example in this repo is focused on AI-driven stock screening.\n\n[Read more about it here.](https://medium.com/p/bbcb9afaef83)\n\n## Warning/Disclaimer\nOptimizing prompts using this framework can be VERY expensive! I accept no liability for any costs incurred. If cost is a consideration, please use local LLMs like Ollama for optimization.\n\n## Check out NexusTrade\n\nTo see the results of the optimized prompt, check out [NexusTrade.io](https://nexustrade.io/). NexusTrade is an AI-Powered automated trading and investment platform that allows users to create, test, optimize, and deploy algorithmic trading strategies.\n\nWith the stock screener, you can ask any of the questions within the groundTruths directory, including:\n\n- Which semiconductor stocks had the highest net income for the fiscal year 2020?\n- What are the latest EBITDA values for the top 10 pharmaceutical stocks (by market cap)?\n- What is the average free cash flow of the top 10 e-commerce stocks by market cap?\n\nAny question about technical or fundamental data, NexusTrade's AI Stock Screener can answer. Try it today for free!\n\n## Features\n\n- Genetic algorithm-based optimization of AI prompts\n- Population management with crossover and mutation operations\n- Training and validation using separate datasets\n- Automated evaluation of prompt performance\n- Multi-generational evolution of prompts\n- Customizable parameters for population size, generations, and more\n\n## Prerequisites\n\nBefore you run this project, ensure you have the following prerequisites set up:\n\n1. **Node.js**: Make sure you have Node.js installed. You can download it from [nodejs.org](https://nodejs.org/).\n\n2. **Populate `inputs.ts`**: Create and populate the `inputs.ts` file with the following format:\n\n```javascript\nconst inputs = [\n  {\n    text: \"\u003cQuestion to ask the model\u003e\",\n    foldername: \"foldername_for_the_input_output\",\n  },\n];\n\nexport default inputs;\n```\n\n3. **Populate `additionalSystemPrompts.ts`**: Create and populate the `additionalSystemPrompts.ts` file with the following format:\n\n```javascript\nconst additionalSystemPrompts = [\n  \"System Prompt 1\",\n  \"System Prompt 2\",\n  \"System Prompt 3\",\n  \"System Prompt 4\",\n  \"System Prompt 5\",\n];\n\nexport default additionalSystemPrompts;\n```\n\n4. **Set Up Environment Variables**: Create a `.env` file in the root directory of the project and add the following environment variables:\n\n```plaintext\nANTHROPIC_API_KEY=your_anthropic_api_key\nOPENAI_API_KEY=your_openai_api_key\nOLLAMA_SERVICE_URL=http://localhost:11434\nGOOGLE_APPLICATION_CREDENTIALS_JSON=path_to_your_google_application_credentials_json\nCLOUD_DB=your_cloud_db_connection_string\nLOCAL_DB=your_local_db_connection_string\nMODEL_NAME=your_model_name\n```\n\nReplace `your_anthropic_api_key`, `your_openai_api_key`, `path_to_your_google_application_credentials_json`, `your_cloud_db_connection_string`, `your_local_db_connection_string`, and `your_model_name` with your actual API keys, the path to your Google application credentials JSON file, your database connection strings, and the model name. You can change the model to an open-source model from Ollama or one from Anthropic by setting the `MODEL_NAME` environment variable.\n\nFor the local db, you can populate it with the following:\n\n```plaintext\nLOCAL_DB=\"mongodb://127.0.0.1:27017/promptoptimizer\"\n```\n\nBy default, the system will use the local database specified in `process.env.LOCAL_DB`. If you want to use a cloud database, ensure `process.env.CLOUD_DB` is populated and modify the code to use it as needed.\n\n## Installation\n\n### Node.js Dependencies\n\nTo install the necessary Node.js dependencies, run the following command in the root directory of the project:\n\n```bash\nnpm install\n```\n\n### TypeScript and ts-node\n\nTo run TypeScript files directly, you need to install `ts-node`. You can install it globally using the following command:\n\n```bash\nnpm install -g ts-node\n```\n\nAlternatively, you can add it as a dev dependency to your project:\n\n```bash\nnpm install --save-dev ts-node\n```\n\n### Python Dependencies\n\nTo install the necessary Python dependencies, ensure you have Python installed and then run the following command in the root directory of the project:\n\n```bash\npip install -r requirements.txt\n```\n\nThis will install the following Python packages:\n\n- `matplotlib`\n- `seaborn`\n- `pandas`\n\n### MongoDB Installation\n\nTo install MongoDB locally, follow these steps:\n\n1. **Download MongoDB**: Go to the [MongoDB Download Center](https://www.mongodb.com/try/download/community) and download the MongoDB Community Server for your operating system.\n\n2. **Install MongoDB**:\n\n- **Windows**: Run the downloaded `.msi` installer and follow the installation instructions.\n- **macOS**: Use Homebrew to install MongoDB. Open a terminal and run:\n\n```bash\nbrew tap mongodb/brew\nbrew install mongodb-community@5.0\n```\n\n- **Linux**: Follow the specific instructions for your Linux distribution on the MongoDB website.\n\n3. **Start MongoDB**:\n\n- **Windows**: MongoDB should start automatically after installation. If not, you can start it manually by running `mongod` in the command prompt.\n- **macOS**: Start MongoDB using Homebrew services:\n\n```bash\nbrew services start mongodb/brew/mongodb-community\n```\n\n- **Linux**: Start MongoDB by running:\n\n```bash\nsudo systemctl start mongod\n```\n\n4. **Verify Installation**: To verify that MongoDB is running, open a terminal or command prompt and run:\n\n```bash\nmongo\n```\n\nThis should open the MongoDB shell, indicating that MongoDB is installed and running correctly.\n\n## Generating Ground Truths\n\nTo steer the model towards the desired behavior, you need to know exactly how you want the model to respond to a wide range of inputs. To do this, you will update the file `input.ts` with filenames and inputs you want the model to understand. Then, you will execute the script `populateGroundTruth.ts`. This script allows you to create ground truths in a semi-automated way.\n\n## Creating a Scoring Heuristic\n\nUsing some method (such as a large language model), you need to be able to quantify how close your output is to your desired output. You can do this using the LLM-based \"Prompt Evaluator\" within the repo. The \"Prompt Evaluator\" takes the output of the model and the expected output and returns a score.\n\n## Setting Up an Alternative Model\n\nTo use Ollama as an alternative to OpenAI, follow these steps:\n\n1. **Download Ollama**: Go to [ollama.com/download](https://ollama.com/download) and download the appropriate version for your operating system.\n\n2. **Download the Model**: Visit [ollama.com/library/llama3.1](https://ollama.com/library/llama3.1) to download the model you want to use.\n\n3. **Set the Environment Variable**: Make sure to set the following environment variable in your `.env` file:\n\n```plaintext\nOLLAMA_SERVICE_URL=http://localhost:11434\n```\n\nIf you have an `ANTHROPIC_API_KEY`, you can alternatively use Anthropic for prompt optimization.\n\n## Running the TypeScript Script\n\nTo run the TypeScript script, you can use `ts-node`. Ensure you have `ts-node` installed as described above, and then run the following command:\n\n```bash\nts-node main.ts\n```\n\nAlternatively, you can compile the TypeScript code to JavaScript and then run it using Node.js:\n\n1. Compile the TypeScript code:\n\n```bash\ntsc\n```\n\n2. Run the compiled JavaScript code:\n\n```bash\nnode dist/main.js\n```\n\n## Visualizing Performance\n\nTo visualize the training and validation fitness over generations, you can use the Python script `graph.py`. This script generates graphs so you can see how the performance of your prompt changed over time. To run the script, use the following command:\n\n```bash\npython graph.py\n```\n\nBy following these instructions, you'll be able to optimize any arbitrary prompt, quantify its performance, and visualize how it improves over time.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faustin-starks%2FPromptimizer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Faustin-starks%2FPromptimizer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faustin-starks%2FPromptimizer/lists"}