{"id":17669382,"url":"https://github.com/trigaten/the_prompt_report","last_synced_at":"2025-04-06T02:10:55.184Z","repository":{"id":203621218,"uuid":"709620206","full_name":"trigaten/The_Prompt_Report","owner":"trigaten","description":null,"archived":false,"fork":false,"pushed_at":"2024-06-15T07:16:33.000Z","size":29133,"stargazers_count":291,"open_issues_count":18,"forks_count":24,"subscribers_count":6,"default_branch":"main","last_synced_at":"2024-10-24T09:47:55.567Z","etag":null,"topics":["ai","chatgpt","generative","llms","prisma","review","systematic"],"latest_commit_sha":null,"homepage":"https://trigaten.github.io/Prompt_Survey_Site/","language":"HTML","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/trigaten.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-10-25T03:57:28.000Z","updated_at":"2024-10-19T18:08:03.000Z","dependencies_parsed_at":"2024-05-03T18:07:54.375Z","dependency_job_id":"12840637-ef89-43e1-9e11-02b396523eee","html_url":"https://github.com/trigaten/The_Prompt_Report","commit_stats":null,"previous_names":["trigaten/prompt_systematic_review","trigaten/the_prompt_report"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/trigaten%2FThe_Prompt_Report","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/trigaten%2FThe_Prompt_Report/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/trigaten%2FThe_Prompt_Report/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/trigaten%2FThe_Prompt_Report/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/trigaten","download_url":"https://codeload.github.com/trigaten/The_Prompt_Report/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247423515,"owners_count":20936626,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai","chatgpt","generative","llms","prisma","review","systematic"],"created_at":"2024-10-24T00:22:33.562Z","updated_at":"2025-04-06T02:10:55.157Z","avatar_url":"https://github.com/trigaten.png","language":"HTML","funding_links":[],"categories":[],"sub_categories":[],"readme":"# The Prompt Report Code Repository \nGenerative Artificial Intelligence (GenAI) systems are being increasingly deployed across all parts of\nindustry and research settings. Developers and end users interact with these systems through the use of\nprompting or prompt engineering. While prompting is a widespread and highly researched concept, there\nexists conflicting terminology and a poor ontological understanding of what constitutes a prompt due to the\narea’s nascency. This repository is the code for The Prompt Report, our research that establishes a structured \nunderstanding of prompts, by assembling a taxonomy of prompting techniques and analyzing their use. This code \nallows for the automated review of papers, the collection of data, and the running of experiments. Our dataset \nis available on [Hugging Face](https://huggingface.co/datasets/PromptSystematicReview/ThePromptReport) and our paper is \navailable on [ArXiv.org](https://arxiv.org/pdf/2406.06608). Information is also available on our [website](https://trigaten.github.io/Prompt_Survey_Site/).\n\n## Table of Contents\n- [Prompt Engineering Survey](#prompt-engineering-survey)\n  - [Install requirements](#install-requirements)\n  - [Setting up API keys](#setting-up-api-keys)\n  - [Setting up keys for running tests](#setting-up-keys-for-running-tests)\n  - [Structure of the Repository](#structure-of-the-repository)\n  - [Running the code](#running-the-code)\n    - [TLDR;](#tldr)\n  - [Notes](#notes)\n\n## Install requirements\n\nafter cloning, run `pip install -r requirements.txt` from root\n\n## Setting up API keys\n\nMake a file at root called `.env`.\n\nFor OpenAI: https://platform.openai.com/docs/quickstart \u003cbr\u003e\nFor Hugging Face: https://huggingface.co/docs/hub/security-tokens, also run `huggingface-cli login` \u003cbr\u003e\nFor Sematic Scholar: https://www.semanticscholar.org/product/api#api-key  \u003cbr\u003e\n\nUse the reference `example.env` file to fill in your API keys/tokens. \n```\nOPENAI_API_KEY=sk.-...\nSEMANTIC_SCHOLAR_API_KEY=...\nHF_TOKEN=...\n```\n\n## Setting up keys for running tests\nThen to load the .env file, type: \u003cbr\u003e\n`pip install pytest-dotenv`\n\nYou can also choose to update the env file by doing: \u003cbr\u003e\n`py.test --envfile path/to/.env`\n\nIn the case that you have several .env files, create a new env_files in the pytest config folder and type:\n```\nenv_files =\n.env\n.test.env\n.deploy.env\n```\n## Structure of the Repository\nThe script `main.py` calls the necessary functions to download all the papers, deduplicate and filter them, and then run all the experiments. \n\nThe core of the repository is in `src/prompt_systematic_review`. The `config_data.py` script contains configurations that are important for running experiments and saving time. You can see in `main.py` how some of these options are used. \n\nThe source folder is divided into 4 main sections: 3 scripts (`automated_review.py`, `collect_papers.py`,`config_data.py`) that deal with collecting the data and running the automated review, the `utils` folder that contains utility functions that are used throughout the repository, the `get_papers` folder that contains the scripts to download the papers, and the `experiments` folder that contains the scripts to run the experiments. \n\nAt the root, there is a `data` folder. It comes pre-loaded with some data that is used for the experiments, however the bulk of the dataset can either be generated by running `main.py` or by downloading the data from Hugging Face. It is in `data/experiments_output` that the results of the experiments are saved.\n\nNotably, the keywords used in the automated review/scraping process are in `src/prompt_systematic_review/utils/keywords.py`. Anyone who wishes to run the automated review can adjust these keywords to their liking in that file. \n\n## Running the code\n\n### TLDR;\n```bash\ngit clone https://github.com/trigaten/Prompt_Systematic_Review.git \u0026\u0026 cd Prompt_Systematic_Review\npip install -r requirements.txt\n# create a .env file with your API keys\nnano .env\ngit lfs install\ngit clone https://huggingface.co/datasets/PromptSystematicReview/ThePromptReport\nmv ThePromptReport/* data/\npython main.py\n```\n\nRunning `main.py` will download the papers, run the automated review, and run the experiments.\nHowever, if you wish to save time and only run the experiments, you can download the data from [Hugging Face](https://huggingface.co/datasets/PromptSystematicReview/ThePromptReport) and move the papers folder and all the csv files in the dataset into the data folder (should look like `data/papers/*.pdf` and `data/master_papers.csv` etc). Adjust main.py accordingly. \n\nEvery experiment script has a `run_experiment` function that is called in `main.py`. The `run_experiment` function is responsible for running the experiment and saving the results. However each script can be run individually by just running `python src/prompt_systematic_review/experiments/\u003cexperiment_name\u003e.py` from root. \n\nThere is one experiment, `graph_internal_references` that, because of weird issues with parallelism, is better run from root as an individual script. To avoid it causing issues with other experiments, it is run last as it is ordered at the bottom of the list in `experiments/__init__.py`.\n\n\n\n## Notes\n\n- Sometimes a paper title may appear differently on the arXiv API. For example, \"Visual Attention-Prompted Prediction and Learning\" (arXiv:2310.08420), according to arXiv API is titled \"A visual encoding model based on deep neural networks and transfer learning\"\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftrigaten%2Fthe_prompt_report","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftrigaten%2Fthe_prompt_report","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftrigaten%2Fthe_prompt_report/lists"}