{"id":18395850,"url":"https://github.com/deepset-ai/biqa-llm","last_synced_at":"2025-04-14T08:54:19.342Z","repository":{"id":209871648,"uuid":"722915421","full_name":"deepset-ai/biqa-llm","owner":"deepset-ai","description":null,"archived":false,"fork":false,"pushed_at":"2023-11-29T14:57:17.000Z","size":146,"stargazers_count":8,"open_issues_count":0,"forks_count":2,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-03-27T22:23:33.642Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/deepset-ai.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2023-11-24T08:54:44.000Z","updated_at":"2024-07-22T10:57:49.000Z","dependencies_parsed_at":"2023-11-29T15:53:02.849Z","dependency_job_id":"3a9d2687-843f-4423-8e4d-dd70f2a9aef8","html_url":"https://github.com/deepset-ai/biqa-llm","commit_stats":null,"previous_names":["deepset-ai/biqa-llm"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deepset-ai%2Fbiqa-llm","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deepset-ai%2Fbiqa-llm/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deepset-ai%2Fbiqa-llm/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deepset-ai%2Fbiqa-llm/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/deepset-ai","download_url":"https://codeload.github.com/deepset-ai/biqa-llm/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248852109,"owners_count":21171839,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-06T02:12:10.056Z","updated_at":"2025-04-14T08:54:19.320Z","avatar_url":"https://github.com/deepset-ai.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"## BIQA with LLMs\n\nExplorations in using latest LLMs for BIQA.\n\n### Data\n\nThe data is from the [Stack Overflow Developer Survey 2023](https://survey.stackoverflow.co/2023/).\n\n- `data/eval_set_multi_answers_res.json`: Question and query pairs as list of `SQLSample`s with possibly more than one valid SQL for a question. Also results included.\n- `data/survey_results_normalized_v2.db`: The main sqlite file. Download from here: [deepset/stackoverflow-survey-2023-text-sql](https://huggingface.co/datasets/deepset/stackoverflow-survey-2023-text-sql).\n\nOr download as:\n```console\nwget -O data/survey_results_normalized_v2.db \"https://drive.google.com/uc?export=download\u0026id=1e_knoK9rYgWe8ADUw3PC8Fp6Jhnjgoms\u0026confirm=t\"\n```\n\n### Environment setup\n\n```console\npip install -r requirements.txt\n```\n\n### Running the Evaluation\n\n_Note: To use the OpenAI models, the `OPENAI_API_KEY` environment variable needs to be set. Can also put in `.env` file to be loaded by `python-dotenv`_\n\nCreate annotations i.e. fill in the predictions for the eval set:\n\n```console\npython create_annotations.py \\\n       -i data/eval_set_multi_answers_res.json \\\n       -o eval_preds.json \\\n```\n\nEvaluate manually i.e. go through each evaluation (where pred available) and label them correct or not. If labelled correct, the prediction is added to the labels/answers.\n```console\npython evaluate_manually.py eval_preds.json eval_preds_manual.json\n```\n\nCalculate the final performance metrics:\n\n```console\npython calculate_metrics.py eval_preds_manual.json\n```\n\n### Different Approaches\n\nSchema + Examples:\n```console\npython create_annotations.py \\\n       -i data/eval_set_multi_answers_res.json \\\n       -o eval_preds.json \\\n```\n\nSchema + raw description\n```console\npython create_annotations.py \\\n -i data/eval_set_multi_answers_res.json\\\n -o eval_preds_base_raw_desc.json \\\n -g base --raw-description \\\n```\n\nSchema + column descriptions + few shot\n```console\npython create_annotations.py \\\n-i data/eval_set_multi_answers_res.json \\\n-o eval_preds_base_col_desc_fs.json \\\n-g base -d per-column --few-shot\n```\n\nAgents:\n```console\npython create_annotations.py \\\n -i data/eval_set_multi_answers_res.json \\\n -o eval_preds_agents.json \\\n -g agent\n```\n\n(Perfect) Retrieval:\n```console\npython create_annotations.py \\ \n -i data/eval_set_multi_answers_res.json \\\n -o eval_preds_retriever.json \\\n -g retriever -d per-column \\\n```\n\n### Running the API\n\nFor the application to work with OpenAI models, the `OPENAI_API_KEY` environment variable needs to be set.\n\nYou can set it directly or put it in the `.env` file.\n\n```console\nuvicorn api.main:app --reload --host=\"0.0.0.0\" --port=8000\n```\n\n### Helper scripts\n\n`column_descriptions.py`: To generate the \"seed\" column-level description file (to be completed manually)\n`count_tokens.py`: Count number of tokens or to view the final prompt\n`retriever_analysis.py`: Analysis of the retriever performance + plot generation\n`view_eval_set.py`: View the data in the eval (or prediction) set\n\n### Appendix\n\n#### Data Creation\n\nCreated with this [Notebook](https://colab.research.google.com/drive/12NUeRMsld0toXMSXKFMaQVAv58XwOAT1?usp=sharing); uses [this spreadsheet](https://docs.google.com/spreadsheets/d/1Xh_TgMbyitvtw08g0byEmBpkwDGZDdBYenthOzcK6qI/edit?usp=sharing) defining manual adjustments.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdeepset-ai%2Fbiqa-llm","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdeepset-ai%2Fbiqa-llm","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdeepset-ai%2Fbiqa-llm/lists"}