{"id":21769956,"url":"https://github.com/mideind/icelandiceval","last_synced_at":"2025-03-21T06:14:34.842Z","repository":{"id":202762643,"uuid":"708081498","full_name":"mideind/IcelandicEval","owner":"mideind","description":"Utilities to generate Icelandic evaluation data sets for LLMs","archived":false,"fork":false,"pushed_at":"2023-10-23T11:34:19.000Z","size":1940,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":7,"default_branch":"main","last_synced_at":"2025-01-26T03:08:13.323Z","etag":null,"topics":["evaluation","grammar","icelandic","inflection","llm","python"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mideind.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2023-10-21T13:22:22.000Z","updated_at":"2023-11-11T13:55:07.000Z","dependencies_parsed_at":null,"dependency_job_id":"c5ead3fc-4230-438c-a1f6-9a3c6e58a7d3","html_url":"https://github.com/mideind/IcelandicEval","commit_stats":null,"previous_names":["mideind/icelandiceval"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mideind%2FIcelandicEval","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mideind%2FIcelandicEval/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mideind%2FIcelandicEval/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mideind%2FIcelandicEval/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mideind","download_url":"https://codeload.github.com/mideind/IcelandicEval/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244745751,"owners_count":20503050,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["evaluation","grammar","icelandic","inflection","llm","python"],"created_at":"2024-11-26T14:10:45.900Z","updated_at":"2025-03-21T06:14:34.824Z","avatar_url":"https://github.com/mideind.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# IcelandicEval\nA repository of utilities to generate Icelandic\nevaluation data sets for LLMs. The data sets are mostly about\nword inflection and grammatical correctness.\n\n## calc-freq.py\n\nThis utility program generates evaluation data\nfor LLMs, typically OpenAI's GPT-4, to test proficiency\nin Icelandic. The data consists of lists of noun phrases,\nwhere each phrase contains an adjective and a noun,\nand the task is to inflect the adjective and noun together\nin all four cases (nominative, accusative, dative, genitive),\nin singular as well as plural.\n\nThe final output of the program is a set of three JSONL\nfiles, each containing a number of samples. The samples are\nbucketed into three categories, easy, medium and hard,\ndepending on the frequency of the adjectives and nouns used\nin each sample. Each sample is an LLM prompt and an ideal\ncompletion.\n\n### Usage\n\nClone this repo into a directory, create a virtualenv and\ninstall the requirements:\n\n```bash\n    git clone https://github.com/mideind/IcelandicEval.git\n    cd IcelandicEval\n    python3 -m venv venv\n    source venv/bin/activate\n    pip install -r requirements.txt\n```\n\nThe `nouns.csv` and `adjectives.csv` files need to be present\nin the `data` directory. They were originally created\nby querying the BÍN database\n([bin.arnastofnun.is](https://bin.arnastofnun.is)), for example\n(in the psql command line):\n\n```bash\n    psql\u003e \\copy (select ord, ofl from bin2023\n        where ofl in ('kk', 'kvk', 'hk')) to 'data/nouns.csv' with csv;\n\n    psql\u003e \\copy (select ord from bin2023 where ofl = 'lo')\n        to 'data/adjectives.csv' with csv;\n```\n\nThen, given those files, this program is run to generate\nrandomly sampled, bucketed lists of nouns and adjectives\nrespectively. The buckets are created by frequency of\noccurrence of the word forms in the `icegrams` database,\nwith bucket 0 containing the least frequent words and bucket\n2 the most frequent. The bucket files are created in the `data`\ndirectory, under the names `nouns-{0,1,2}.txt` and `adj-{0,1,2}.txt`.\n\n```bash\n    python calc-freq.py --nouns\n    python calc-freq.py --adjectives\n```\n\nFinally, after the buckets 0-2 have been created, the\nfinal evaluation samples can be generated. The number\nof samples desired from each bucket can be passed in as\na command line parameter, defaulting to 20.\n\n```bash\n    python calc-freq.py --generate [N, default 20]\n```\n\nThe results are found in three JSONL files, named\n`data/icelandic-inflection-{easy,medium,hard}/samples.jsonl`.\nThey are in a format that is suitable for use with OpenAI's\nevals suite (see [github.com/openai/evals](https://github.com/openai/evals)).\n\n# License\n\nCopyright (C) 2023 Miðeind ehf. All rights reserved.\n\nThis software is under the MIT License. Consult the LICENSE.md file\nfor details.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmideind%2Ficelandiceval","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmideind%2Ficelandiceval","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmideind%2Ficelandiceval/lists"}