{"id":22199797,"url":"https://github.com/betagouv/comparia","last_synced_at":"2025-08-02T04:34:53.832Z","repository":{"id":243675758,"uuid":"810918226","full_name":"betagouv/ComparIA","owner":"betagouv","description":"Interroger à l'aveugle deux modèles de langage conversationnels sur des tâches exprimées en français et comparer les résultats. ","archived":false,"fork":false,"pushed_at":"2025-07-31T20:12:08.000Z","size":213263,"stargazers_count":39,"open_issues_count":12,"forks_count":6,"subscribers_count":3,"default_branch":"develop","last_synced_at":"2025-07-31T20:37:22.244Z","etag":null,"topics":["fastapi","gradio","llm","svelte"],"latest_commit_sha":null,"homepage":"https://beta.gouv.fr/startups/languia.html","language":"CSS","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/betagouv.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2024-06-05T15:41:15.000Z","updated_at":"2025-07-31T20:12:13.000Z","dependencies_parsed_at":"2024-08-19T11:01:32.048Z","dependency_job_id":"051d74be-d839-47ee-aee0-b98e913b2706","html_url":"https://github.com/betagouv/ComparIA","commit_stats":null,"previous_names":["betagouv/languia","betagouv/comparia"],"tags_count":193,"template":false,"template_full_name":null,"purl":"pkg:github/betagouv/ComparIA","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/betagouv%2FComparIA","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/betagouv%2FComparIA/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/betagouv%2FComparIA/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/betagouv%2FComparIA/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/betagouv","download_url":"https://codeload.github.com/betagouv/ComparIA/tar.gz/refs/heads/develop","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/betagouv%2FComparIA/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":268334630,"owners_count":24233794,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-02T02:00:12.353Z","response_time":74,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["fastapi","gradio","llm","svelte"],"created_at":"2024-12-02T15:18:03.050Z","updated_at":"2025-08-02T04:34:53.802Z","avatar_url":"https://github.com/betagouv.png","language":"CSS","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cbr\u003e\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"https://comparia.beta.gouv.fr/\"\u003e\n  \u003cimg src=\"https://github.com/user-attachments/assets/bd071ffd-1253-486d-ad18-9f5b371788b0\" width=300px alt=\"compar:IA logo\" /\u003e  \u003c/a\u003e\n\u003c/p\u003e\n\n\n\u003ch2 align=\"center\" \u003eComparateur d’IA conversationnelles / Conversational AI comparator\u003c/h3\u003e\n\u003cp align=\"center\"\u003eCompar:IA est un outil permettant de comparer à l’aveugle différents modèles d'IA conversationnelle pour sensibiliser aux enjeux de l'IA générative (biais, impact environmental) et constituer des jeux de données de préférence en français.\u003c/p\u003e\n\u003cp align=\"center\"\u003eCompar:IA is a tool for blindly comparing different conversational AI models to raise awareness about the challenges of generative AI (bias, environmental impact) and to build up French-language preference datasets.\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\u003ca href=\"https://comparia.beta.gouv.fr/\"\u003e🌐 comparia.beta.gouv.fr\u003c/a\u003e · \u003ca href=\"https://www.comparia.beta.gouv.fr/a-propos\"\u003e📚 À propos\u003c/a\u003e · \u003ca href=\"https://beta.gouv.fr/startups/languia.html\"\u003e🚀 Description de la startup d'Etat\u003c/a\u003e\u003cp\u003e\n\u003cdiv align=\"center\"\u003e\n  \u003ca href=\"https://comparia.beta.gouv.fr/\" \n     aria-label=\"Cliquez pour se rendre sur la plateforme hébergée\"\n     title=\"Capture d'écran du comparateur\"\u003e\n    \u003cimg \n      src=\"https://github.com/user-attachments/assets/6c8257fc-a2e5-4ee1-8052-dbf14a0419ea\" \n      alt=\"Aperçu du comparateur\" \n      width=\"800\"\n    /\u003e\n  \u003c/a\u003e\n\u003c/div\u003e\n\u003cdiv align=\"center\"\u003e\n  \u003csub\u003e\n    \u003ci\u003eCliquez sur l'image ci-dessus pour consulter le site (s'ouvre dans un nouvel onglet)\u003c/i\u003e\n  \u003c/sub\u003e\n\u003c/div\u003e\n\n\u003cbr\u003e\n\nLe comparateur est basé sur [Gradio](https://www.gradio.app/) et [FastChat](https://github.com/lm-sys/FastChat/), le code de l'arène Chatbot Arena par LMSYS (voir [Project architecture and rationale (en)](https://github.com/betagouv/ComparIA?tab=readme-ov-file#project-architecture-and-rationale) plus bas).\n\nThe comparator is based on [Gradio](https://www.gradio.app/) and [FastChat](https://github.com/lm-sys/FastChat/), the Chatbot Arena code by LMSYS (see [Project architecture and rationale](https://github.com/betagouv/ComparIA?tab=readme-ov-file#project-architecture-and-rationale) below).\n\n\n\n## Run the arena\n\n1. Rename `register-api-endpoint-file.json.dist` to `register-api-endpoint-file.json` and add valid API keys\n\n### Without Docker\n\nDue to how Gradio's Custom Components work and because they haven't been published as Python packages, building them manually is a bit tedious. At the moment we use 4 custom components: \n\n```bash\npip install -r requirements.txt\ngradio cc install custom_components/frinput\ngradio cc build --no-generate-docs custom_components/frinput\ngradio cc install custom_components/customradiocard;gradio cc build --no-generate-docs custom_components/customradiocard\ngradio cc install custom_components/customdropdown;gradio cc build --no-generate-docs custom_components/customdropdown\ngradio cc install custom_components/customchatbot; npm install @gouvfr/dsfr;\ngradio cc build --no-generate-docs custom_components/customchatbot\n```\nthen `export LANGUIA_DEBUG=True; uvicorn main:app --reload --timeout-graceful-shutdown 1` or simply `uvicorn main:app`\nFor arena only, you can also launch: `gradio demo.py`\n\n#### To dev for a custom component\n\nYou can launch this to work on the first custom component (first screen):\n`gradio cc dev --component-directory custom_components/customdropdown demo.py`\n\n### With Docker\n\n`cd docker/; docker compose up -d`\n\nFor image building only you can use: `docker build -t comparia . -f docker/Dockerfile`\n\n## Project architecture and rationale\n\n### LMSys fork\nWe initially forked LMSYS' FastChat codebase, used at https://lmarena.ai to get an immediately running arena. Its architecture was composed of:\n- the arena (a Gradio project with 2-3 Python files)\n- a controller to register model workers\n\nBut as it was easier to run models in vLLM Docker containers or by using external APIs, the controller / model workers architecture ended up being unused code. Furthermore, we needed a dashboard for the controller so it got recoded.\n\n### Custom Components\nOur main focus with compar:IA is to invest heavily on overall design and UX/UI. Thanks to Gradio's [Custom Components](https://www.gradio.app/guides/custom-components-in-five-minutes) we can customize any Gradio component as a Svelte app, and control the user interface look and feel.\n\nWe currently use 4 distinct (and sometimes poorly named) Custom Components:\n- `FrInput`: the [DSFR](https://www.systeme-de-design.gouv.fr/) input component\n- `CustomDropdown`: encompasses most of the first screen, with mode selection, models selection, and initial textarea\n- `CustomRadioCard`: used in the first screen for suggestions and later for voting\n- `CustomChatbot`: a component crafted for the specific compar:IA experience, allowing you to compare two chatbots' response to one user message, and receive user's feedback\n\n\n\n### Mounted `gradio.Blocks` within a FastAPI app\n\nBecause we needed a static website as well, we used Gradio's [`mount_gradio_app`](https://www.gradio.app/docs/gradio/mount_gradio_app`) feature, allowing you to customize how FastAPI serves the gradio app (Gradio is based on FastAPI), while using the underlying FastAPI app to serve other pages. This lives in `main.py` while most of the Gradio code is split between `languia/block_arena.py` and `languia/listeners.py`.\nThe static site's pages are in the `templates/` folder, which also hosts the complex Jinja2 template files needed in the arena (especially after the \"reveal\" step).\n\n### Future evolutions\n\nAfter 8 months of intensive development, the Gradio framework may show some limits, especially when it comes to fully custom CSS. Ugly CSS overrides are used heavily throughout this repo (especially in the infamous `assets/custom-arena.css`), while the integration of the French design system (DSFR) is made difficult by how Gradio adds a lot of Svelte-generated CSS everywhere.\nFurthermore, since the app is now more stable, we don't need to iterate quickly anymore, which is what Gradio allowed, and we could gain some snappiness by using a Svelte SPA (Single Page App) and a lighter frontend-backend communication.\nI feel there is a gradual path consisting in decapsulating the Custom Components one by one into a basic Svelte app, and replace Gradio with a basic FastAPI endpoint, screen-by-screen and iterating. If you have opinions on this, I warmly welcome you to open an issue on the matter 🙃\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbetagouv%2Fcomparia","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbetagouv%2Fcomparia","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbetagouv%2Fcomparia/lists"}