{"id":20484420,"url":"https://github.com/sap-samples/llm-round-trip-correctness","last_synced_at":"2025-04-13T14:42:54.145Z","repository":{"id":243064219,"uuid":"807487257","full_name":"SAP-samples/llm-round-trip-correctness","owner":"SAP-samples","description":"This repo provides code for evaluation of llm round-trip-correctness on text to process model and vice versa","archived":false,"fork":false,"pushed_at":"2025-03-07T13:59:49.000Z","size":4666,"stargazers_count":3,"open_issues_count":2,"forks_count":1,"subscribers_count":6,"default_branch":"main","last_synced_at":"2025-03-27T05:41:36.461Z","etag":null,"topics":["benchmarking","business","evaluation-framework","genai","processes","round-trip-correctness"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/SAP-samples.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-05-29T07:41:19.000Z","updated_at":"2025-02-25T19:07:26.000Z","dependencies_parsed_at":"2024-06-06T14:44:05.062Z","dependency_job_id":"2008a909-3130-417c-9100-73d6234e467e","html_url":"https://github.com/SAP-samples/llm-round-trip-correctness","commit_stats":null,"previous_names":["sap-samples/model-to-model-evaluation-code"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SAP-samples%2Fllm-round-trip-correctness","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SAP-samples%2Fllm-round-trip-correctness/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SAP-samples%2Fllm-round-trip-correctness/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SAP-samples%2Fllm-round-trip-correctness/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/SAP-samples","download_url":"https://codeload.github.com/SAP-samples/llm-round-trip-correctness/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248731679,"owners_count":21152838,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["benchmarking","business","evaluation-framework","genai","processes","round-trip-correctness"],"created_at":"2024-11-15T16:22:17.726Z","updated_at":"2025-04-13T14:42:54.092Z","avatar_url":"https://github.com/SAP-samples.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![REUSE status](https://api.reuse.software/badge/github.com/SAP-samples/llm-round-trip-correctness)](https://api.reuse.software/info/github.com/SAP-samples/llm-round-trip-correctness)\n\n# Round-trip-correctness to evaluate BPMN generation\n\n## Description\nThis repository tests the idea of a proxy evaluation method for text to BPMN model pipeline.\nThe proxy evaluation involves a round-trip pipeline, \"text to bpmn to text\" or \"bpmn to text to bpmn\" and calculating an average similarity metric we call RTC in the absence of a ground truth BPMN/text.\nTo show if the proxy method is effective, we first must investigate how the existing BPMN to BPMN evaluation from [model_evaluation](./model_evaluation) module correlates with the text to text similarity methods from [text_evaluation](./text_evaluation) by running the pipelines in this repository. \nThis work is inspired by [this](https://arxiv.org/abs/2402.08699) publication on text to code round-tripping.  \n\n\n\n## Requirements and set up\n\nThe requirements are in this [pyproject.toml](./pyproject.toml) file. After cloning the repository, run:\n\n```shell\npoetry install\n```\n\n## Getting started\n\nTo run the LLM specific pipeline, use a command similar to this:\n```shell\nscreen -d -m python genai_gpt_pipeline.py --model-path ./data/pet/ground_json --text-path ./data/pet/process_descriptions --example pet --direction t2t \n```\nor run the LLM agnostic pipeline as:\n```shell\nscreen -d -m python universal_pipeline.py --llm gemini --model-path ./data/pet/ground_json --text-path ./data/pet/process_descriptions --example pet --direction t2t \n```\nThe csv files are written to the results directory. The jupyter notebooks are used to visualize the results. \n\n\n## Known Issues\nNo known issue.\n\n## How to obtain support\n[Create an issue](https://github.com/SAP-samples/model-to-model-evaluation-code/issues) in this repository if you find a bug or have questions about the content.\n\n\n\n## Contributing\nIf you wish to contribute code, offer fixes or improvements, please send a pull request. Due to legal reasons, contributors will be asked to accept a DCO when they create the first pull request to this project. This happens in an automated fashion during the submission process. SAP uses [the standard DCO text of the Linux Foundation](https://developercertificate.org/).\n\n## License\nCopyright (c) 2024 SAP SE or an SAP affiliate company. All rights reserved. This project is licensed under the Apache Software License, version 2.0.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsap-samples%2Fllm-round-trip-correctness","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsap-samples%2Fllm-round-trip-correctness","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsap-samples%2Fllm-round-trip-correctness/lists"}