{"id":28206912,"url":"https://github.com/tamohannes/urartu","last_synced_at":"2025-06-11T22:32:09.985Z","repository":{"id":188327077,"uuid":"677753887","full_name":"tamohannes/urartu","owner":"tamohannes","description":"🦁 An open-source NLP framework that offers high-level wrappers designed for effortless launch, enhanced reproducibility, superior control, and unmatched flexibility for your experiments.","archived":false,"fork":false,"pushed_at":"2025-05-13T09:53:44.000Z","size":185,"stargazers_count":11,"open_issues_count":1,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-05-17T12:14:51.260Z","etag":null,"topics":["aim","aimstack","deep-learning","experiment-tracking","machine-learning","natural-language-processing","python","pytorch","reproducible-research","slurm-workload-manager"],"latest_commit_sha":null,"homepage":"https://tamohannes.com","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/tamohannes.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2023-08-12T14:25:58.000Z","updated_at":"2025-05-13T09:53:47.000Z","dependencies_parsed_at":null,"dependency_job_id":"ed6fd066-7f89-4aa6-8eb7-bb8c93fdb530","html_url":"https://github.com/tamohannes/urartu","commit_stats":null,"previous_names":["tmynn/urartu","tamohannes/urartu"],"tags_count":2,"template":false,"template_full_name":null,"purl":"pkg:github/tamohannes/urartu","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tamohannes%2Furartu","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tamohannes%2Furartu/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tamohannes%2Furartu/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tamohannes%2Furartu/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/tamohannes","download_url":"https://codeload.github.com/tamohannes/urartu/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tamohannes%2Furartu/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":259354893,"owners_count":22844947,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aim","aimstack","deep-learning","experiment-tracking","machine-learning","natural-language-processing","python","pytorch","reproducible-research","slurm-workload-manager"],"created_at":"2025-05-17T12:14:49.027Z","updated_at":"2025-06-11T22:32:09.978Z","avatar_url":"https://github.com/tamohannes.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003c!--- BADGES: START ---\u003e\n\n[![PyPI - Package Version](https://img.shields.io/pypi/v/urartu?logo=pypi\u0026style=flat\u0026color=orange)](https://pypi.org/project/urartu/)\n[![PyPI - Python Version](https://img.shields.io/badge/Python-3.10-blue.svg?style=flat\u0026logo=python\u0026logoColor=white)](https://www.python.org/)\n[![GitHub - License](https://img.shields.io/github/license/tamohannes/urartu)](https://opensource.org/licenses/Apache-2.0)\n\n\u003c!--- BADGES: END ---\u003e\n\n🚀 New Release Available!\nWe're excited to announce a new version of our project! 🎉 Our README.md is currently being updated to reflect all the fantastic changes. In the meantime, please explore the release notes and dive in. We'd love to hear your feedback! ❤️\n\n# **Urartu 🦁**\n\nWelcome to Urartu, your go-to NLP framework designed to simplify your work. Urartu provides easy-to-use abstractions for common pipeline components, making your machine learning journey smoother.\nWith a `.yaml` file-based configuration system and seamless `slurm` job submission capabilities on clusters, Urartu removes the technical hassle so you can focus on making impactful NLP work! 🚀\n\n![urartu_schema drawio](https://github.com/tamohannes/urartu/assets/23078323/9d747c2d-9856-4dbe-85ab-74a595f86603)\n\n# **Installation**\n\nGetting started with Urartu is super easy! 🌀 Just run:\n```bash\npip install urartu\n```\n\nOr, if you prefer to install directly from the source:\n\n- Clone the repository:\n    ```bash\n    git clone git@github.com:tamohannes/urartu.git`\n    ```\n- Navigate to the project directory:\n    ```bash\n    cd urartu\n    ```\n- Execute the magic command:\n    ```bash\n    pip install -e .\n    ```\n\nAnd just like that, you're all set! ✨ Use the following command anywhere in your system to access Urartu:\n\n```bash\nurartu --help\n```\n\n# **Getting started**\n\nTo jump right in with Urartu, start with our `starter_template`. You can copy this to begin your project or check out the steps in our [Starter Template Setup](./starter_template_setup.md). Your setup will mirror what's found in this directory.\n\nThink of Urartu as the foundational framework for your projects, similar to an abstract class in object-oriented programming (OOP).\nYour project acts as the concrete implementation, where Urartu provides the foundational scaffolding.\nIt includes high-level abstractions, configuration through `.yaml` files powered by [Hydra](https://github.com/facebookresearch/hydra), and `slurm` job management utilizing the [Submitit](https://github.com/facebookincubator/submitit) library.\nThis setup ensures your projects are both flexible and robust, making your machine learning workflow efficient and scalable.\nIt also includes key NLP features such as dataset readers, model loaders, and device handlers.\n\nHere's how to get started:\n1. Extend Urartu: Inherit the structure of Urartu and customize it by writing your own actions and configurations, akin to implementing methods from an abstract class in OOP.\n2. Utilize Core Functionalities: Jumpstart your project with pre-defined functionalities:\n    - Datasets:\n        - Load a HF (Hugging Face) dataset from a dictionary, a file, or directly from the HF hub.\n    - Models:\n        - Use a HF model as a causal language model or integrate it into a pipeline.\n        - Incorporate the OpenAI API for advanced modeling.\n3. Customize Further: Develop and place your own classes within the corresponding directories of your project to meet your specific needs.\n\nBy following these steps, you can efficiently set up and customize your machine learning projects with Urartu.\n\n# **Firing Up 🔥**\n\nOnce you've cloned the `starter_template`, head over to that directory in your terminal:\n```bash\ncd starter_template\n```\n\nTo launch a single run with predefined configurations, execute the following command:\n```bash\nurartu action_config=generate aim=aim slurm=slurm\n```\n\nIf you're looking to perform multiple runs, simply use the `--multirun` flag. To configure multiple runs, add a sweeper at the end of your `generate.yaml` config file like this:\n\n```yaml\n...\n\nhydra:\n  sweeper:\n    params:\n      action_config.task.model.generate.num_beams: 1,5,10\n```\nThis setup initiates 3 separate runs, each utilizing different `num_beams` settings to adjust the model's behavior.\n\nThen, start your multi-run session with the same command:\n\n```bash\nurartu action_config=generate aim=aim slurm=slurm\n```\n\nWith these steps, you can effortlessly kickstart your machine learning experiments with Urartu, whether for a single test or comprehensive multi-run analyses!\n\n# **Navigating the Urartu Architecture**\n\nDive into the structured world of Urartu, where managing NLP components becomes straightforward and intuitive.\n\n## **Configs: Tailoring Your Setup**\n\nSet up your environment effortlessly with our configuration templates found in the `urartu/config` directory:\n- `urartu/config/main.yaml`: This primary configuration file lays the groundwork with default settings for all system keys.\n- `urartu/config/action_config` This space is dedicated to configurations specific to various actions.\n\n## **Crafting Customizations**\n\nConfiguring Urartu to meet your specific needs is straightforward. You have two easy options:\n\n1. **Custom Config Files**: Store your custom configuration files in the configs directory to adjust the settings. This directory aligns with `urartu/config`, allowing you to maintain project-specific settings in files like `generate.yaml` for your `starter_template` project.\n\n    - **Personalized User Configs**: For an even more tailored experience, create a `configs_{username}` directory at the same level as configs, replacing `{username}` with your system username. This setup automatically loads and overrides default settings without extra steps. ✨\n\nConfiguration files are prioritized in the following order: `urartu/config`, `starter_template/configs`, `starter_template/configs_{username}`, ensuring your custom settings take precedence.\n\n2. **CLI Approach**: If you prefer using the command-line interface (CLI), Urartu supports enhancing commands with key-value pairs directly in the CLI, such as:\n\n    ```bash\n    urartu action_config=example action_config.experiment_name=NAME_OF_EXPERIMENT\n    ```\n\nSelect the approach that best fits your workflow and enjoy the customizability that Urartu offers.\n\n## **Actions: Shaping Functionality**\n\nAt the heart of Urartu is the `Action` class, which orchestrates all operations. This script manages everything from CLI arguments to the execution of the main function based on the `action_name` parameter.\n\n## **Logging: Capture Every Detail**\n\nUrartu is equipped with a comprehensive logging system to ensure no detail of your project's execution is missed. Here's how it works:\n- Standard Runs: Every execution is meticulously logged and stored in a structured directory within your current working directory. The path format is:\n`.runs/${action_name}/${now:%Y-%m-%d}_${now:%H-%M-%S}`\n- Debug Mode: If the debug flag is enabled, logs are saved under: `.runs/debug/${action_name}/${now:%Y-%m-%d}_${now:%H-%M-%S}`\n- Multi-run Sessions: For runs involving multiple configurations or tests, logs are appended with a `.runs/debug/${action_name}/${now:%Y-%m-%d}_${now:%H-%M-%S}_multirun` suffix to differentiate them.\n\nEach run directory is organized to contain essential files such as:\n- output.log: Captures all output from the run.\n- notes.md: Allows for manual annotations and observations.\n- cfg.yaml: Stores the configuration used for the run.\n\nAdditional files may be included depending on the type of run, ensuring you have all the data you need at your fingertips.\n\n## **Effortless Launch**\n\nLaunching with Urartu is a breeze, offering you two launch options:\n\n- Local Marvel: Execute jobs right on your local machine.\n- Cluster Voyage: Set sail to the slurm cluster by toggling the `slurm.use_slurm` in `config_{username}/slurm/slurm.yaml` to switch between local and cluster executions.\n\nChoose your adventure and launch your projects with ease! 🚀\n\nEncountered any issues or have suggestions? Feel free to open an issue for support.\n\n# **Exploring the Experiments**\nUnveil insights with ease using Urartu in partnership with [Aim](https://github.com/aimhubio/aim), the intuitive and powerful open-source AI metadata tracker. To access a rich trove of metrics captured by Aim, simply:\n- Navigate to the directory containing the .aim repository.\n- Fire up the magic with:\n```bash\naim up\n```\nWatch as Aim brings your experiments into sharp relief, providing the clarity needed to drive informed decisions and pioneering efforts in machine learning. 📈\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftamohannes%2Furartu","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftamohannes%2Furartu","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftamohannes%2Furartu/lists"}