{"id":20759352,"url":"https://github.com/matsmoll/aligned","last_synced_at":"2025-08-13T17:34:18.740Z","repository":{"id":64526146,"uuid":"486155120","full_name":"MatsMoll/aligned","owner":"MatsMoll","description":"The DBT of ML, as Aligned describes data dependencies in ML systems, and reduce technical data debt","archived":false,"fork":false,"pushed_at":"2024-06-11T19:58:51.000Z","size":1835,"stargazers_count":48,"open_issues_count":1,"forks_count":1,"subscribers_count":12,"default_branch":"main","last_synced_at":"2024-06-19T09:50:32.728Z","etag":null,"topics":["ai","data-contracts","data-lake","datacontracts","dbt","feature-engineering","feature-store","ml","ml-ops","mlops"],"latest_commit_sha":null,"homepage":"https://www.aligned.codes","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/MatsMoll.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-04-27T10:57:19.000Z","updated_at":"2024-07-19T01:56:31.380Z","dependencies_parsed_at":"2023-08-31T12:42:29.326Z","dependency_job_id":"7a9b033e-d1df-42e3-ae5e-f319c02897c0","html_url":"https://github.com/MatsMoll/aligned","commit_stats":{"total_commits":314,"total_committers":4,"mean_commits":78.5,"dds":0.2420382165605095,"last_synced_commit":"84203c6629a21f82c73c2ddbf30691af09374b14"},"previous_names":["otovo/aladdin","matsmoll/aligned","otovo/aligned"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MatsMoll%2Faligned","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MatsMoll%2Faligned/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MatsMoll%2Faligned/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MatsMoll%2Faligned/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/MatsMoll","download_url":"https://codeload.github.com/MatsMoll/aligned/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":229773618,"owners_count":18122034,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai","data-contracts","data-lake","datacontracts","dbt","feature-engineering","feature-store","ml","ml-ops","mlops"],"created_at":"2024-11-17T09:56:38.075Z","updated_at":"2025-08-13T17:34:18.696Z","avatar_url":"https://github.com/MatsMoll.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Aligned\n\nA data management tool with additional features tailured for ML applications.\n\nSimilar to how DBT is a data management tool for business analytics, will Aligned manage ML projects.\n\nThis is done by defining data contracts and model contracts.\n\nAligned does this through two things.\n1. A light weight data management system. Making it possible to query a data lake and databases.\n2. Tooling to define a `model_contract`. Clearing up common unanswerd questions through code.\n\nTherefore, aligned can also be seen as a way to implement interfaces or the [strategy pattern](https://en.wikipedia.org/wiki/Strategy_pattern) into data applications.\n\nFurthermore, Aligned collect data lineage between models, basic feature transformations. While also making it easy to reduce data leakage with point-in-time valid data and fix other problems described in [Sculley et al. [2015]](https://papers.nips.cc/paper/2015/file/86df7dcfd896fcaf2674f757a2463eba-Paper.pdf).\n\n## Examples\n\nBelow are some examples of how Aligned can be used.\n\n### Aligned UI\n\nAligned provides an UI to view which data exists, the expectations we have and find faults.\n\n[View the example UI](https://aligned-catalog.azurewebsites.net/).\nHowever, this is still under development, so sign up for a [wait list](https://aligned-managed-web.vercel.app/) to get access.\n\n\n### Example Repo\n\nWant to look at examples of how to use `aligned`?\nView the [`MatsMoll/aligned-example` repo](https://github.com/MatsMoll/aligned-example).\n\nOr see how you could query a file in a data lake.\n\n```python\nstore = await ContractStore.from_dir(\".\")\ndf = await store.execute_sql(\"SELECT * FROM titanic LIMIT 10\").to_polars()\n```\n\n## Docs\n\nCheck out the [Aligned Docs](https://www.aligned.codes), but keep in mind that they are still work in progress.\n\n---\n\n### Available Features\n\nBelow are some of the features Aligned offers:\n\n- [Data Catalog](https://aligned-managed-web.vercel.app/)\n- [Data Lineage](https://aligned-managed-web.vercel.app/)\n- [Model Performance Monitoring](https://aligned-managed-web.vercel.app/)\n- [Data Freshness](#data-freshness)\n- [Data Quality Assurance](#data-quality)\n- [Feature Store](https://matsmoll.github.io/posts/understanding-the-chaotic-landscape-of-mlops#feature-store)\n- [Exposing Models](#exposed-models)\n\n\nAll from the simple API of defining\n- [Data Sources](#data-sources)\n- [Data Contracts](#data-contracts)\n- [Models](#describe-models)\n\nAs a result, loading model features is as easy as:\n\n```python\nentities = {\"passenger_id\": [1, 2, 3, 4]}\nawait store.model(\"titanic\").features_for(entities).to_pandas()\n```\n\nAligned is still in active development, so changes are likely.\n\n## Model Contract\n\nAligned introduces a new concept called the \"model contract\", which tries to answer the following questions.\n\n- What is predicted?\n- What is associated with a prediction? - A user id?\n- Where do we store predictions?\n- Do a model depend on other models?\n- Is the model exposed through an API?\n- What needs to be sent in, to use the model?\n- Is it classification, regression, gen ai?\n- Where is the ground truth stored? - if any\n- Who owns the model?\n- Where do we store data sets?\n\nAll this is described through a `model_contract`, as shown below.\n\n```python\n@model_contract(\n    name=\"eta_taxi\",\n    input_features=[\n        trips.eucledian_distance,\n        trips.number_of_passengers,\n        traffic.expected_delay\n    ],\n    output_source=FileSource.delta_at(\"titanic_model/predictions\")\n)\nclass EtaTaxi:\n    trip_id = Int32().as_entity()\n    predicted_at = ValidFrom()\n    predicted_duration = trips.duration.as_regression_target()\n```\n\n## Data Sources\n\nAligned makes handling data sources easy, as you do not have to think about how it is done.\n\nFurthermore, Aligned makes it easy to switch parts of the business logic to a local setup for debugging purposes.\n\n```python\nfrom aligned import FileSource, AwsS3Config, AzureBlobConfig\n\ndir_type: Literal[\"local\", \"aws\", \"azure\"] = ...\n\nif dir_type == \"aws\":\n    aws_config = AwsS3Config(...)\n    root_directory = aws_config.directory(\"my-awesome-project\")\n\nelif dir_type == \"azure\":\n    azure_config = AzureBlobConfig(...)\n    root_directory = azure_config.directory(\"my-awesome-project\")\nelse:\n    root_directory = FileSource.directory(\"my-awesome-project\")\n\n\ntaxi_project = root_directory.sub_directory(\"eta_taxi\")\n\ncsv_source = taxi_project.csv_at(\"predictions.csv\")\nparquet_source = taxi_project.parquet_at(\"predictions.parquet\")\ndelta_source = taxi_project.delta_at(\"predictions\")\n```\n\n### Date Formatting\nManaging a data lake can be hard. However, a common problem when using file formats can be managing date formats. As a result do Aligned provide a way to standardise this, so you can focus on what matters.\n\n```python\nfrom aligned import FileSource\nfrom aligned.schemas.date_formatter import DateFormatter\n\niso_formatter = DateFormatter.iso_8601()\nunix_formatter = DateFormatter.unix_timestamp(time_unit=\"us\", time_zone=\"UTC\")\ncustom_strtime_formatter = DateFormatter.string_format(\"%Y/%m/%d %H:%M:%S\")\n\nFileSource.csv_at(\"my/file.csv\", date_formatter=unix_formatter)\n```\n\n## Data Contracts\n\nAligned also makes it possible to define data and features through `data_contract`s.\nThen get code completion and typesafety by referencing them in other features.\n\nThis makes the features light weight, data source independent, and flexible.\n\n```python\n@data_contract(\n    name=\"passenger\",\n    description=\"Some features from the titanic dataset\",\n    source=FileSource.csv_at(\"titanic.csv\"),\n    materialized_source=FileSource.parquet_at(\"titanic.parquet\"),\n)\nclass TitanicPassenger:\n\n    passenger_id = Int32().as_entity()\n\n    age = (\n        Float()\n            .description(\"A float as some have decimals\")\n            .lower_bound(0)\n            .upper_bound(110)\n    )\n\n    name = String()\n    sex = String().accepted_values([\"male\", \"female\"])\n    did_survive = Bool().description(\"If the passenger survived\")\n    sibsp = Int32().lower_bound(0).description(\"Number of siblings on titanic\")\n    cabin = String().is_optional()\n\n    # Creates two one hot encoded values\n    is_male, is_female = sex.one_hot_encode(['male', 'female'])\n```\n\n#### Load data\n\nTo load the data use the following code.\n\n```python\ndf = await TitanicPassenger.query().all().to_polars()\n```\n\n### Exposed models\n\nAligned mainly focuses on defining the expected input and output of different models. However, this in itself makes it hard to use the models. This is why Aligned makes it possible to define how our ML models are exposed by setting an `exposed_model` attribute.\n\n\n```python\nfrom aligned.exposed_model.mlflow import mlflow_server\n\n@model_contract(\n    name=\"eta_taxi\",\n    exposed_model=mlflow_server(\n        host=\"http://localhost:8000\",\n    ),\n    ...\n)\nclass EtaTaxi:\n    trip_id = Int32().as_entity()\n    predicted_at = EventTimestamp()\n    predicted_duration = trips.duration.as_regression_target()\n```\n\nThis also makes it possible to get predictions with the following command:\n\n```python\nawait store.model(\"eta_taxi\").predict_over({\n    \"trip_id\": [...]\n}).to_polars()\n```\n\nOr store them directly in the `output_source` with something like:\n\n```python\nawait store.model(\"eta_taxi\").predict_over({\n    \"trip_id\": [...]\n}).upsert_into_output_source()\n```\n\nSome of the existing implementations are:\n- MLFlow Server\n- Run MLFLow model in memory\n- Ollama completion endpoint\n- Ollama embedded endpoint\n- Send entities to generic endpoint\n\n## Data Freshness\nMaking sure a source contains fresh data is a crucial part to create proper ML applications.\nTherefore, Aligned provides an easy way to check how fresh a source is.\n\n```python\n@data_contract(\n    name=\"departures\",\n    description=\"Features related to the departure of a taxi ride\",\n    source=taxi_db.table(\"departures\"),\n)\nclass TaxiDepartures:\n\n    trip_id = UUID().as_entity()\n\n    pickuped_at = EventTimestamp()\n\n    number_of_passengers = Int32().is_optional()\n\n    dropoff_latitude = Float()\n    dropoff_longitude = Float()\n\n    pickup_latitude = Float()\n    pickup_longitude = Float()\n\n\nfreshness = await TaxiDepartures.freshness_in_batch_source()\n\nif freshness \u003c datetime.now() - timedelta(days=2):\n    raise ValueError(\"To old data to create an ML model\")\n```\n\n## Data quality\nAligned will make sure all the different features gets formatted as the correct datatype.\nIn addition will aligned also make sure that the returned features align with defined constraints.\n\n```python\n@data_contract(...)\nclass TitanicPassenger:\n    ...\n    age = (\n        Float()\n            .lower_bound(0)\n            .upper_bound(110)\n    )\n    sibsp = Int32().lower_bound(0).is_optional()\n```\n\nThen since our data contract have a `is_optional` and a `lower_bound`, will the `.validate(...)` command filter out the entities that do not follow that behavior.\n\n```python\ndf = await store.model(\"titanic_model\").features_for({\n    \"passenger_id\": [1, 50, 110]\n}).drop_invalid().to_pandas()\n```\n\n## Contract Store\n\nAligned collects all the data contracts and model contracts in a contract store. You can generate this in a few different ways, and each method serves some different use-cases.\n\nFor experimentational use-cases will the `await ContractStore.from_dir(\".\")` probably make the most sense. However, this will scan the full directory which can lead to slow startup times.\n\nTherefore, it is also possible to manually add the different data contracts and contracts with the following.\n\n```python\nstore = ContractStore.empty()\nstore.add(MyView)\nstore.add(MyModel)\n```\n\nThis makes it possible to define different contracts per project, or team. As a result, you can also combine different stores with.\n\n```python\nforecasting_store = await ContractStore.from_dir(\"path/for/forecasting\")\nrecommendation_store = await ContractStore.from_dir(\"path/for/recommendation\")\n\ncombined_store = recommendation_store.combined_with(forecasting_store)\n```\n\nLastly, we can also load the all features from a serializable format, such as a JSON file.\n\n```python\nawait FileSource.json_at(\"contracts.json\").as_contract_store()\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmatsmoll%2Faligned","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmatsmoll%2Faligned","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmatsmoll%2Faligned/lists"}