{"id":46795437,"url":"https://github.com/strondata/smart-data","last_synced_at":"2026-07-01T05:01:08.568Z","repository":{"id":343086245,"uuid":"1174805361","full_name":"strondata/smart-data","owner":"strondata","description":"Framework Python extensível para pipelines de dados. Abstrai a complexidade do processamento em Sistemas, Fluxos e Componentes, com suporte nativo a qualidade de dados, governança e execução via CLI.","archived":false,"fork":false,"pushed_at":"2026-06-27T03:02:41.000Z","size":1697,"stargazers_count":1,"open_issues_count":27,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-06-27T04:16:17.246Z","etag":null,"topics":["cli","data-engineering","data-pipelines","data-quality","data-structures","pandas","pyspark","python","python3"],"latest_commit_sha":null,"homepage":"https://strondata.github.io/smart-data/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/strondata.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-03-06T21:30:48.000Z","updated_at":"2026-06-27T03:02:00.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/strondata/smart-data","commit_stats":null,"previous_names":["strondata/smart-data"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/strondata/smart-data","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/strondata%2Fsmart-data","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/strondata%2Fsmart-data/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/strondata%2Fsmart-data/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/strondata%2Fsmart-data/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/strondata","download_url":"https://codeload.github.com/strondata/smart-data/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/strondata%2Fsmart-data/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34993438,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-01T02:00:05.325Z","response_time":130,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cli","data-engineering","data-pipelines","data-quality","data-structures","pandas","pyspark","python","python3"],"created_at":"2026-03-10T04:06:14.901Z","updated_at":"2026-07-01T05:01:08.542Z","avatar_url":"https://github.com/strondata.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# aptdata\n\n\u003e **v0.0.3** · A declarative, extensible framework for building smart data pipelines in Python.\n\n[![Python](https://img.shields.io/badge/python-3.10%2B-blue)](https://www.python.org/)\n[![License](https://img.shields.io/badge/license-MIT-green)](LICENSE)\n[![Version](https://img.shields.io/badge/version-0.0.3-orange)](CHANGELOG.md)\n\n---\n\n## Overview\n\n**aptdata** is built around three universal abstractions — **System**,\n**Flow**, and **Component** — that cover every data-processing paradigm in a\nsingle, coherent model:\n\n```mermaid\nflowchart TD\n    I[\"IComponent / IFlow / ISystem\\n@dataclass + ABC — pure interfaces\"]\n    B[\"BaseComponent / BaseFlow / BaseSystem\\n@pydantic_dataclass — validated fields\"]\n    Y[\"Your concrete implementations\"]\n\n    I --\u003e B --\u003e Y\n```\n\nDatasets remain the fundamental data-exchange contract (`IDataset` /\n`BaseDataset`).  Every outcome from the CLI is emitted as a machine-readable\nJSON line, making aptdata a natural fit for AI orchestrators, CI/CD\npipelines and scripted workflows.\n\n---\n\n## Requirements\n\n- Python ≥ 3.10\n- [Poetry](https://python-poetry.org/) (for development)\n\n---\n\n## Installation\n\n### From PyPI\n\n```bash\npip install aptdata\n```\n\n### Optional extras\n\n```bash\npip install aptdata[pandas]   # pandas support\npip install aptdata[spark]    # PySpark support\npip install aptdata[plugins]  # REST, PostgreSQL, Parquet I/O\npip install aptdata[ai]       # MCP server for AI agents\npip install aptdata[all]      # everything\n```\n\n### From source (development)\n\n```bash\ngit clone https://github.com/strondata/smart-data.git\ncd aptdata\npoetry install\n```\n\n---\n\n## Quick start\n\n```python\nfrom pydantic.dataclasses import dataclass as pydantic_dataclass\nfrom aptdata.core import (\n    BaseDataset, IDataset,\n    BaseComponent, ComponentMeta, ComponentKind,\n    BaseFlow, IFlow,\n    BaseSystem,\n)\n\n@pydantic_dataclass\nclass MemoryDataset(BaseDataset):\n    def __post_init__(self): self._data = None\n    def read(self): return self._data\n    def write(self, data): self._data = data\n\n@pydantic_dataclass\nclass DoubleComponent(BaseComponent):\n    def validate_inputs(self, inputs: list[IDataset]) -\u003e bool:\n        return len(inputs) == 1\n    def execute(self, inputs: list[IDataset]) -\u003e list[IDataset]:\n        out = MemoryDataset(uri=\"memory://out\")\n        out.write([x * 2 for x in inputs[0].read()])\n        return [out]\n\n@pydantic_dataclass\nclass ETLFlow(BaseFlow):\n    def __post_init__(self):\n        self._nodes = {}\n        self._edges = []\n        self._compiled = False\n    def add_component(self, c): self._nodes[c.component_id] = c\n    def connect(self, src, tgt, condition=None): ...\n    def compile(self): self._compiled = True\n    def run(self, inputs): return inputs  # wire your logic here\n\n@pydantic_dataclass\nclass MySystem(BaseSystem):\n    def __post_init__(self): self._flows: list[IFlow] = []\n    def register_flow(self, flow): self._flows.append(flow)\n    def run(self):\n        for flow in self._flows:\n            flow.run([])\n\n# Register and run via CLI\nfrom aptdata.plugins import registry\nregistry.register(\"my_system\", MySystem)\n```\n\n```bash\naptdata run my_system\n# {\"event\": \"pipeline.started\", \"pipeline\": \"my_system\", \"env\": \"dev\", \"dry_run\": false, \"trace_id\": null}\n# {\"event\": \"pipeline.completed\", \"pipeline\": \"my_system\", \"env\": \"dev\", \"dry_run\": false, \"elapsed_seconds\": 0.001, \"trace_id\": null}\n```\n\n---\n\n## CLI reference\n\n```\naptdata run SYSTEM_NAME [--env ENV] [--dry-run]\naptdata monitor [--refresh SECONDS]\naptdata scaffold PROJECT_NAME [--template TEMPLATE] [--output PATH]\naptdata schema export --output schema.json\naptdata system list [--json]\naptdata system info NAME [--json]\naptdata system validate NAME\naptdata plugin list [--json]\naptdata plugin inspect NAME [--json]\naptdata plugin preview READER [--limit N]\naptdata plugin load MODULE_PATH\naptdata config validate PATH\naptdata config init [--output PATH]\naptdata config show PATH\naptdata config run PATH [--env ENV]\naptdata telemetry status [--json]\naptdata telemetry export [--format json]\naptdata mesh list [--dir DIR] [--json]\naptdata mesh run COMPONENT [--dir DIR] [--dry-run] [--json]\naptdata mesh build COMPONENT [--dir DIR] [--json]\naptdata mcp-start [--transport TRANSPORT]\naptdata interactive\n```\n\nEvery static command supports `--json` for machine-readable JSON line output\n(backward compatible). Without `--json`, commands render Rich tables, panels,\nand syntax-highlighted output.\n\n### Scaffold templates\n\n| Template              | Description                                         |\n|-----------------------|-----------------------------------------------------|\n| `hello-world`         | Minimal pandas pipeline (default)                   |\n| `medallion`           | Bronze → Silver → Gold data lakehouse               |\n| `rag-ingestion`       | RAG pipeline: extract → chunk → embed → load        |\n| `data-quality-test`   | Schema contract + expectation suite                  |\n| `job-wheel`           | Python wheel executor for portable job packaging     |\n| `docker-compose-app`  | Multi-service Docker Compose application             |\n\n```bash\naptdata scaffold my_lakehouse --template medallion\naptdata scaffold my_job --template job-wheel\naptdata scaffold my_service --template docker-compose-app\n```\n\n---\n\n## Processing Engines\n\nEngine-agnostic transformation wrappers for pandas and PySpark:\n\n```python\nfrom aptdata.plugins.transform import PandasTransformer\n\ndef clean(df):\n    return df.dropna().drop_duplicates()\n\ntransformer = PandasTransformer(\"clean\", clean)\nresult = transformer.transform(my_dataset)\n```\n\nSee [Transform Engines docs](docs/transform-engines.md) for PySpark usage.\n\n---\n\n## Data Quality \u0026 Contracts\n\n```python\nfrom aptdata.plugins.quality import (\n    EnforcementMode, ExpectColumnToNotBeNull,\n    QualityValidator, SchemaContract,\n)\n\nvalidator = QualityValidator(\n    expectations=[ExpectColumnToNotBeNull(\"id\")],\n    enforcement=EnforcementMode.ABORT,\n)\nclean_data = validator.validate(raw_df)\n```\n\nSee [Quality docs](docs/quality.md) for all built-in expectations.\n\n---\n\n## Data Governance\n\n```python\nfrom aptdata.plugins.governance import (\n    BusinessRule, DatasetCatalog, DatasetCatalogEntry, LineageStore,\n)\nfrom aptdata.core.lineage import LineageGraph, LineageNode, LineageEventType\n\n# Lineage tracking\ngraph = LineageGraph(run_id=\"run-1\", workflow_name=\"etl\")\ngraph.add_node(LineageNode(dataset_uri=\"s3://raw/data\", event_type=LineageEventType.READ))\n\nstore = LineageStore()\nstore.save(graph)\n```\n\nSee [Governance docs](docs/governance.md) for the full API.\n\n---\n\n## AI Agents \u0026 MCP Server\n\naptdata ships with a built-in [Model Context Protocol](https://modelcontextprotocol.io/) server (`mcp-start`). This transforms AI assistants (like Claude, Copilot, or Devin) into autonomous data engineers with direct access to:\n\n- **Pipeline Execution:** Trigger and monitor data flows (`run_flow`).\n- **Data Quality:** Audit the latest quality test results (`quality://reports/...`).\n- **Data Governance:** Read business rules to prevent violations (`governance://rules`).\n- **Lineage:** Trace upstream dependencies and column-level provenance (`get_pipeline_lineage`).\n\n```bash\naptdata mcp-start --transport stdio\n```\n\nSee the [MCP Documentation](docs/mcp.md) for setup instructions.\n\n---\n\n## Release process\n\nReleases are automated via the [Release workflow](.github/workflows/release.yml).\nAfter a PR is merged into `main`, the CI reads its labels and bumps the version\naccordingly.\n\n| Label | Effect |\n|---|---|\n| `release:patch` | `0.0.1 → 0.0.2` |\n| `release:minor` | `0.0.1 → 0.1.0` |\n| `release:major` | `0.0.1 → 1.0.0` |\n| `release:skip` | no release (explicit opt-out) |\n| *(no label)* | no release (silent skip) |\n\nThe workflow will:\n1. Detect the merged PR and its labels.\n2. Run `bump-my-version bump \u003cpart\u003e` to update `pyproject.toml` and\n   `aptdata/__init__.py`.\n3. Create a `chore(release): bump version to X.Y.Z` commit and a `vX.Y.Z` tag.\n4. Push the commit and tag to `main`.\n5. The tag push automatically triggers the **Publish to PyPI** workflow.\n\n\u003e **Branch protection note:** GitHub Actions must have *read and write\n\u003e permissions* (Settings → Actions → General → Workflow permissions) and, if\n\u003e branch protection is enabled on `main`, the rule must allow GitHub Actions\n\u003e to bypass it.\n\n---\n\n## Development\n\n```bash\nmake install   # install all dependencies\nmake test      # run the test suite\nmake lint      # lint with ruff\nmake docs      # build the documentation\n```\n\n---\n\n## Documentation\n\nFull documentation is available in the [`docs/`](docs/) directory and can be\nserved locally with:\n\n```bash\nmkdocs serve\n```\n\n---\n\n## License\n\n[MIT](LICENSE)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstrondata%2Fsmart-data","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fstrondata%2Fsmart-data","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstrondata%2Fsmart-data/lists"}