https://github.com/zillow/intake-nested-yaml-catalog

Supports a single YAML file hierarchical catalog to organize datasets and avoid a data swamp.
https://github.com/zillow/intake-nested-yaml-catalog

data-access data-catalog intake python

Last synced: 9 days ago
JSON representation

Supports a single YAML file hierarchical catalog to organize datasets and avoid a data swamp.

• Host: GitHub
• URL: https://github.com/zillow/intake-nested-yaml-catalog
• Owner: zillow
• License: apache-2.0
• Created: 2019-07-25T17:31:46.000Z (almost 6 years ago)
• Default Branch: master
• Last Pushed: 2020-04-06T13:18:03.000Z (about 5 years ago)
• Last Synced: 2025-03-30T03:11:47.372Z (about 1 month ago)
• Topics: data-access, data-catalog, intake, python
• Language: Python
• Size: 49.8 KB
• Stars: 7
• Watchers: 2
• Forks: 2
• Open Issues: 2
• Metadata Files:
  • Readme: README.rst
  • Contributing: CONTRIBUTING.md
  • License: LICENSE

Awesome Lists containing this project

README

        
.. image:: https://travis-ci.org/zillow/intake-nested-yaml-catalog.svg?branch=master

    :target: https://travis-ci.org/zillow/intake-nested-yaml-catalog

.. image:: https://coveralls.io/repos/github/zillow/intake-nested-yaml-catalog/badge.svg?branch=master

    :target: https://coveralls.io/github/zillow/intake-nested-yaml-catalog?branch=master

.. image:: https://readthedocs.org/projects/intake-nested-yaml-catalog/badge/?version=latest

    :target: https://intake-nested-yaml-catalog.readthedocs.io/en/latest/?badge=latest

    :alt: Documentation Status

Welcome to Intake plugin for nested YAML catalogs

==================================================

This is an `Intake `_ plugin supporting a

single YAML hierarchical catalog to organize datasets and avoid a data swamp.

Example of organizing the datasets by business domain entities:

.. code-block:: yaml

    metadata:

      hierarchical_catalog: true

    entity:

      customer:

        customer_attributes:

          args:

            urlpath: s3://foo

          driver: parquet

      user:

        user_profile:

          args:

            urlpath: s3://foo

          driver: parquet

Can be accessed as:

.. code-block:: python

  df = catalog.entity.customer.customer_attributes.read()