{"id":18508677,"url":"https://github.com/noteable-io/airflow-provider-noteable","last_synced_at":"2026-01-23T05:36:31.696Z","repository":{"id":238717833,"uuid":"627138423","full_name":"noteable-io/airflow-provider-noteable","owner":"noteable-io","description":"A Noteable-\u003c\u003eAirflow provider for running Airflow DAGs in Astronomer Airflow clusters.","archived":false,"fork":false,"pushed_at":"2023-04-12T21:33:57.000Z","size":9,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-02-17T02:23:49.389Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/noteable-io.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-04-12T21:33:54.000Z","updated_at":"2023-04-12T21:34:00.000Z","dependencies_parsed_at":"2024-05-07T18:03:20.315Z","dependency_job_id":"6ef8e86a-1511-4b89-b990-826610fc7333","html_url":"https://github.com/noteable-io/airflow-provider-noteable","commit_stats":null,"previous_names":["noteable-io/airflow-provider-noteable"],"tags_count":0,"template":false,"template_full_name":"astronomer/airflow-provider-sample","purl":"pkg:github/noteable-io/airflow-provider-noteable","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/noteable-io%2Fairflow-provider-noteable","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/noteable-io%2Fairflow-provider-noteable/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/noteable-io%2Fairflow-provider-noteable/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/noteable-io%2Fairflow-provider-noteable/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/noteable-io","download_url":"https://codeload.github.com/noteable-io/airflow-provider-noteable/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/noteable-io%2Fairflow-provider-noteable/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28681000,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-23T04:33:33.518Z","status":"ssl_error","status_checked_at":"2026-01-23T04:33:30.433Z","response_time":59,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-06T15:15:03.931Z","updated_at":"2026-01-23T05:36:31.674Z","avatar_url":"https://github.com/noteable-io.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cp align=\"center\"\u003e\n  \u003ca href=\"https://www.airflow.apache.org\"\u003e\n    \u003cimg alt=\"Airflow\" src=\"https://cwiki.apache.org/confluence/download/attachments/145723561/airflow_transparent.png?api=v2\" width=\"60\" /\u003e\n  \u003c/a\u003e\n\u003c/p\u003e\n\u003ch1 align=\"center\"\u003e\n  Airflow Sample Provider\n\u003c/h1\u003e\n  \u003ch3 align=\"center\"\u003e\n  Guidelines on building, deploying, and maintaining provider packages that will help Airflow users interface with external systems. Maintained with ❤️ by Astronomer.\n\u003c/h3\u003e\n\n\u003cbr/\u003e\n\nThis repository provides best practices for building, structuring, and deploying Airflow provider packages as independent python modules available on PyPI.\n\nProvider repositories must be public on Github and follow the structural and technical guidelines laid out in this Readme. Ensure that all of these requirements have been met before submitting a provider package for community review.\n\nHere, you'll find information on requirements and best practices for key aspects of your project:\n\n- File formatting\n- Development\n- Airflow integration\n- Documentation\n- Testing\n\n## Formatting Standards\n\nBefore writing and testing the functionality of your provider package, ensure that your project follows these formatting conventions.\n\n### Package name\n\nThe highest level directory in the provider package should be named in the following format:\n\n```\nairflow-provider-\u003cprovider-name\u003e\n```\n\n### Repository structure\n\nAll provider packages must adhere to the following file structure:\n\n```bash\n├── LICENSE # A license is required, MIT or Apache is preferred.\n├── README.md\n├── sample_provider # Your package import directory. This will contain all Airflow modules and example DAGs.\n│   ├── __init__.py\n│   ├── example_dags\n│   │   └── sample.py\n│   ├── hooks\n│   │   ├── __init__.py\n│   │   └── sample.py\n│   ├── operators\n│   │   ├── __init__.py\n│   │   └── sample.py\n│   └── sensors\n│       ├── __init__.py\n│       └── sample.py\n├── setup.py # A setup.py file to define dependencies and how the package is built and shipped. If you'd like to use setup.cfg, that is fine as well.\n└── tests # Unit tests for each module.\n    ├── __init__.py\n    ├── hooks\n    │   ├── __init__.py\n    │   └── test_sample_hook.py\n    ├── operators\n    │   ├── __init__.py\n    │   └── test_sample_operator.py\n    └── sensors\n        ├── __init__.py\n        └── test_sample_sensor.py\n```\n\n\n## Development Standards\n\nIf you followed the formatting guidelines above, you're now ready to start editing files to include standard package functionality.\n\n### Python Packaging Scripts\n\nYour `setup.py` file should contain all of the appropriate metadata and dependencies required to build your package. Use the [sample `setup.py` file](https://github.com/astronomer/airflow-provider-sample/blob/main/setup.py) in this repository as a starting point for your own project.\n\nIf some of the options for building your package are variables or user-defined, you can specify a `setup.cfg` file instead.\n\nTo improve discoverability of your provider package on PyPI, it is recommended to [add classifiers](https://packaging.python.org/en/latest/tutorials/packaging-projects/#configuring-metadata) to the package's metadata. The following standard classifiers should be used in addition to any others you may choose to include:\n\n- Framework :: Apache Airflow\n- Framework :: Apache Airflow :: Provider\n\n### Managing Dependencies\n\nWhen building providers, these guidelines will help you avoid potential for dependency conflicts:\n\n- It is important that the providers do not include dependencies that conflict with the underlying dependencies for a particular Airflow version. All of the default dependencies included in the core Airflow project can be found in the Airflow [setup.py file](https://github.com/apache/airflow/blob/master/setup.py#L705).\n- Keep all dependencies relaxed at the upper bound. At the lower bound, specify minor versions (for example, `depx \u003e=2.0.0, \u003c3`).\n\n### Versioning\n\nUse standard semantic versioning for releasing your package. When cutting a new release, be sure to update all of the relevant metadata fields in your setup file.\n\n### Building Modules\n\nAll modules must follow a specific set of best practices to optimize their performance with Airflow:\n\n- **All classes should always be able to run without access to the internet.** The Airflow Scheduler parses DAGs on a regular schedule. Every time that parse happens, Airflow will execute whatever is contained in the `init` method of your class. If that `init` method contains network requests, such as calls to a third party API, there will be problems due to repeated network calls.\n- **Init methods should never call functions which return valid objects only at runtime**. This will cause a fatal import error when trying to import a module into a DAG. A common best practice for referencing connectors and variables within DAGs is to use [Jinja Templating](https://airflow.apache.org/docs/apache-airflow/stable/concepts.html#jinja-templating).\n- **All operator modules need an `execute` method.** This method defines the logic that the operator will implement.\n\nModules should also take advantage of native Airflow features that allow your provider to:\n\n- Register custom connection types, which improve the user experience when connecting to your tool.\n- Include `extra-links` that link your provider back to its page on the Astronomer Registry. This provides users easy access to documentation and example DAGs.\n\nRefer to the `Airflow Integration Standards` section for more information on how to build in these extra features.\n\n### Unit testing\n\nYour top-level `tests/` folder should include unit tests for all modules that exist in the repository. You can write tests in the framework of your choice, but the Astronomer team and Airflow community typically use [pytest](https://docs.pytest.org/en/stable/).\n\nYou can test this package by running: `python3 -m unittest` from the top-level of the directory.\n\n## Airflow Integration Standards\n\nAirflow exposes a number of plugins to interface from your provider package. We highly encourage provider maintainers to add these plugins because they significantly improve the user experience when connecting to a provider.\n\n### Defining an entrypoint\n\nTo enable custom connections, you first need to define an `apache_airflow_provider ` entrypoint in your `setup.py` or `setup.cfg` file:\n\n```\nentry_points={\n  \"apache_airflow_provider\": [\n      \"provider_info=sample_provider.__init__:get_provider_info\"\n        ]\n    }\n```\n\nNext, you need to add a `get_provider_info` method to the `__init__` file in your top-level provider folder. This function needs to return certain metadata associated with your package in order for Airflow to use it at runtime:\n\n```python\ndef get_provider_info():\n    return {\n        \"name\": \"Sample Apache Airflow Provider\",  # Required\n        \"description\": \"A sample template for Apache Airflow providers.\",  # Required\n        \"connection-types\": [\n            {\"connection-type\": \"sample\", \"hook-class-name\": \"sample_provider.hooks.sample.SampleHook\"}\n        ],\n        \"versions\": [\"1.0.0\"] # Required\n    }\n```\n\nOnce you define the entrypoint, you can use native Airflow features to expose custom connection types in the Airflow UI, as well as additional links to relevant documentation.\n\n### Adding Custom Connection Forms\n\nAirflow enables custom connection forms through discoverable hooks. The following is an example of a custom connection form for the Fivetran provider:\n\n\u003cimg src=\"https://user-images.githubusercontent.com/63181127/112921463-d07b2880-90d8-11eb-871b-fc4e1c6cade9.png\" width=\"600\" /\u003e\n\nAdd code to the hook class to initiate a discoverable hook and create a custom connection form. The following code defines a hook and a custom connection form:\n\n```python\nclass ExampleHook(BaseHook):\n    \"\"\"ExampleHook docstring...\"\"\"\n\n    conn_name_attr = 'example_conn_id'\n    default_conn_name = 'example_default'\n    conn_type = 'example'\n    hook_name = 'Example'\n\n    @staticmethod\n    def get_connection_form_widgets() -\u003e Dict[str, Any]:\n        \"\"\"Returns connection widgets to add to connection form\"\"\"\n        from flask_appbuilder.fieldwidgets import BS3PasswordFieldWidget, BS3TextFieldWidget\n        from flask_babel import lazy_gettext\n        from wtforms import PasswordField, StringField, BooleanField\n\n        return {\n            \"bool\": BooleanField(lazy_gettext('Example Boolean')),\n            \"account\": StringField(\n                lazy_gettext('Account'), widget=BS3TextFieldWidget()\n            ),\n            \"secret_key\": PasswordField(\n                lazy_gettext('Secret Key'), widget=BS3PasswordFieldWidget()\n            ),\n        }\n\n    @staticmethod\n    def get_ui_field_behaviour() -\u003e Dict:\n        \"\"\"Returns custom field behaviour\"\"\"\n        import json\n\n        return {\n            \"hidden_fields\": ['port'],\n            \"relabeling\": {},\n            \"placeholders\": {\n                'extra': json.dumps(\n                    {\n                        \"example_parameter\": \"parameter\",\n                    },\n                    indent=4,\n                ),\n                'host': 'example hostname',\n                'schema': 'example schema',\n                'login': 'example username',\n                'password': 'example password',\n                'account': 'example account name',\n                'secret_key': 'example secret key',\n            },\n        }\n ```\n\nSome notes about using custom connections:\n\n- `get_connection_form_widgets()` creates extra fields using flask_appbuilder. A variety of field types can be created using this function, such as strings, passwords, booleans, and integers.\n\n- `get_ui_field_behaviour()` is a JSON schema describing the form field behavior. Fields can be hidden, relabeled, and given placeholder values.\n\n- To connect a form to Airflow, add the hook class name and connection type of a discoverable hook to `\"connection-types\"` in the `get_provider_info` method as mentioned in `Defining an entrypoint`.\n\n### Adding Custom Links\n\nOperators can add custom links that users can click to reach an external source when interacting with an operator in the Airflow UI. This link can be created dynamically based on the context of the operator. The following code example shows how to initiate an extra link within an operator:\n\n```python\nfrom airflow.models import BaseOperator, BaseOperatorLink\n\nclass ExampleLink(BaseOperatorLink):\n    \"\"\"Link for ExmpleOperator\"\"\"\n\n    name = 'Example Link'\n\n    def get_link(self, operator: BaseOperator, *, ti_key=None):\n        \"\"\"Get link to registry page.\"\"\"\n\n        registry_link = \"https://{example}.com\"\n        return registry_link.format(example='example')\n\nclass ExampleOperator(BaseOperator):\n    \"\"\"ExampleOperator docstring...\"\"\"\n\n    operator_extra_links = (Example_Link(),)\n```\n\nTo connect custom links to Airflow, add the operator class name to `\"extra-links\"` in the `get_provider_info` method mentioned above.\n\n## Documentation Standards\n\nCreating excellent documentation is essential for explaining the purpose of your provider package and how to use it.\n\n### Inline Module Documentation\n\nEvery Python module, including all hooks, operators, sensors, and transfers, should be documented inline via [sphinx-templated docstrings](https://pythonhosted.org/an_example_pypi_project/sphinx.html). These docstrings should be included at the top of each module file and contain three sections separated by blank lines:\n- A one-sentence description explaining what the module does.\n- A longer description explaining how the module works. This can include details such as code blocks or blockquotes. For more information Sphinx markdown directives, read the [Sphinx documentation](https://www.sphinx-doc.org/en/master/usage/restructuredtext/directives.html#directive-code-block).\n- A declarative definition of parameters that you can pass to the module, templated per the example below.\n\nFor a full example of inline module documentation, see the [example operator in this repository](https://github.com/astronomer/airflow-provider-sample/blob/main/sample_provider/operators/sample_operator.py#L11).\n\n### README\n\nThe README for your provider package should give users an overview of what your provider package does. Specifically, it should include:\n\n- High-level documentation about the provider's service.\n- Steps for building a connection to the service from Airflow.\n- What modules exist within the package.\n- An exact set of dependencies and versions that your provider has been tested with.\n- Guidance for contributing to the provider package.\n\n## Functional Testing Standards\n\nTo build your repo into a python wheel that can be tested, follow the steps below:\n\n1. Clone the provider repo.\n2. `cd` into provider directory.\n3. Run `python3 -m pip install build`.\n4. Run `python3 -m build` to build the wheel.\n5. Find the .whl file in `/dist/*.whl`.\n6. Download the [Astro CLI](https://github.com/astronomer/astro-cli).\n7. Create a new project directory, cd into it, and run `astro dev init` to initialize a new astro project.\n8. Ensure the Dockerfile contains the Airflow 2.0 image:\n\n   ```\n   FROM quay.io/astronomer/ap-airflow:2.0.0-buster-onbuild\n   ```\n\n9. Copy the `.whl` file to the top level of your project directory.\n10. Install `.whl` in your containerized environment by adding the following to your Dockerfile:\n\n   ```\n   RUN pip install --user airflow_provider_\u003cPROVIDER_NAME\u003e-0.0.1-py3-none-any.whl\n   ```\n\n11. Copy your sample DAG to the `dags/` folder of your astro project directory.\n12. Run `astro dev start` to build the containers and run Airflow locally (you'll need Docker on your machine).\n13. When you're done, run `astro dev stop` to wind down the deployment. Run `astro dev kill` to kill the containers and remove the local Docker volume. You can also use `astro dev kill` to stop the environment before rebuilding with a new `.whl` file.\n\n\u003e Note: If you are having trouble accessing the Airflow webserver locally, there could be a bug in your wheel setup. To debug, run `docker ps`, grab the container ID of the scheduler, and run `docker logs \u003cscheduler-container-id\u003e` to inspect the logs.\n\n## Publishing your Provider repository for the Astronomer Registry\n\nIf you have never submitted your Provider repository for publication to the Astronomer Registry, [create a new release/tag for your repository](https://docs.github.com/en/repositories/releasing-projects-on-github/managing-releases-in-a-repository) on the `main` branch. Ultimately, the backend of the Astronomer Registry will check for new tags for a Provider repository to trigger adding the new version of the Provider on the Registry.\n\n\u003e **NOTE:** Tags for the repository must follow typical [semantic versioning](https://semver.org/).\n\nNow that you've created a release/tag, head over to the [Astronomer Registry](https://registry.astronomer.io) and [fill out the form](https://registry.astronomer.io/publish) with your shiny new Provider repo details!\n\nIf your Provider is currently on the Astronomer Registry, simply create a new release/tag will trigger an update to the Registry and the new version will be published.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnoteable-io%2Fairflow-provider-noteable","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnoteable-io%2Fairflow-provider-noteable","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnoteable-io%2Fairflow-provider-noteable/lists"}