https://github.com/frictionlessdata/datapackage-pipelines-github

Datapackage-pipelines github
https://github.com/frictionlessdata/datapackage-pipelines-github

Last synced: 7 months ago
JSON representation

Datapackage-pipelines github

• Host: GitHub
• URL: https://github.com/frictionlessdata/datapackage-pipelines-github
• Owner: frictionlessdata
• License: mit
• Created: 2017-04-18T07:30:17.000Z (about 9 years ago)
• Default Branch: main
• Last Pushed: 2021-06-01T12:52:13.000Z (almost 5 years ago)
• Last Synced: 2025-09-30T11:17:23.881Z (7 months ago)
• Language: Python
• Homepage:
• Size: 11.7 KB
• Stars: 2
• Watchers: 9
• Forks: 0
• Open Issues: 1
• Metadata Files:
  • Readme: README.md
  • Contributing: CONTRIBUTING.md
  • License: LICENSE.md

Awesome Lists containing this project

README

          
# datapackage-pipelines-github

Extension for `datapackage-pipelines` for pulling stuff from GitHub.

- **Pipelines and Processor Code** from a specific branch will be pulled 

- **Issues** will be presented, using their title and description, as 'unable to start' pipelines.

- **Pull Requests** will be shown and executed, based on a predefined policies defining:

  - Should pipeline specs be pulled from PRs

  - Should processor code be pulled from PRs

  - Which processors are allowed to run, when executing PRs

  

  You can define separate policies for local PRs (from the same fork) or remote PRs (from another fork.)

note: When pulling code from PRs, we will only bring processors referenced from the 

pipeline spec and residing in the same directory as the `pipeline-spec.yaml` file. 

 

## Source spec

Place files named `github.source-spec.yaml` in your pipelines directory.

Each one should be of the form:

```yaml

:

    repository: 

    base-path: 

    code: 

      ref: 

    issues: 

      closed: 

      pipeline-id-format:     

    pull-requests: 

      local:  

        specs:  

        code: 

        disallow-processors: 

         - 

      remote: 

        specs:  

        code: 

        disallow-processors: 

         - 

      pipeline-id-format: 

```

`pipeline-id-format` is a Python format string with two placeholders:

- `issue-id`: The issue/pr number 

- `title-slug`: The issue title slug

The default format for issues is "issue/{issue-id:03}_{title-slug}"

The default format for pull-requests is "pr/{pr-id:03}_{title-slug}"

#### Example:

```yaml

dpp-github:

    repository: firctionlessdata/datapackage-pipelines-github

    code:

      ref: master

    pull-requests:

      local:

        specs: yes

        code: yes

        disallow-processors:

         - "dump\..*"

      remote:

        specs: no    

      pipeline-id-format: "{title-slug}__{issue-id}"

    issues:

      closed: no

      pipeline-id-format: "{title-slug}__{issue-id}"

```