{"id":31482312,"url":"https://github.com/snowflake-labs/nf-snowflake","last_synced_at":"2026-02-19T03:04:26.433Z","repository":{"id":259188812,"uuid":"807770470","full_name":"Snowflake-Labs/nf-snowflake","owner":"Snowflake-Labs","description":null,"archived":false,"fork":false,"pushed_at":"2025-09-15T05:41:54.000Z","size":185,"stargazers_count":9,"open_issues_count":1,"forks_count":2,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-09-15T07:22:25.083Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Groovy","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Snowflake-Labs.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2024-05-29T18:28:26.000Z","updated_at":"2025-09-15T05:37:38.000Z","dependencies_parsed_at":"2025-06-10T06:21:44.160Z","dependency_job_id":"64abb405-4dcc-433a-ba05-292f5e33eba6","html_url":"https://github.com/Snowflake-Labs/nf-snowflake","commit_stats":null,"previous_names":["snowflake-labs/nf-snowflake"],"tags_count":12,"template":false,"template_full_name":null,"purl":"pkg:github/Snowflake-Labs/nf-snowflake","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Snowflake-Labs%2Fnf-snowflake","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Snowflake-Labs%2Fnf-snowflake/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Snowflake-Labs%2Fnf-snowflake/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Snowflake-Labs%2Fnf-snowflake/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Snowflake-Labs","download_url":"https://codeload.github.com/Snowflake-Labs/nf-snowflake/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Snowflake-Labs%2Fnf-snowflake/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":277974415,"owners_count":25908396,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-02T02:00:08.890Z","response_time":67,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-10-02T07:47:33.487Z","updated_at":"2026-02-19T03:04:26.427Z","avatar_url":"https://github.com/Snowflake-Labs.png","language":"Groovy","funding_links":[],"categories":[],"sub_categories":[],"readme":"# nf-snowflake plugin\n\n## Overview\nnf-snowflake is a [Nextflow](https://www.nextflow.io/docs/latest/overview.html) plugin that enables Nextflow pipelines to run inside [Snowpark Container Service](https://docs.snowflake.com/en/developer-guide/snowpark-container-services/overview).\n\nEach Nextflow task is translated to a [Snowflake Job Service](https://docs.snowflake.com/en/sql-reference/sql/execute-job-service) and executed as an SPCS job. The Nextflow main/driver program can run in two modes:\n\n1. **Locally** - Running on your local machine or CI/CD environment, connecting to Snowflake via JDBC\n2. **Inside SPCS** - Running as a separate SPCS job within Snowpark Container Services\n\nThese two execution modes correspond to the two authentication methods supported by the plugin. When the main/driver program runs inside an SPCS job, Snowflake automatically injects the required environment variables (such as `SNOWFLAKE_ACCOUNT`, `SNOWFLAKE_HOST`, etc.) and the session token file (`/snowflake/session/token`). The plugin automatically discovers and uses these credentials for authentication.\n\nIntermediate results between different Nextflow processes are shared via [Snowflake stages](https://docs.snowflake.com/en/user-guide/data-load-local-file-system-create-stage), which must be configured as the working directory.\n\n## Prerequisites\n\nBefore using this plugin, you should have:\n\n- **Nextflow** (version 23.04.0 or later)\n- **Snowflake account** with access to:\n  - Snowpark Container Services (Compute Pools/Image Registries)\n  - Internal stages\n- **Familiarity with**:\n  - Nextflow pipelines and configuration\n  - Docker/container images\n  - Snowflake authentication methods\n\n## Authentication\n\nThe plugin supports two authentication methods, corresponding to the two execution modes for the main/driver program:\n\n### 1. Session Token Authentication (Main/Driver Running Inside SPCS)\n\nWhen the Nextflow main/driver program runs inside an SPCS job, Snowflake automatically injects the session token file at `/snowflake/session/token` and the following environment variables:\n\n- `SNOWFLAKE_ACCOUNT`\n- `SNOWFLAKE_HOST`\n- `SNOWFLAKE_DATABASE`\n- `SNOWFLAKE_SCHEMA`\n- `SNOWFLAKE_WAREHOUSE` (optional)\n\nThe plugin automatically discovers and uses these credentials for authentication. No additional configuration is required.\n\n### 2. Connections.toml Authentication (Main/Driver Running Locally)\n\nWhen the Nextflow main/driver program runs locally (on your machine or in CI/CD), the plugin uses the Snowflake [connections.toml](https://docs.snowflake.com/en/developer-guide/jdbc/jdbc-configure#connecting-using-the-connections-toml-file) configuration file for authentication.\n\n**File Locations** (searched in order):\n1. `~/.snowflake/connections.toml` (if directory exists)\n2. Location specified in `SNOWFLAKE_HOME` environment variable\n3. OS-specific defaults:\n   - Linux: `~/.config/snowflake/connections.toml`\n   - macOS: `~/Library/Application Support/snowflake/connections.toml`\n   - Windows: `%USERPROFILE%\\AppData\\Local\\snowflake\\connections.toml`\n\n**Example connections.toml:**\n```toml\n[default]\naccount = \"myaccount\"\nuser = \"myuser\"\npassword = \"mypassword\"\ndatabase = \"mydb\"\nschema = \"myschema\"\nwarehouse = \"mywh\"\n\n[production]\naccount = \"prodaccount\"\nauthenticator = \"externalbrowser\"\ndatabase = \"proddb\"\nschema = \"public\"\n```\n\n**Specify a connection in nextflow.config:**\n```groovy\nsnowflake {\n    connectionName = 'production'\n    computePool = 'MY_COMPUTE_POOL'\n}\n```\n\nIf no `connectionName` is specified, the plugin will use:\n1. Connection name from `SNOWFLAKE_DEFAULT_CONNECTION_NAME` environment variable\n2. The `default` connection from connections.toml\n\n## Configuration Reference\n\nAll plugin configurations are defined under the `snowflake` scope in your `nextflow.config`:\n\n### computePool\n\nThe name of the Snowflake compute pool to use for executing jobs.\n\n```groovy\nsnowflake {\n    computePool = 'MY_COMPUTE_POOL'\n}\n```\n\n### registryMappings\n\nDocker registry mappings for container images. Snowflake does not support pulling images directly from arbitrary external registries. Instead, you must first replicate container images from external registries (such as Docker Hub, GitHub Container Registry, etc.) to Snowflake image repositories.\n\nThe `registryMappings` configuration allows you to automatically replace external registry hostnames with Snowflake image repository names in your pipeline's container specifications.\n\n**Format:** Comma-separated list of mappings in the form `external_registry:snowflake_repository`\n\n```groovy\nsnowflake {\n    registryMappings = 'docker.io:my_registry,ghcr.io:github_registry'\n}\n```\n\n**How it works:**\n1. First, replicate images to your Snowflake image repository:\n   ```bash\n   docker pull docker.io/alpine:latest\n   docker tag docker.io/alpine:latest \u003csnowflake_repo_url\u003e/alpine:latest\n   docker push \u003csnowflake_repo_url\u003e/alpine:latest\n   ```\n\n2. Then, when your process uses `container 'docker.io/alpine:latest'`, the plugin automatically replaces `docker.io` with your Snowflake image repository URL, resulting in the correct Snowflake image reference.\n\n### connectionName\n\nThe name of the connection to use from the connections.toml file. When specified, the JDBC driver will use the connection configuration defined under this name.\n\n```groovy\nsnowflake {\n    connectionName = 'production'\n}\n```\n\n**Note:** This is only used when the session token file is not available (i.e., when running outside Snowpark Container Services).\n\n## Quick Start\n\nThis guide assumes you are familiar with both Nextflow and Snowpark Container Services.\n\n### 1. Create a Compute Pool\n\n```sql\nCREATE COMPUTE POOL my_compute_pool\nMIN_NODES = 2\nMAX_NODES = 5\nINSTANCE_FAMILY = CPU_X64_M\nAUTO_SUSPEND_SECS = 3600;\n```\n\n### 2. Create a Snowflake Internal Stage for Working Directory\n\n```sql\nCREATE OR REPLACE STAGE nxf_workdir\nENCRYPTION = (TYPE = 'SNOWFLAKE_SSE');\n```\n\n### 3. Set Up Image Repository\n\n```sql\nCREATE IMAGE REPOSITORY IF NOT EXISTS my_images;\n```\n\n### 4. Build and Upload Container Images\n\nBuild the container image for each Nextflow [process](https://www.nextflow.io/docs/latest/process.html), upload the image to [Snowflake Image Registry](https://docs.snowflake.com/en/developer-guide/snowpark-container-services/working-with-registry-repository), and update each process's [container](https://www.nextflow.io/docs/latest/reference/process.html#process-container) field.\n\n**Example process definition:**\n```groovy\nprocess INDEX {\n    tag \"$transcriptome.simpleName\"\n    container '/mydb/myschema/my_images/salmon:1.10.0'\n\n    input:\n    path transcriptome\n\n    output:\n    path 'index'\n\n    script:\n    \"\"\"\n    salmon index --threads $task.cpus -t $transcriptome -i index\n    \"\"\"\n}\n```\n\n### 5. Configure Nextflow\n\nAdd a Snowflake profile to your `nextflow.config` file and enable the nf-snowflake plugin:\n\n```groovy\nplugins {\n    id 'nf-snowflake@1.0.0'\n}\n\nprofiles {\n    snowflake {\n        process.executor = 'snowflake'\n\n        snowflake {\n            computePool = 'my_compute_pool'\n            registryMappings = 'docker.io:my_images'\n        }\n    }\n}\n```\n\n### 6. Run Your Pipeline\n\nExecute the Nextflow pipeline with the Snowflake profile:\n\n```bash\nnextflow run . -profile snowflake -work-dir snowflake://stage/nxf_workdir/\n```\n\n## Snowflake Filesystem and Working Directory\n\n### Snowflake Stage URI\n\nThe plugin uses a custom URI scheme to access Snowflake internal stages:\n\n```\nsnowflake://stage/\u003cstage_name\u003e/\u003cpath\u003e\n```\n\n**Components:**\n- `snowflake://` - URI scheme identifier\n- `stage/` - Literal prefix indicating a Snowflake stage\n- `\u003cstage_name\u003e` - The name of your Snowflake internal stage\n- `\u003cpath\u003e` - Optional path within the stage\n\n**Examples:**\n```groovy\n// Access root of a stage\nworkDir = 'snowflake://stage/my_stage/'\n\n// Access a subdirectory within a stage\nworkDir = 'snowflake://stage/my_stage/workflows/pipeline1/'\n```\n\n### Working Directory Requirement\n\n**IMPORTANT:** The Nextflow working directory (`workDir`) **must** be a Snowflake stage using the `snowflake://` URI scheme. This is a strict requirement for the plugin to function correctly.\n\nThe working directory is used to:\n- Store intermediate task results\n- Share data between pipeline processes\n- Store task execution metadata and logs\n\n**Correct configuration:**\n```groovy\nprofiles {\n    snowflake {\n        process.executor = 'snowflake'\n        workDir = 'snowflake://stage/nxf_workdir/'  // ✓ Valid\n\n        snowflake {\n            computePool = 'my_compute_pool'\n        }\n    }\n}\n```\n\n**Or specify on the command line:**\n```bash\nnextflow run . -profile snowflake -work-dir snowflake://stage/nxf_workdir/\n```\n\n**Invalid configurations:**\n```groovy\nworkDir = 's3://my-bucket/work/'              // ✗ Invalid - not a Snowflake stage\nworkDir = '/local/path/work/'                 // ✗ Invalid - local filesystem\nworkDir = 'snowflake://my_stage/work/'        // ✗ Invalid - missing 'stage/' prefix\n```\n\n### Stage Setup\n\nBefore running your pipeline, ensure your stage is properly configured:\n\n```sql\n-- Create an internal stage with encryption\nCREATE OR REPLACE STAGE my_workdir\nENCRYPTION = (TYPE = 'SNOWFLAKE_SSE');\n\n-- Verify stage exists\nSHOW STAGES LIKE 'my_workdir';\n\n-- Optional: Test stage access\nLIST @my_workdir;\n```\n\n## Additional Resources\n\n- [Nextflow Documentation](https://www.nextflow.io/docs/latest/index.html)\n- [Snowpark Container Services Documentation](https://docs.snowflake.com/en/developer-guide/snowpark-container-services/overview)\n- [Snowflake JDBC Configuration](https://docs.snowflake.com/en/developer-guide/jdbc/jdbc-configure)\n- [Nextflow Plugin Development](https://www.nextflow.io/docs/latest/plugins.html)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsnowflake-labs%2Fnf-snowflake","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsnowflake-labs%2Fnf-snowflake","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsnowflake-labs%2Fnf-snowflake/lists"}