Summary

Enterprise Databricks extension for Nextflow, provided by Seqera Labs. This plugin extends the public nf-databricks plugin with enterprise features including Unity Catalog filesystem (uc:// scheme), Unity Catalog functions, and planned Unity Catalog executor and UC secrets provider capabilities.

Get Started

To use the xpack-databricks plugin, set your license and configure the plugin in your Nextflow configuration:

export NXF_XPACK_LICENSE=<your license string>

Add the plugin to your nextflow.config:

plugins {
    id 'xpack-databricks@0.2.0'
}

The nf-databricks plugin is automatically loaded as a dependency.

Configure authentication using the databricks block (or environment variables):

databricks {
    host = 'https://<workspace>.cloud.databricks.com'
    token = '<token>'
}

Authentication precedence (highest to lowest):

1. Per-call overrides (opts.host/opts.token or opts.serverUrl/opts.accessToken)
2. databricks { ... } session config
3. Environment variables DATABRICKS_HOST / DATABRICKS_TOKEN
4. Legacy environment variables UNITY_CATALOG_URL / UNITY_CATALOG_TOKEN

Examples

Unity Catalog Filesystem Access

Access Unity Catalog volumes using the uc:// URI scheme:

Channel.fromPath('uc://catalog.schema.volume/data/*.csv')
    .view()

process example {
    publishDir 'uc://catalog.schema.volume/results'
    // ...
}

Unity Catalog Functions

Query Unity Catalog metadata:

include { listCatalogs; listSchemas; listTables; describeTable } from 'plugin/xpack-databricks'

workflow {
    // List catalogs
    catalogs = listCatalogs()

    // List schemas in a catalog
    schemas = listSchemas('my_catalog')

    // List tables
    tables = listTables('my_catalog', 'my_schema')

    // Describe table structure
    desc = describeTable('my_catalog', 'my_schema', 'my_table')
}

License

Proprietary software provided by Seqera Labs. Copyright 2021-2025, Seqera Labs, S.L. All Rights Reserved.