Summary

This plugin integrates Nextflow with lakeFS, allowing you to use lakefs:// URIs as inputs and outputs in your pipelines. It brings the power of data versioning, reproducibility, and atomic operations to your Nextflow workflows. The plugin treats lakeFS repositories and branches as a native file system, enabling seamless data management without requiring manual pre-loading or post-processing steps.

Key Features:

• Native URI Support: Use lakefs://<repo>/<branch>/<path> URIs directly in your Nextflow scripts and nextflow.config
• Read Inputs: Stage input files from a lakeFS repository for use in your processes
• Publish Outputs: Publish output files from your processes directly to a lakeFS repository branch
• Transparent Authentication: Securely configure your lakeFS credentials
• Multiple Transfer Modes: Choose between signed_url or physical_path modes for data access
• Auto-Create Branches: Optionally create new branches automatically when writing to non-existent branches

Get Started

Installation

Add the following to your nextflow.config file. Replace <version> with the desired plugin version (e.g., 0.1.0):

plugins {
    id 'nf-lakefs@<version>'
}

Configuration

Configure the plugin with your lakeFS server details and credentials in your nextflow.config:

lakefs {
    apiUrl    = 'https://your-lakefs-server.example.com/api/v1'
    accessKey = 'YOUR_LAKEFS_ACCESS_KEY'
    secretKey = 'YOUR_LAKEFS_SECRET_KEY'
    transferMode = 'signed_url'
    autoCreateBranch = false
    autoCreateBranchSource = 'main'
}

Configuration Options:

Option	Required	Default	Description
apiUrl	Yes	-	lakeFS API endpoint URL (must end with /api/v1)
accessKey	Yes	-	lakeFS access key
secretKey	Yes	-	lakeFS secret key
transferMode	No	signed_url	Transfer mode: signed_url or physical_path
autoCreateBranch	No	false	Automatically create branch if it doesn't exist when writing
autoCreateBranchSource	No	main	Source branch to create new branches from
readTimeout	No	60s	HTTP read timeout for lakeFS API calls
connectTimeout	No	30s	HTTP connect timeout for lakeFS API calls
allowedSchemaBucketsForLinking	No	[]	Whitelist of cloud storage scheme+bucket prefixes allowed as link sources

Transfer Modes

The plugin supports two modes for reading and writing data:

1. signed_url - The plugin requests a signed URL for file access from lakeFS or generates a cloud provider storage signed URL for write requests
2. physical_path - The plugin requests the cloud provider specific physical path (s3:// or gs://) and delegates the request to the relevant Nextflow plugin. Ensure your Nextflow process has permissions to use the cloud provider API requests

Advanced Configuration: Auto-Create Branch

When autoCreateBranch is enabled, the plugin automatically creates a new branch if it doesn't exist when you attempt to write to it:

lakefs {
    apiUrl    = 'https://your-lakefs-server.example.com/api/v1'
    accessKey = 'YOUR_LAKEFS_ACCESS_KEY'
    secretKey = 'YOUR_LAKEFS_SECRET_KEY'
    autoCreateBranch = true
    autoCreateBranchSource = 'main'
}

Writing to lakefs://my-repo/new-branch/file.txt will automatically create new-branch from main if it doesn't already exist.

AWS Configuration (for physical_path mode with S3)

When using physical_path with an S3-backed lakeFS repository, also configure your AWS credentials:

aws {
    accessKey = '<AWS_ACCESS_KEY_ID>'
    secretKey = '<AWS_SECRET_ACCESS_KEY>'
    region    = 'eu-central-1'
}

allowedSchemaBucketForLinking Configuration

When using java.nio.file.Files.createLink() or Link mode on publishing, whitelist cloud storage scheme+bucket combinations allowed to be treated as equivalent backends:

lakefs {
    allowedSchemaBucketForLinking = [
            'gs://bucket1',
            'gs://bucket2'
    ]
}

For S3-backed repos:

lakefs {
    allowedSchemaBucketForLinking = [
            's3://bucket1',
            's3://bucket2'
    ]
}

Examples

Basic Pipeline with lakeFS Input and Output

This example reads a configuration file from a lakeFS repository, processes it, and publishes the result back to the repository:

main.nf

params.input = 'lakefs://my-repo/main/path/to/**/config.yaml'
params.output_dir = 'lakefs://my-repo/output/path/to/'

process EXAMPLE_PROCESS {

    input:
    path my_file

    output:
    path 'result.txt'

    script:
    """
    echo "Processing ${my_file}..."
    cat "${my_file}" | wc -l > result.txt
    """
}

workflow {
    Channel.fromPath(params.input, type: 'file')
            | view
            | EXAMPLE_PROCESS
}

Run the pipeline:

nextflow run main.nf 

License

This Nextflow plugin is currently published for read-only usage.