Summary

The xpack-platform plugin extends Seqera Platform with features that ship outside the open-source Nextflow distribution. This version provides the global cache: a cross-pipeline, cross-run task reuse system backed by cloud object storage. The global cache turns a cloud storage prefix into a shared, content-addressable cache where the work directory itself is the cache, shared across runs, pipelines and users. Unlike -resume, which reuses work from a previous run of the same pipeline, tasks are reused whenever any other run has already executed the same work, regardless of who ran it or which pipeline it belonged to. Supported on Amazon S3, Azure Blob Storage and Google Cloud Storage. Requires a license (NXF_XPACK_LICENSE).

Get Started

Add the plugin and point it at a cloud prefix:

plugins {
    id 'xpack-platform'
}

globalCache {
    enabled      = true
    path         = 's3://my-bucket/cache'
    fileIdentity = 'provenance'
}

Then run as usual — there is no -resume flag to pass, because reuse is not scoped to a session:

nextflow run main.nf

Note: globalCache.path derives the work directory, so do not set -w or workDir.

Configuration Options

globalCache.enabled — Enable the global cache (default: false). Requires Nextflow 26.09.1-edge or later.

globalCache.path — The shared cloud storage prefix used as the cache, e.g. s3://bucket/cache. Must be a cloud URI and also derives the run work directory.

globalCache.fileIdentity — How a task input is identified across runs (default: provenance):

• provenance — an input produced by a task is identified by that task using metadata
• sample — an input is identified by a fixed-cost sample of its content (head, middle, tail windows plus byte size)

globalCache.secondaryPaths — An ordered list of read-only secondary cache prefixes consulted when a task is not found in the primary cache, e.g. ['s3://shared/cache'] (default: []).

globalCache.allowUnverifiedEndpoint — Skip the startup check for S3-compatible endpoints (default: false).

globalCache.sample — Options for the sample file identity strategy:

• inlineMembers — directory member threshold (default: 5)
• threshold — file size above which sampling is used (default: 1 MB)
• size — size of each sampled window (default: 128 KB)

Examples

Amazon S3

plugins {
    id 'xpack-platform'
}

globalCache {
    enabled      = true
    path         = 's3://my-bucket/cache'
    fileIdentity = 'sample'

    sample {
        threshold = '1 MB'
        size      = '128 KB'
    }
}

aws {
    region = 'eu-west-1'
}

Google Cloud Storage

plugins {
    id 'xpack-platform'
}

globalCache {
    enabled = true
    path    = 'gs://my-bucket/cache'
}

google {
    project  = 'my-project'
    location = 'us-central1'
}

Azure Blob Storage

plugins {
    id 'xpack-platform'
}

globalCache {
    enabled = true
    path    = 'az://my-container/cache'
}

azure {
    storage {
        accountName = 'mystorageaccount'
        accountKey  = System.getenv('AZURE_STORAGE_KEY')
    }
}

Sharing a read-only cache

globalCache {
    enabled        = true
    path           = 's3://my-bucket/cache'
    fileIdentity   = 'sample'
    secondaryPaths = ['s3://org-shared/cache']
}

Maintenance Commands

Push cache entries from source to target:

nextflow plugin xpack-platform:push-cache <source-cache> <target-cache> \
    [--dry-run=true] [--yes=true]

Clean cache entries by state or age:

nextflow plugin xpack-platform:clean-cache <cache-path> \
    [--failed=true] [--orphans=true] [--older-than=<age>] \
    [--dry-run=true] [--yes=true]

License

This is proprietary software provided by Seqera Labs. A valid license key is required for use. Unauthorized copying, modification, or distribution is prohibited.