Summary

nf-blocks is a Nextflow plugin that publishes pipeline outputs into a content-addressed store and records their lineage there. It owns the cas URI scheme, allowing users to configure outputDir = 'cas://lab' to send published files through the plugin instead of a traditional results/ tree. The plugin provides a lineage store for lineage.store.location = 'cas://lab', integrating with Nextflow's own lid:// records. It writes DAG-CBOR lineage records into the store, maintains a SQLite index, and provides a fromStore channel factory for reading outputs in downstream pipelines. An explorer tool (nf-blocks:explore) enables browsing and managing Selections. The plugin is currently in beta (0.5.0-beta.1) and supports cloud deployment on AWS Batch with S3, retention policies, and migration from publishDir to workflow outputs.

Get Started

Enable the plugin and point both the lineage store and output directory at the same store alias:

plugins {
    id 'nf-blocks@0.5.0-beta.1'
}

lineage.enabled = true
lineage.store.location = 'cas://lab'   // alias of the writable member
outputDir = 'cas://lab'                // optional: unset, it is the lineage alias

cas {
    stores {
        lab {
            location = '/data/cas'     // the writable member
        }
    }
    resolve = ['lab']                  // optional; default is every alias, writable first
    asserted_by = 'anonymous'          // optional opaque label; never the OS user name
}

An alias matches ^[a-z][a-z0-9_-]{0,31}$ and must not parse as a content address. outputDir may be left out: unset, it means the alias in lineage.store.location. When set, it must name that alias, or the run aborts. Nextflow downloads the plugin from the Nextflow Registry the first time the pipeline runs.

To add nf-blocks to someone else's pipeline, such as an nf-core one, with a -c config file: a plugins { } block in that file replaces the pipeline's own plugin list rather than adding to it. Repeat the pipeline's pinned plugins beside nf-blocks, or Nextflow installs the latest release of each one the pipeline includes (measured with nf-core/sarek 3.10.0, whose nf-schema@2.7.2 pin was replaced by nf-schema 3.0.0, and the run failed).

For S3 deployment with AWS Batch:

plugins {
    id 'nf-blocks@0.5.0-beta.1'
}

lineage.enabled = true
lineage.store.location = 'cas://lab'

workDir = 's3://my-work-bucket/work'
process.executor = 'awsbatch'
process.queue = 'my-queue'

aws {
    region = 'us-east-1'
    profile = 'my-profile'             // or any credential source nf-amazon accepts
}

cas {
    stores {
        lab { location = 's3://my-cas-bucket/cas' }
    }
    tmpDir = '/scratch/nf-blocks'      // optional; default java.io.tmpdir
}

Examples

Any pipeline with a workflow output block publishes into the store:

workflow {
    aligned_bam = ALIGN(input_samples)
    qc_summary = QC(aligned_bam)
    
    output {
        aligned_bam
        qc_summary
    }
}

Published files are stored under the store's coordinate tree:

/data/cas/coords/aligned/A/A.bam
/data/cas/coords/qc/A/A_qc/summary.txt
/data/cas/nf/<run hash>/aligned/A/A.bam/.data.json

The lineage record for a published file names its coordinate:

{"kind":"FileOutput","spec":{"path":"cas://lab/aligned/A/A.bam", ...}}

Reading outputs in a downstream pipeline using fromStore:

include { fromStore } from 'plugin/nf-blocks'

workflow {
    main:
    picked  = channel.fromStore(selection: '<selection address>')
    aligned = channel.fromStore(run: 'latest', pipeline: 'cas-test-pipeline', output: 'aligned', where: [sample: 'B'])

    picked.mix(aligned).view()
}

For typed scripts with Nextflow 26.04.6:

nextflow.enable.types = true

include { fromStore } from 'plugin/nf-blocks'

record Sample {
    sample: String
    lane: Integer
}

process COUNT_LINES {
    input:
    tuple(meta: Sample, bam: Path)

    output:
    stdout()

    script:
    """
    printf '%s\\t' '${meta.sample}'
    wc -l < '${bam}'
    """
}

workflow {
    COUNT_LINES(nextflow.Channel.fromStore(selection: '<selection address>', records: true))
        .view()
}

Configuring a downstream pipeline to read from a producer's store:

plugins {
    id 'nf-blocks@0.5.0-beta.1'
}

manifest.name = 'downstream'

lineage.enabled = true
lineage.store.location = 'cas://mine'

cas {
    stores {
        mine { location = '/data/cas-downstream' }   // writable: this pipeline's own outputs
        lab  { location = '/data/cas' }              // the producer's store, read-only here
    }
    resolve = ['mine', 'lab']
}

Command-line operations for managing Selections and retention:

nextflow plugin nf-blocks:items aligned nested.kit=truseq --run latest --pipeline cas-test-pipeline
nextflow -q plugin nf-blocks:items aligned lane=2 --run lid://<run hash> --format selection \
  | nextflow -q plugin nf-blocks:put /dev/stdin --name lane-2

nextflow plugin nf-blocks:prune --keep-last 3
nextflow plugin nf-blocks:prune --keep-last 3 --apply true

nextflow plugin nf-blocks:sweep
nextflow plugin nf-blocks:sweep --apply true --budget '10 GB'

nextflow plugin nf-blocks:untrash <cid>

Exploring and creating snapshots:

nextflow plugin nf-blocks:explore
nextflow plugin nf-blocks:snapshot

License

Apache License 2.0. See the COPYING file for details.

What's new

• Beta release (0.5.0-beta.1): Requires Nextflow 26.04.6 and nf-amazon plugin (>=3.9.2)
• Milestone 4 (cloud): Support for local directories and S3 locations as writable members; AWS Batch deployment with or without Fusion; server-side SHA-256 hashing on S3 with optional head-node hashing under Fusion
• Milestone 5 (nf-core pipelines): Lineage recording from workflow outputs only; warnings for publishDir usage; Output Index File support via index { } declarations
• Milestone 6 (retention): Content retention by default as roots; sweep operations with age floor and grace period; pinning and release mechanisms; Trash ledger for safe deletion
• On-disk format (block kinds, index, Store Log) subject to change before 1.0; changes include migrations to preserve existing stores

Breaking changes

• Typed script limitation: In Nextflow 26.04.6 with nextflow.enable.types = true, plugin channel factories are not accessible via channel.fromStore(). Use nextflow.Channel.fromStore(..., records: true) instead. This is being tracked in nextflow-io/nextflow#7694
• Plugin pinning required: When adding nf-blocks to an external pipeline (e.g., nf-core) via -c config file, a plugins { } block replaces the pipeline's plugin list entirely. All pipeline's pinned plugins must be repeated alongside nf-blocks, otherwise Nextflow installs the latest release of each plugin, which may cause compatibility issues
• publishDir migration: Pipelines using publishDir instead of workflow output { } blocks will have files stored but lineage not recorded; migration to workflow outputs is required to maintain lineage tracking
• S3 bucket policy implications: Enforcing conditional writes on the blocks/ prefix prevents CopyObject operations, forcing all S3-sourced outputs through the head node and requiring cas.tmpDir capacity for the largest outputs