Summary

nf-blocks is a Nextflow plugin that publishes pipeline outputs into a content-addressed store and records their lineage there. It owns the cas URI scheme, allowing users to configure outputDir = 'cas://lab' to send published files through the plugin into a content-addressed store instead of a traditional results/ directory. The plugin provides a lineage store for lineage.store.location = 'cas://lab', integrating with Nextflow's own lid:// records. It maintains a SQLite index of lineage records in DAG-CBOR format, supports reading outputs from other pipelines via the fromStore channel factory, and includes an explorer tool (nf-blocks:explore) for browsing stores and managing Selections. The plugin is currently in beta (0.8.0-beta.2) and is built and tested against Nextflow 26.04.6.

Get Started

Enable the plugin and point both the lineage store and the output directory at the same store alias:

plugins {
    id 'nf-blocks@0.8.0-beta.2'
}

lineage.enabled = true
lineage.store.location = 'cas://lab'   // alias of the writable member
outputDir = 'cas://lab'                // optional: unset, it is the lineage alias

cas {
    stores {
        lab {
            location = '/data/cas'     // the writable member
        }
    }
    resolve = ['lab']                  // optional; default is every alias, writable first
    asserted_by = 'anonymous'          // optional opaque label; never the OS user name
}

An alias must match ^[a-z][a-z0-9_-]{0,31}$ and must not parse as a content address. outputDir may be left unset, in which case it defaults to the alias in lineage.store.location. When set, it must name that alias, or the run aborts. Nextflow downloads the plugin from the Nextflow Registry automatically on first use.

When adding nf-blocks to someone else's pipeline (such as an nf-core pipeline) with a -c config file, note that a plugins { } block replaces rather than adds to the pipeline's plugin list. Repeat the pipeline's pinned plugins alongside nf-blocks, or Nextflow will install the latest release of each plugin the pipeline includes, potentially breaking compatibility.

Examples

Any pipeline with a workflow output block publishes into the store. With the configuration above, running a pipeline writes each published path under the store's coordinate tree:

/data/cas/coords/aligned/A/A.bam
/data/cas/coords/qc/A/A_qc/summary.txt
/data/cas/nf/<run hash>/aligned/A/A.bam/.data.json

The lineage record for a published file names its coordinate rather than a work directory path:

{"kind":"FileOutput","spec":{"path":"cas://lab/aligned/A/A.bam", ...}}

To read outputs from a run in another pipeline, configure a read-only member alongside your writable store:

plugins {
    id 'nf-blocks@0.8.0-beta.2'
}

manifest.name = 'downstream'

lineage.enabled = true
lineage.store.location = 'cas://mine'

cas {
    stores {
        mine { location = '/data/cas-downstream' }
        lab  { location = '/data/cas' }
    }
    resolve = ['mine', 'lab']
}

Then use fromStore in your workflow:

include { fromStore } from 'plugin/nf-blocks'

workflow {
    main:
    picked  = channel.fromStore(selection: '<selection address>')
    aligned = channel.fromStore(run: 'latest', pipeline: 'cas-test-pipeline', output: 'aligned', where: [sample: 'B'])

    picked.mix(aligned).view()
}

Each item arrives in the shape the producer published it, with every file a cas:// path that Nextflow stages automatically. The where: parameter filters by metadata paths using equality or comparison operators (>, >=, <, <=).

For S3 storage with a cloud executor:

plugins {
    id 'nf-blocks@0.8.0-beta.2'
}

lineage.enabled = true
lineage.store.location = 'cas://lab'

workDir = 's3://my-work-bucket/work'
process.executor = 'awsbatch'
process.queue = 'my-queue'

aws {
    region = 'us-east-1'
    profile = 'my-profile'
}

cas {
    stores {
        lab { location = 's3://my-cas-bucket/cas' }
    }
    tmpDir = '/scratch/nf-blocks'
}

Browse a store with:

nextflow plugin nf-blocks:explore

Manage retention and perform sweeps:

nextflow plugin nf-blocks:prune --keep-last 3
nextflow plugin nf-blocks:prune --keep-last 3 --apply true
nextflow plugin nf-blocks:sweep
nextflow plugin nf-blocks:sweep --apply true

What's new

This is a beta release (0.8.0-beta.2). Key milestones implemented include:

• Milestone 4 (cloud): Any store member, including the writable one, can be a local directory or S3 location. Runs on AWS Batch with or without Fusion publish to writable S3 members. Under Fusion, task nodes hash their own outputs; between S3 buckets, S3 computes SHA-256 server-side, so most files never pass through the head node.
• Milestone 5 (nf-core pipelines): nf-blocks records lineage from workflow output { } blocks only. Processes using publishDir are warned once; their files are stored but not recorded against any run (counted as unjoined). Outputs declaring index { } record their Output Index File.
• Milestone 6 (retention): Every run's content is a root by default and kept until released. A sweep only deletes released, unpinned content past its grace period.

The plugin requires nf-amazon >= 3.9.2. The on-disk format (block kinds, index, Store Log) may change before 1.0.0, with migrations provided for block addresses. Warnings from the plugin go to .nextflow.log and the terminal.

Breaking changes

The on-disk format (block kinds, the index, the Store Log) may still change before version 1.0. Block format changes will include migrations, since a block's address is its content and existing stores keep their blocks.

In typed scripts with nextflow.enable.types = true, Nextflow 26.04.6 cannot reach plugin channel factories through channel. (see nextflow-io/nextflow#7694). Call nextflow.Channel.fromStore(..., records: true) instead until this is fixed.

License

Apache License 2.0