Summary

nf-blocks is a Nextflow plugin that publishes pipeline outputs into a content-addressed store and records their lineage there. It owns the cas URI scheme, allowing users to configure outputDir = 'cas://lab' to send published files through the plugin instead of onto a traditional results/ directory tree. The plugin provides a lineage store via lineage.store.location = 'cas://lab', enabling Nextflow's lid:// records to live in the same store. It supports content-addressed storage with lineage tracking, including DAG-CBOR lineage records and SQLite indexing. The plugin enables reading outputs in downstream pipelines through the fromStore channel factory and browsing stored outputs via nf-blocks:explore. It is built and tested against Nextflow 26.04.6 and is currently in beta (0.6.0-beta.1).

Get Started

Enable the plugin and point both the lineage store and the output directory at the same store alias:

plugins {
    id 'nf-blocks@0.6.0-beta.1'
}

lineage.enabled = true
lineage.store.location = 'cas://lab'   // alias of the writable member
outputDir = 'cas://lab'                // optional: unset, it is the lineage alias

cas {
    stores {
        lab {
            location = '/data/cas'     // the writable member
        }
    }
    resolve = ['lab']                  // optional; default is every alias, writable first
    asserted_by = 'anonymous'          // optional opaque label; never the OS user name
}

An alias matches ^[a-z][a-z0-9_-]{0,31}$ and must not parse as a content address. outputDir may be left out: unset, it means the alias in lineage.store.location. When set, it must name that alias, or the run aborts. Nextflow downloads the plugin from the Nextflow Registry the first time the pipeline runs.

To add nf-blocks to someone else's pipeline, such as an nf-core one, with a -c config file: a plugins { } block in that file replaces the pipeline's own plugin list rather than adding to it. Repeat the pipeline's pinned plugins beside nf-blocks, or Nextflow installs the latest release of each one the pipeline includes.

Examples

Any pipeline with a workflow output block publishes into the store. Running a pipeline with the configuration above writes each published path under the store's coordinate tree:

/data/cas/coords/aligned/A/A.bam
/data/cas/coords/qc/A/A_qc/summary.txt
/data/cas/nf/<run hash>/aligned/A/A.bam/.data.json

The lineage record for a published file names its coordinate rather than a work directory path:

{"kind":"FileOutput","spec":{"path":"cas://lab/aligned/A/A.bam", ...}}

To read outputs in another pipeline, use fromStore with the producer's store configured as a read-only member:

plugins {
    id 'nf-blocks@0.6.0-beta.1'
}

manifest.name = 'downstream'

lineage.enabled = true
lineage.store.location = 'cas://mine'

cas {
    stores {
        mine { location = '/data/cas-downstream' }
        lab  { location = '/data/cas' }
    }
    resolve = ['mine', 'lab']
}

In your Nextflow script, include and use fromStore:

include { fromStore } from 'plugin/nf-blocks'

workflow {
    main:
    picked  = channel.fromStore(selection: '<selection address>')
    aligned = channel.fromStore(run: 'latest', pipeline: 'cas-test-pipeline', output: 'aligned', where: [sample: 'B'])

    picked.mix(aligned).view()
}

Each item arrives in the shape the producer published it, with every file a cas:// path that Nextflow stages like any other.

For S3 storage, configure a writable S3 member with the nf-amazon plugin:

plugins {
    id 'nf-blocks@0.6.0-beta.1'
}

lineage.enabled = true
lineage.store.location = 'cas://lab'

workDir = 's3://my-work-bucket/work'
process.executor = 'awsbatch'
process.queue = 'my-queue'

aws {
    region = 'us-east-1'
    profile = 'my-profile'
}

cas {
    stores {
        lab { location = 's3://my-cas-bucket/cas' }
    }
    tmpDir = '/scratch/nf-blocks'
}

Browse a store using the explorer:

nextflow plugin nf-blocks:explore

This serves a live view at http://127.0.0.1:<port>/. Create and manage Selections from the command line:

nextflow plugin nf-blocks:items aligned nested.kit=truseq --run latest --pipeline cas-test-pipeline
nextflow -q plugin nf-blocks:items aligned lane=2 --run lid://<run hash> --format selection \
  | nextflow -q plugin nf-blocks:put /dev/stdin --name lane-2

Manage data retention:

nextflow plugin nf-blocks:prune --keep-last 3
nextflow plugin nf-blocks:prune --keep-last 3 --apply true

nextflow plugin nf-blocks:sweep
nextflow plugin nf-blocks:sweep --apply true

What's new

Status: beta (0.6.0-beta.1)

Current support includes:

• Milestone 4 (cloud): Any member, including the writable one, can be a local directory or S3 location. Runs on AWS Batch with or without Fusion publish into writable S3 members. Under Fusion, task nodes hash their own outputs; between S3 buckets, S3 computes SHA-256 server-side, so most files never pass through the head node.
• Milestone 5 (nf-core pipelines): nf-blocks records lineage from workflow outputs (output { }) only. Processes using publishDir are warned once; files are stored but counted as unjoined. Outputs declaring index { } record their Output Index File linked from the Output Collection.
• Milestone 6 (retention): Every run's content is a root by default and kept until released. Sweeps only delete released, unpinned, and past-grace-period content.
• nf-blocks requires the nf-amazon plugin (>=3.9.2).
• In typed scripts (nextflow.enable.types = true), use nextflow.Channel.fromStore(..., records: true) instead of channel.fromStore due to a Nextflow limitation.
• The on-disk format (block kinds, index, Store Log) may change before 1.0, with migrations provided for existing stores.

Breaking changes

• The on-disk format (block kinds, the index, the Store Log) may still change before version 1.0.0. A change to the blocks will come with a migration since a block's address is its content and existing stores keep their blocks.
• In typed scripts, Nextflow 26.04.6 cannot reach plugin factories through channel. — use nextflow.Channel.fromStore(..., records: true) instead until this is fixed in Nextflow.
• When adding nf-blocks to someone else's pipeline via a -c config file, the plugins { } block replaces the pipeline's plugin list rather than adding to it. You must repeat all the pipeline's pinned plugins alongside nf-blocks to avoid unexpected version updates.

License

Apache License 2.0. See the COPYING file for details.