Summary

nf-blocks is a Nextflow plugin that publishes pipeline outputs into a content-addressed store and records their lineage there. It owns the cas URI scheme, allowing users to configure outputDir = 'cas://lab' to send every published file through the plugin instead of onto a results/ tree. It provides a lineage store for lineage.store.location = 'cas://lab', so Nextflow's own lid:// records live in the same store. The plugin writes DAG-CBOR lineage records into the store alongside Nextflow's lid:// records and maintains a SQLite index. Another pipeline can read the outputs back with the fromStore channel factory, and the nf-blocks:explore command browses them and saves Selections.

Status: Beta (0.2.0-beta.1). The plugin supports Milestone 4 (cloud) with any member being a local directory or an S3 location. It requires the nf-amazon plugin (>=3.9.2) and is built and tested against Nextflow 26.04.6.

Get Started

Enable the plugin and point both the lineage store and the output directory at the same store alias:

plugins {
    id 'nf-blocks@0.2.0-beta.1'
}

lineage.enabled = true
lineage.store.location = 'cas://lab'   // alias of the writable member
outputDir = 'cas://lab'                // optional: unset, it is the lineage alias

cas {
    stores {
        lab {
            location = '/data/cas'     // the writable member
        }
    }
    resolve = ['lab']                  // optional; default is every alias, writable first
    asserted_by = 'anonymous'          // optional opaque label; never the OS user name
}

An alias matches ^[a-z][a-z0-9_-]{0,31}$ and must not parse as a content address. outputDir may be left out: unset, it means the alias in lineage.store.location. When set, it must name that alias, or the run aborts.

Nextflow downloads the plugin from the Nextflow Registry the first time the pipeline runs.

To add nf-blocks to someone else's pipeline, such as an nf-core one, with a -c config file: a plugins { } block in that file replaces the pipeline's own plugin list rather than adding to it. Repeat the pipeline's pinned plugins beside nf-blocks, or Nextflow installs the latest release of each one the pipeline includes.

Examples

Basic Local Storage

Any pipeline with a workflow output block publishes into the store. With the configuration above, running a pipeline writes each published path under the store's coordinate tree:

/data/cas/coords/aligned/A/A.bam
/data/cas/coords/qc/A/A_qc/summary.txt
/data/cas/nf/<run hash>/aligned/A/A.bam/.data.json

The lineage record for a published file names its coordinate rather than a work directory path:

{"kind":"FileOutput","spec":{"path":"cas://lab/aligned/A/A.bam", ...}}

Publishing into S3

A writable S3 member takes an s3://<bucket>[/<prefix>] location; the S3 client is nf-amazon's, configured from the aws scope:

plugins {
    id 'nf-blocks@0.2.0-beta.1'
}

lineage.enabled = true
lineage.store.location = 'cas://lab'

workDir = 's3://my-work-bucket/work'
process.executor = 'awsbatch'
process.queue = 'my-queue'

aws {
    region = 'us-east-1'
    profile = 'my-profile'
}

cas {
    stores {
        lab { location = 's3://my-cas-bucket/cas' }
    }
    tmpDir = '/scratch/nf-blocks'
}

Reading Outputs in Another Pipeline

A second pipeline reads a run's outputs back out of the store with fromStore:

plugins {
    id 'nf-blocks@0.2.0-beta.1'
}

manifest.name = 'downstream'

lineage.enabled = true
lineage.store.location = 'cas://mine'

cas {
    stores {
        mine { location = '/data/cas-downstream' }
        lab  { location = '/data/cas' }
    }
    resolve = ['mine', 'lab']
}

include { fromStore } from 'plugin/nf-blocks'

workflow {
    main:
    picked  = channel.fromStore(selection: '<selection address>')
    aligned = channel.fromStore(run: 'latest', pipeline: 'cas-test-pipeline', output: 'aligned', where: [sample: 'B'])

    picked.mix(aligned).view()
}

Typed Script Usage

In a typed script, use nextflow.Channel.fromStore with records: true:

nextflow.enable.types = true

include { fromStore } from 'plugin/nf-blocks'

record Sample {
    sample: String
    lane: Integer
}

process COUNT_LINES {
    input:
    tuple(meta: Sample, bam: Path)

    output:
    stdout()

    script:
    """
    printf '%s\\t' '${meta.sample}'
    wc -l < '${bam}'
    """
}

workflow {
    COUNT_LINES(nextflow.Channel.fromStore(selection: '<selection address>', records: true))
        .view()
}

Command-Line Selection Management

From the command line, use nf-blocks:items to find items and save them as a named Selection:

nextflow plugin nf-blocks:items aligned nested.kit=truseq --run latest --pipeline cas-test-pipeline
nextflow -q plugin nf-blocks:items aligned lane=2 --run lid://<run hash>,lid://<other run hash> --format selection \
  | nextflow -q plugin nf-blocks:put /dev/stdin --name lane-2

Plugin Development and Building

make assemble      # Build the plugin
make test          # Run unit tests
make check         # Unit tests plus dependency check and memory bound
make smoke         # Build, install, and run a real pipeline
make release       # Publish to Nextflow Registry

License

Apache License 2.0. See the COPYING file for details.

What's new

• Status: Beta release (0.2.0-beta.1). Pin the version in plugins { } as the on-disk format (block kinds, the index, the Store Log) may still change before 1.0.
• Cloud Support: Milestone 4 (cloud) with S3 support for any member. Requires nf-amazon plugin (>=3.9.2).
• Index Snapshots: A head node starting cold can seed the cache from the member's Index Snapshot and read only Store Log entries written after it, avoiding the need to read every block of a large member.
• Typed Scripts: Support for typed Nextflow scripts with nextflow.Channel.fromStore(..., records: true) until nextflow-io/nextflow#7694 is fixed.
• Selections: Named groups of Output Items across runs composed from the explorer or command line with nf-blocks:items and nf-blocks:put.
• Explorer: Browse stores with nf-blocks:explore and generate Index Snapshots with nf-blocks:snapshot.