Summary

nf-blocks is a Nextflow plugin that publishes pipeline outputs into a content-addressed store and records their lineage there. It owns the cas URI scheme, enabling users to direct outputs to a content-addressed store (e.g., outputDir = 'cas://lab') and maintain lineage records alongside Nextflow's own lid:// records. The plugin records DAG-CBOR lineage and maintains a SQLite index, allowing subsequent pipelines to read outputs back using the fromStore channel factory and browse stores with nf-blocks:explore.

Status: Beta (0.1.0-beta.1) — Currently supports local writable members with read-only S3 access, and the on-disk format may change before version 1.0.

Get Started

Enable the plugin and configure the lineage store and output directory to use the same store alias:

plugins {
    id 'nf-blocks@0.1.0-beta.1'
}

lineage.enabled = true
lineage.store.location = 'cas://lab'   // alias of the writable member
outputDir = 'cas://lab'                // optional: unset, it is the lineage alias

cas {
    stores {
        lab {
            location = '/data/cas'     // the writable member
        }
    }
    resolve = ['lab']                  // optional; default is every alias, writable first
    asserted_by = 'anonymous'          // optional opaque label; never the OS user name
}

Store aliases must match ^[a-z][a-z0-9_-]{0,31}$ and must not parse as a content address. When outputDir is unset, it defaults to the alias specified in lineage.store.location. Nextflow automatically downloads the plugin from the Nextflow Registry on first run.

Examples

Publishing outputs

Any pipeline with a workflow output block publishes into the store. With the configuration above, running a pipeline writes each published path to the store's coordinate tree:

/data/cas/coords/aligned/A/A.bam
/data/cas/coords/qc/A/A_qc/summary.txt
/data/cas/nf/<run hash>/aligned/A/A.bam/.data.json

The lineage record for a published file names its coordinate rather than a work directory path:

{"kind":"FileOutput","spec":{"path":"cas://lab/aligned/A/A.bam", ...}}

Reading outputs in another pipeline

A second pipeline reads a run's outputs back from the store. Configure it with the producer's store as a read-only member:

plugins {
    id 'nf-blocks@0.1.0-beta.1'
}

manifest.name = 'downstream'           // or cas.pipeline = 'downstream'

lineage.enabled = true
lineage.store.location = 'cas://mine'

cas {
    stores {
        mine { location = '/data/cas-downstream' }   // writable: this pipeline's own outputs
        lab  { location = '/data/cas' }              // the producer's store, read-only here
    }
    resolve = ['mine', 'lab']
}

Use fromStore to retrieve outputs:

include { fromStore } from 'plugin/nf-blocks'

workflow {
    main:
    picked  = channel.fromStore(selection: '<selection address>')
    aligned = channel.fromStore(run: 'latest', pipeline: 'cas-test-pipeline', output: 'aligned', where: [sample: 'B'])

    picked.mix(aligned).view()
}

Each item arrives in the shape the producer published it with cas:// paths that Nextflow stages like any other.

Using typed scripts

In a typed script with nextflow.enable.types = true, call fromStore through nextflow.Channel with records: true:

nextflow.enable.types = true

include { fromStore } from 'plugin/nf-blocks'

record Sample {
    sample: String
    lane: Integer
}

process COUNT_LINES {
    input:
    tuple(meta: Sample, bam: Path)

    output:
    stdout()

    script:
    """
    printf '%s\\t' '${meta.sample}'
    wc -l < '${bam}'
    """
}

workflow {
    COUNT_LINES(nextflow.Channel.fromStore(selection: '<selection address>', records: true))
        .view()
}

Browsing and creating Selections

Browse stores with the explorer:

nextflow plugin nf-blocks:explore

This serves a live view at http://127.0.0.1:<port>/ (port specified with --port or ephemeral). Create snapshots without starting a server:

nextflow plugin nf-blocks:snapshot

Curate outputs into named Selections from the command line:

nextflow plugin nf-blocks:items aligned nested.kit=truseq --run latest --pipeline cas-test-pipeline
nextflow -q plugin nf-blocks:items aligned lane=2 --run lid://<run hash> --format selection \
  | nextflow -q plugin nf-blocks:put /dev/stdin --name lane-2

License

Apache License 2.0. See the COPYING file for details.

What's new

• Beta release (0.1.0-beta.1) built and tested against Nextflow 26.04.6
• Support for content-addressed stores with DAG-CBOR lineage records and SQLite indexing
• fromStore channel factory for reading outputs from previous pipeline runs
• nf-blocks:explore web interface for browsing stores and managing Selections
• Command-line tools: nf-blocks:items, nf-blocks:put, and nf-blocks:snapshot
• Support for Selections that curate outputs across runs without copying

Breaking changes

• The on-disk format (block kinds, the index, the Store Log) may change before version 1.0. Blocks are content-addressed, so existing stores retain their blocks; format changes will include migrations.
• In typed scripts, channel.fromStore is not yet available; use nextflow.Channel.fromStore(..., records: true) instead (requires Nextflow fix for nextflow-io/nextflow#7694).
• S3 support is currently read-only; runs cannot publish to S3, through Fusion, or from cloud executors yet. Only local directories are writable members.