Summary

nf-blocks is a Nextflow plugin that publishes pipeline outputs into a content-addressed store and records their lineage there. It owns the cas URI scheme, enabling users to direct outputs to a content-addressed store via outputDir = 'cas://lab' and configure lineage storage with lineage.store.location = 'cas://lab'. The plugin records DAG-CBOR lineage records alongside Nextflow's lid:// records, maintains a SQLite index, and provides the fromStore channel factory for reading outputs in downstream pipelines. It also includes nf-blocks:explore for browsing the store and managing Selections. Currently in beta (0.1.0-beta.2), the plugin supports local writable stores with read-only S3 support, and the on-disk format may change before version 1.0.

Get Started

Enable the plugin and configure the lineage store and output directory to point at the same store alias:

plugins {
    id 'nf-blocks@0.1.0-beta.2'
}

lineage.enabled = true
lineage.store.location = 'cas://lab'   // alias of the writable member
outputDir = 'cas://lab'                // optional: unset, it is the lineage alias

cas {
    stores {
        lab {
            location = '/data/cas'     // the writable member
        }
    }
    resolve = ['lab']                  // optional; default is every alias, writable first
    asserted_by = 'anonymous'          // optional opaque label; never the OS user name
}

An alias must match ^[a-z][a-z0-9_-]{0,31}$ and must not parse as a content address. outputDir may be left unset, in which case it defaults to the alias in lineage.store.location. When set, it must name that alias, or the run aborts. Nextflow downloads the plugin from the Nextflow Registry on first run.

Examples

Any pipeline with a workflow output block publishes into the store. Running a configured pipeline writes each published path under the store's coordinate tree:

/data/cas/coords/aligned/A/A.bam
/data/cas/coords/qc/A/A_qc/summary.txt
/data/cas/nf/<run hash>/aligned/A/A.bam/.data.json

The lineage record for a published file names its coordinate rather than a work directory path:

{"kind":"FileOutput","spec":{"path":"cas://lab/aligned/A/A.bam", ...}}

Reading outputs in another pipeline

A second pipeline reads a run's outputs or a Selection back out of the store with fromStore. Its config names the producer's store as a read-only member beside its own writable one:

plugins {
    id 'nf-blocks@0.1.0-beta.2'
}

manifest.name = 'downstream'           // or cas.pipeline = 'downstream'

lineage.enabled = true
lineage.store.location = 'cas://mine'

cas {
    stores {
        mine { location = '/data/cas-downstream' }   // writable: this pipeline's own outputs
        lab  { location = '/data/cas' }              // the producer's store, read-only here
    }
    resolve = ['mine', 'lab']
}

The script includes fromStore and uses it:

include { fromStore } from 'plugin/nf-blocks'

workflow {
    main:
    picked  = channel.fromStore(selection: '<selection address>')
    aligned = channel.fromStore(run: 'latest', pipeline: 'cas-test-pipeline', output: 'aligned', where: [sample: 'B'])

    picked.mix(aligned).view()
}

For typed scripts with nextflow.enable.types = true, call through nextflow.Channel with records: true:

nextflow.enable.types = true

include { fromStore } from 'plugin/nf-blocks'

record Sample {
    sample: String
    lane: Integer
}

process COUNT_LINES {
    input:
    tuple(meta: Sample, bam: Path)

    output:
    stdout()

    script:
    """
    printf '%s\\t' '${meta.sample}'
    wc -l < '${bam}'
    """
}

workflow {
    COUNT_LINES(nextflow.Channel.fromStore(selection: '<selection address>', records: true))
        .view()
}

Browsing and managing the store

Browse the store with nextflow plugin nf-blocks:explore, which serves a live view at http://127.0.0.1:<port>/. Create snapshots without a server using nextflow plugin nf-blocks:snapshot. Create Selections from the command line:

nextflow plugin nf-blocks:items aligned nested.kit=truseq --run latest --pipeline cas-test-pipeline
nextflow -q plugin nf-blocks:items aligned lane=2 --run lid://<run hash>,lid://<other run hash> --format selection \
  | nextflow -q plugin nf-blocks:put /dev/stdin --name lane-2

License

Apache License 2.0. See the COPYING file for details.

What's new

• Beta release (0.1.0-beta.2) with support for local writable stores and read-only S3 access
• DAG-CBOR lineage records recorded alongside Nextflow's lid:// records
• fromStore channel factory for reading outputs in downstream pipelines
• nf-blocks:explore for browsing the store and managing Selections
• nf-blocks:put and nf-blocks:items command-line tools for creating and exporting Selections

Breaking changes

• The on-disk format (block kinds, the index, the Store Log) may change before version 1.0. A change to blocks will include migration support, as a block's address is its content and existing stores keep their blocks.
• In typed scripts (nextflow.enable.types = true), use nextflow.Channel.fromStore(..., records: true) instead of channel.fromStore until nextflow-io/nextflow#7694 is fixed.
• S3 support is read-only; runs cannot publish to S3, through Fusion, or from cloud executors yet.