Summary

nf-blocks is a Nextflow plugin that publishes pipeline outputs into a content-addressed store and records their lineage there. It owns the cas URI scheme, so outputDir = 'cas://lab' sends every published file through the plugin instead of onto a results/ tree, and it provides a lineage store for lineage.store.location = 'cas://lab', so Nextflow's own lid:// records live in the same store.

It is built and tested against released Nextflow 26.04.6. A run writes DAG-CBOR lineage records into the store beside Nextflow's own lid:// records, and keeps a SQLite index of them; another pipeline reads the outputs back with the fromStore channel factory, and nf-blocks:explore browses them and saves Selections. The plugin supports cloud storage (AWS S3), nf-core pipelines, and retention policies with garbage collection.

Status: beta (0.10.0-beta.1). Pin the version in plugins { } as the examples below show.

Get Started

Enable the plugin and point both the lineage store and the output directory at the same store alias:

plugins {
    id 'nf-blocks@0.10.0-beta.1'
}

lineage.enabled = true
lineage.store.location = 'cas://lab'   // alias of the writable member
outputDir = 'cas://lab'                // optional: unset, it is the lineage alias

cas {
    stores {
        lab {
            location = '/data/cas'     // the writable member
        }
    }
    resolve = ['lab']                  // optional; default is every alias, writable first
    asserted_by = 'ada@example.org'    // optional; who the blocks say asserted them
}

asserted_by is optional. Left unset, nf-blocks uses your Seqera Platform identity, then NXF_BLOCKS_USER, then git config user.email, and warns if none of them gives a name. A value of anonymous counts as unset.

An alias matches ^[a-z][a-z0-9_-]{0,31}$ and must not parse as a content address. outputDir may be left out: unset, it means the alias in lineage.store.location. When set, it must name that alias, or the run aborts.

Nextflow downloads the plugin from the Nextflow Registry the first time the pipeline runs.

To add nf-blocks to someone else's pipeline with a -c config file: a plugins { } block in that file replaces the pipeline's own plugin list rather than adding to it. Repeat the pipeline's pinned plugins beside nf-blocks, or Nextflow installs the latest release of each one the pipeline includes.

nf-blocks requires the nf-amazon plugin (>=3.9.2). If you install nf-blocks by unpacking a local build into NXF_PLUGINS_DIR rather than from the registry, Nextflow 26.04.6 does not fetch that dependency, so install it alongside: nextflow plugin install nf-amazon@3.9.2.

Examples

Any pipeline with a workflow output block publishes into the store. With the configuration above, running it writes each published path under the store's coordinate tree:

/data/cas/coords/aligned/A/A.bam
/data/cas/coords/qc/A/A_qc/summary.txt
/data/cas/nf/<run hash>/aligned/A/A.bam/.data.json

and the lineage record for a published file names its coordinate rather than a work directory path:

{"kind":"FileOutput","spec":{"path":"cas://lab/aligned/A/A.bam", ...}}

Publishing into S3

A writable S3 member takes an s3://<bucket>[/<prefix>] location; the S3 client is nf-amazon's, configured from the aws scope. With a cloud executor the work dir is on S3 too:

plugins {
    id 'nf-blocks@0.10.0-beta.1'
}

lineage.enabled = true
lineage.store.location = 'cas://lab'

workDir = 's3://my-work-bucket/work'
process.executor = 'awsbatch'
process.queue = 'my-queue'

aws {
    region = 'us-east-1'
    profile = 'my-profile'             // or any credential source nf-amazon accepts, SSO included
}

cas {
    stores {
        lab { location = 's3://my-cas-bucket/cas' }
    }
    tmpDir = '/scratch/nf-blocks'      // optional; default java.io.tmpdir
}

Reading outputs in another pipeline

A second pipeline reads a run's outputs, or a Selection, back out of the store with fromStore. Its config names the producer's store as a read-only member beside its own writable one:

plugins {
    id 'nf-blocks@0.10.0-beta.1'
}

manifest.name = 'downstream'           // or cas.pipeline = 'downstream'

lineage.enabled = true
lineage.store.location = 'cas://mine'

cas {
    stores {
        mine { location = '/data/cas-downstream' }   // writable: this pipeline's own outputs
        lab  { location = '/data/cas' }              // the producer's store, read-only here
    }
    resolve = ['mine', 'lab']
}

fromStore comes from the plugin, so the script includes it:

include { fromStore } from 'plugin/nf-blocks'

workflow {
    main:
    picked  = channel.fromStore(selection: '<selection address>')
    aligned = channel.fromStore(run: 'latest', pipeline: 'cas-test-pipeline', output: 'aligned', where: [sample: 'B'])

    picked.mix(aligned).view()
}

Each item arrives in the shape the producer published it, here [meta, bam], with every file a cas:// path Nextflow stages like any other.

In a typed script (nextflow.enable.types = true), Nextflow 26.04.6 cannot reach a plugin factory through channel. Use nextflow.Channel instead, with records: true:

nextflow.enable.types = true

include { fromStore } from 'plugin/nf-blocks'

record Sample {
    sample: String
    lane: Integer
}

process COUNT_LINES {
    input:
    tuple(meta: Sample, bam: Path)

    output:
    stdout()

    script:
    """
    printf '%s\\t' '${meta.sample}'
    wc -l < '${bam}'
    """
}

workflow {
    COUNT_LINES(nextflow.Channel.fromStore(selection: '<selection address>', records: true))
        .view()
}

Building and testing the plugin

make assemble

Unit tests:

make test    # unit tests
make check   # unit tests plus the dependency check and the memory bound
make smoke   # builds, installs into a temp NXF_PLUGINS_DIR and runs a real pipeline

make smoke needs Nextflow 26.04.6 on the path; override the binary with NEXTFLOW=/path/to/nextflow and the pipeline with PIPELINE_SRC=....

License

Apache License 2.0. See the COPYING file for details.