Summary

nf-blocks is a Nextflow plugin that publishes pipeline outputs into a content-addressed store and records their lineage there. It owns the cas URI scheme, allowing users to configure outputDir = 'cas://lab' to send every published file through the plugin instead of onto a traditional results/ directory structure. The plugin provides a lineage store for lineage.store.location = 'cas://lab', so Nextflow's lid:// records live in the same store. It writes DAG-CBOR lineage records into the store alongside Nextflow's own records and maintains a SQLite index of them. Additional pipelines can read outputs back using the fromStore channel factory, and the nf-blocks:explore command browses outputs and manages Selections. The plugin is currently in beta (0.9.0-beta.1) and is built and tested against Nextflow 26.04.6.

Get Started

Enable the plugin and point both the lineage store and the output directory at the same store alias:

plugins {
    id 'nf-blocks@0.9.0-beta.1'
}

lineage.enabled = true
lineage.store.location = 'cas://lab'   // alias of the writable member
outputDir = 'cas://lab'                // optional: unset, it is the lineage alias

cas {
    stores {
        lab {
            location = '/data/cas'     // the writable member
        }
    }
    resolve = ['lab']                  // optional; default is every alias, writable first
    asserted_by = 'anonymous'          // optional opaque label; never the OS user name
}

An alias must match the pattern ^[a-z][a-z0-9_-]{0,31}$ and must not parse as a content address. outputDir may be left unset, in which case it defaults to the alias in lineage.store.location. When set, it must name that alias, or the run will abort. Nextflow downloads the plugin from the Nextflow Registry the first time the pipeline runs.

When adding nf-blocks to someone else's pipeline using a -c config file, note that a plugins { } block in the config file replaces the pipeline's own plugin list rather than adding to it. Repeat the pipeline's pinned plugins alongside nf-blocks, or Nextflow will install the latest release of each plugin the pipeline includes, which may cause compatibility issues.

Examples

Any pipeline with a workflow output block publishes into the store. Running a configured pipeline writes each published path under the store's coordinate tree:

/data/cas/coords/aligned/A/A.bam
/data/cas/coords/qc/A/A_qc/summary.txt
/data/cas/nf/<run hash>/aligned/A/A.bam/.data.json

The lineage record for a published file names its coordinate rather than a work directory path:

{"kind":"FileOutput","spec":{"path":"cas://lab/aligned/A/A.bam", ...}}

To publish into S3 with a cloud executor:

plugins {
    id 'nf-blocks@0.9.0-beta.1'
}

lineage.enabled = true
lineage.store.location = 'cas://lab'

workDir = 's3://my-work-bucket/work'
process.executor = 'awsbatch'
process.queue = 'my-queue'

aws {
    region = 'us-east-1'
    profile = 'my-profile'
}

cas {
    stores {
        lab { location = 's3://my-cas-bucket/cas' }
    }
    tmpDir = '/scratch/nf-blocks'
}

To read a run's outputs in another pipeline:

plugins {
    id 'nf-blocks@0.9.0-beta.1'
}

manifest.name = 'downstream'

lineage.enabled = true
lineage.store.location = 'cas://mine'

cas {
    stores {
        mine { location = '/data/cas-downstream' }
        lab  { location = '/data/cas' }
    }
    resolve = ['mine', 'lab']
}

And in the workflow script:

include { fromStore } from 'plugin/nf-blocks'

workflow {
    main:
    picked  = channel.fromStore(selection: '<selection address>')
    aligned = channel.fromStore(run: 'latest', pipeline: 'cas-test-pipeline', output: 'aligned', where: [sample: 'B'])

    picked.mix(aligned).view()
}

For typed scripts, use:

nextflow.enable.types = true

include { fromStore } from 'plugin/nf-blocks'

workflow {
    COUNT_LINES(nextflow.Channel.fromStore(selection: '<selection address>', records: true))
        .view()
}

To build and test the plugin locally:

make assemble
make test
make smoke

To publish a Selection from the command line:

nextflow plugin nf-blocks:items aligned nested.kit=truseq --run latest --pipeline cas-test-pipeline
nextflow -q plugin nf-blocks:items aligned lane=2 --run lid://<run hash> --format selection \
  | nextflow -q plugin nf-blocks:put /dev/stdin --name lane-2

To manage retention and perform sweeps:

nextflow plugin nf-blocks:prune --keep-last 3
nextflow plugin nf-blocks:prune --keep-last 3 --apply true

nextflow plugin nf-blocks:sweep
nextflow plugin nf-blocks:sweep --apply true --budget '10 GB'

License

Apache License 2.0. See the COPYING file for details.

What's new

This is a beta release (0.9.0-beta.1). Pin the version in plugins { }, as changes to the on-disk format (block kinds, the index, the Store Log) may still occur before 1.0. A change to blocks will come with a migration, since a block's address is its content and existing stores keep their blocks.

Supported milestones:

• Milestone 4 (cloud): Any store member, including the writable one, can be a local directory or an S3 location. Runs on AWS Batch with or without Fusion publish into a writable S3 member. Under Fusion, the task node hashes its own outputs; between S3 buckets, S3 computes the SHA-256 of a server-side copy.
• Milestone 5 (nf-core pipelines): Records lineage from workflow outputs only. Processes using publishDir are warned once; files are still stored but counted as unjoined.
• Milestone 6 (retention): Every run's content is a root by default and kept until released. A sweep only deletes what is released, unpinned, and past its grace period.

Key requirements:

• Requires the nf-amazon plugin (≥3.9.2)
• In typed scripts, call nextflow.Channel.fromStore(..., records: true) instead of channel.fromStore until nextflow-io/nextflow#7694 is fixed
• Warnings from the plugin go to .nextflow.log and the terminal