Summary

nf-blocks is a Nextflow plugin that publishes pipeline outputs into a content-addressed store and records their lineage there. It owns the cas URI scheme, enabling outputs to be stored with outputDir = 'cas://lab' and lineage records to be maintained at lineage.store.location = 'cas://lab'. The plugin records DAG-CBOR lineage records alongside Nextflow's lid:// records and maintains a SQLite index. It supports reading outputs back with the fromStore channel factory and includes an explorer tool for browsing and managing Selections. Current status: beta (0.3.0-beta.2).

Get Started

Enable the plugin and configure the lineage store and output directory to point to the same store alias:

plugins {
    id 'nf-blocks@0.3.0-beta.2'
}

lineage.enabled = true
lineage.store.location = 'cas://lab'   // alias of the writable member
outputDir = 'cas://lab'                // optional: unset, it is the lineage alias

cas {
    stores {
        lab {
            location = '/data/cas'     // the writable member
        }
    }
    resolve = ['lab']                  // optional; default is every alias, writable first
    asserted_by = 'anonymous'          // optional opaque label; never the OS user name
}

An alias must match ^[a-z][a-z0-9_-]{0,31}$ and must not parse as a content address. When adding nf-blocks to someone else's pipeline via a -c config file, repeat the pipeline's pinned plugins alongside nf-blocks in the plugins { } block, as that block replaces rather than adds to the pipeline's plugin list.

Examples

Any pipeline with a workflow output block publishes into the store. Running a pipeline writes each published path under the store's coordinate tree:

/data/cas/coords/aligned/A/A.bam
/data/cas/coords/qc/A/A_qc/summary.txt
/data/cas/nf/<run hash>/aligned/A/A.bam/.data.json

The lineage record for a published file names its coordinate rather than a work directory path:

{"kind":"FileOutput","spec":{"path":"cas://lab/aligned/A/A.bam", ...}}

Reading outputs in another pipeline with fromStore:

include { fromStore } from 'plugin/nf-blocks'

workflow {
    main:
    picked  = channel.fromStore(selection: '<selection address>')
    aligned = channel.fromStore(run: 'latest', pipeline: 'cas-test-pipeline', output: 'aligned', where: [sample: 'B'])

    picked.mix(aligned).view()
}

Publishing to S3:

plugins {
    id 'nf-blocks@0.3.0-beta.2'
}

lineage.enabled = true
lineage.store.location = 'cas://lab'

workDir = 's3://my-work-bucket/work'
process.executor = 'awsbatch'
process.queue = 'my-queue'

aws {
    region = 'us-east-1'
    profile = 'my-profile'
}

cas {
    stores {
        lab { location = 's3://my-cas-bucket/cas' }
    }
    tmpDir = '/scratch/nf-blocks'
}

License

Apache License 2.0. See the COPYING file for details.

What's new

• Milestone 4 (cloud): any store member, including the writable one, can be a local directory or S3 location. Runs on AWS Batch with or without Fusion publish into writable S3 members with optimized hashing.
• Milestone 5 (nf-core pipelines): nf-blocks records lineage from workflow outputs only. Processes using publishDir are warned once; their files are stored but counted as unjoined rather than recorded against any run. Outputs declaring index { } record their Output Index File linked from the Output Collection.
• Support for reading outputs in downstream pipelines through fromStore with filtering by run, pipeline, output name, and custom conditions.
• Explorer tool (nf-blocks:explore) for browsing store members and managing Selections, with static file server support and snapshot generation.
• Command-line tools: nf-blocks:items for finding items and nf-blocks:put for building and writing Selection or Claim blocks from DAG-JSON.

Breaking changes

• On-disk format (block kinds, index, Store Log) may change before 1.0. Changes to blocks include migrations since a block's address is its content and existing stores retain their blocks.
• In typed scripts (nextflow.enable.types = true), Nextflow 26.04.6 cannot reach plugin channel factories through channel. until nextflow-io/nextflow#7694 is fixed. Use nextflow.Channel.fromStore(..., records: true) instead.
• nf-blocks requires the nf-amazon plugin (>=3.9.2). When installing from a local build, manually install nf-amazon alongside as Nextflow does not fetch dependencies automatically.