Summary

nf-blocks is a Nextflow plugin that publishes pipeline outputs into a content-addressed store and records their lineage there. It owns the cas URI scheme, allowing users to direct outputs to a content-addressed store (e.g., outputDir = 'cas://lab') instead of a traditional results/ directory. The plugin integrates with Nextflow's lineage system, storing DAG-CBOR lineage records alongside Nextflow's lid:// records in a SQLite-indexed store. It supports local and S3 storage backends, cloud executors like AWS Batch, and provides lineage tracking for pipeline outputs with optional index files.

Status: beta (0.3.0-beta.1) – Pin the version in your plugin configuration. Current support includes:

• Cloud storage (local directories and S3 locations)
• nf-core pipelines with workflow outputs
• AWS Batch execution with or without Fusion
• Server-side hashing via S3 to avoid head-node bottlenecks
• Output Index File recording and linking

Get Started

Enable the plugin and configure store aliases pointing to your content-addressed storage:

plugins {
    id 'nf-blocks@0.3.0-beta.1'
}

lineage.enabled = true
lineage.store.location = 'cas://lab'   // alias of the writable member
outputDir = 'cas://lab'                // optional: unset, it is the lineage alias

cas {
    stores {
        lab {
            location = '/data/cas'     // the writable member
        }
    }
    resolve = ['lab']                  // optional; default is every alias, writable first
    asserted_by = 'anonymous'          // optional opaque label; never the OS user name
}

Store aliases must match ^[a-z][a-z0-9_-]{0,31}$ and must not parse as a content address. If outputDir is unset, it defaults to the alias in lineage.store.location. Nextflow downloads the plugin from the Nextflow Registry automatically.

Important: When adding nf-blocks to another pipeline (e.g., an nf-core pipeline) via -c config file, a plugins { } block replaces rather than extends the pipeline's existing plugin list. Repeat the pipeline's pinned plugins alongside nf-blocks to avoid Nextflow auto-installing the latest versions.

Examples

Any pipeline with a workflow output block publishes into the store. Published paths are organized under the store's coordinate tree:

/data/cas/coords/aligned/A/A.bam
/data/cas/qc/A/A_qc/summary.txt
/data/cas/nf/<run hash>/aligned/A/A.bam/.data.json

Lineage records reference the content address rather than work directory paths:

{"kind":"FileOutput","spec":{"path":"cas://lab/aligned/A/A.bam", ...}}

Publishing to S3

Configure S3 as a writable member with nf-amazon's S3 client:

plugins {
    id 'nf-blocks@0.3.0-beta.1'
}

lineage.enabled = true
lineage.store.location = 'cas://lab'

workDir = 's3://my-work-bucket/work'
process.executor = 'awsbatch'
process.queue = 'my-queue'

aws {
    region = 'us-east-1'
    profile = 'my-profile'
}

cas {
    stores {
        lab { location = 's3://my-cas-bucket/cas' }
    }
    tmpDir = '/scratch/nf-blocks'      // optional; default java.io.tmpdir
}

Reading outputs in another pipeline

A downstream pipeline reads a run's outputs from another pipeline's store using fromStore:

include { fromStore } from 'plugin/nf-blocks'

workflow {
    main:
    picked  = channel.fromStore(selection: '<selection address>')
    aligned = channel.fromStore(run: 'latest', pipeline: 'cas-test-pipeline', output: 'aligned', where: [sample: 'B'])

    picked.mix(aligned).view()
}

Configuration for the downstream pipeline:

plugins {
    id 'nf-blocks@0.3.0-beta.1'
}

manifest.name = 'downstream'

lineage.enabled = true
lineage.store.location = 'cas://mine'

cas {
    stores {
        mine { location = '/data/cas-downstream' }   // writable: this pipeline's outputs
        lab  { location = '/data/cas' }              // the producer's store, read-only
    }
    resolve = ['mine', 'lab']
}

For typed scripts, use nextflow.Channel.fromStore() with records: true:

nextflow.enable.types = true

include { fromStore } from 'plugin/nf-blocks'

record Sample {
    sample: String
    lane: Integer
}

process COUNT_LINES {
    input:
    tuple(meta: Sample, bam: Path)

    output:
    stdout()

    script:
    """
    printf '%s\\t' '${meta.sample}'
    wc -l < '${bam}'
    """
}

workflow {
    COUNT_LINES(nextflow.Channel.fromStore(selection: '<selection address>', records: true))
        .view()
}

Curating Selections

Create and manage named Selections (curated groups of outputs) from the command line:

nextflow plugin nf-blocks:items aligned nested.kit=truseq --run latest --pipeline cas-test-pipeline
nextflow -q plugin nf-blocks:items aligned lane=2 --run lid://<run hash>,lid://<other run hash> --format selection \
  | nextflow -q plugin nf-blocks:put /dev/stdin --name lane-2

Or use the web explorer:

nextflow plugin nf-blocks:explore

The explorer serves at http://127.0.0.1:<port>/ (port auto-assigned or via --port). Selections can be exported as samplesheets from the web interface.

License

Apache License 2.0. See the COPYING file for details.