Summary

nf-blocks is a Nextflow plugin that publishes pipeline outputs into a content-addressed store and records their lineage there. It owns the cas URI scheme, allowing users to configure outputDir = 'cas://lab' to send published files through the plugin instead of onto a results/ tree. It provides a lineage store for lineage.store.location = 'cas://lab', so Nextflow's own lid:// records live in the same store. The plugin writes DAG-CBOR lineage records into the store beside Nextflow's own lid:// records and maintains a SQLite index. Another pipeline can read the outputs back with the fromStore channel factory, and nf-blocks:explore browses them and saves Selections. This is a beta release (0.4.0-beta.1) supporting cloud deployments on AWS Batch with or without Fusion, nf-core pipelines, and retention policies.

Get Started

Enable the plugin and point both the lineage store and the output directory at the same store alias:

plugins {
    id 'nf-blocks@0.4.0-beta.1'
}

lineage.enabled = true
lineage.store.location = 'cas://lab'   // alias of the writable member
outputDir = 'cas://lab'                // optional: unset, it is the lineage alias

cas {
    stores {
        lab {
            location = '/data/cas'     // the writable member
        }
    }
    resolve = ['lab']                  // optional; default is every alias, writable first
    asserted_by = 'anonymous'          // optional opaque label; never the OS user name
}

An alias matches ^[a-z][a-z0-9_-]{0,31}$ and must not parse as a content address. outputDir may be left out: unset, it means the alias in lineage.store.location. When set, it must name that alias, or the run aborts. Nextflow downloads the plugin from the Nextflow Registry the first time the pipeline runs.

To add nf-blocks to someone else's pipeline with a -c config file: a plugins { } block in that file replaces the pipeline's own plugin list rather than adding to it. Repeat the pipeline's pinned plugins beside nf-blocks, or Nextflow installs the latest release of each one.

Examples

Any pipeline with a workflow output block publishes into the store. With the configuration above, running the test pipeline writes each published path under the store's coordinate tree:

/data/cas/coords/aligned/A/A.bam
/data/cas/coords/qc/A/A_qc/summary.txt
/data/cas/nf/<run hash>/aligned/A/A.bam/.data.json

The lineage record for a published file names its coordinate rather than a work directory path:

{"kind":"FileOutput","spec":{"path":"cas://lab/aligned/A/A.bam", ...}}

A second pipeline reads a run's outputs back out of the store with fromStore. Its config names the producer's store as a read-only member beside its own writable one:

plugins {
    id 'nf-blocks@0.4.0-beta.1'
}

manifest.name = 'downstream'           // or cas.pipeline = 'downstream'

lineage.enabled = true
lineage.store.location = 'cas://mine'

cas {
    stores {
        mine { location = '/data/cas-downstream' }   // writable: this pipeline's own outputs
        lab  { location = '/data/cas' }              // the producer's store, read-only here
    }
    resolve = ['mine', 'lab']
}

fromStore comes from the plugin, so the script includes it:

include { fromStore } from 'plugin/nf-blocks'

workflow {
    main:
    picked  = channel.fromStore(selection: '<selection address>')
    aligned = channel.fromStore(run: 'latest', pipeline: 'cas-test-pipeline', output: 'aligned', where: [sample: 'B'])

    picked.mix(aligned).view()
}

For S3 publishing with AWS Batch:

plugins {
    id 'nf-blocks@0.4.0-beta.1'
}

lineage.enabled = true
lineage.store.location = 'cas://lab'

workDir = 's3://my-work-bucket/work'
process.executor = 'awsbatch'
process.queue = 'my-queue'

aws {
    region = 'us-east-1'
    profile = 'my-profile'             // or any credential source nf-amazon accepts, SSO included
}

cas {
    stores {
        lab { location = 's3://my-cas-bucket/cas' }
    }
    tmpDir = '/scratch/nf-blocks'      // optional; default java.io.tmpdir
}

License

Apache License 2.0. See the COPYING file for details.