Summary

nf-quilt is a Nextflow plugin developed by Quilt Data, Inc. that enables you to read and write directly to Quilt packages wherever your Nextflow pipeline currently uses S3 URIs. It works with any Amazon S3-compatible object store. Quilt packages are versioned, immutable, and shareable data containers that store data, metadata, and documentation as a single atomic unit, accessible via the Quilt Platform graphical web catalog.

Get Started

Basic Usage

To add the nf-quilt plugin to your Nextflow command:

nextflow run nf-core/sarek -profile test,docker -plugins nf-quilt --outdir "s3://$WRITE_BUCKET/nf_quilt/sarek"

Configuration

Add the plugin to your nextflow.config file to avoid manually specifying it each time:

plugins {
    id 'nf-quilt'
}

As of v0.9, you can also configure the plugin behavior:

quilt {
    catalog = 'open.quiltdata.com'
    force = false  // replace (instead of update) existing packages
    meta = [pipeline: 'nf-core/sarek']
}

Configuration Parameters

• catalog: DNS hostname of the Quilt catalog (default: None)
• force: completely replace the existing package, rather than updating it (default: false)
• meta: map of metadata to add to the package (default: {})
• message: commit message template when saving the package
• package: name of the package to read or write when using an S3 URI (default: first two path components)
• readme: template string for the package README_NF_QUILT.md file
• summarize: files to display on the package front page via quilt_summarize.json (default: *.md,*.html,*.?sv,*.pdf,igv.json,**/multiqc_report.html); use false to disable
• workflow: name of a Quilt workflow for metadata validation (default: None)

Examples

Writing to Quilt Packages

Transform an S3 output by adding the plugin to automatically create a formatted Quilt package:

nextflow run nf-core/sarek -profile test,docker -plugins nf-quilt --outdir "s3://$WRITE_BUCKET/nf_quilt/sarek"

Using Quilt+ URIs for Input and Output

Specify versioned input with precise data tracking:

nextflow run main.nf --input "quilt+s3://$READ_BUCKET#package=nf-core/sarek@8a2164f48be8e0d6385f64b76b74a8543e9fb1b12a8eff6daeaffa653d52fcf7" --outdir "quilt+s3://$WRITE_BUCKET?key=value#package=test/my-sarek-processor"

Advanced Configuration with Template Strings

Customize commit messages and README files:

quilt {
    meta = [pipeline: 'nf-core/sarek']
    msg = "${meta['config']?.get('runName')}: ${meta['cmd']}"
    readme = '''
# ${pkg}

## ${now}

## Run Command

${cmd}

### Workflow

- **workflow run name**: ```${meta['workflow']?.get('runName')}```
- **scriptFile**: ```${meta['workflow']?.get('scriptFile')}```
- **sessionId**: ```${meta['workflow']?.get('sessionId')}```
- **start**: ```${meta['time_start']}```
- **complete**: ```${meta['time_complete']}```

### Nextflow

${nextflow}

### Processes

`${meta['workflow']?.get('stats')?.getAt('processes')}`
    '''
}

License

The plugin license is not explicitly specified in the provided documentation. Please refer to the nf-quilt GitHub repository for license information.