Summary

nf-quilt is a Nextflow plugin developed by Quilt Data, Inc. that enables you to read and write directly to Quilt packages wherever your Nextflow pipeline currently uses s3 URIs. It works with any Amazon S3-compatible object store, as long as you have the appropriate credentials.

Quilt packages are versioned, immutable, and shareable data containers that store data, metadata, and documentation as a single atomic unit. They are accessible even by non-technical users via the Quilt Platform, a graphical web catalog which runs either in your private AWS cloud or on open.quiltdata.com.

Get Started

Basic Usage

To add the nf-quilt plugin to your Nextflow pipeline, specify it on the command line:

nextflow run nf-core/sarek -profile test,docker -plugins nf-quilt --outdir "s3://$WRITE_BUCKET/nf_quilt/sarek"

Configuration via nextflow.config

To avoid manually specifying the plugin, add it to your nextflow.config file:

plugins {
    id 'nf-quilt'
}

As of v0.9, you can also add a quilt section to configure metadata and plugin behaviors:

quilt {
    catalog = 'open.quiltdata.com'
    force = false  // replace (instead of update) existing packages
    meta = [pipeline: 'nf-core/sarek']
}

Examples

Writing to Quilt Packages

Transform an S3-based pipeline to write to Quilt packages by adding the plugin:

nextflow run nf-core/sarek -profile test,docker -plugins nf-quilt --outdir "s3://$WRITE_BUCKET/nf_quilt/sarek"

Using Quilt+ URIs for Output

Specify a Quilt+ URI as the --outdir parameter with optional metadata:

nextflow run nf-core/sarek -profile test,docker --outdir "quilt+s3://$WRITE_BUCKET?key=value#package=nf_quilt/sarek"

Using Quilt+ URIs for Input and Output

Use versioned Quilt+ URIs as input to ensure precise data versioning:

nextflow run main.nf --input "quilt+s3://$READ_BUCKET#package=nf-core/sarek@8a2164f48be8e0d6385f64b76b74a8543e9fb1b12a8eff6daeaffa653d52fcf7" --outdir "quilt+s3://$WRITE_BUCKET?key=value#package=test/my-sarek-processor"

Advanced Configuration with Template Strings

Customize commit messages and README files using template variables:

quilt {
    meta = [pipeline: 'nf-core/sarek']
    msg = "${meta['config']?.get('runName')}: ${meta['cmd']}"
    readme = '''
# ${pkg}

## ${now}

## Run Command

${cmd}

### Workflow

- **workflow run name**: ```${meta['workflow']?.get('runName')}```
- **scriptFile**: ```${meta['workflow']?.get('scriptFile')}```
- **sessionId**: ```${meta['workflow']?.get('sessionId')}```
- **start**: ```${meta['time_start']}```
- **complete**: ```${meta['time_complete']}```

### Nextflow

${nextflow}
    '''
}

License

Not explicitly specified in the provided documentation. Please refer to the nf-quilt GitHub repository for license information.