Summary

The xpack-amzn plugin enhances the standard Nextflow AWS Batch executor with the following features:

• Amazon EFS Support: Mount Amazon Elastic File System (EFS) volumes directly in your Batch jobs
• POSIX Shared File Systems: Support for FSx for Lustre, Qumulo, Weka, and other POSIX-compatible shared file systems as work directories
• AWS Secrets Manager Integration: Native support for Nextflow secrets backed by AWS Secrets Manager, enabling secure credential management in your pipelines

The plugin requires a license key to be used. If you are interested, please contact us for an evaluation license at sales@seqera.io.

Get Started

Prerequisites

• Nextflow 24.01.0-edge or later (see compatibility table below)
• The nf-amazon plugin version 3.4.0 or later
• AWS credentials configured with appropriate permissions

License Configuration

Configure the Nextflow XPACK license by defining the NXF_XPACK_LICENSE variable in your environment:

export NXF_XPACK_LICENSE=<your license string>

Installation

Add the plugin to your nextflow.config file:

plugins {
    id 'xpack-amzn@2.1.2'
}

Make sure to use a version matching your Nextflow version according to the compatibility table below.

Configuration

The plugin automatically enhances the awsbatch executor. Configure it using the standard AWS Batch settings:

process {
    executor = 'awsbatch'
    queue = 'my-batch-queue'
}

aws {
    region = 'us-east-1'
    batch {
        cliPath = '/home/ec2-user/miniconda/bin/aws'
    }
}

Amazon EFS Configuration

To mount EFS volumes in your Batch jobs, add the following to your configuration:

aws.batch.efsVolumes.'fs-1234567890'.mountPath = '/mnt/efs'

Replace fs-1234567890 with the ID of your EFS instance. Repeat this configuration for each EFS instance you want to use.

Available options:

Option	Description	Default
aws.batch.efsVolumes.''.mountPath	The host path where the file system will be mounted	(required)
aws.batch.efsVolumes.''.rootPath	The file system directory to expose through the mount point	/
aws.batch.efsVolumes.''.readOnly	Mount the file system as read-only	false

Examples

Using EFS as Work Directory

Configure your pipeline to use an EFS mount as the work directory:

workDir = '/mnt/efs/work'

aws.batch.efsVolumes.'fs-0123456789abcdef0'.mountPath = '/mnt/efs'

Using FSx for Lustre

For high-performance computing workloads with FSx for Lustre or other POSIX shared file systems:

workDir = '/fsx/work'

aws {
    batch {
        volumes = ['/fsx:/fsx']
    }
}

The provisioning and mounting of such file systems in the launch template is your responsibility. See the Nextflow documentation for more details on volume mounts.

Using AWS Secrets Manager

Store secrets in AWS Secrets Manager and reference them in your pipeline:

process SECRET_PROCESS {
    secret 'MY_API_KEY'

    script:
    """
    echo "Using secret: \$MY_API_KEY"
    """
}

License

This is proprietary software provided by Seqera Labs. A valid license key is required for use. Unauthorized copying, modification