Summary

The xpack-google plugin enhances the standard Nextflow Google Cloud executors with the following features:

• Google Filestore (NFS) Support: Mount NFS volumes from Google Filestore directly in your Batch jobs, enabling shared file system workflows
• POSIX Shared File Systems: Use local POSIX-compatible paths as work directories when backed by NFS mounts
• Google Cloud Batch Support: Extended capabilities for the Google Batch executor with improved volume management

The plugin requires a license key to be used. If you are interested, please contact us for an evaluation license at sales@seqera.io.

Get Started

Prerequisites

• Java 11 or later
• Nextflow 23.01.0-edge or later (see compatibility table below)
• The nf-google plugin version 1.22.0 or later
• Google Cloud credentials configured with appropriate permissions
• Google Cloud Batch API enabled
• Google Filestore instance

License Configuration

Configure the Nextflow XPACK license by defining the NXF_XPACK_LICENSE variable in your environment:

export NXF_XPACK_LICENSE=<your license string>

Instance Preparation

1. Install Java and the required NFS system dependencies:

   sudo apt-get -y install openjdk-11-jdk-headless nfs-common
   

2. Configure Google credentials:

   export GOOGLE_APPLICATION_CREDENTIALS=$PWD/google.json
   

3. Mount the shared file system in the launching environment:

   sudo mkdir /nfs
   sudo mount 10.195.15.250:/share1 /nfs/
   sudo chmod go+rw /nfs
   

   Replace 10.195.15.250:/share1 with your Filestore instance mount point. See the Google Filestore documentation for details.

Installation

Add the plugin to your nextflow.config file:

plugins {
    id 'xpack-google@2.1.1'
}

Make sure to use a version matching your Nextflow version according to the compatibility table below.

Configuration for Google Cloud Batch

plugins {
    id 'xpack-google@2.1.1'
}

google.location = 'europe-west2'
google.batch.nfsVolumes.nfs1.target = '10.195.15.250:/share1'
google.batch.nfsVolumes.nfs1.mountPath = '/nfs'

process.executor = 'google-batch'

Replace the location, NFS target, and mountPath with values matching your Filestore instance configuration.

NFS Volume Options

Option	Description
google.batch.nfsVolumes..target	The NFS server address in format IP-ADDRESS:/SHARE-NAME (required)
google.batch.nfsVolumes..mountPath	The local path where the NFS share will be mounted (required)
google.batch.nfsVolumes..readOnly	Mount the file system as read-only (default: false)

Multiple Filestore instances can be configured by providing unique names for each volume (e.g., nfs1, nfs2).

Examples

Using Filestore as Work Directory

Configure your pipeline to use a Filestore NFS mount as the work directory:

plugins {
    id 'xpack-google@2.1.1'
}

workDir = '/nfs/work'

google {
    project = 'my-project'
    location = 'us-central1'
    batch {
        nfsVolumes {
            'pipeline-storage' {
                target = '10.128.0.2:/vol1'
                mountPath = '/nfs'
            }
        }
    }
}

process.executor = 'google-batch'

Multiple NFS Volumes

Mount multiple NFS volumes for different purposes: