Summary

This plugin implements a Nextflow executor for Hashicorp Nomad, enabling Nextflow pipelines to submit and manage tasks on Nomad clusters. It supports distributed work directories, custom job naming, security configurations, and comprehensive process-level Nomad options for fine-grained task placement and resource management.

Get Started

To use this plugin in your Nextflow pipeline, add the following to your nextflow.config file:

plugins {
    id 'nf-nomad'
}

process {
    executor = 'nomad'
}

nomad {
    client {
        address = 'http://your-nomad-server:4646'
    }
}

For testing environments, run unit tests with:

./gradlew check

Test environment selection supports different configurations:

./gradlew test                                    # unit tests only
./gradlew test -PtestEnv=mock                    # unit + mock tests
./gradlew test -PtestEnv=local                   # unit + local Nomad integration tests
./gradlew test -PtestEnv=oci                     # unit + OCI Nomad integration tests
./gradlew test -PtestEnv=local -PnomadAddr=http://<host>:4646  # custom endpoint

Examples

The following example demonstrates how to configure a Nextflow pipeline to target a specific process to run on Nomad with customized datacenters and resource limits:

// nextflow.config
plugins {
  id 'nf-nomad'
}

process {
  executor = 'nomad'
  
  withName: runAlignment {
    nomadOptions = [
      datacenters: ['dc-analytics'],
      priority: 'high',
      resources: [
        cores: 8,
        memoryMax: '32 GB'
      ]
    ]
  }
}

Advanced process-level Nomad options with comprehensive configuration:

process {
  withName: sayHello {
    nomadOptions = [
      datacenters: ['dc1', 'dc2'],
      namespace: 'bio',
      constraints: { node { unique = [name: params.RUN_IN_NODE] } },
      affinity: [attribute: '${meta.workload}', operator: '=', value: 'batch', weight: 25],
      meta: [owner: 'team-x', step: 'align'],
      shutdownDelay: '15s',
      failures: [
        restart: [attempts: 1, delay: '5s', mode: 'fail'],
        reschedule: [attempts: 2, delay: '10s']
      ],
      volumes: [[type: 'host', name: 'ref-data', path: '/ref', readOnly: true]],
      secretsPath: 'secret/projects/team-x',
      secrets: ['MY_ACCESS_KEY', 'MY_SECRET_KEY'],
      spread: [name: 'node.datacenter', weight: 50, targets: ['us-east1': 70, 'us-east2': 30]],
      priority: 'high',
      resources: [memoryMax: '64 GB', cores: 4, device: [[name: 'nvidia/gpu', count: 1]]]
    ]
  }
}

To run end-to-end tests with nf-test:

./gradlew clean installPlugin -Pversion=99.99.99
cd docker && sudo ./start.sh
source /mtp/nomad/nomad_temp/.env
cd ../src/e2e && NXF_ASSETS=/tmp/nomad/nomad_temp/scratchdir/assets ./nf-test test

License

This project is licensed under the Apache License 2.0 - see the LICENSE file for details.