Summary

This plugin implements a Nextflow executor for Hashicorp Nomad, enabling Nextflow pipelines to submit and manage tasks on Nomad clusters. It supports distributed work directories via SPI, custom job naming, security defaults, and process-level Nomad options for fine-grained control over task execution.

Get Started

To use this plugin in your Nextflow pipeline, add the following to your nextflow.config file:

plugins {
    id 'nf-nomad'
}

process {
    executor = 'nomad'
}

nomad {
    client {
        address = 'http://your-nomad-server:4646'
    }
}

For local integration testing, the default Nomad address is http://localhost:4646. If you need a different endpoint, override it explicitly:

./gradlew test -PtestEnv=local -PnomadAddr=http://<host>:4646

Run unit tests with:

./gradlew check

Examples

The following example demonstrates how to configure a Nextflow pipeline to target a specific process to run on Nomad with customized datacenters and resource limits:

// nextflow.config
plugins {
  id 'nf-nomad'
}

process {
  executor = 'nomad'
  
  withName: runAlignment {
    nomadOptions = [
      datacenters: ['dc-analytics'],
      priority: 'high',
      resources: [
        cores: 8,
        memoryMax: '32 GB'
      ]
    ]
  }
}

Process-level Nomad options with advanced features:

process {
  withName: sayHello {
    nomadOptions = [
      datacenters: ['dc1', 'dc2'],
      namespace: 'bio',
      constraints: { node { unique = [name: params.RUN_IN_NODE] } },
      affinity: [attribute: '${meta.workload}', operator: '=', value: 'batch', weight: 25],
      meta: [owner: 'team-x', step: 'align'],
      shutdownDelay: '15s',
      volumes: [[type: 'host', name: 'ref-data', path: '/ref', readOnly: true]],
      secrets: ['MY_ACCESS_KEY', 'MY_SECRET_KEY'],
      resources: [memoryMax: '64 GB', cores: 4, device: [[name: 'nvidia/gpu', count: 1]]]
    ]
  }
}

License

This project is licensed under the Apache License 2.0 - see the LICENSE file for details.