Summary

This plugin implements a Nextflow executor for Hashicorp Nomad, enabling Nextflow pipelines to submit and manage tasks on Nomad clusters. It provides comprehensive support for Nomad job configuration, resource management, distributed work directories, and advanced scheduling options.

Get Started

To use this plugin in your Nextflow pipeline, add the following to your nextflow.config file:

plugins {
    id 'nf-nomad'
}

process {
    executor = 'nomad'
}

nomad {
    client {
        address = 'http://your-nomad-server:4646'
    }
}

For local integration tests, the default Nomad address is http://localhost:4646. If you need a different endpoint, override explicitly:

./gradlew test -PtestEnv=local -PnomadAddr=http://<host>:4646

To run and test the plugin in a development environment:

1. Clone the Nextflow repository:

   git clone --depth 1 https://github.com/nextflow-io/nextflow _resources/nextflow
   

2. Generate the nextflow class path:

   cd _resources/nextflow && ./gradlew exportClasspath
   

3. Compile the plugin alongside the Nextflow code:

   cd ../../ && ./gradlew compileGroovy
   

4. Run Nextflow with the plugin:

   ./launch.sh run main.nf -plugins nf-nomad
   

Examples

The following example demonstrates how to configure a Nextflow pipeline to target a specific process to run on Nomad with customized datacenters and resource limits:

// nextflow.config
plugins {
  id 'nf-nomad'
}

process {
  executor = 'nomad'
  
  withName: runAlignment {
    nomadOptions = [
      datacenters: ['dc-analytics'],
      priority: 'high',
      resources: [
        cores: 8,
        memoryMax: '32 GB'
      ]
    ]
  }
}

Advanced process-level Nomad options with comprehensive configuration:

process {
  withName: sayHello {
    nomadOptions = [
      datacenters: ['dc1', 'dc2'],
      namespace: 'bio',
      constraints: { node { unique = [name: params.RUN_IN_NODE] } },
      affinity: [attribute: '${meta.workload}', operator: '=', value: 'batch', weight: 25],
      meta: [owner: 'team-x', step: 'align'],
      shutdownDelay: '15s',
      failures: [
        restart: [attempts: 1, delay: '5s', mode: 'fail'],
        reschedule: [attempts: 2, delay: '10s']
      ],
      volumes: [[type: 'host', name: 'ref-data', path: '/ref', readOnly: true]],
      secretsPath: 'secret/projects/team-x',
      secrets: ['MY_ACCESS_KEY', 'MY_SECRET_KEY'],
      spread: [name: 'node.datacenter', weight: 50, targets: ['us-east1': 70, 'us-east2': 30]],
      priority: 'high',
      resources: [memoryMax: '64 GB', cores: 4, device: [[name: 'nvidia/gpu', count: 1]]]
    ]
  }
}

To opt-in to privileged Docker jobs:

nomad {
  jobs {
    privileged = true
  }
}

License

This project is licensed under the Apache License 2.0 - see the LICENSE file for details.