nf-flyte
Provider:
Union AI
Claimed:
25 Sep 2026 21:29:14 (UTC)
Description:
nf-flyte (provider: unionai) · https://github.com/unionai/nf-flyte
FUNCTIONALITY
nf-flyte adds a `flyte` executor that runs each Nextflow task as an action on Flyte v2, an open-source workflow orchestration platform. A pipeline shows up in Flyte as one run with one action per Nextflow task, so it can be monitored, cached and aborted like any other Flyte workload. Pipelines can be launched two ways:
- From a Flyte task: a Nextflow pipeline becomes one step in a larger Flyte workflow, and its results feed directly into downstream analysis or ML steps.
- With a plain `nextflow run` from a laptop or CI: the plugin creates the Flyte run itself and prints its URL. The run ends with the pipeline's result, stopping Nextflow (Ctrl+C) aborts it, and a heartbeat ensures tasks are cleaned up if the Nextflow process disappears.
Process directives (container, cpus, memory, disk, accelerator, time) map to the Flyte action's image, resources and timeout.
KEY TECHNICAL CHOICES
- Nextflow keeps control. nf-flyte is only an executor. Retries through errorStrategy, including exit-code rules such as retrying with more memory, keep working, and so do -resume and all pipeline logic. Flyte-level retries are off for task actions, so the two systems never compete.
- Flyte stages the data. Task inputs are passed as Flyte file and directory inputs, and the task directory comes back as an output, all moved by Flyte's data layer. The task runs Nextflow's own standard wrapper script, rendered for the task container. So process images need only bash, with no cloud CLI or storage credentials in task containers, and the work directory can be on any object store Flyte supports.
- Accurate failure reporting. A failed task is reported through Flyte's error channel. It shows as failed in Flyte, is never cached, and returns its exit code, stdout and stderr to Nextflow like any other executor.
- Cross-run caching. Flyte caches task results under a key built from each task's script, container, environment and inputs. Inputs are identified by the task that produced them, or by content for files Nextflow downloaded from outside. Unlike Nextflow's task hash, the key doesn't include the session, so a task completed in one run is reused by later runs without pinning a work directory. This complements -resume.
- Lightweight integration. The plugin talks to Flyte's public API over HTTP/JSON and has no dependencies beyond Nextflow itself. It requires Nextflow 25.10.0 or later and has been tested on 25.10 and 26.04.
BENEFITS TO THE NEXTFLOW COMMUNITY
- A new home for Nextflow pipelines. Organizations that standardized on Flyte for data and ML workloads can run Nextflow and nf-core pipelines on their existing infrastructure, instead of building and operating a second platform. That lowers the cost of adopting Nextflow and brings nf-core to new users.
- nf-core pipelines, unmodified. Pipelines run as published, with their standard biocontainers images. We tested with nf-core/demo end to end.
- Nextflow inside wider workflows. Pipelines can be steps in larger workflows that mix genomics with downstream analysis and machine learning, with results passed between steps.
- Result reuse across runs at the task level, beyond a single session's -resume.
- Visibility: each task gets its own view in Flyte, with logs and status, and Nextflow's execution report, timeline and DAG can be rendered into the Flyte run's report.
LICENSE
nf-flyte is released under the Apache License, Version 2.0. See COPYING in the repository.
Project URL:
https://github.com/unionai/nf-flyte
Latest version:
-
Total downloads:
0 View trends
No Releases Yet
This plugin has been registered but no releases have been published yet.