Summary

Nextflow's own -with-dag gives you the abstract process/channel graph that's known before anything runs. nf-bigbrother records the graph of what actually ran. After each task it writes a JSON snapshot and a matching Graphviz .dot, so the graph is available at every point during the run, not just at the end.

Each JSON snapshot follows a WfCommons-style layout (schemaVersion 1.5):

• workflow.specification holds the tasks (id, name, parents/children, input/output files) and the files with their sizes.
• workflow.execution holds per-task runtime, CPU, I/O and memory counters plus the machines the tasks ran on.

Parents and children come from the files: if task A writes a file that task B reads, A is a parent of B.

Get Started

Requires Nextflow 25.10 or newer.

In a pipeline you own, enable the plugin from its nextflow.config:

plugins {
    id 'nf-bigbrother@1.0.1'
}

bigbrother {
    outputDir    = 'bigbrother'   // defaults shown
    emitPartials = true           // write a snapshot after every task
    emitDot      = true           // also write a .dot next to each json
    prefix       = ''
}

Observing a Pipeline You Don't Own

For a pipeline you only run — an nf-core workflow, say — enable the plugin with NXF_PLUGINS_DEFAULT rather than a plugins block:

NXF_PLUGINS_DEFAULT=nf-bigbrother@1.0.1 \
  nextflow run nf-core/demo -r 1.2.0 -profile test,docker --outdir results

Do not reach for -plugins nf-bigbrother@1.0.1, and do not put a plugins block in a -c config. Both replace the pipeline's own plugins declaration rather than adding to it, which silently unpins whatever the pipeline depends on.

If you do need the plugin declared in config, restate the pipeline's own plugins alongside it in a single block:

plugins {
    id 'nf-schema@2.7.2'      // whatever the pipeline already pinned
    id 'nf-bigbrother@1.0.1'
}

The bigbrother settings work the same way in a -c config file; only the plugins block has this replace-not-merge behaviour.

A run fills outputDir with partial_000_<name>_<uuid>.json / .dot (one per completed task), a final complete_<name>_<uuid>.*, and an error_* snapshot instead if it dies partway.

Examples

Running any pipeline with the plugin enabled produces the snapshots described above. Every snapshot already has a plain .dot beside it; the bundled Python tools turn them into something nicer.

tools/bb_dag.py renders a snapshot (grouping by process, colour, node metrics). It uses only the standard library, plus the dot binary for images:

tools/bb_dag.py bigbrother/complete_*.json -f svg --metrics -o graph.svg
tools/bb_dag.py --watch bigbrother -f svg -o live.svg   # re-render as a run goes

tools/bb_animate.py turns the whole sequence of snapshots into an animation:

tools/bb_animate.py bigbrother -o dag-build.gif --fps 14

There is a small container-free pipeline under examples/local-pipeline that produces a correct graph in a few seconds:

cd examples/local-pipeline
nextflow run main.nf
../../tools/bb_dag.py bb_out/complete_*.json -f png --metrics -o graph.png

The workflow used as the real reference is nf-core/rnaseq on the test profile:

NXF_PLUGINS_DEFAULT=nf-bigbrother@1.0.1 \
  nextflow run nf-core/rnaseq -r 3.21.0 -profile test,docker --outdir results

License

Apache License 2.0. See the COPYING file.

If you are publishing a paper and use this plugin, please consider citing the publication:

@article{kharma2026comprehensive,
  title={Comprehensive Plugin-Based Monitoring of Nexflow Workflow Executions},
  author={Kharma, Sami and Wies, Tobias and Schintke, Florian},
  journal={arXiv preprint arXiv:2603.28783},
  year={2026}
}