Summary

Nextflow's own -with-dag gives you the abstract process/channel graph that's known before anything runs. nf-bigbrother records the graph of what actually ran. It appends one JSON line per task to an event log as the run proceeds, and writes a full JSON snapshot plus a matching Graphviz .dot at the end, so the graph is available at every point during the run and not just when it finishes.

The final snapshot is a valid WfCommons WfFormat instance (schemaVersion 1.6):

• workflow.specification holds the tasks (id, name, parents/children, input/output files) and the files with their sizes.
• workflow.execution holds per-task runtime, CPU, I/O and memory counters, what each task requested of the scheduler, and the machines they ran on.

Parents and children come from the files: if task A writes a file that task B reads, A is a parent of B.

Get Started

Requires Nextflow 25.10 or newer.

In a pipeline you own, enable the plugin from its nextflow.config:

plugins {
    id 'nf-bigbrother@1.1.0'
}

bigbrother {
    outputDir    = 'bigbrother'   // defaults shown
    emitEvents   = true           // append a JSON line per task to events_*.jsonl
    emitPartials = false          // also write a whole-graph snapshot during the run
    snapshotEvery = 1             // ...every N tasks, when emitPartials is on
    emitDot      = true           // write a .dot beside every json snapshot
    prefix       = ''
}

For a pipeline you only run, an nf-core workflow say, enable the plugin with NXF_PLUGINS_DEFAULT rather than a plugins block:

NXF_PLUGINS_DEFAULT=nf-bigbrother@1.1.0 \
  nextflow run nf-core/demo -r 1.2.0 -profile test,docker --outdir results

Do not reach for -plugins nf-bigbrother@1.1.0, and do not put a plugins block in a -c config. Both replace the pipeline's own plugins declaration rather than adding to it, which silently unpins whatever the pipeline depends on. If you do need the plugin declared in config, restate the pipeline's own plugins alongside it in a single block:

plugins {
    id 'nf-schema@2.7.2'      // whatever the pipeline already pinned
    id 'nf-bigbrother@1.1.0'
}

A run then fills outputDir with events_<name>_<uuid>.jsonl (appended to as it goes), a final complete_<name>_<uuid>.json / .dot, and an error_* snapshot instead if it dies partway.

Examples

Running any pipeline with the plugin enabled produces output including an event log and final JSON snapshot with a matching .dot file. The bundled Python tools turn either that or the event log into visualizations:

tools/bb_dag.py bigbrother/complete_*.json -f svg --metrics -o graph.svg
tools/bb_dag.py bigbrother/events_*.jsonl -f svg -o graph.svg   # same graph
tools/bb_dag.py --watch bigbrother -f svg -o live.svg   # re-render as a run goes

tools/bb_animate.py replays a run into an animation, one frame per task:

tools/bb_animate.py bigbrother -o dag-build.gif --fps 14

A small container-free pipeline under examples/local-pipeline produces a correct graph in a few seconds:

cd examples/local-pipeline
nextflow run main.nf
../../tools/bb_dag.py bb_out/complete_*.json -f png --metrics -o graph.png

The workflow used as the real reference is nf-core/rnaseq on the test profile:

NXF_PLUGINS_DEFAULT=nf-bigbrother@1.1.0 \
  nextflow run nf-core/rnaseq -r 3.21.0 -profile test,docker --outdir results

License

Apache License 2.0. See the COPYING file.

If you are publishing a paper and use this plugin, please consider citing the publication:

@article{kharma2026comprehensive,
  title={Comprehensive Plugin-Based Monitoring of Nexflow Workflow Executions},
  author={Kharma, Sami and Wies, Tobias and Schintke, Florian},
  journal={arXiv preprint arXiv:2603.28783},
  year={2026}
}

What's new

emitPartials defaulted to true up to 1.0.1, which wrote a partial_NNN_*.json after every task. The event log supersedes it, carrying the same information one line per task instead of a full rewrite, so it now defaults to false. Turn it back on if you have tooling that reads the partial files, and use snapshotEvery to thin them out on a large run.