Running DataCrumbs

This page shows the supported runtime flow after probe generation.

Single-node direct run

Start DataCrumbs directly:

sudo datacrumbs /path/to/probes.json.gz 1

Then run an application through the client library:

datacrumbs_wrap dd if=/dev/zero of=/tmp/out.bin bs=1M count=16 status=none

Stop the runtime with Ctrl-C or:

kill -INT <pid>

Systemd-managed run

When using the installed datacrumbs@.service, first write the per-run environment file through the scheduler service scripts or the test wrapper, then start the unit:

systemctl start datacrumbs@1.service

Logs for the service go to:

${DATACRUMBS_LOG_DIR}/datacrumbs_${DATACRUMBS_USER}_${DATACRUMBS_SERVICE_RUN_ID}_$(hostname).log

Use standard systemd tools for debugging:

systemctl status datacrumbs@1.service
journalctl -u datacrumbs@1.service

Testing and smoke runs

The active utility smoke test in datacrumbs-utils runs dd through datacrumbs_wrap.

Example:

ctest --test-dir <utils-build-dir> --output-on-failure -R datacrumbs_utils_dd_preload

IOR example

Generate a probes file that includes the workload you care about, start the runtime, and then launch IOR through the client library.

sudo datacrumbs /tmp/ior-probes.json.gz ior-test
datacrumbs_wrap ior -a POSIX -b 1m -t 256k -s 16 -F

Scheduler examples

Flux:

flux run --datacrumbs-enable --datacrumbs-probe-file /tmp/probes.json.gz -N 2 -n 4 ./myapp

SLURM interactive allocation:

datacrumbs_salloc --datacrumbs-enable --datacrumbs-probe-file /tmp/probes.json.gz -N 2

SLURM batch:

datacrumbs_sbatch --datacrumbs-enable --datacrumbs-probe-file /tmp/probes.json.gz job.sh