DataCrumbs: eBPF-based I/O Profiling and Tracing for HPC Applications

DataCrumbs is the runtime side of the DataCrumbs stack. It loads signed probe descriptions, attaches generic BPF programs to runtime-selected functions, and writes DFTracer output.

The current split is:

• datacrumbs: runtime daemon, BPF programs, system configuration generation, and scheduler service integration

• datacrumbs-utils: probe builder, client library, wrappers, tests, and developer tooling

Getting Started

• Introduction
  • Overview
  • Key Features
  • Use Cases
  • Architecture
  • Target Applications
  • License
  • Citation
• eBPF Background
  • What is eBPF?
  • How eBPF Works
  • eBPF in DataCrumbs
  • Advantages of eBPF for I/O Tracing
  • eBPF Limitations
  • eBPF Tools and Ecosystem
  • Further Reading
• Dependencies
  • System Requirements
  • Build Dependencies
  • Core eBPF Dependencies
  • Additional Library Dependencies
  • Optional Dependencies
  • Python Dependencies (for Analysis)
  • Kernel Headers
  • Verifying Dependencies
  • Complete Dependency Installation
  • Troubleshooting

Build and Operation

• Building DataCrumbs
  • What Gets Built Where
  • Quick Start
  • Important Configure Options
  • Install-Time System Configuration
  • Login-node signing service
  • Dependencies
  • Notes
• Runtime Setup
  • Environment and Module Setup
  • Important Runtime Inputs
  • Direct Runtime Invocation
  • Trace and Run Files
  • Resource Limits
  • Service Logs
  • Runtime Probe Status Database
  • Probe signing service
• Using DataCrumbs
  • Quick Start
  • Direct Runtime Mode
  • Application Integration Modes
  • Systemd service mode
  • Scheduler metadata
• Application Integration
  • Supported Integration Paths
  • datacrumbs_wrap
  • datacrumbs_track
  • Scheduler-managed service flow
  • Multi-node service control
• Running DataCrumbs
  • Single-node direct run
  • Systemd-managed run
  • Testing and smoke runs
  • IOR example
  • Scheduler examples

Scheduler Integration

• Flux Integration
  • Current Flow
  • Installed components
  • Login-node prerequisite
  • Deploying the unit
  • Plugin behavior
  • Testing without a real Flux prolog
• SLURM Integration
  • Current Flow
  • Installed components
  • Login-node prerequisite
  • Deploying the unit
  • Notes
  • Testing without a real SLURM prolog

Legacy Notes

• Overview

Indices and tables

• Index

• Module Index

• Search Page