Robustmq Metrics

RobustMQ Metrics

Purpose

Add observability with minimal but complete coverage of a target pipeline:

• process count and process duration
• retry and terminal outcomes (if applicable)
• key failure points in read/process/write path
• liveness/health

Do not over-instrument. Prefer compact metrics that support operations and debugging.

Metric Design Rules

1. Cover chain, not everything:

   • success/failure + duration
   • retry/terminal path (when strategy exists)
   • read/commit/write failures
   • up/down

2. Always include count and latency for processing path.

3. Label policy:

   • required: stable low-cardinality dimensions only
   • optional only when bounded enum (for example result, strategy, protocol)
   • service/entity name labels are optional and must pass cardinality review
   • forbidden high-cardinality labels: topic, error_message, payload-derived labels

4. Naming:

   • use module prefix (for example mqtt_, raft_, handler_)
   • counters end with _total
   • duration histogram uses _ms
   • liveness gauge uses _up

Minimal Metric Set Template

Adapt this template to the target module:

• <module>_messages_processed_success_total{...}
• <module>_messages_processed_failure_total{...}
• <module>_process_duration_ms{...} (histogram)
• <module>_retry_total{...,strategy} (if retry exists)
• <module>_terminal_total{...,result} (discard/dlq/drop etc., if applicable)
• <module>_critical_step_failure_total{...} (read/write/commit/etc.)
• <module>_up{...} (gauge)

Implementation Workflow

1. Define metrics in src/common/metrics

   • register counters/histogram/gauge
   • add record helper APIs
   • keep API signatures consistent with chosen low-cardinality labels

2. Insert runtime instrumentation

   • add metrics at success, failure, retry, terminal, and liveness points

3. Fix call sites impacted by signature changes

   • update all metric helper users
   • ensure compile passes across dependent crates

4. Validation

   • run targeted cargo check for impacted crates
   • run lints for touched files

Grafana Decision Rule

After metrics are added, decide whether to update grafana/robustmq-broker.json:

• Update dashboard if metrics are operationally critical and stable.
• Skip dashboard only when user explicitly says not to update or metrics are temporary.

When updating dashboard:

• add/extend compact row
• prioritize low-cardinality dimensions and trend panels
• avoid panel explosion; 4-6 panels for first iteration

Output Format

Return:

1. Metrics added/changed (names + labels)
2. Instrumentation points (files/functions)
3. Whether Grafana was updated and why
4. Validation commands and results

Guardrails

• Do not add metrics without clear operational use.
• Do not introduce high-cardinality labels.
• Do not duplicate semantically equivalent metrics.
• Do not break existing metric names unless migration is requested.