When a log stream loses its leader, OceanBase elects a new leader from followers with quorum intact. Confirm the alert is stream-scoped, verify OBProxy routing, inspect __all_virtual_log_stat, and only run ALTER SYSTEM SWITCH REPLICA LEADER if automatic election stalls.

Symptom

Monitoring reports no stable leader for one or more log streams, or applications see write errors scoped to specific shards.

Impact

Writes targeting affected log streams fail until a new leader is elected. Reads may fail if clients bypass OBProxy.

Preconditions

  • OCP or Grafana access to log stream metrics.
  • Documented OBProxy endpoint for application teams.
  • Read-only SSH to observer nodes for diagnostics.

Immediate safety checks

  1. Confirm whether the cluster still has quorum in each zone (read-only queries).
  2. Avoid destructive DDL until leader stability returns.
  3. Notify application owners if write error rate increases.

Diagnostic commands (read-only)

SELECT ls_id, role, zone, replay_lag FROM oceanbase.__all_virtual_log_stat;

Expected: one LEADER per active log stream; followers show FOLLOWER with bounded replay lag.

Decision points

ObservationAction
Automatic election completesMonitor; no manual switch
Election flappingEscalate; check network and disk
Quorum lossFollow major outage procedure; do not force writes

Remediation (modifying)

Only if automatic election stalled beyond your tested threshold:

ALTER SYSTEM SWITCH REPLICA LEADER ls_id = <ls_id> TO SERVER = '<observer_addr>';

Replace placeholders with values from your cluster. This changes leadership; run during an approved window.

Validation

  • Writes succeed through OBProxy without application restart.
  • __all_virtual_log_stat shows stable roles for 15+ minutes.
  • Application error rate returns to baseline.

Escalation

  • Split-brain symptoms (conflicting leader reports).
  • Quorum loss or minority partition.
  • Repeated elections on the same log stream within one hour.

Rollback considerations

If manual switch worsened lag, capture state and engage vendor support with Paxos logs. Do not delete replicas without backup verification.

Sources and further reading