documenso-incident-runbook

Documenso Incident Runbook

Overview

Step-by-step procedures for responding to Documenso integration incidents.

Prerequisites

• Access to monitoring dashboards
• kubectl access (if using Kubernetes)
• Documenso dashboard access
• On-call escalation contacts

Incident Severity Levels

Quick Diagnostic Commands

# Check Documenso API status
curl -s https://status.documenso.com/api/v2/status.json | jq '.status'

# Test API connectivity
curl -v -H "Authorization: Bearer $DOCUMENSO_API_KEY" \
  https://app.documenso.com/api/v2/documents?perPage=1

# Check your service health
curl -s https://your-app.com/health | jq

# View recent error logs
kubectl logs -l app=signing-service --since=10m | grep -i error

# Check pod status
kubectl get pods -l app=signing-service

Incident Response Procedures

Procedure 1: Documenso API Unresponsive

Symptoms:

• Connection timeouts
• 5xx errors from Documenso
• Health checks failing

Steps:

1. Confirm the issue (2 min)

   # Check Documenso status page
   curl -s https://status.documenso.com/api/v2/status.json | jq
   
   # Direct API test
   curl -v --max-time 10 \
     -H "Authorization: Bearer $DOCUMENSO_API_KEY" \
     https://app.documenso.com/api/v2/documents?perPage=1
   

2. Enable graceful degradation (5 min)

   # If using feature flags
   curl -X POST https://your-feature-flags.com/api/flags \
     -d '{"flag": "documenso_enabled", "value": false}'
   
   # Or via kubectl
   kubectl set env deployment/signing-service DOCUMENSO_ENABLED=false
   

3. Notify stakeholders

   Subject: [P1] Documenso Integration Degraded
   
   Status: Documenso API is unresponsive
   Impact: Document signing temporarily unavailable
   Action: Graceful degradation enabled
   ETA: Monitoring for recovery
   

4. Monitor for recovery

   # Watch for recovery
   watch -n 30 'curl -s https://status.documenso.com/api/v2/status.json | jq .status'
   

5. Re-enable after recovery

   kubectl set env deployment/signing-service DOCUMENSO_ENABLED=true
   

Procedure 2: High Error Rate (4xx/5xx)

Symptoms:

• Error rate > 5%
• Mixed successful and failed requests
• Specific operations failing

Steps:

1. Identify error pattern (5 min)

   # Check recent errors
   kubectl logs -l app=signing-service --since=30m | \
     grep -E "statusCode.*[45][0-9]{2}" | \
     sort | uniq -c | sort -rn | head -20
   

2. Check specific error types

   # 401 Unauthorized - API key issue
   # 403 Forbidden - Permission issue
   # 404 Not Found - Resource deleted
   # 422 Validation - Bad request data
   # 429 Rate Limited - Too many requests
   # 500+ Server Error - Documenso issue
   

3. For 401/403 errors:

   # Verify API key
   echo $DOCUMENSO_API_KEY | head -c 10
   
   # Test key directly
   curl -H "Authorization: Bearer $DOCUMENSO_API_KEY" \
     https://app.documenso.com/api/v2/documents?perPage=1
   

4. For 429 errors:

   # Check request rate
   kubectl logs -l app=signing-service --since=5m | \
     grep "documenso" | wc -l
   
   # Reduce concurrency
   kubectl set env deployment/signing-service DOCUMENSO_CONCURRENCY=1
   

5. For 5xx errors:

   • Check Documenso status page
   • Enable circuit breaker
   • Implement retry with backoff

Procedure 3: Document Stuck in Pending

Symptoms:

• Documents not completing
• Recipients not receiving emails
• Webhooks not firing

Steps:

1. Check document status

   curl -H "Authorization: Bearer $DOCUMENSO_API_KEY" \
     https://app.documenso.com/api/v2/documents/{documentId}
   

2. Check recipient status

   # Look for recipient signing status
   curl -H "Authorization: Bearer $DOCUMENSO_API_KEY" \
     https://app.documenso.com/api/v2/documents/{documentId} | \
     jq '.recipients[] | {email, signingStatus}'
   

3. Verify webhook delivery

   • Check Documenso dashboard for webhook logs
   • Check your webhook endpoint logs
   • Test webhook endpoint manually

4. Manual intervention

   # Resend document
   curl -X POST \
     -H "Authorization: Bearer $DOCUMENSO_API_KEY" \
     https://app.documenso.com/api/v2/documents/{documentId}/resend
   

Procedure 4: Webhook Not Receiving Events

Symptoms:

• No webhook events received
• Events delayed significantly
• Partial event delivery

Steps:

1. Verify webhook configuration

   • Check Documenso dashboard webhook settings
   • Verify URL is correct and HTTPS
   • Confirm events are subscribed

2. Test webhook endpoint

   curl -X POST https://your-app.com/webhooks/documenso \
     -H "Content-Type: application/json" \
     -H "X-Documenso-Secret: your-secret" \
     -d '{"event":"test","payload":{}}'
   

3. Check webhook logs in Documenso

   • Dashboard -> Team Settings -> Webhooks
   • View delivery attempts and responses

4. Check your endpoint

   # Verify endpoint is responding
   curl -I https://your-app.com/webhooks/documenso
   
   # Check for errors in logs
   kubectl logs -l app=signing-service | grep webhook
   

5. Resend failed webhooks

   • Use Documenso dashboard to retry failed deliveries

Circuit Breaker Implementation

// Emergency circuit breaker
class DocumensoCircuitBreaker {
  private failures = 0;
  private lastFailure = 0;
  private state: "closed" | "open" | "half-open" = "closed";

  private readonly threshold = 5;
  private readonly timeout = 60000; // 1 minute

  async execute<T>(operation: () => Promise<T>): Promise<T> {
    if (this.state === "open") {
      if (Date.now() - this.lastFailure > this.timeout) {
        this.state = "half-open";
      } else {
        throw new Error("Circuit breaker is open");
      }
    }

    try {
      const result = await operation();
      this.onSuccess();
      return result;
    } catch (error) {
      this.onFailure();
      throw error;
    }
  }

  private onSuccess(): void {
    this.failures = 0;
    this.state = "closed";
  }

  private onFailure(): void {
    this.failures++;
    this.lastFailure = Date.now();
    if (this.failures >= this.threshold) {
      this.state = "open";
      console.error("Circuit breaker opened!");
    }
  }

  getState(): string {
    return this.state;
  }
}

Post-Incident Review Template

## Incident Report: [Title]

**Date:** YYYY-MM-DD
**Duration:** X hours Y minutes
**Severity:** P1/P2/P3/P4
**Author:** [Name]

### Summary
Brief description of what happened.

### Timeline
- HH:MM - Initial alert triggered
- HH:MM - Investigation started
- HH:MM - Root cause identified
- HH:MM - Mitigation applied
- HH:MM - Service restored

### Root Cause
Technical explanation of why the incident occurred.

### Impact
- X documents affected
- Y users impacted
- Z minutes of downtime

### Resolution
What was done to resolve the incident.

### Action Items
- [ ] Implement additional monitoring
- [ ] Update runbook
- [ ] Add circuit breaker

### Lessons Learned
What we learned from this incident.

Emergency Contacts

Output

• Incident diagnosed
• Mitigation applied
• Stakeholders notified
• Post-incident review scheduled

Resources

• Documenso Status
• Documenso Support
• [Internal Wiki - Signing Service]

Next Steps

For data handling procedures, see documenso-data-handling.