Webhook Implementation Guide

Implement a production-ready webhook system with signature verification, event processing, idempotency, and retry logic for the following requirements:

Webhook Direction

Receiving Webhooks (Incoming)

Webhook Providers: [Stripe / GitHub / Shopify / Twilio / Custom / Multiple]
Expected Volume: [10 events/day / 1000 events/day / 100k events/day]
Event Types: [List expected event types]

Sending Webhooks (Outgoing)

Send To: [Customer endpoints / Partner systems / Internal services]
Expected Volume: [Events per day]
Event Types: [List event types to send]

Implementation Scope

Direction: [Receiving only / Sending only / Both]

Incoming Webhooks Configuration

Webhook Provider 1: [Name, e.g., Stripe]

Endpoint Path: [/webhooks/stripe / /api/webhooks/stripe]
HTTP Method: [POST / PUT / Custom]
Authentication Method: [Signature verification / API key / Basic auth / Custom]
Signature Header: [stripe-signature / x-hub-signature / Custom]
Signature Algorithm: [HMAC-SHA256 / HMAC-SHA512 / Custom]
Webhook Secret: [Environment variable name]
Payload Format: [JSON / Form-encoded / XML]
Event Types: [List all event types to handle]

Webhook Provider 2: [Name, e.g., GitHub]

Endpoint Path: [Path]
HTTP Method: [Method]
Authentication Method: [Method]
Signature Header: [Header name]
Signature Algorithm: [Algorithm]
Webhook Secret: [Secret variable]
Payload Format: [Format]
Event Types: [Event types]

[Define 1-5 webhook providers]

Event Processing

Event Handler 1: [EventType, e.g., payment.succeeded]

Provider: [Which webhook provider]
Event Type: [Exact event type string]
Processing Logic: [What to do with this event]
Database Operations: [Insert / Update / Delete / None]
External API Calls: [Which APIs to call]
Notifications: [Email / SMS / Push / None]
Priority: [High / Medium / Low]
Async Processing: [Process immediately / Queue for later]

Event Handler 2: [EventType, e.g., order.created]

Provider: [Provider]
Event Type: [Type]
Processing Logic: [Logic]
Database Operations: [Operations]
External API Calls: [APIs]
Notifications: [Notifications]
Priority: [Priority]
Async Processing: [Strategy]

[Define 5-20 event handlers]

Signature Verification

Verification Strategy

Verification Required: [Yes for all / Yes for specific providers / No]
Signature Location: [Header / Query param / Body]
Verification Timing: [Before parsing / After parsing]
Failed Verification Response: [400 / 401 / 403]

Provider-Specific Verification

Provider 1: [Name]

Algorithm: [HMAC-SHA256 / HMAC-SHA512 / RSA / Custom]
Secret Storage: [Environment variable / Secrets manager / Database]
Signature Format: [Hex / Base64 / Custom]
Timestamp Validation: [Required / Optional / None]
Tolerance Window: [5 minutes / 10 minutes / Custom]

Provider 2: [Name]

Algorithm: [Algorithm]
Secret Storage: [Storage method]
Signature Format: [Format]
Timestamp Validation: [Required/Optional]
Tolerance Window: [Window]

Idempotency

Duplicate Detection

Idempotency Key: [Event ID / Custom header / Combination]
Storage: [Redis / Database / In-memory / None]
TTL: [24 hours / 7 days / 30 days]
Duplicate Handling: [Return 200 / Return cached response / Custom]

Event Tracking

Track Processed Events: [Yes / No]
Storage Method: [Redis SET / Database table / Both]
Cleanup Strategy: [TTL-based / Periodic cleanup / Manual]
Event History: [Keep for auditing / Discard after processing]

Error Handling

Validation Errors

Invalid Signature: [400 error / Log and ignore / Alert]
Invalid Payload: [400 error / Log and ignore / Alert]
Unknown Event Type: [200 OK and log / 400 error / Custom]
Missing Required Fields: [400 error / Use defaults / Alert]

Processing Errors

Database Errors: [Retry / Return 500 / Queue for manual review]
External API Failures: [Retry / Skip / Queue]
Timeout Handling: [Max processing time: X seconds]
Partial Failures: [Rollback / Continue / Custom]

Error Responses

Success Response: [200 OK with {received: true} / 204 No Content]
Validation Error: [400 with error details / 400 generic]
Processing Error: [500 with details / 500 generic]
Rate Limit: [429 with retry-after header]

Outgoing Webhooks Configuration

Webhook Subscription Management

Subscription Storage: [Database / Config file / API]
Subscription Fields:
- URL (customer endpoint)
- Events (which events to send)
- Secret (for signature generation)
- Active status
- Retry configuration

Event 1: [EventName, e.g., order.created]

Trigger: [When this event occurs in your system]
Payload Schema: [JSON schema of webhook payload]
Required Fields: [List required fields]
Optional Fields: [List optional fields]
Subscribers: [Who receives this event]

Event 2: [EventName, e.g., payment.completed]

Trigger: [Trigger condition]
Payload Schema: [Schema]
Required Fields: [Fields]
Optional Fields: [Fields]
Subscribers: [Subscribers]

[Define 3-10 outgoing event types]

Sending Webhooks

Delivery Configuration

Signature Algorithm: [HMAC-SHA256 / HMAC-SHA512 / Custom]
Signature Header: [X-Webhook-Signature / X-Hub-Signature / Custom]
Timestamp Header: [X-Webhook-Timestamp / None]
Event ID Header: [X-Webhook-ID / X-Event-ID / None]
Retry Header: [X-Webhook-Retry-Count / None]

Retry Logic

Max Retries: [3 / 5 / 10 / Custom]
Retry Strategy: [Exponential backoff / Linear / Custom]
Backoff Base: [2 seconds / 5 seconds / Custom]
Max Backoff: [1 hour / 24 hours / Custom]
Retry On: [5xx errors / Timeouts / Connection errors / All failures]
Give Up After: [24 hours / 7 days / Never]

Failure Handling

Failed Delivery: [Queue for manual retry / Disable webhook / Alert / Custom]
Dead Letter Queue: [Store failed events / Discard / Custom]
Notification: [Email admin / Slack alert / Dashboard / None]
Webhook Deactivation: [After X failures / Manual only / Never]

Queue System (if async processing)

Queue Configuration

Queue System: [Bull / BullMQ / RabbitMQ / SQS / Redis / None]
Queue Strategy: [One queue per event type / Single queue / Priority queues]
Concurrency: [Process X events simultaneously]
Priority Levels: [High / Medium / Low / None]

Job Processing

Retry Policy: [Exponential backoff / Fixed delay / Custom]
Max Job Attempts: [3 / 5 / Unlimited]
Job Timeout: [30 seconds / 5 minutes / Custom]
Failed Job Handling: [Move to DLQ / Log / Alert]

Monitoring & Logging

Metrics to Track

Incoming Webhooks:
- Webhooks received per provider
- Signature verification failures
- Processing success/failure rate
- Processing duration
- Duplicate events received
Outgoing Webhooks:
- Webhooks sent per event type
- Delivery success/failure rate
- Retry attempts
- Average delivery time
- Failed deliveries

Logging

Log Level: [Debug / Info / Warn / Error]
Log Contents:
- Event ID
- Provider/Subscriber
- Event type
- Processing status
- Error details (if failed)
- Processing duration
PII Handling: [Redact sensitive data / Log everything / Custom]

Alerting

Critical Alerts:
- Signature verification failures spike
- Processing failures > X%
- Queue backlog > Y events
- Webhook endpoint down
Warning Alerts:
- High retry rate
- Slow processing
- Approaching rate limits

Security

Rate Limiting

Incoming Webhooks: [X requests per minute per provider]
Outgoing Webhooks: [Y requests per minute per subscriber]
Rate Limit Strategy: [Token bucket / Fixed window / Sliding window]
Rate Limit Response: [429 with retry-after / 503 / Custom]

IP Whitelisting

Incoming Webhooks: [Whitelist provider IPs / No restriction]
Provider IPs: [List known IP ranges for each provider]
IP Verification: [Before signature check / After / None]

Secret Management

Secret Storage: [Environment variables / AWS Secrets Manager / Vault / Custom]
Secret Rotation: [Manual / Automated / Not supported]
Secret Per Subscriber: [Yes / Shared secret / None]

Testing

Test Webhooks

Test Endpoint: [Separate /webhooks/test endpoint / Use production]
Test Events: [Sample payloads for each event type]
Signature Generation: [Tool to generate test signatures]

Validation

Payload Validation: [JSON schema / Zod / Joi / Custom]
Required Fields Check: [Strict / Lenient]
Type Checking: [Strict / Coerce types]

Code Generation Requirements

Generate a complete webhook system including:

Incoming Webhook Endpoints:
- Express/Fastify routes for each provider
- Raw body parsing for signature verification
- Signature verification middleware
- Event routing to handlers
- Error handling and responses
Signature Verification:
- Verification functions for each provider
- HMAC signature generation/comparison
- Timestamp validation
- Secret retrieval from secure storage
Event Handlers:
- Handler functions for each event type
- Database operations
- External API calls
- Error handling and logging
- Async processing (if queued)
Idempotency Layer:
- Event ID tracking in Redis/Database
- Duplicate detection logic
- TTL-based cleanup
- Cached response handling
Outgoing Webhook System:
- Webhook subscription management
- Event triggering logic
- Payload construction
- Signature generation
- HTTP client for delivery
Retry Mechanism:
- Exponential backoff implementation
- Retry queue (if using queue system)
- Failure tracking
- Dead letter queue
- Webhook deactivation logic
Queue Integration (if async):
- Queue setup (Bull/BullMQ/SQS)
- Job definitions for each event type
- Worker processes
- Job retry configuration
- Failed job handling
Monitoring & Logging:
- Structured logging for all events
- Metrics collection (Prometheus/StatsD)
- Grafana dashboards
- Alert rules
- Webhook health checks
Testing Suite:
- Unit tests for signature verification
- Integration tests for each event handler
- Mock webhook payloads
- Signature generation utilities
- End-to-end webhook flow tests
Documentation:
- Webhook endpoint documentation
- Event type catalog
- Signature verification guide
- Retry policy documentation
- Troubleshooting guide
- Webhook testing guide
Admin Interface (optional):
- View webhook logs
- Retry failed webhooks
- Manage subscriptions
- Test webhook delivery
- View metrics

Output production-ready webhook infrastructure following best practices with:

Cryptographically secure signature verification
Idempotent event processing
Comprehensive error handling
Exponential backoff retry logic
Dead letter queue for failed events
Rate limiting and IP whitelisting
Structured logging and monitoring
Async processing with queues
Secret rotation support
Testing utilities and mock data
Clear documentation
Admin tools for troubleshooting