Skip to main content

Scaling & Compliance Analysis for Postchi Batch Sending

This document analyzes the requirements for running Postchi at scale in production, managing multiple SMTP servers, handling rate limits, and ensuring compliance with email regulations.

Table of Contents​

  1. Rate Limiting & Traffic Management
  2. Compliance Requirements
  3. IP Warmup Strategy
  4. Implementation Priorities

Rate Limiting & Traffic Management​

Current State​

  • Queue Level: BullMQ limiter set to 100 jobs/second
  • Worker Level: Configurable concurrency via env.worker.concurrency
  • Limitation: No per-domain or per-IP rate limiting

Industry Rate Limits​

AWS SES​

  • Sandbox: 200 emails/day, 1 email/second
  • Production: 50,000 emails/day (default), customizable
  • Rate: 14-50 emails/second (account-dependent)
  • Quota: Based on recipients, not messages

SendGrid​

  • API: No rate limit on v3 mail/send endpoint
  • Request Rate: 600 requests/minute for v2+ APIs
  • IP Throttling: High volumes (250k-500k/day) from single IP cause deferrals

Mailgun​

  • New Accounts: 100 emails/hour (requires verification)
  • Per IP: Automatically adjusts based on ESP feedback
  • Warmup: Stage-based (e.g., 100 emails/hour for first 1,000)

Challenges at Scale​

1. Multiple SMTP Servers​

  • Different rate limits per server/IP
  • Need intelligent routing to available servers
  • Failover when one server hits limits

2. Domain-Level Limits​

  • Each domain may have different sending profiles
  • ISPs track reputation per domain + IP combination
  • Need to distribute sends across multiple IPs per domain

3. Recipient ISP Limits​

  • Gmail/Yahoo have per-sender hourly limits
  • Need to track sends per recipient domain (e.g., @gmail.com)
  • Implement backoff when ISP returns temporary failures (450 errors)

Proposed Solutions​

Solution 1: Per-Domain Rate Limiting​

// Add to Domain model
interface DomainSendingLimits {
  maxPerHour: number;
  maxPerDay: number;
  currentHourCount: number;
  currentDayCount: number;
  resetHourAt: Date;
  resetDayAt: Date;
}

Solution 2: IP Pool Management​

// Track multiple SMTP servers/IPs
interface SmtpServer {
  id: string;
  host: string;
  port: number;
  maxPerHour: number;
  currentHourCount: number;
  status: 'active' | 'warming' | 'cooldown' | 'blocked';
  warmupSchedule?: WarmupSchedule;
}

Solution 3: Dynamic Queue Routing​

  • Use BullMQ's group rate limiting feature
  • Create separate queues per SMTP server
  • Route jobs to least-loaded server
  • Implement circuit breaker pattern for failed servers

Solution 4: Recipient Domain Throttling​

// Track sends per recipient domain
interface RecipientDomainLimits {
  domain: string; // e.g., "gmail.com"
  maxPerHour: number;
  currentCount: number;
  lastResetAt: Date;
}

Compliance Requirements​

CAN-SPAM Act (United States)​

Mandatory Requirements​

  1. Unsubscribe Mechanism

    • Must honor opt-out within 10 business days
    • Clear and conspicuous explanation required
    • Cannot require login to unsubscribe

  2. Physical Address

    • Valid postal address in every email

  3. Accurate Headers

    • From/To/Reply-To must be accurate
    • Subject line cannot be deceptive

  4. Commercial Identification

    • Must identify message as advertisement

Penalties​

  • $43,792 per email in violation

Current Status​

  • ✅ Accurate headers (from, reply-to)
  • ❌ No unsubscribe mechanism
  • ❌ No physical address in templates
  • ❌ No commercial identification

GDPR (European Union)​

Mandatory Requirements​

  1. Explicit Consent

    • Must obtain before sending
    • Record of consent required
    • Clear explanation of data usage

  2. Right to Be Forgotten

    • Delete personal data on request
    • Process within 30 days

  3. Unsubscribe Processing

    • Process "without undue delay"
    • Must be as easy as subscribing

  4. Data Protection

    • Encryption of personal data
    • Audit logs of access

Penalties​

  • €20 million or 4% of global annual turnover (whichever is higher)

Current Status​

  • ✅ Suppression list (partial GDPR compliance)
  • ❌ No consent tracking
  • ❌ No audit logs
  • ❌ No one-click unsubscribe

Gmail/Yahoo Requirements (2024)​

For Senders >5,000 emails/day​

  1. List-Unsubscribe Header

    • Both mailto and HTTPS URLs required
    • One-click unsubscribe (RFC 8058)
    • List-Unsubscribe-Post header

  2. SPF/DKIM/DMARC

    • All three must be configured
    • DMARC policy required

  3. Spam Rate

    • Must stay below 0.3% spam complaint rate
    • Monitor via Google Postmaster Tools

Current Status​

  • ✅ DKIM signing
  • ✅ SPF/DMARC (domain-level)
  • ❌ No List-Unsubscribe header
  • ❌ No spam rate monitoring

List-Unsubscribe Header Implementation​

Required Headers​

List-Unsubscribe: <mailto:unsubscribe@postchi.io?subject=unsubscribe>, <https://postchi.io/unsubscribe?token=xyz>
List-Unsubscribe-Post: List-Unsubscribe=One-Click

Implementation Requirements​

  1. HTTP Endpoint: POST to unsubscribe URL
  2. Token Security: Unique, time-limited, signed tokens
  3. Immediate Processing: Add to suppression list instantly
  4. Confirmation Page: Thank you page with re-subscribe option
  5. Not for Transactional: Only marketing emails

IP Warmup Strategy​

Timeline​

  • Minimum: 4 weeks
  • Recommended: 6-8 weeks
  • Target Volume: 2-5 million emails/day per IP (after warmup)

Warmup Schedule​

Week 1-2: Highly Engaged (30-day active)​

  • Day 1-3: 5,000 emails/day per IP
  • Day 4-7: 10,000 emails/day per IP
  • Week 2: 25,000 emails/day per IP

Week 3-4: Moderately Engaged (60-day active)​

  • Week 3: 50,000 emails/day per IP
  • Week 4: 100,000 emails/day per IP

Week 5-6: Less Engaged (90-day active)​

  • Week 5: 250,000 emails/day per IP
  • Week 6: 500,000 emails/day per IP

Week 7-8: All Recipients​

  • Week 7: 1,000,000 emails/day per IP
  • Week 8: 2,000,000+ emails/day per IP (full capacity)

Warmup Best Practices​

  1. Send Daily: Consistent sending is critical
  2. Monitor Metrics: Watch bounce rate, complaint rate, engagement
  3. Separate IPs: Different IPs for transactional vs marketing
  4. Avoid Peak Seasons: Don't start warmup Nov-Dec
  5. Maintain Activity: Sending gap >30 days resets reputation

Database Schema for IP Warmup​

interface SmtpServerWarmup {
  serverId: string;
  status: 'warming' | 'complete';
  startDate: Date;
  currentDay: number;
  dailyLimit: number;
  sentToday: number;
  targetSegment: 'engaged_30' | 'engaged_60' | 'engaged_90' | 'all';
  metrics: {
    bounceRate: number;
    complaintRate: number;
    openRate: number;
  };
}

Implementation Priorities​

Phase 1: Critical Compliance (MUST HAVE for Production)​

1.1 Unsubscribe Mechanism​

Priority: 🔴 CRITICAL Effort: Medium Components:

  • Add List-Unsubscribe and List-Unsubscribe-Post headers
  • Create unsubscribe endpoint (POST /v1/unsubscribe)
  • Generate secure, time-limited tokens
  • Auto-add to suppression list
  • Unsubscribe link in email footer

Database Changes:

-- Add unsubscribe token to messages
ALTER TABLE Message ADD COLUMN unsubscribeToken VARCHAR(255);
ALTER TABLE Message ADD INDEX idx_unsubscribe_token (unsubscribeToken);

-- Track unsubscribe events
CREATE TABLE UnsubscribeEvent (
  id VARCHAR(255) PRIMARY KEY,
  email VARCHAR(255) NOT NULL,
  messageId VARCHAR(255),
  method ENUM('link', 'one_click', 'reply'),
  unsubscribedAt TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
  INDEX idx_email (email)
);

1.2 Template Compliance Fields​

Priority: 🔴 CRITICAL Effort: Small Components:

  • Add physical address to Organization model
  • Add commercial identification toggle
  • Update email templates to include footer

Database Changes:

-- Add to Organization
ALTER TABLE Organization ADD COLUMN physicalAddress TEXT;
ALTER TABLE Organization ADD COLUMN companyName VARCHAR(255);

Priority: 🔴 CRITICAL (for GDPR) Effort: Medium Components:

  • Track when/how consent was obtained
  • Store consent proof (IP, timestamp, form data)
  • Allow withdrawal of consent

Database Changes:

CREATE TABLE ContactConsent (
  id VARCHAR(255) PRIMARY KEY,
  contactId VARCHAR(255) NOT NULL,
  organizationId VARCHAR(255) NOT NULL,
  consentType ENUM('marketing', 'transactional'),
  granted BOOLEAN DEFAULT FALSE,
  grantedAt TIMESTAMP,
  withdrawnAt TIMESTAMP,
  source VARCHAR(255), -- 'api', 'form', 'import'
  ipAddress VARCHAR(45),
  metadata JSON,
  FOREIGN KEY (contactId) REFERENCES Contact(id),
  INDEX idx_contact_org (contactId, organizationId)
);

Phase 2: Scale & Deliverability (SHOULD HAVE)​

2.1 Per-Domain Rate Limiting​

Priority: 🟡 HIGH Effort: Medium Components:

  • Add rate limit tracking to Domain model
  • Implement Redis-based counters (sliding window)
  • Delay jobs when limit reached

2.2 IP Pool Management​

Priority: 🟡 HIGH Effort: Large Components:

  • Multiple SMTP server configuration
  • Health checking and failover
  • Load balancing across IPs
  • Per-IP rate limiting

Database Changes:

CREATE TABLE SmtpServer (
  id VARCHAR(255) PRIMARY KEY,
  organizationId VARCHAR(255) NOT NULL,
  host VARCHAR(255) NOT NULL,
  port INT NOT NULL,
  username VARCHAR(255),
  password VARCHAR(255), -- encrypted
  maxPerHour INT DEFAULT 1000,
  maxPerDay INT DEFAULT 10000,
  status ENUM('active', 'warming', 'cooldown', 'blocked'),
  createdAt TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
  INDEX idx_org_status (organizationId, status)
);

CREATE TABLE SmtpServerMetrics (
  id VARCHAR(255) PRIMARY KEY,
  serverId VARCHAR(255) NOT NULL,
  hour TIMESTAMP NOT NULL,
  sentCount INT DEFAULT 0,
  bouncedCount INT DEFAULT 0,
  failedCount INT DEFAULT 0,
  FOREIGN KEY (serverId) REFERENCES SmtpServer(id),
  UNIQUE KEY idx_server_hour (serverId, hour)
);

2.3 Recipient Domain Throttling​

Priority: 🟡 HIGH Effort: Medium Components:

  • Detect recipient domain (Gmail, Yahoo, Outlook, etc.)
  • Apply per-domain limits
  • Implement exponential backoff on 450 errors

2.4 IP Warmup Automation​

Priority: 🟢 MEDIUM Effort: Large Components:

  • Warmup schedule management
  • Automatic daily limit adjustment
  • Segment-based targeting (engaged users first)
  • Metrics monitoring and alerts

Phase 3: Advanced Features (NICE TO HAVE)​

3.1 Spam Complaint Monitoring​

Priority: 🟢 MEDIUM Effort: Medium Components:

  • Integrate with Google Postmaster Tools
  • Yahoo Feedback Loop
  • Microsoft SNDS

3.2 Engagement-Based Throttling​

Priority: 🟢 LOW Effort: Large Components:

  • Slow down sends to low-engagement segments
  • Speed up for high-engagement segments
  • Machine learning for optimal send times

3.3 Multi-Region Support​

Priority: 🟢 LOW Effort: Large Components:

  • Multiple Redis instances
  • Geo-distributed workers
  • Regional SMTP servers

Immediate (This Sprint)​

  1. ✅ Batch sending feature (DONE)
  2. 🔴 Unsubscribe mechanism (headers + endpoint + UI)
  3. 🔴 Template compliance fields (physical address, company name)

Next Sprint​

  1. 🔴 Consent tracking system
  2. 🟡 Per-domain rate limiting
  3. 🟡 Basic IP pool management (multi-SMTP support)

Following Sprint​

  1. 🟡 Recipient domain throttling
  2. 🟡 IP warmup automation
  3. 🟢 Spam complaint monitoring

Technical Implementation Notes​

BullMQ Rate Limiting Patterns​

Global Queue Rate Limiting​

// Current implementation - applies to ALL workers
const worker = new Worker('email-sending', async (job) => {...}, {
  connection,
  concurrency: 10,
  limiter: {
    max: 100,      // 100 jobs
    duration: 1000 // per second
  }
});

Per-Group Rate Limiting (for IP pools)​

// Create groups per SMTP server
await emailQueue.add('send-email', data, {
  group: {
    id: `smtp-${serverId}`,
    rate: {
      max: 50,     // 50 emails
      duration: 60000 // per minute per server
    }
  }
});

Dynamic Rate Limiting (for ISP throttling)​

// In worker, when receiving 450 error from ISP
if (error.responseCode === 450) {
  await worker.rateLimit(60000); // Pause for 1 minute
  throw new Error('Rate limited by recipient ISP');
}

Unsubscribe Token Generation​

import crypto from 'crypto';

function generateUnsubscribeToken(messageId: string, email: string): string {
  const payload = JSON.stringify({
    messageId,
    email,
    expiresAt: Date.now() + (90 * 24 * 60 * 60 * 1000) // 90 days
  });

  const cipher = crypto.createCipheriv('aes-256-gcm', SECRET_KEY, IV);
  const encrypted = Buffer.concat([
    cipher.update(payload, 'utf8'),
    cipher.final()
  ]);

  const authTag = cipher.getAuthTag();
  return Buffer.concat([encrypted, authTag]).toString('base64url');
}

List-Unsubscribe Header Format​

// In email worker
const unsubscribeToken = generateUnsubscribeToken(data.messageId, recipient);
const unsubscribeUrl = `https://postchi.io/api/v1/unsubscribe?token=${unsubscribeToken}`;

mailOptions.headers = {
  ...mailOptions.headers,
  'List-Unsubscribe': `<mailto:unsubscribe@postchi.io?subject=${unsubscribeToken}>, <${unsubscribeUrl}>`,
  'List-Unsubscribe-Post': 'List-Unsubscribe=One-Click'
};

Cost Implications​

Dedicated IPs​

  • Cost: $5-50/month per IP (varies by provider)
  • Requirement: 100k+ emails/month to justify
  • Benefit: Full reputation control

Shared IPs​

  • Cost: Included in most plans
  • Limitation: Reputation shared with other senders
  • Risk: Can be affected by other senders' behavior

Redis for Rate Limiting​

  • Memory: ~1KB per tracked key
  • Estimate: 10,000 domains = 10MB
  • Cost: Negligible on most plans

Storage for Compliance​

  • Consent Records: ~1KB per contact
  • Estimate: 1M contacts = 1GB
  • Retention: GDPR requires 3+ years

Monitoring & Alerts​

Key Metrics to Track​

  1. Sending Metrics

    • Emails sent per hour/day per IP
    • Queue depth and processing rate
    • Failed sends by error code

  2. Deliverability Metrics

    • Bounce rate (target: <5%)
    • Complaint rate (target: <0.3%)
    • Open rate (benchmark: 15-25%)
    • Click rate (benchmark: 2-5%)

  3. Compliance Metrics

    • Unsubscribe rate (normal: 0.1-0.5%)
    • Time to process unsubscribe (target: <10 min)
    • Consent withdrawal requests

  4. Infrastructure Metrics

    • SMTP server health
    • Redis connection status
    • Queue worker errors
  • 🚨 Bounce rate >5% for any IP
  • 🚨 Complaint rate >0.3% for any IP
  • 🚨 SMTP server unreachable
  • ⚠️ Queue depth >10,000 jobs
  • ⚠️ Unsubscribe processing delayed >1 hour
  • ℹ️ Daily sending summary

References​