Redis for application developers: caching, queues, and pub/sub.

Redis is one of those tools where the surface area feels small — it's a key-value store — until you start discovering what you can do with its data structures. Lists become job queues. Sorted sets become leaderboards and rate limiters. Pub/sub becomes a lightweight event bus. Streams become durable message logs. This post covers the patterns that show up most often in real application development, with the configuration decisions that actually matter in production.

Connection management: don't open a new connection per request

The most common Redis mistake is creating a new connection for every operation. Redis connections are cheap compared to PostgreSQL, but still not free — at scale, connection overhead adds up fast. Use a connection pool:

// Node.js with ioredis
import Redis from 'ioredis';

const redis = new Redis({
  host:           process.env.REDIS_HOST || 'localhost',
  port:           6379,
  password:       process.env.REDIS_PASSWORD,
  db:             0,
  maxRetriesPerRequest: 3,
  retryStrategy: (times) => Math.min(times * 100, 2000), // Exponential backoff, max 2s
  enableOfflineQueue: true,   // Queue commands while reconnecting
  lazyConnect: false,         // Connect on startup, fail fast if Redis is down
});

redis.on('error', (err) => console.error('Redis error:', err));
redis.on('connect', () => console.log('Redis connected'));

export default redis;

For multi-threaded runtimes (Go, Java, Python) use a proper connection pool. ioredis handles pooling internally for Node.js. The key parameter is maxRetriesPerRequest — set this to a small number so failed commands fail fast rather than hanging indefinitely when Redis is briefly unavailable.

Caching: the patterns that actually work

Cache-aside (lazy loading)

The most common pattern: check cache first, fall back to database, populate cache on miss:

async function getUserById(id: string): Promise {
  const cacheKey = `user:${id}`;
  
  // Try cache first
  const cached = await redis.get(cacheKey);
  if (cached) return JSON.parse(cached);
  
  // Cache miss — fetch from database
  const user = await db.users.findById(id);
  if (!user) throw new NotFoundError(`User ${id} not found`);
  
  // Populate cache with TTL
  // Add jitter to prevent cache stampede (all keys expiring at the same second)
  const ttl = 3600 + Math.floor(Math.random() * 300); // 60-65 minutes
  await redis.setex(cacheKey, ttl, JSON.stringify(user));
  
  return user;
}

Cache invalidation

Cache invalidation is hard. The two patterns that work reliably:

// Pattern 1: Delete on write (simple, always consistent)
async function updateUser(id: string, data: Partial): Promise {
  const updated = await db.users.update(id, data);
  await redis.del(`user:${id}`);  // Invalidate immediately
  return updated;
}

// Pattern 2: Write-through (populate cache on write, expensive but always fresh)
async function updateUser(id: string, data: Partial): Promise {
  const updated = await db.users.update(id, data);
  await redis.setex(`user:${id}`, 3600, JSON.stringify(updated));
  return updated;
}

// Pattern 3: Tag-based invalidation (invalidate groups of related keys)
// Store cache tags as Redis sets
async function cacheWithTags(key: string, value: unknown, tags: string[], ttl: number) {
  await redis.setex(key, ttl, JSON.stringify(value));
  for (const tag of tags) {
    await redis.sadd(`tag:${tag}`, key);
    await redis.expire(`tag:${tag}`, ttl + 60);
  }
}

async function invalidateTag(tag: string) {
  const keys = await redis.smembers(`tag:${tag}`);
  if (keys.length) {
    await redis.del(...keys, `tag:${tag}`);
  }
}

Job queues with BullMQ

BullMQ uses Redis sorted sets and lists to implement a durable, distributed job queue with priorities, retries, delays, and rate limiting. It's what Sven Agent uses for mission orchestration:

import { Queue, Worker } from 'bullmq';

const connection = { host: process.env.REDIS_HOST, port: 6379 };

// Producer: add jobs to the queue
const emailQueue = new Queue('email', { connection });

await emailQueue.add('send-welcome', {
  to:      'user@example.com',
  subject: 'Welcome to 47Network',
  template: 'welcome',
}, {
  attempts: 3,               // Retry up to 3 times on failure
  backoff: {
    type: 'exponential',
    delay: 1000,             // 1s, 2s, 4s backoff
  },
  removeOnComplete: { age: 86400 },  // Remove completed jobs after 24h
  removeOnFail: { age: 604800 },     // Keep failed jobs for 7 days (for debugging)
});

// Consumer: process jobs
const worker = new Worker('email', async (job) => {
  const { to, subject, template } = job.data;
  await sendEmail({ to, subject, template });
  // Throwing an error triggers a retry
}, {
  connection,
  concurrency: 5,            // Process 5 jobs simultaneously
  limiter: {
    max:      100,           // Rate limit: max 100 jobs per minute per worker
    duration: 60_000,
  },
});

Pub/sub for real-time events

Redis pub/sub is useful for broadcasting events to multiple subscribers without persistent message storage. It's not a replacement for a message broker (messages are lost if no subscriber is connected) but is excellent for real-time UI updates, cache invalidation across multiple app instances, and lightweight event broadcasting:

// Publisher (fires events when things change)
async function publishUserUpdate(userId: string, change: object) {
  await redis.publish(`user:${userId}:updates`, JSON.stringify({
    type: 'user_updated',
    userId,
    change,
    timestamp: Date.now(),
  }));
}

// Subscriber (one connection per subscriber — can't run commands on a subscribed connection)
const subscriber = new Redis(redisConfig);

await subscriber.subscribe('user:*:updates', (err) => {
  if (err) console.error('Subscribe failed:', err);
});

subscriber.on('pmessage', (pattern, channel, message) => {
  const event = JSON.parse(message);
  // Push to WebSocket clients, invalidate cache, update UI, etc.
  broadcastToWebSocketClients(event);
});

Sorted sets: rate limiters and leaderboards

Redis sorted sets (ZADD, ZRANGEBYSCORE, ZREMRANGEBYSCORE) are the data structure behind sliding window rate limiters:

// Sliding window rate limiter using sorted sets
// Allows `limit` requests per `windowMs` milliseconds
async function checkRateLimit(
  key: string,
  limit: number,
  windowMs: number
): Promise<{ allowed: boolean; remaining: number }> {
  const now = Date.now();
  const windowStart = now - windowMs;
  const redisKey = `ratelimit:${key}`;

  const pipeline = redis.pipeline();
  pipeline.zremrangebyscore(redisKey, '-inf', windowStart); // Remove old entries
  pipeline.zadd(redisKey, now, `${now}-${Math.random()}`); // Add current request
  pipeline.zcard(redisKey);                                 // Count requests in window
  pipeline.pexpire(redisKey, windowMs);                     // Auto-cleanup
  
  const results = await pipeline.exec();
  const count = results![2][1] as number;
  
  return {
    allowed:   count <= limit,
    remaining: Math.max(0, limit - count),
  };
}

Persistence: RDB vs AOF

Redis offers two persistence mechanisms. For application caches, neither is strictly necessary — the cache can be rebuilt from the database. For job queues or session stores, you need at least one:

• RDB (snapshots): point-in-time snapshot saved every N seconds or after M writes. Fast to restore, may lose up to minutes of data. Good for caches and leaderboards.
• AOF (append-only file): logs every write command. Can be configured to fsync every second (lose up to 1 second) or every write (safe but slow). Good for queues and sessions.

# redis.conf — recommended for job queues
save ""              # Disable RDB snapshots (we're using AOF)
appendonly yes
appendfsync everysec # fsync every second — balance of safety and performance
no-appendfsync-on-rewrite yes  # Don't fsync during BGREWRITEAOF
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb