Redis

Area: In-memory data store
Related: PostgreSQL, graphile-worker, WebSocket, Authentication vs Authorization, Docker

What is it?

Redis is a key-value server that keeps all its data in RAM and talks over TCP. The values aren’t only strings. They’re data structures: lists, hashes, sets, sorted sets, streams, bitmaps, HyperLogLogs. You don’t fetch a value and change it in your app. You send a command that changes the structure inside the server, like “push to this list” or “add 5 to this member’s score”.

SET session:abc123 '{"userId":42}' EX 3600   # string with a 1 hour TTL
INCR rate:ip:1.2.3.4                         # atomic counter
ZADD leaderboard 1500 alice                  # sorted set: member + score
ZREVRANGE leaderboard 0 9 WITHSCORES         # top 10

Every command is atomic, because one thread runs commands one at a time. That’s the main reason Redis is useful for counters, locks, queues and rate limits. Two clients can’t interleave inside a command.

Redis is a shared, single-threaded set of data structures in RAM. Each command runs alone, and each one is cheap.

Why it exists

Around 2009 the usual answer to “my database is too slow for this” was memcached: a cache for opaque blobs, with no persistence and no types. If you wanted a leaderboard you had to read the blob, deserialize it, sort it, write it back, and hope nobody else wrote at the same time.

Salvatore Sanfilippo wrote Redis to do that work on the server: keep the structures in memory, give each one its own commands, and make each command atomic. Persistence and replication came later, which is how it went from “faster memcached” to something people use as a primary store for some workloads.

Licensing note. In 2024 Redis moved from BSD to source-available licenses (RSALv2/SSPL). The Linux Foundation forked the last BSD version as Valkey, and AWS, Google and others now offer Valkey. Redis 8 later added AGPLv3 as an option. For an app developer the commands are the same, but check which one your cloud provider actually runs.

How it works underneath

The event loop

Redis runs one main thread with an event loop (epoll on Linux, kqueue on macOS). It reads a command from a socket, runs it, queues the reply, and moves on. Since Redis 6, optional I/O threads (io-threads) can parse requests and write replies, but commands still execute on one thread.

Consequences:

• A command that touches one key in O(1) takes microseconds. A single instance usually handles somewhere in the low hundreds of thousands of simple ops per second, depending on hardware and pipelining.
• One slow command blocks every client. KEYS * on 10 million keys, DEL on a 5 GB hash, LRANGE of a huge list, a long Lua script. Nothing else runs until it’s done.
• More CPU cores don’t speed up one instance. You scale with more instances (Cluster) or read replicas.

The wire protocol (RESP)

Clients send commands as arrays of bulk strings. SET k v on the wire is:

*3\r\n$3\r\nSET\r\n$1\r\nk\r\n$1\r\nv\r\n

It’s simple enough to type into telnet. Redis 6 added RESP3 with richer reply types (maps, sets, push messages for client-side caching).

Because of the request/reply round trip, sending 1,000 commands one by one costs 1,000 RTTs. Pipelining sends them all and reads all replies at once:

const results = await redis.multi()   // node-redis: batch commands
  .incr("a").incr("b").expire("a", 60)
  .execAsPipeline();                  // pipeline only. .exec() wraps them in MULTI/EXEC.

On a 1 ms network that turns a second into a few milliseconds.

Encodings: why small collections are cheap

Each type has a compact encoding for small values and a general one for big values. Redis converts automatically when a threshold is crossed.

Type	Small encoding	Large encoding	Default thresholds (redis.conf)
Hash	listpack	hashtable	hash-max-listpack-entries 128, hash-max-listpack-value 64
Sorted set	listpack	skiplist + hashtable	zset-max-listpack-entries 128, zset-max-listpack-value 64
Set	intset (all ints) or listpack	hashtable	set-max-intset-entries 512, set-max-listpack-entries 128
List	quicklist of listpacks	same	list-max-listpack-size -2 (8 KB per node)
String	int / embstr / raw

A listpack is one contiguous block of memory scanned linearly. For 100 fields that’s faster and much smaller than a hash table. This is why the classic trick of “store a million small objects as small hashes instead of separate string keys” saves a lot of memory. Check what you have with OBJECT ENCODING key and MEMORY USAGE key.

Older docs say “ziplist”. Redis 7 replaced ziplists with listpacks and renamed the config options.

Sorted sets use a skip list (ordered by score, O(log n) insert and range) plus a hash table (member → score, O(1) lookup). That pair is why ZADD, ZRANK and ZRANGEBYSCORE are all fast.

Expiry

A key with a TTL is removed in two ways:

• Lazily, when someone accesses it and Redis sees it’s expired.
• Actively, by a background cycle that samples keys with TTLs. It runs hz times per second (default 10), takes a sample of 20, deletes the expired ones, and repeats if more than 25% of the sample was expired.

So expired keys can sit in memory for a while, and a mass expiry of millions of keys at the same second creates CPU spikes. Add jitter to TTLs (3600 + random(0..300)).

Eviction when memory is full

maxmemory caps RAM. maxmemory-policy decides what happens at the cap:

Policy	Evicts
noeviction (default)	Nothing. Writes fail with OOM command not allowed.
allkeys-lru / allkeys-lfu	Any key, least recently / least frequently used
volatile-lru / volatile-lfu	Only keys with a TTL
allkeys-random / volatile-random	Random
volatile-ttl	Keys with the shortest remaining TTL

LRU and LFU are approximate. Redis samples maxmemory-samples keys (default 5) and evicts the best candidate among them, instead of keeping a full linked list. Raising the sample count gets closer to true LRU at more CPU cost.

For a pure cache, use allkeys-lru or allkeys-lfu. For anything where losing a key is a bug (queues, locks, sessions you can’t recreate), use noeviction and alert on memory, or keep them on a separate instance.

Persistence

Redis is in-memory but can write to disk two ways:

RDB snapshots. Redis fork()s. The child writes the whole dataset to a file while the parent keeps serving. Copy-on-write means pages are shared until the parent modifies them.

• Compact, fast to restart from.
• You lose everything since the last snapshot.
• fork() on a large instance can take noticeable time, and with heavy writes during the snapshot, memory use can approach double. Size the box for it. Redis warns at startup if Linux Transparent Huge Pages are on, because THP makes copy-on-write copy 2 MB pages instead of 4 KB ones.

AOF (append-only file). Every write command is appended to a log and replayed on restart.

• appendfsync everysec (default when AOF is on): fsync once per second, lose about a second on a crash.
• always: fsync every write. Durable, much slower.
• no: let the OS decide.
• The log is compacted by an AOF rewrite (another fork). Redis 7 splits it into a base file plus incremental files (“multi-part AOF”).

Most production setups use both, or use managed Redis and accept its defaults. Either way, Redis replication is asynchronous, so a failover can still lose acknowledged writes.

Data types and what they’re for

Strings and counters

await redis.set("cache:user:42", JSON.stringify(user), { EX: 300 });
const n = await redis.incr(`views:post:${id}`);       // atomic, no read-modify-write race
await redis.set("feature:new-editor", "1");

Max value size is 512 MB. Don’t go anywhere near it. Large values block the thread while they’re copied.

Hashes

A small object with fields you can update individually.

HSET user:42 name "Alice" plan "pro" seats 5
HINCRBY user:42 seats 1
HGETALL user:42

TTLs used to apply only to whole keys. Newer versions (7.4+) add per-field expiry with HEXPIRE.

Lists

Linked lists of listpacks. Push and pop at both ends in O(1). BLPOP blocks until an item arrives, which made lists the original Redis job queue. The weak point: once an item is popped, it’s gone. If the worker crashes, the job is lost. LMOVE to a “processing” list helps, but Streams solve it properly.

Sets and sorted sets

SADD online:room:7 user:1 user:2
SINTER tags:redis tags:database          # set intersection
ZADD leaderboard 1500 alice 1320 bob
ZINCRBY leaderboard 25 bob
ZREVRANGE leaderboard 0 9 WITHSCORES
ZRANGEBYSCORE delayed:jobs -inf 1729500000   # "jobs due before now"

Sorted sets with a timestamp as the score are the standard way to build delayed jobs, sliding-window rate limits and “recent activity” feeds.

Streams

An append-only log with ids like 1729500000000-0, and consumer groups that track what each consumer has acknowledged. Closer to a small Kafka than to a list.

XADD events * type signup userId 42
XGROUP CREATE events emailers $ MKSTREAM
XREADGROUP GROUP emailers worker-1 COUNT 10 BLOCK 5000 STREAMS events >
XACK events emailers 1729500000000-0
XAUTOCLAIM events emailers worker-2 60000 0   # take over messages idle > 60s

Unacked messages stay in the group’s pending entries list, so a crashed worker’s messages can be claimed by another. Cap growth with XADD events MAXLEN ~ 100000 ....

Pub/Sub

PUBLISH channel msg delivers to whoever is subscribed right now. No storage, no replay, no ack. A subscriber that disconnects for one second misses everything in that second. Good for “tell all WebSocket servers to push this update”. Bad for anything that must arrive. A connection in subscribe mode can’t run normal commands, so clients use a separate connection for it.

The probabilistic ones

• HyperLogLog: counts unique items in at most 12 KB with about 0.81% standard error. PFADD visitors:2026-10-21 user:42, PFCOUNT.
• Bitmaps: SETBIT active:2026-10-21 42 1. A million users’ daily activity in about 125 KB.
• Geo: a sorted set with geohash scores. GEOADD, GEOSEARCH.

Atomicity: transactions, scripts, locks

MULTI/EXEC is not a SQL transaction

MULTI
DECRBY account:1 100
INCRBY account:2 100
EXEC

The commands queue and then run back to back with nothing interleaved. But there’s no rollback. If the second command fails at runtime (wrong type), the first one stays applied. And you can’t read a value inside the transaction and branch on it.

For read-then-write, use WATCH (optimistic locking): WATCH key, read, MULTI, write, EXEC. If anyone changed the key in between, EXEC returns null and you retry.

Lua scripts and functions

A script runs atomically on the server, so it can read, decide and write in one step:

// release a lock only if we still own it
const RELEASE = `
  if redis.call("GET", KEYS[1]) == ARGV[1] then
    return redis.call("DEL", KEYS[1])
  end
  return 0`;
await redis.eval(RELEASE, { keys: ["lock:invoice:9"], arguments: [token] });

Scripts block the server while they run, so keep them short. Redis 7 added Functions (FUNCTION LOAD, FCALL), which are named, persisted scripts that survive restarts and replicate as part of the dataset.

A lock that mostly works

const token = crypto.randomUUID();
const ok = await redis.set("lock:invoice:9", token, { NX: true, PX: 30_000 });
if (!ok) return; // someone else has it
try { await doWork(); } finally { await redis.eval(RELEASE, { keys: ["lock:invoice:9"], arguments: [token] }); }

• NX makes acquire atomic. PX stops a crashed holder from keeping it forever.
• The random token stops you from deleting a lock that expired and was taken by someone else.
• It is not safe for correctness under GC pauses, clock jumps or failover. Your process can pause past the 30 s, the lock expires, someone else takes it, and you both write. Martin Kleppmann’s “How to do distributed locking” explains why, including the multi-node Redlock algorithm. If correctness matters, use a fencing token that the storage layer checks, or a database lock (SELECT ... FOR UPDATE, Postgres advisory locks).

Common patterns

Cache-aside

async function getUser(id: string) {
  const hit = await redis.get(`user:${id}`);
  if (hit) return JSON.parse(hit);
  const user = await db.users.find(id);
  await redis.set(`user:${id}`, JSON.stringify(user), { EX: 300 + Math.floor(Math.random() * 60) });
  return user;
}
// on write: update the DB, then DEL the key (don't SET it, to avoid racing a stale read)

The trap is the stampede: a hot key expires and 500 requests all miss and hit the database together. Fixes: a short lock so only one request rebuilds (SET rebuild:user:42 1 NX EX 5), serve-stale-while-refreshing, or early random refresh.

Fixed-window rate limit

const key = `rl:${userId}:${Math.floor(Date.now() / 60_000)}`;
const [count] = await redis.multi().incr(key).expire(key, 60).exec();
if (Number(count) > 100) throw new TooManyRequests();

Allows bursts at window edges (100 at 0:59 and 100 at 1:00). A sliding window uses a sorted set: ZADD the request timestamp, ZREMRANGEBYSCORE old ones, ZCARD to count, all in one script.

Sessions. SET sess:<id> <json> EX 86400. Revoking is DEL. Run it with noeviction or on its own instance so a cache flood can’t log everyone out.

Fan-out between app servers. Several WebSocket servers, each holding some clients. Publish document updates to a channel and every server forwards to its local clients. If a missed message matters, use a stream instead.

Scaling and high availability

Replication. Replicas copy the primary asynchronously. Reads from replicas can be stale. WAIT 1 100 blocks until at least one replica acknowledges, which narrows (not closes) the loss window.

Sentinel. Separate processes that watch a primary, agree it’s down, promote a replica, and tell clients the new address. Clients must ask Sentinel, not hard-code the host.

Cluster. Shards the keyspace into 16,384 hash slots. A key’s slot is CRC16(key) mod 16384, and each primary owns a range of slots.

• A client that hits the wrong node gets -MOVED 3999 10.0.0.5:6379 and should update its slot map. Cluster-aware clients handle this.
• Multi-key commands (MGET, SINTER, transactions, scripts) only work if all keys are in the same slot, otherwise CROSSSLOT. Force it with hash tags: only the part inside {} is hashed, so {user:42}:profile and {user:42}:cart share a slot.
• Overusing one hash tag creates a hot shard.

Redis vs the alternatives

Need	Redis	Alternative
Cache	Rich types, TTL, eviction	memcached (simpler, multithreaded, blobs only)
Job queue	Streams or BullMQ, very fast	Postgres queue (transactional with your data), SQS, RabbitMQ
Source of truth	Possible with AOF, but async replication and RAM-bound	Postgres
Pub/sub	Easy, but no delivery guarantee	Streams, Kafka, NATS
Locks	Fine for efficiency, not for correctness	Postgres advisory locks, etcd, ZooKeeper
Search / vectors	Redis Query Engine exists	Postgres + pgvector, a search engine

If you already run Postgres and the load is modest, a Postgres table often does the job with one less moving part. Add Redis when you need sub-millisecond latency, very high write rates on counters, or the specific structures (sorted sets, streams, HLL).

A real-world example

A collaborative notes app with several stateless API servers and a few WebSocket servers:

• Sessions live in Redis with a 7 day TTL, refreshed on activity. Logging out everywhere is DEL on each of the user’s session keys, tracked in a set user:<id>:sessions.
• Rate limits per user and per IP use a sliding window in a sorted set, in one Lua script per request.
• Presence (“who’s viewing this note”) is a sorted set per note with the last heartbeat time as score. Clients ping every 20 s, and readers drop members older than 60 s with ZREMRANGEBYSCORE.
• Real-time fan-out: a server that receives an edit publishes it on note:<id>. Other WebSocket servers subscribed to that channel push it to their clients. Durable history lives in Postgres, so a missed pub/sub message just means the client refetches on reconnect.
• Caching of expensive permission lookups uses cache-aside with a 60 s TTL. Sharing changes DEL the affected keys.
• The cache and the session store are separate instances: the cache is allkeys-lru, the session store noeviction.

Gotchas

• KEYS * blocks the server. Use SCAN with a cursor.
• Deleting a huge key with DEL blocks too. Use UNLINK, which frees memory in a background thread.
• Default maxmemory-policy is noeviction: a full cache starts failing writes instead of evicting.
• Expired keys aren’t freed exactly on time. Mass expiry at the same second causes CPU spikes.
• MULTI/EXEC has no rollback.
• Pub/Sub drops messages for disconnected subscribers.
• Async replication means a failover can lose acknowledged writes.
• CROSSSLOT errors appear only after moving to Cluster. Design key names with hash tags early if you might.
• Big values and huge collections (a list with 10 million items) cause latency spikes and slow replication. Find them with redis-cli --bigkeys or --memkeys.
• fork() for RDB or AOF rewrite can briefly stall a large instance and spike memory.
• A Redis with no password bound to a public interface gets taken over by bots within hours. Use requirepass or ACLs, bind to private networks, enable TLS on managed services.
• Connection per request in serverless functions exhausts maxclients. Reuse a client per container or use a pooled HTTP-based service.
• SELECT databases (0 to 15) don’t exist in Cluster. Use key prefixes instead.

Practice

1. Run redis-server in Docker, connect with redis-cli, and type a command in raw RESP through nc localhost 6379.
2. Create a hash with 100 fields and check OBJECT ENCODING and MEMORY USAGE. Add a 129th field and check again.
3. Build cache-aside for a slow function. Simulate a stampede with 200 concurrent requests on an expired key, then fix it with a rebuild lock.
4. Write a sliding-window rate limiter as one Lua script. Test it with a burst at a window edge and compare with the fixed-window version.
5. Build a job queue with Streams and a consumer group. Kill a worker mid-job and recover its messages with XAUTOCLAIM.
6. Implement the lock above. Add an artificial 35 s pause inside the critical section and show two holders at once. Then add a fencing token checked by a Postgres UPDATE ... WHERE version = $n.
7. Run a 3-primary Cluster locally and trigger CROSSSLOT with MGET a b. Fix it with hash tags.

Sources worth reading

• redis.io docs: data types, Redis persistence, Key eviction, Redis cluster specification
• The default redis.conf, which documents every setting inline
• redis-cli --latency, SLOWLOG GET, LATENCY DOCTOR docs for diagnosing stalls
• Martin Kleppmann, How to do distributed locking (2016), and antirez’s reply Is Redlock safe?
• Valkey project docs, for the open-source fork
• BullMQ docs, for a production queue built on Redis