Building Redis from scratch

A codecrafter challenge to build redis tool from scratch

· 17 min read

Setting the Stage

For last few weeks, I was working on a codecrafters challenge where I had to build redis from scratch. In this age of AI, it is now more important than before to understand how tools are designed & built and do it the hard way. Even though the challenge is not complete yet, I got to learn a lot of things and fill some gaps in my knowledge.

Building Redis from scratch might sound intimidating (well I was intimtidated when i started), but here’s the secret: Redis is actually a surprisingly simple piece of software at its core. Let’s break it down.

So, what is Redis?

Let’s start with understanding what redis actually is. This is what google gave me.

Let’s summarize what redis actually is in simple terms:

Core idea: Data lives in RAM, so reads/writes are extremely fast (microsecond latency, infact a single redis server can handle 100,000 queries per second).

Data structures it supports: Strings, Lists, Sets, Sorted sets, Hashes, Streams etc

Common use cases:

  • Caching - store expensive DB query results with a TTL
  • Sessions - fast user session storage
  • Rate limiting - atomic counters per user/IP
  • Pub/Sub - lightweight message passing between services
  • Queues - using Lists or Streams
  • Leaderboards - sorted sets make ranking trivial

Key properties:

  • Single-threaded command execution (no lock contention)
  • Optional persistence (RDB snapshots or AOF logs)
  • Replication and clustering for scale

In simple words, it stores key, value pair just like a HashMap in memory. Moreover, it spins off a TCP server on port 6379 by default. Whenever a client sends request with a key, it responds back the corresponding value in the store.

What we’re Building

First of all, let’s understand what redis does exactly does. As I said earlier, redis has a HashMap like structure to store key value pairs. We can store and retrive values using keys. Let’s see it in action by using the official redis-server.

# This will spin off the server on port 6379
$ redis-server

In a new terminal window, we will first store a key value pair { "foo": "bar" } inside the redis store and try to retrive value by passing key “foo”. It will look something like this:

# PING command to check if redis server is running
$ redis-cli PING
PONG

# SET command to store a key value pair
$ redis-cli SET foo bar
OK

# GET command to retrieve a value by key
$ redis-cli GET foo
"bar"

The Plan

We’re going to build Redis from the ground up in layers. Our redis implementation needs three main components.

Layer 1: TCP Server

  • Create a TCP server that accepts connections
  • Understand buffered I/O and why it matters
  • Set up the basic project structure

Layer 2: Protocol Parser

  • Learn why Redis uses RESP instead of JSON
  • Understand how RESP encodes data
  • Build a parser that converts bytes into commands

Layer 3: Command Handler

  • Implement a key-value store in memory
  • Add support for basic commands (PING, GET, SET)
  • Handle concurrent clients

By the end, we’ll have a working Redis server that real Redis clients can connect to. More importantly, you’ll understand the design decisions that make Redis so fast.

Scaffold the Project

Let’s start with a new Rust project:

cargo new redis-server
cd redis-server

I am planning to build it in rust. You can choose to use language that you are comfortable with.

Let’s Build It

Let’s start with the first layer i.e. spinning off a TCP server on port 6379.

Build the TCP server

use std::net::{TcpListener, TcpStream};
use std::io::{BufReader, Write};

fn main() -> std::io::Result<()> {
    let listener = TcpListener::bind("127.0.0.1:6379")?;
    println!("Redis server listening on port 6379");

    for stream in listener.incoming() {
        let stream = stream?;
        println!("New client connected!");
        handle_client(stream)?;
    }

    Ok(())
}

fn handle_client(mut stream: TcpStream) -> std::io::Result<()> {
    // We'll implement this next
    Ok(())
}

Let’s break down what’s happening:

  • TcpListener::bind("127.0.0.1:6379"): This tells the OS “reserve port 6379 for me”. The OS will now route any TCP connections to this port to our program.

  • listener.incoming(): This is a blocking iterator. Each iteration waits for a new client to connect. When one does, it gives us a TcpStream.

  • TcpStream: This is a bidirectional pipe to the client. We can read their requests and write our responses.

Here’s where it gets interesting. When a client connects and sends data, it might come in chunks:

// Client sends: SET key value
// We might receive:
Chunk 1: "*3\r\n$3\r\n"
Chunk 2: "SET\r\n$3\r\nkey\r\n$5\r\n"
Chunk 3: "value\r\n"

Why does this happen? TCP is a stream protocol. It guarantees bytes arrive in order, but not that they arrive in the same groupings you sent them.

This is like receiving a letter one word at a time through the mail slot. You need to collect all the words before you can read the full message. This is why we use BufReader:

fn handle_client(stream: TcpStream) -> std::io::Result<()> {
    let mut reader = BufReader::new(stream);

    loop {
        // We'll parse RESP commands here
        // For now, let's just print what we receive
    }
}

What does BufReader do? It maintains an internal buffer. When you ask for data, it:

  1. Checks if it already has enough buffered
  2. If not, reads a big chunk from the TCP stream
  3. Returns what you asked for and keeps the rest

This means fewer system calls and easier parsing. Instead of reading byte-by-byte from the network (slow!), we read in chunks. Now let’s add a simple echo to test it:

use std::io::BufRead;

fn handle_client(stream: TcpStream) -> std::io::Result<()> {
    let reader = BufReader::new(stream.try_clone()?);
    let mut writer = stream;

    for line in reader.lines() {
        let line = line?;
        println!("Received: {}", line);
        writeln!(writer, "Echo: {}", line)?;
    }

    Ok(())
}

Run this and connect with telnet:

cargo run
# In another terminal:
telnet localhost 6379

Type anything and you’ll see it echoed back. This proves our TCP server works!

The Protocol Parser

Now next step is to build the protocol parser. Rust usesRESP protocol to communicate. But first, let’s understand why Redis needs its own protocol. You might be thinking: Why not just use JSON over HTTP like every other API?

Why a custom protocol ?

Before we start coding, let’s understand why Redis needs its own protocol. You might be thinking: “Why not just use JSON over HTTP like every other API?”

Here’s the thing - Redis is designed for speed. Every microsecond matters when you’re serving millions of requests per second. HTTP adds overhead with headers, connection setup, and JSON parsing is relatively slow.

RESP (Redis Serialization Protocol) solves this by being:

  • Simple: You can literally type it by hand and debug it with telnet
  • Fast to parse: No complex state machines or lookahead needed
  • Binary-safe: Can handle any bytes, not just UTF-8 strings
  • Explicit about types: You know exactly what you’re getting

Think of it like this: HTTP is a formal letter with an envelope, stamps, and return address. RESP is a quick note passed across the table. Both work, but one is way faster when you need it.

RESP basics

RESP uses a clever trick: the first byte tells you what type of data follows. Let’s explore each type with real examples.

Simple strings — the “OK” response

When Redis wants to say “yep, got it”, it sends:

+OK\r\n

The + means “simple string”, then comes the text, then \r\n (carriage return + newline) to mark the end.

Why have this? Simple strings are fast - no length counting needed. Redis uses them for simple confirmations.

Errors — when things go wrong

Errors look similar but start with -:

-ERR unknown command 'TYPO'\r\n
Integers — counting things

When Redis needs to return a number (like “how many items in this list?”):

:42\r\n

The : prefix says “this is an integer”, then the number as ASCII digits.

Why not just send the binary number? Because RESP is designed to be human-readable when you’re debugging. You can see :42 in a network trace and know exactly what it means.

Bulk strings — the real workhorses

This is where it gets interesting. When Redis stores actual data (which could be anything - text, images, binary data), it uses bulk strings:

$5\r\nhello\r\n

Let’s break this down:

  • $ = “bulk string incoming”
  • 5 = “it’s exactly 5 bytes long”
  • \r\n = “now the data starts”
  • hello = the actual 5 bytes
  • \r\n = “data ended”

Why declare length upfront? This is brilliant. The parser knows exactly how many bytes to read. No scanning for delimiters, no escaping issues. If you’re sending binary data with \r\n in it, no problem - the parser just reads 5 bytes and moves on.

Special case — Null: What if a key doesn’t exist? Redis sends:

$-1\r\n

The length of -1 means “null” (no data follows).

Arrays — commands and responses

Here’s where it all comes together. Redis commands are sent as arrays of bulk strings:

*3\r\n$3\r\nSET\r\n$3\r\nkey\r\n$5\r\nvalue\r\n

Let’s decode this step by step:

  1. *3\r\n - Array with 3 elements
  2. $3\r\nSET\r\n - First element: “SET” (3 bytes)
  3. $3\r\nkey\r\n - Second element: “key” (3 bytes)
  4. $5\r\nvalue\r\n - Third element: “value” (5 bytes)

This is the command: SET key value

Why arrays of bulk strings? Because it’s unambiguous. Each argument has an explicit length. No worrying about spaces in filenames or escaping quotes.

Try this mental exercise: How would you send a filename with spaces using a simple space-delimited protocol? You’d need escaping. What if the filename contains your escape character? More escaping. RESP sidesteps all of this.

Building the parser

Now that we understand why RESP is designed this way, let’s think about how to parse it.

Here’s the thing: parsing RESP is just following the instructions. The first byte tells you exactly what to do:

  • Read the first byte - this tells you the type
  • Based on the type, read the appropriate format
  • For arrays, recursively parse each element

Let’s think through parsing that SET key value command:

*3\r\n$3\r\nSET\r\n$3\r\nkey\r\n$5\r\nvalue\r\n

Step 1: Read first byte = *. It’s an array!

Step 2: Read until \r\n to get count = 3. We need to parse 3 elements.

Step 3: Parse first element:

  • Read byte = $. It’s a bulk string!
  • Read until \r\n = 3. Length is 3.
  • Read exactly 3 bytes = SET
  • Skip the trailing \r\n

Step 4: Parse second element (same process):

  • $, then 3, then read 3 bytes = key

Step 5: Parse third element:

  • $, then 5, then read 5 bytes = value

Result: Array of [“SET”, “key”, “value”]

Here’s an approach to implementing this:

// First, define what we can parse
enum RespValue {
    SimpleString(String),
    Error(String),
    Integer(i64),
    BulkString(Option<Vec<u8>>),  // Option handles null case
    Array(Option<Vec<RespValue>>), // Can nest arrays!
}

The parsing function follows the mental model:

fn parse(reader: &mut impl BufRead) -> Result<RespValue, Error> {
    // Read one byte to know the type
    let type_byte = read_one_byte(reader)?;

    match type_byte {
        b'+' => parse_simple_string(reader),
        b'-' => parse_error(reader),
        b':' => parse_integer(reader),
        b'$' => parse_bulk_string(reader),
        b'*' => parse_array(reader),
        _    => Err("Unknown type!"),
    }
}

For bulk strings, the interesting part is:

fn parse_bulk_string(reader: &mut impl BufRead) -> Result<RespValue, Error> {
    let length = read_line_as_number(reader)?;

    if length == -1 {
        return Ok(RespValue::BulkString(None)); // Null!
    }

    // Read exactly `length` bytes
    let mut buffer = vec![0u8; length as usize];
    reader.read_exact(&mut buffer)?;

    // Don't forget to consume the trailing \r\n
    skip_crlf(reader)?;

    Ok(RespValue::BulkString(Some(buffer)))
}

For arrays, we recursively parse each element:

fn parse_array(reader: &mut impl BufRead) -> Result<RespValue, Error> {
    let count = read_line_as_number(reader)?;
    
    if count == -1 {
        return Ok(RespValue::Array(None)); // Null array
    }
    
    let mut elements = Vec::new();
    for _ in 0..count {
        elements.push(parse(reader)?); // Recursive!
    }
    
    Ok(RespValue::Array(Some(elements)))
}

This recursion is what makes RESP powerful. Arrays can contain arrays, which can contain arrays… Redis uses this for complex responses.

The Command Handler

Now we have a TCP server that can parse RESP. The final piece is actually doing something with those commands. This is where we implement the Redis logic.

The in-memory store

At its core, Redis is just a HashMap:

use std::collections::HashMap;
use std::sync::{Arc, Mutex};

type Store = Arc<Mutex<HashMap<String, Vec<u8>>>>;

fn main() -> std::io::Result<()> {
    let store = Arc::new(Mutex::new(HashMap::new()));
    let listener = TcpListener::bind("127.0.0.1:6379")?;
    
    for stream in listener.incoming() {
        let stream = stream?;
        let store = Arc::clone(&store);
        
        std::thread::spawn(move || {
            handle_client(stream, store).unwrap();
        });
    }
    
    Ok(())
}

What’s happening here?

  • Arc (Atomic Reference Counted) lets multiple threads share ownership
  • Mutex ensures only one thread modifies the HashMap at a time
  • Arc::clone creates a new reference (not a deep copy!)
  • Each client gets its own thread with a reference to the shared store

Implementing commands

Now let’s handle actual Redis commands:

fn handle_client(stream: TcpStream, store: Store) -> std::io::Result<()> {
    let reader = BufReader::new(stream.try_clone()?);
    let mut writer = stream;
    
    loop {
        let value = match parse(reader) {
            Ok(v) => v,
            Err(_) => break, // Client disconnected or sent invalid data
        };
        
        let response = match value {
            RespValue::Array(Some(elements)) => {
                execute_command(&elements, &store)
            }
            _ => RespValue::Error("Expected array".to_string()),
        };
        
        write_resp(&mut writer, &response)?;
    }
    
    Ok(())
}

The command executor pattern matches on the command name:

fn execute_command(elements: &[RespValue], store: &Store) -> RespValue {
    if elements.is_empty() {
        return RespValue::Error("Empty command".to_string());
    }
    
    let command = match &elements[0] {
        RespValue::BulkString(Some(bytes)) => {
            String::from_utf8_lossy(bytes).to_uppercase()
        }
        _ => return RespValue::Error("Invalid command".to_string()),
    };
    
    match command.as_str() {
        "PING" => handle_ping(&elements[1..]),
        "ECHO" => handle_echo(&elements[1..]),
        "GET" => handle_get(&elements[1..], store),
        "SET" => handle_set(&elements[1..], store),
        _ => RespValue::Error(format!("Unknown command '{}'", command)),
    }
}

Let’s implement each command:

PING — the simplest command:

fn handle_ping(args: &[RespValue]) -> RespValue {
    if args.is_empty() {
        RespValue::SimpleString("PONG".to_string())
    } else {
        // PING with argument echoes it back
        args[0].clone()
    }
}

ECHO — return what you receive:

fn handle_echo(args: &[RespValue]) -> RespValue {
    if args.is_empty() {
        return RespValue::Error("ECHO requires an argument".to_string());
    }
    args[0].clone()
}

GET — retrieve a value:

fn handle_get(args: &[RespValue], store: &Store) -> RespValue {
    if args.len() != 1 {
        return RespValue::Error("GET requires exactly one argument".to_string());
    }
    
    let key = match &args[0] {
        RespValue::BulkString(Some(bytes)) => bytes,
        _ => return RespValue::Error("Invalid key".to_string()),
    };
    
    let store = store.lock().unwrap();
    match store.get(key) {
        Some(value) => RespValue::BulkString(Some(value.clone())),
        None => RespValue::BulkString(None), // Null
    }
}

SET — store a value:

fn handle_set(args: &[RespValue], store: &Store) -> RespValue {
    if args.len() < 2 {
        return RespValue::Error("SET requires key and value".to_string());
    }
    
    let key = match &args[0] {
        RespValue::BulkString(Some(bytes)) => bytes.clone(),
        _ => return RespValue::Error("Invalid key".to_string()),
    };
    
    let value = match &args[1] {
        RespValue::BulkString(Some(bytes)) => bytes.clone(),
        _ => return RespValue::Error("Invalid value".to_string()),
    };
    
    let mut store = store.lock().unwrap();
    store.insert(key, value);
    
    RespValue::SimpleString("OK".to_string())
}

Writing responses

Finally, we need to serialize our RespValue back into bytes:

fn write_resp(writer: &mut impl Write, value: &RespValue) -> std::io::Result<()> {
    match value {
        RespValue::SimpleString(s) => {
            write!(writer, "+{}\r\n", s)?;
        }
        RespValue::Error(e) => {
            write!(writer, "-{}\r\n", e)?;
        }
        RespValue::Integer(i) => {
            write!(writer, ":{}\r\n", i)?;
        }
        RespValue::BulkString(None) => {
            write!(writer, "$-1\r\n")?;
        }
        RespValue::BulkString(Some(bytes)) => {
            write!(writer, "${}\r\n", bytes.len())?;
            writer.write_all(bytes)?;
            write!(writer, "\r\n")?;
        }
        RespValue::Array(None) => {
            write!(writer, "*-1\r\n")?;
        }
        RespValue::Array(Some(elements)) => {
            write!(writer, "*{}\r\n", elements.len())?;
            for element in elements {
                write_resp(writer, element)?;
            }
        }
    }
    writer.flush()
}

Testing it out

Now run your server:

cargo run

In another terminal, use the real redis-cli:

$ redis-cli PING
PONG

$ redis-cli SET foo bar
OK

$ redis-cli GET foo
"bar"

$ redis-cli GET nonexistent
(nil)

It works! You’ve built a Redis server that real Redis clients can talk to.

What We Learned

Building Redis from scratch reveals why it’s designed the way it is:

  • TCP over HTTP — No connection overhead, no header parsing. Just raw bytes.

  • RESP over JSON — Simpler to parse, binary-safe, explicit types. No ambiguity about what "123" means (string or number?).

  • In-memory HashMap — The simplest data structure that could possibly work. Redis’s speed comes from keeping everything in RAM and avoiding disk I/O.

But at its core, it’s still just a TCP server parsing RESP and manipulating a HashMap. Everything else is optimization and features built on top of this foundation.