“Data is a precious thing and will last longer than the systems themselves."

— Tim Berners-Lee, Inventor of the World Wide Web

In the digital era, the world is defined with data. From daily commute to the global markets, every event, every action, and every decision leaves a digital footprint.

The core challenge—and indeed, the primary purpose of technology itself—has always revolved around this reality. It all begins with assimilation: the art of gathering and storing this vast, chaotic torrent of data. This raw material is then transformed through analysis and computation, refined from simple facts into coherent information, and ultimately, into actionable intelligence. Finally, this refined intelligence is disseminated, powering the applications and systems that shape our experiences.

But this is not the end of the line. The intelligence we derive, the predictions we make, and the insights we gain are fed back into the system, influencing future actions and, in turn, generating new data. This is the great big loop: information is derived from data, and all this derived intelligence is fed back to the data.

At the heart of this cycle lies a fundamental challenge. The problem is no longer just about storage; it's about how beautifully we can manage this flow. How elegantly can we store it, how swiftly can we analyze it, and how clearly can we present it? It’s the craft of creating systems that are not just functional, but efficient, reliable, and simple.

This article is structured around building one such system from the ground up, an exploration of the principles, the problems, and the patterns that emerge.

Introduction

At first glance, the architecture of getMe might sound familiar. It’s a system where every change is recorded as an entry in a journal, much like a database's transaction log. But it’s not quite a database. It’s simpler, more fundamental.

Imagine the entire system as a financial ledger. When a “new” piece of data is to be stored, a new record or entry is added to the current end of the ledger (which is divided into segments for better storage, access and management) . An "update" is just another new transaction at a later time that supersedes the old one. A "delete" is a final entry marking the account as closed. This append-only approach makes writing new data incredibly fast, as it's always a simple, sequential operation.

Of course, a ledger that you have to read from cover to cover to find the latest balance is useless. That's where the second key component comes in: an in-memory index. It doesn't hold the transaction details itself, but it tells the exact page and line number of the most recent entry for any given account. This combination gives us the best of both worlds: lightning-fast writes to the end of the log and lightning-fast lookups via the in-memory index. More on the specifics of this flow later.

But getMe wasn't started with a grand picture in mind. What started as a simple, isolated exercise, i.e. to create a basic template for thread-safe read/write operations for a key-value storage, grew quickly. How do you make the storage persistent across restarts? How do you handle more data than can fit in memory? How do you manage the log so it doesn't grow forever? And how do you do all of this efficiently, with concurrency in mind?

getMe has evolved from a simple template into a complete ecosystem. Today, it features a dedicated storage engine, a background compaction mechanism, a formal client-server architecture, an HTTP proxy for external access, and multi-language SDKs to seamlessly integrate into various applications.

Core Components

The system is not a monolith; it's a modular architecture where each component has a clear and distinct responsibility. This separation of concerns is key to its simplicity and robustness.

The Server & Unix Domain Socket (UDS)

This is the front door to the entire system. The Server's primary job is to listen for and accept client connections. Critically, it uses a Unix Domain Socket (UDS) instead of a traditional network port. A UDS is a file on the filesystem that acts as a high-performance communication channel between processes on the same machine. This choice makes getMe ideal for local-first applications, as it bypasses the entire network stack, resulting in lower latency and higher throughput.

• Interaction: The Server is the entry point. It receives raw requests from clients and passes them inward to the Store for processing.

The Store

The Store is the central orchestrator that provides the public API of the storage engine (Get, Put, Delete, BatchPut, BatchGet, BatchDelete, and ClearStore). It doesn't manage files or indexes itself; instead, it intelligently delegates these tasks to specialized components. Crucially, the Store avoids a monolithic top-level lock, pushing concurrency control down to the granular level of the HashTable and SegmentManager to maximize throughput.

• Interaction: The Store is the central hub. It receives commands from the Server, directs write operations to the SegmentManager, and updates the HashTable with the results. For reads, it queries the HashTable first, then requests the specific data from the SegmentManager.

type Store struct {
	basePath                string
	hashTable               *core.HashTable
	segmentManager          *core.SegmentManager
	compactionResultChannel chan *core.CompactionResult
	doneChannel             chan struct{}
	wg                      sync.WaitGroup
}

The HashTable (In-Memory Index)

The hash table is a Go map held entirely in memory that maintains the crucial link between a key and its physical location on disk. Each entry in this map contains a pointer with the SegmentId, the byte Offset within that file, the ValueSize, and a TimeStamp.

• Interaction: The HashTable is primarily a passive component. It is updated by the Store after a successful write and is queried by the Store during a read. Its performance is paramount for fast lookups, and it is protected by a sync.RWMutex to allow many concurrent readers but only one writer at a time.

type HashTableEntry struct {
	TimeStamp int64
	SegmentId uint32
	Offset    uint32
	ValueSize uint32
}

// a hash table does not to be concerned about how the raw data is stored in the disk, 
// it only deals with the mapping from key to (segmentId, offset)
type HashTable struct {
	mu    sync.RWMutex
	table map[string]*HashTableEntry
}

The SegmentManager

The segmentManager is responsible for all direct interactions with the log files on disk. It abstracts away the complexities of file management, keeping track of all segment files and, most importantly, which segment is currently "active" and open for new writes.

• Interaction: The SegmentManager is commanded by the Store. It receives a serialized Entry, appends it to the active segment, and returns the precise disk location (segment ID and offset) back to the Store so the HashTable can be updated.

type SegmentManager struct {
	mu         sync.RWMutex
	basePath   string
	segmentMap map[uint32]*Segment
	// stores the index of the next segment to be created
	nextSegmentCounter      *AtomicCounter
	compactedSegmentManager *CompactedSegmentManager
	isCompacting            atomic.Bool
	compactionResultChannel chan *CompactionResult
}

The Entry and Segment

These are not active components but the fundamental data structures of the system.

• The Entry: This is the atom of data. It’s a structured record containing a TimeStamp, key/value sizes, and the key/value pair itself. Even deletions are represented as a special type of Entry called a "tombstone."

• The Segment: This is a single, append-only log file on disk. Think of it as a single book in the library of data, containing a sequence of entries. Once a segment reaches its maximum size, it becomes immutable.

type Segment struct {
	// mu         sync.RWMutex
	id         uint32
	path       string
	file       *os.File
	entryCount uint32
	size       uint32
	isActive   bool
	maxCount   uint32
	maxSize    uint32
}

type Entry struct {
	TimeStamp int64
	KeySize   uint32
	ValueSize uint32
	Key       []byte
	Value     []byte
}

The CompactedSegmentManager & Compaction

This is the essential cleanup crew. Over time, segments accumulate stale data from updates and deletions. The compaction process, orchestrated by the CompactedSegmentManager, is the background garbage collector that reclaims this wasted space.

• Interaction: This process runs as a background goroutine to avoid blocking live requests. It reads from old, inactive segments, consults the main HashTable to identify which entries are still "live," and writes this live data into new, dense segments. The outcome—a freshly compacted index and a list of old segments to be deleted—is communicated back to the Store asynchronously and safely via a Go channel (compactionResultChannel), ensuring a non-blocking, clean integration of the results.
  This process is explained in detail later in the article.

Concepts Applied

The architecture of getMe is a practical application of several powerful computer science concepts. General programming patterns are combined with Go's specific strengths in concurrency to create something robust and performant.

To understand the design, the best way is to follow the journey of a single piece of data as it enters, lives, and is eventually cleaned up from the system.

The Write Path: Structuring and Storing Data

The journey begins when a client requests to store a key-value pair. This triggers a highly optimized, multi-step process designed for speed and durability.

1. The Entry: The Atom of Data: Before the disk is brought into consideration, the key-value pair is wrapped in a structured Entry object. It’s enriched with crucial metadata: a TimeStamp, the KeySize, and the ValueSize. This timestamp is the ultimate source of truth, allowing us to resolve which version of a key is the latest.

2. Serialization: Preparing for the Physical World: This structured Entry object, which lives in memory, must be converted into a sequence of bytes before being written to a file. This process, serialization, follows a strict, predictable format. The resulting byte stream is a concatenation of the metadata and the data itself, ensuring that we can later read from any point in the file and perfectly reconstruct the original entry.

3. Segments: The Append-Only Log: The serialized data is written to a file on disk called a Segment. getMe doesn't write to a single, monolithic file; it partitions its data into multiple segment files. This is managed by the SegmentManager, a dedicated component that keeps track of all segments and knows which one is currently "active" for writes. When the active segment reaches a predefined maximum size or a maximum no of entries, it is closed (becoming immutable), and a new active segment is created (segment manager now deals with this new segment for storing new entries). This segmentation is crucial for making maintenance tasks like compaction manageable.

4. The Write: Fast, Sequential I/O: The SegmentManager takes the serialized entry and appends it to the end of the active segment file. This is the heart of the log-structured design: all writes are transformed into fast, sequential appends. This approach works in harmony with the underlying hardware, as both spinning disks and SSDs are highly optimized for sequential I/O.

The In-Memory Index: The Key to Fast Reads

Writing data is only half the battle. Interaction with the system is what give it value.
Retrieving the data quickly is paramount.

1. The HashTable: The Store’s Directory: After an entry is successfully written to a segment, its location must be recorded. This is the job of the in-memory HashTable. For every key, it stores a HashTableEntry containing the essential pointer: the SegmentId, the byte Offset within that segment where the entry begins, its ValueSize, and its TimeStamp. This hash table is the directory for the entire database.

2. The Get Operation: A Two-Step Flow: When a client requests to Get a key, the store performs a swift, two-step lookup:

   • Step 1 (Memory): It queries the HashTable. If the key doesn't exist in this map, the data doesn't exist in the store, and we can return "not found" immediately without ever touching the disk.

   • Step 2 (Disk): If the key is found, the hash table provides its exact disk address. The SegmentManager then performs a single, direct read from the correct segment at the correct offset to retrieve the serialized entry. If the segment is not found (which can happen momentarily if compaction just deleted it), the store automatically retries the hash table lookup to fetch the updated location.

3. The BatchGet Operation: Parallel Disk I/O: For bulk reads, the store doesn't just loop over Get. Instead, it first resolves all key locations in memory, groups them by their target SegmentId, and sorts them by Offset. It then spawns parallel goroutines to perform efficient, sequential disk reads for each segment, dramatically accelerating bulk retrievals.

Updates and Deletes: The "Tombstone" Elegance

In getMe, data is never modified in place. An "update" is simply a Put operation for an existing key; a new entry is appended to the log, and the HashTable is updated to point to this new location, leaving the old data stale but untouched.

A Delete operation is handled with similar elegance. First, the store checks the HashTable to ensure the key actually exists, guaranteeing the delete is idempotent. If it does, instead of removing data from disk, it writes a special entry known as a tombstone. This is an Entry with the key and a special marker (like a value size of zero). This tombstone is appended to the log, and the key is removed from the HashTable. This maintains the append-only model, deferring the actual data removal to a later cleanup process.

Concurrency and High-Throughput Operations

getMe is designed to be concurrent, leveraging Go's powerful features.

1. Parallel Reads, Granular Writes: The HashTable, being a critical shared resource, is protected by a sync.RWMutex. This allows any number of read operations (Get) to happen in parallel, maximizing read throughput. However, any write operation (Put, Delete) must acquire an exclusive lock at the HashTable level, which serializes writes to ensure the index remains consistent. Crucially, the Store itself holds no monolithic lock, allowing different subsystems to operate with fine-grained concurrency.

2. Zero-Allocation Conversions: Go's strong typing often requires copying data between string and []byte. To squeeze out maximum performance on the hot path, getMe leverages unsafe.String and unsafe.Slice to perform zero-allocation conversions, avoiding unnecessary memory pressure during frequent reads and writes.

3. Batching Writes for Efficiency: To amortize the cost of locking and disk I/O, getMe implements a highly optimized BatchPut operation. Instead of performing one lock-write-unlock cycle for each entry, it serializes a batch of entries into a shared, reusable buffer (via bytebufferpool). Large payloads are automatically chunked at a configured boundary (e.g., 64KB) to avoid memory bloat. The engine then performs a single large sequential write to the active segment, and updates the HashTable for all the new entries in one go. This significantly increases write throughput while keeping garbage collection pauses to an absolute minimum.

Startup and Recovery: Rebuilding from the Log

When getMe starts, the in-memory HashTable is empty. Its durability model means it can fully recover its state from the segment files on disk. It iterates through all segments, reading every entry and rebuilding the hash table. By processing segments in order and using the entry timestamps, it ensures that for any given key, only the pointer to the absolute latest version is retained in the final index. The log itself is the ultimate source of truth.

Our SegmentManager orchestrates this process in its populateSegmentMap function. Here’s how the Map-Reduce pattern unfolds:

1. Discovery (The Input Split):
The process begins by scanning the data directory for all segment files (e.g., segment_*.log). Each file is treated as an independent unit of work.

2. The "Map" Phase: Parallel Segment Processing
Instead of reading segments one by one, we process them all concurrently. For each segment file discovered, we launch a dedicated goroutine.

• Task: Each "mapper" goroutine is responsible for reading a single segment file from start to finish.

• Output: As it reads, it builds a small, isolated HashTable containing only the key-value locations found within its assigned segment. Because it's processing just one segment, it naturally captures the latest value for any key within that segment's scope.

• Communication: Once a goroutine finishes reading its segment, it sends its completed, isolated hash table over a shared Go channel.

Bootstrapping the Store: A Concurrent Map-Reduce Approach to Rebuilding State

One of the most critical phases in a persistent database's lifecycle is its startup. When our key-value store initializes, it must reconstruct its in-memory state—the hash table index—by reading all the data it has ever stored on disk. A naive, sequential read of every segment file would create a significant bottleneck, leading to slow startup times, especially as the database grows.

To solve this, we've implemented a highly efficient, parallelized startup process modeled on the classic Map-Reduce pattern, leveraging Go's powerful concurrency primitives.

The Challenge: Fast and Accurate State Reconstruction

Our architecture separates data from its index. The raw key-value pairs (as Entry structs) are stored in append-only log files called "segments," while a central HashTable provides the fast, in-memory lookup from a key to its physical location (segment ID and offset).

Upon startup, this HashTable is empty. The challenge is to populate it by processing every segment on disk, ensuring that for any given key, only the most recent value is indexed.

func (sm *SegmentManager) populateSegmentMap(basePath string, centralHashTable *HashTable) error {

	// find all segment files in the base path
	paths, err := filepath.Glob(filepath.Join(basePath, "segment_*.log"))
	if err != nil {
		logger.Error("failed to list segment files basePath", basePath, "error", err)
		return fmt.Errorf("failed to list segment files in %s: %w", basePath, err)
	}
	if paths == nil {
		logger.Warn("no segments found in " + basePath)
		return nil // No segments found is not an error
	}

	logger.Info("Segments already exist, loading them from the disk to the current kv instance...")

	var wg sync.WaitGroup
	ch := make(chan *HashTable, len(paths))

	maxSegmentId := uint32(0)

	// for all the paths, open the segment and add it to the segment map, based on their IDs
	for _, path := range paths {
		var id uint32
		_, err := fmt.Sscanf(filepath.Base(path), "segment_%d.log", &id)
		if err != nil {
			return err
		}

		segment, err := OpenSegment(id, basePath)
		if err != nil {
			return err
		}

		logger.Info("opened segment with id:", id, " at path:", path, "segment details:", *segment)

		maxSegmentId = max(maxSegmentId, id)

		// assign the segment mapped to its id
		sm.segmentMap[uint32(id)] = segment

		wg.Add(1)

		segment.readAllEntriesAsync(&wg, ch)

	}

	// a goroutine to close the channel when all segment reads are done
	go func() {
		wg.Wait()
		close(ch)
	}()

	// Reducer
	for ht := range ch {
		centralHashTable.Merge(ht)
	}

	// remove all the deletion entries from the central hash table
	// since they are not actual entries
	centralHashTable.DeleteDeletionEntries()

	logger.Success("all the segments have been loaded into the central hash table")

	logger.Info("current segment map : ", sm.segmentMap)

	logger.Info("current hash table : ", centralHashTable.Entries())

	logger.Info("loaded segments from the disk")
	// increment the nextSegmentId to be one more than the max id found
	sm.nextSegmentCounter.Set(maxSegmentId + 1)
	return nil

}

The Solution: Parallel Mapping, Sequential Reducing

Our SegmentManager orchestrates this process in its populateSegmentMap function. Here’s how the Map-Reduce pattern unfolds:

1. Discovery (The Input Split):
The process begins by scanning the data directory for all segment files (e.g., segment_*.log). Each file is treated as an independent unit of work.

2. The "Map" Phase: Parallel Segment Processing
Instead of reading segments one by one, we process them all concurrently. For each segment file discovered, we launch a dedicated goroutine.

• Task: Each "mapper" goroutine is responsible for reading a single segment file from start to finish.

• Output: As it reads, it builds a small, isolated HashTable containing only the key-value locations found within its assigned segment. Because it's processing just one segment, it naturally captures the latest value for any key within that segment's scope.

• Communication: Once a goroutine finishes reading its segment, it sends its completed, isolated hash table over a shared Go channel.

This is the "Map" phase: transforming a list of segment files into a stream of in-memory hash tables, all happening in parallel. We use a sync.WaitGroup to ensure the main process waits for every segment-reading goroutine to complete its job.

3. The "Reduce" Phase: Merging for a Single Source of Truth
The "Reduce" phase begins once the mappers start sending their results. The main startup goroutine listens on the channel, receiving the isolated hash tables one by one.

• Task: Its job is to merge these smaller tables into the single, central HashTable of the store.

• Conflict Resolution: The HashTable.Merge method is the heart of the reducer. When a key from a new table already exists in the central table, it resolves the conflict based on the entry's timestamp. The entry with the newer timestamp always wins, ensuring the final index points to the absolute latest value for each key across all segments. If timestamps are identical, it further disambiguates using segment ID and offset as tie-breakers.

Why This Architecture Shines

• Scalability and Speed: The startup time is no longer proportional to the sum of all segment sizes, but rather to the size of the largest segment, as all files are read in parallel. This makes the startup process incredibly fast and scalable.

• Isolation: Each goroutine works on its own data and produces an isolated result. This avoids the need for complex locking during the read-intensive "Map" phase, simplifying the concurrency model.

• Simplicity and Elegance: By leveraging Go's channels and goroutines, we've implemented a powerful data processing pattern in a way that is both concise and easy to reason about.

Compaction: The Background Cleanup Crew

The accumulation of stale data and tombstones requires a garbage collection mechanism, known as Compaction. This process runs in the background as a separate goroutine, ensuring it has minimal impact on live requests.

1. The core logic is abstracted to a CompactedSegmentManager. It selects a set of old, inactive segments.

2. It meticulously reads these segments, checking each entry against the main HashTable to see if it's still "live."

3. Only the live data is written to new, dense, compacted segments. Stale data is discarded.

4. Communication of the result—the new index data and the list of old segments to be deleted—is handled safely via a Go channel (compactionResultChannel). This allows the main store to atomically integrate the results of the compaction when it's ready.

In getMe store, compaction is designed as a non-blocking, asynchronous background process. It ensures that the database can continue to serve reads and writes at full speed, even while it's cleaning up after itself. The entire lifecycle is managed through a careful orchestration between the Store, SegmentManager, and a dedicated CompactedSegmentManager.

The Trigger: Crossing the Threshold

Compaction isn't a constant process. It's triggered intelligently when the number of segment files crosses a predefined threshold (constants.ThresholdForCompaction). This check happens right before a new segment is created, inside the SegmentManager's appendEntryToLatestSegment method.

When the threshold is met, instead of blocking the incoming write, the SegmentManager does two things in quick succession:

1. It reserves a block of future segment IDs for the compacted data using an atomic operation.

2. It launches the entire compaction process in a separate goroutine.

This "fire-and-forget" approach is key to the system's performance, ensuring that write latency is unaffected by the housekeeping process.