Chapter 2
Prometheus and Its Scaling Limitations

Prometheus has become synonymous with cloud-native monitoring, yet its architecture is not without boundaries-especially when confronting the relentless growth in scale, complexity, and availability demanded by modern infrastructure. This chapter dissects the inner mechanics of Prometheus, exposes its performance ceilings, and critically evaluates the architectural trade-offs engineers face as they push beyond single-cluster limits. By unraveling how Prometheus performs under pressure, readers are primed to architect systems that are both observant and resilient at scale.

2.1 Prometheus Architecture Deep Dive

Prometheus is a highly modular and efficient monitoring system designed to operate in dynamic, cloud-native environments. Its core architecture comprises four primary components: the scrape engine, the time series database (TSDB), the query layer, and the alerting mechanism. Each of these components plays a distinctive role in ensuring robust and scalable metric collection, storage, querying, and notification workflows.

At the forefront of data acquisition is the scrape engine, responsible for continuously retrieving metrics from configured targets via HTTP endpoints. Targets expose metrics in a standardized text-based exposition format, enabling Prometheus to collect not only infrastructure metrics but also application-specific insights. The scrape engine operates under a pull model, which contrasts with the push-based modalities of traditional monitoring systems. This design decision enhances reliability, as Prometheus controls when and how metrics are gathered, allowing for adaptation to target availability and network conditions. Scrape configurations define which endpoints to collect from, the intervals at which collection occurs, and parameters such as timeout periods. The engine performs metric discovery either through static configurations or via dynamic service discovery integrations with systems like Kubernetes, Consul, or DNS, facilitating automatic adjustments within ephemeral cloud environments.

Once scraped, metrics are ingested into the time series database. Internally, Prometheus stores data as streams of timestamped value samples, each identified by a unique metric name and a set of key-value pairs called labels. This multi-dimensional label model allows Prometheus to represent complex systems and their relationships with rich metadata, enabling powerful and flexible querying. The TSDB is optimized for writes at the typical scrape frequency (e.g., every 15 seconds), employing an append-only architecture to minimize disk seek overhead and ensure high write throughput.

The internal storage utilizes a custom block format designed to balance compression efficiency and query performance. Each block, usually containing samples from a two-hour time window, comprises a sequence of chunks encoded with delta-of-delta compression for timestamps and simple integer compression for sample values. Index files within these blocks facilitate rapid label matching and retrieval of relevant series without scanning the entire dataset. Historical data is retained on disk, with configurable retention policies, ensuring cost-effective storage without compromising data fidelity.

Data retention housekeeping and compaction processes organize stored blocks, merging smaller segments to improve query efficiency. Prometheus also supports local caching to reduce expensive disk loads during query execution. This storage model facilitates both long-term retrieval and near-real-time analysis, critical for operational monitoring and post-mortem investigations.

Above the storage layer sits the query layer, exposing the PromQL (Prometheus Query Language) interface. PromQL is a domain-specific language that empowers users to select and aggregate time series data using expression matching on metric names, labels, and temporal operators. It supports a variety of aggregation functions such as rate calculations, histograms, and statistical summaries, essential for measuring dynamic system behaviors.

The query engine resolves PromQL expressions by translating label matchers and operators into efficient read operations against the TSDB, retrieving and combining data streams as specified. Queries can be performed via an HTTP API or the interactive web UI, enabling flexible integration with dashboards and other visualization tools. The query range requests support Prometheus's use cases of troubleshooting, alert evaluation, and capacity forecasting.

The architectural data flow can be summarized as follows: the scrape engine continuously polls configured endpoints, feeding recently collected metrics into the TSDB. Retained data blocks serve user queries and drive alert evaluations. Alerts, which are defined using PromQL expressions with specified thresholds or conditions, are periodically evaluated against the current datastore during rule intervals. When conditions trigger, alerts pass to the alert manager, a separate component responsible for deduplicating, grouping, and routing notifications to external systems such as email, PagerDuty, or Slack.

This end-to-end data pipeline ensures resilient and near real-time observability, despite the ephemeral and distributed nature of modern cloud-native applications. The modularity inherent in Prometheus's architecture enables scalability by horizontally sharding scrape targets and vertically scaling query and storage resources as needed. Furthermore, integration with remote storage backends permits offloading older data or handling increased ingestion workloads beyond local capacity.

The interplay between the scrape engine, TSDB, query processing, and alert handling forms a cohesive architecture. The scrape engine's controlled data acquisition prevents overloads and missing data, the TSDB's label-based model ensures rich multi-dimensional indexing and efficient storage, the query layer's expressive language facilitates comprehensive analysis, and the alert subsystem enables rapid and automated issue detection. Collectively, these components empower Prometheus to deliver a robust monitoring framework tailored for highly dynamic and scalable cloud environments.

2.2 Storage and Retention

Prometheus's local storage is fundamentally centered around a custom time series database (TSDB) optimized for high ingestion rates and efficient querying of recent metrics. The design priorities include compact on-disk storage, fast writes, and the ability to retain data over configurable time windows. At the core, the Prometheus TSDB organizes time series data into blocks that are sequentially written and periodically compacted to reduce overhead and maintain query performance.

Time series ingestion proceeds through an append-only write path. Each unique time series, identified by its metric name and label set, is mapped to an in-memory series object. Samples are appended sequentially to a write-ahead log (WAL) to ensure durability against process or system crashes. The WAL is segmented and written to disk in relatively small increments, balancing data safety with write amplification.

Simultaneously, samples are buffered in memory and periodically flushed into immutable disk blocks known as chunks. These chunks represent continuous time intervals-typically two hours in Prometheus's default configuration-and store samples encoded in a highly efficient format. Each block also includes extensive metadata, such as label indexes and chunk summaries, facilitating rapid label-based queries and time range selection. This architecture supports a predominantly write- and prune-heavy workload, where new data arrives continuously, and old data must be efficiently discarded.

Prometheus employs a multi-level compaction strategy similar in spirit to Log-Structured Merge Trees (LSM-trees), adapted for time series data. Newly created blocks are compacted in stages, merging smaller blocks into larger ones while discarding overwritten or obsolete entries. Compaction serves several purposes:

• Reducing the total number of blocks to minimize open file descriptors and metadata overhead.
• Increasing block time span, thus improving query efficiency over larger time windows.
• Reclaiming disk space by pruning samples corresponding to expired series or stale data.

The compaction process occurs in the background and is designed to be I/O-friendly, avoiding excessive latency spikes. However, excessive compaction or poor tuning can still introduce write amplification and increased disk activity, which are critical factors for storage performance and reliability.

Retention is implemented through a time-based pruning mechanism that deletes blocks older than the configured retention period. By default, Prometheus supports retention times ranging from hours to several months, adjusted via the -storage.tsdb.retention.time configuration. When a block passes...