Chapter 2
Pre-Training: Data, Algorithms, and Infrastructure

What does it take to forge a model as powerful as BERT? This chapter peels back the curtain on the immense engineering, data strategy, and technological orchestration required. From assembling massive text corpora to designing resilient distributed training pipelines, discover the complex and fascinating machinery that enables BERT's remarkable capabilities, and learn how best practices push efficiency and scalability to their very limits.

2.1 Corpus Construction at Scale

The efficacy of BERT pre-training hinges critically on the construction of a high-quality, large-scale text corpus that adequately represents the linguistic diversity and complexity required for robust language modeling. To this end, methodologies for corpus construction must address several intertwined challenges: methodical source selection, rigorous text cleaning, effective deduplication techniques, and thoughtful language balancing. Each of these facets contributes directly to the representativeness and quality of the dataset, which ultimately shapes the model's generalization capabilities.

Source selection forms the foundation of corpus assembly, balancing scale with diversity and domain relevance. Diverse textual sources enhance the model's ability to generalize across genres, styles, and topics. Commonly employed sources include:

• Web crawled data, such as Common Crawl, provides vast quantities of raw text but is characterized by high noise and heterogeneity.
• Curated corpora, like Wikipedia and newswire datasets, supply higher-quality and topic-structured content, though often more limited in size.
• Books and scholarly articles, which contribute domain-rich language and complex syntactic structures.
• Social media and forums, which introduce informal and conversational language patterns.

A balance must be struck to ensure the corpus comprises both breadth and depth, avoiding disproportionate representation from any single domain or register. For BERT, domain-agnostic pre-training corpora like BookCorpus and English Wikipedia have pioneered this approach, supplemented often by large web datasets. When targeting multilingual pre-training, source selection must also consider language coverage and script diversity, incorporating varied language-specific resources and aligned content where available.

Raw text data, especially from web crawls, inherently contains noise manifesting as HTML markup, advertisements, boilerplate, corrupted encodings, duplicated segments, or non-linguistic content. Text cleaning aims to eliminate these artifacts to enhance the semantic purity and syntactic coherence of the corpus.

Key cleaning steps often include:

• HTML and markup stripping: Removing tags, scripts, styles, and embedded media descriptions via robust parsers or regular expression filters.
• Removal of boilerplate and navigation text: Identifying repeated template elements common to multiple web pages through heuristics or machine learning classifiers.
• Normalization: Converting multiple encodings to Unicode normalization forms (NFC or NFKC), unifying whitespace, and transforming punctuation consistently.
• Character filtering: Excluding non-textual tokens, emoji, or control characters that do not contribute linguistic content or impair tokenization.
• Language identification and filtering: Applying language detection algorithms at the sentence or document level to isolate intended language content and discard misclassified text.

Implementing these steps at scale requires distributed processing and scalable pipelines capable of efficiently parsing terabytes of raw data. Open-source tools such as langid.py, FastText's language identifiers, and heuristics developed for large web corpora act as critical components of this cleaning strategy.

Redundancy in large-scale corpora is widespread due to content replication across domains, mirrors, or aggregators. Deduplication prevents model overfitting on repeated text segments, which can skew language representations and lead to memorization rather than generalization.

Deduplication techniques operate primarily at document or segment granularity:

• Exact deduplication involves removing fully identical documents or paragraphs, often implemented using hash functions such as SHA-1 or MD5 on text blocks.
• Near-duplicate detection employs more sophisticated approaches, including:

• Fingerprinting methods such as MinHash or simhash to generate compact sketches of text content that approximate similarity.
• Locality-sensitive hashing (LSH) to efficiently cluster similar text segments.
• Vector embeddings with thresholded cosine similarity for semantic-level deduplication.

The deduplication process can be algorithmically expressed as follows:

from datasketch import MinHash, MinHashLSH

def get_minhash(doc, num_perm=128):
tokens = doc.split()
m = MinHash(num_perm=num_perm)
for token in tokens:
m.update(token.encode('utf8'))
return m

lsh = MinHashLSH(threshold=0.9, num_perm=128)
corpus = [...] # list of documents
filtered_docs = []

for i, doc in enumerate(corpus):
m = get_minhash(doc)
result = lsh.query(m)
if not result:
lsh.insert(f"doc{i}", m)
filtered_docs.append(doc)

At scale, deduplication demands distributed approaches often implemented with MapReduce or Spark, segmenting the corpus to hash or embed chunks in parallel before merging hash tables to identify duplicates.

Language balancing is paramount in multilingual BERT pre-training or corpora that include multiple dialects and registers. Imbalanced corpora can bias model parameters towards dominant languages or styles, impairing performance on underrepresented languages or domains.

Strategies to achieve a balanced corpus distribution include:

• Controlled sampling: Downsampling overrepresented languages or domains while upsampling or supplementing data for less represented ones.
• Data augmentation: Synthesizing data for underrepresented languages via back-translation or domain transfer.
• Weighted training: Applying sampling weights during batch construction to maintain proportional exposure to languages.
• Segmentation harmonization: Normalizing document or sentence lengths across languages to avoid bias toward shorter or longer texts.

Quantitatively, language balance can be measured via metrics such as language-wise token counts, normalized entropy over the distribution of languages, or divergence metrics relative to a target distribution. For example, a desired language token distribution p = {pi} can be achieved by minimizing the Kullback-Leibler divergence DKL(q?p), where q is the observed distribution after sampling:

Automated pipelines often recalibrate sampling ratios dynamically to achieve this balance, adapting to corpus growth and newly incorporated sources.

Maintaining quality and representativeness throughout the corpus construction process demands continuous validation. Automated metrics provide statistical evidence of corpus composition and quality, including:

• Vocabulary coverage: Monitoring rare and frequent token distributions to detect overfitting or vocabulary collapse.
• Syntactic complexity: Analyzing sentence length distributions and parse tree depth statistics.
• Topic diversity: Employing topic modeling techniques (e.g., Latent Dirichlet Allocation) to ensure thematic balance.
• Readability indices: Applying measures such as Flesch-Kincaid grade level to...