Abstract

Scaling laws are useful guides for developing language models, but there are still gaps between current scaling studies and how language models are ultimately trained and evaluated. For instance, scaling is usually studied in the compute-optimal training regime (i.e., “Chinchilla optimal” regime); however, in practice, models are often over-trained to reduce inference costs. Moreover, scaling laws mostly predict loss on next-token prediction, but ultimately models are compared based on downstream task performance. In this paper, we address both shortcomings. To do so, we create a testbed of 104 models with 0.011B to 6.9B parameters trained with various numbers of tokens on three data distributions. First, we investigate scaling in the over-trained regime. We fit scaling laws that extrapolate in both the number of model parameters and the ratio of training tokens to parameters. This enables us to predict the validation loss of a 1.4B parameter, 900B token run (i.e., 32 over-trained) and a 6.9B parameter, 138B token run—each from experiments that take 300 less compute. Second, we relate the perplexity of a language model to its downstream task performance via a power law. We use this law to predict top-1 error averaged over downstream tasks for the two aforementioned models using experiments that take 20 less compute. Our experiments are available at https://github.com/mlfoundations/scaling.

Introduction

Training large language models is expensive. Moreover, training high-quality models requires a complex recipe of algorithmic techniques and training data. To reduce the cost of finding successful training recipes, researchers first evaluate ideas with small experiments and then extrapolate their efficacy to larger scales. With reliable extrapolation, it is possible to quickly iterate at small scale and still pick the method that will perform best for the final large training run. Indeed, this workflow has become commonplace for training state-of-the-art language models such as Chinchilla 70B [43], PaLM 540B [17], and GPT-4 [74].

Despite their importance for model development, published scaling laws differ from the goals of training state-of-the-art models in important ways. For instance, scaling studies usually focus on the compute-optimal training regime (“Chinchilla optimality” [43]), while widely used models are now often over-trained to reduce inference costs. Another potential mismatch between scaling laws and eventual applications of the models is that most scaling laws quantify model performance by perplexity in next-token prediction instead of accuracy on widely used benchmark datasets. As a result, it is unclear whether following scaling laws leads to truly better models, or merely to models with lower perplexity in the compute-optimal training regime.

In this paper, we address both topics: scaling in the over-trained regime and downstream performance prediction, with an extensive set of experiments.

Motivated by the practice of training beyond compute-optimal, we first investigate whether scaling follows reliable trends in the over-trained regime. We find that for a set of model configurations with a constant ratio of training tokens to parameters, the models’ reducible loss [41, 43] follows consistent power laws () in the amount of training compute . As one increases the ratio of tokens to parameters, corresponding to more over-training, the scaling exponent remains about the same, while the scalar changes. We consider the extent to which our observations are explainable and find a promising approach by reparameterizing forms for scaling laws.

To establish if and when scaling is predictable in the over-trained regime, we experiment with a testbed of 104 models, trained from scratch on three different datasets: RedPajama [110], C4 [86, 25], and RefinedWeb [80]. We find that scaling laws fit on small models, trained closer to compute-optimal, can accurately predict the performance of larger models that undergo more over-training. Figure (left) illustrates our main over-training result, where we invest FLOPs to extrapolate the C4 validation performance of a 1.4B parameter model trained on 900B tokens, which requires more compute to train.

In addition to over-training, we also investigate if scaling laws can predict the performance of a model on downstream tasks. We establish a power law relationship between language modeling perplexity and the average top-1 error on a suite of downstream tasks. While it can be difficult to predict the error on individual tasks, we find that aggregate performance can be accurately predicted from a model’s perplexity among models trained on the same training data. Figure (right) presents our main downstream error prediction result, where we invest FLOPs to predict the average top-1 error over a set of downstream tasks to within 1 percentage point for a 6.9B compute-optimal model, which requires more compute to train.

Scaling and over-training

In this section, we describe empirical observations and their potential mathematical descriptions. First, we provide key definitions (Section ). We next present a phenomenon wherein training a collection of models for increasing token multipliers—ratios of tokens to parameters—follows similar scaling trends (Section ). We then show that these patterns are consistent with previously proposed power-law scaling when reparameterizing in terms of training compute and token multipliers (Section ). Towards connecting loss scaling and downstream performance, we revisit our collection of models, plot their average top-1 error vs. validation loss, and notice that error decays exponentially with lower loss (Section ). To describe these observations, we propose a scaling law for downstream error as a function of loss (Section ).

Preliminaries

Scaling laws for loss.

We examine scaling laws that predict the loss of a model as a function of the compute in FLOPs used to train the model. If one increases the number of parameters in a model or the number of tokens that a model is trained on, compute requirements naturally increase. Hence, is assumed to be a function of . Following Kaplan et al. [49], we use the approximation , which Hoffmann et al. [43] independently verify. We consider scaling laws,

where is an irreducible loss and is the reducible loss. captures the Bayes error or minimum possible loss achievable on the validation domain. The term captures what can possibly be learned about the validation domain by training on a source domain. should go to zero with increased training data and model capacity. is often assumed to follow a power law: (i.a., Hestness et al. [41], OpenAI [74]). It is also often helpful to consider a power law in a - plot, where it appears as a line with slope and -intercept .

Token multipliers.

We define a token multiplier as the ratio of training tokens to model parameters. We introduce for notational convenience as it allows us to consider fixed relationships between and even as a model gets bigger (i.e., as becomes larger).

Compute-optimal training.

Hoffmann et al. [43] establish compute-optimal training, where, for any compute budget , the allocation of parameters and tokens that minimize training or validation loss is given by,

To solve for the optimal , one can sweep for each , retaining the best configurations. Hoffmann et al. [43] find that as increases, and scale roughly evenly. Assuming equal scaling, there is a fixed compute-optimal token multiplier per training distribution.

Over-training.

We define over-training as the practice of allocating compute sub-optimally, so smaller models train on a disproportionately large number of tokens (i.e., ). While loss should be higher than in the compute-optimal allocation for a given training budget, the resulting models have fewer parameters and thus are cheaper at inference.

Observation: Over-trained models follow consistent trends

We begin our scaling investigation by training models with 0.011B to 0.411B parameters for token multipliers between 20 and 640, where points lie roughly on the compute-optimal frontier, and corresponds to over-training. We defer experimental details to Section to focus on our observations. In Figure , we plot loss against compute on a - scale for the models trained on three datasets and evaluated on the C4 eval set. We notice a phenomenon of “parallel lines” when fitting power laws to the reducible loss, which suggests a near-constant scaling exponent even with increased over-training. This indicates that scaling in the over-trained regime may be predictable given training runs closer to compute-optimal.

Deriving scaling laws for over-trained behavior

In search of an analytic expression for the observations in Figure , we turn to the scaling literature. A common functional form for the risk of a model, as proposed in prior work [91, 43] is,

Recall from Section , is the number of parameters, the number of training tokens and the irreducible loss. The constants are fit from data. By fitting this parametric form, Hoffmann et al. [43] find that scaling exponents and are close, suggesting that one should scale equally as compute increases. Hence, we assume . With this assumption, we reparameterize Equation () in terms of compute and a token multiplier . We get the following form,

where , , gives the relation to Equation (). For a complete derivation, see Appendix .

Equation () has the following interpretation: (i) The scaling exponent is not dependent on . Thus, we always expect lines with the same slope in the - plot—as in Figure . (ii) The term determines the offsets between curves with different token multipliers. Hence, we expect non-overlapping, parallel lines in the - plot for the range of we consider—also consistent with Figure .

Recall that we make the assumption , which implies equal scaling of parameters and tokens as more compute is available. However, as explained in Appendix , even if , we get a parameterization that implies the power-law exponent remains constant with over-training.

Observation: Loss tracks average top-1 error

Scaling is typically studied in the context of loss [49, 43, 70], which Schaeffer et al. [98] note is smoother than metrics like accuracy. However, practitioners ultimately care about downstream, in-the-wild task performance. To better connect scaling laws and over-training to task prediction, we revisit the suite of models plotted in Figure . In Figure , we plot average downstream top-1 errors over evaluations sourced from LLM-Foundry [67] against the C4 eval loss. We defer details of the setup to Section to focus here on a few key observations. The average errors appear to follow exponential decay as loss decreases. Additionally, the particular relationship between loss and error is dataset-dependent. For instance, models trained on C4 result in the lowest C4 eval loss, but this does not translate to downstream gains compared to models trained on RedPajama or RefinedWeb.

Proposing a scaling law for average top-1 error

Based on the exponential decay we observe in Figure , we propose the following relationship between downstream average top-1 error and loss ,

where are fit from data. Equation () also has an appealing interpretation in terms of model perplexity ,

Namely, follows a power law in with maximum error , where intuitively should be close to the random chance performance.

Equation () in conjunction with () suggests a two-step method to predict as a function of compute and the amount of over-training. For choices of training and validation distributions, (i) fit a scaling law to Equation () using pairs to yield . (ii) Fit a scaling law to Equation () using pairs to get .

Experimental setup

Towards testing the analytic predictions in Equations (), (), we discuss our experimental setup. We first present language modeling details (Section ). Next, we discuss our strategy for deciding which models to include in our scaling investigation and our procedure for fitting scaling trends (Section ). We then present metrics to validate how well scaling laws predict loss and downstream performance (Section ).

Training setup

Architecture.

We train transformers [114], based on auto-regressive, decoder-only, pre-normalization architectures like GPT-2 [83] and LLaMA [111]. We adopt OpenLM [37] as our core modeling library, which utilizes PyTorch [78, 6], xformers [52], triton [73], FlashAttention [22], FSDP [127], and bfloat16 automatic mixed precision. Like LLaMA, we omit bias terms, but replace RMSNorm [125] with LayerNorm [8], which has readily available fused implementations. Following Wortsman et al. [120], we apply qk-LayerNorm [23], which adds robustness to otherwise poor hyperparameter choices (e.g., for learning rate). We use SwiGLU [100] activations and depth-scaled initialization [126]. We use a sequence length of 2048, rotary positional embeddings [104], and the GPT-NeoX-20B tokenizer [13], which yields a vocabulary size of 50k. We do not use weight tying [82, 44].

Objectives and optimization.

We train with a standard causal language modeling objective (i.e., next token prediction) with an additive z-loss [17] (coefficient 1-4), which mitigates output logit norm growth [65] instabilities. We use the AdamW optimizer [60] (PyTorch defaults except = 0.95), with independent weight decay [120] (coefficient 1-4). For the learning rate schedule, we use linear warmup and cosine decay. We cool down to a low learning rate (3-5).

Training datasets.

To ensure our conclusions are not particular to a training distribution, we train models on C4 [86, 25], RedPajama [110], and RefinedWeb [80]. They are open-source and have 138B, 1.15T, and 600B tokens respectively. We sample without replacement and employ sequence packing without attention masking. We separate documents in our training corpora with end-of-text tokens.

Creating scaling laws for validation loss and downstream error prediction

Recipe for model configuration selection.

To create a testbed of models for our scaling experiments, we grid search over a total of 435 models, trained from scratch, in the 0.01B to 0.5B parameter range as seen in Figure (left). We train on the OpenLM data mix [37], which largely consists of tokens from RedPajama [110] and The Pile [29]. We train on 20 tokens per parameter (), which we find in early experiments gives models near the compute-optimal frontier for the data mix. This is similar to findings presented in Hoffmann et al. [43]’s Table 3, which suggests that roughly 20 tokens per parameter are optimal in their experimental setup. Our validation set, OpenLM eval, contains tokens from recent arXiv papers, the OpenLM codebase itself, and news articles. To find maximally performant models on validation data, we tune model width, number of layers, number of attention heads, warmup steps, and batch size. We find in early experiments that qk-LayerNorm makes models less sensitive to learning rate, which is a phenomenon Wortsman et al. [120] report in their Figure 1. Hence, we fix the learning rate for our sweeps. We also perform smaller grid searches over 1.4B and 6.9B parameter model configurations at , retaining the best configurations.

In Figure (center), we plot the efficient frontier of minimum loss configurations. While there appears to be a general trend, configurations between and FLOPs lie below the frontier established by other models. We hypothesize these models over-perform as they are trained for more optimization steps than their neighbors based on our power-of-two batch sizes. We provide support for this hypothesis in Appendix , but opt to remove these models from our investigation.

In Figure (right), we fit trends to the remaining models and to a subset of four models. We notice that the trends hit both the 1.4B and 6.9B models suggesting that our small-scale configurations are reasonable for extrapolation to larger parameter and compute regimes. We retain the four model configuration subsets as a representative sample.

We do not tune hyperparameters for other token multipliers (i.e., ), other training or evaluation distributions, or on downstream task validation sets. For more details, see Appendix . We present our final hyperparameters in Table given their importance.

Models have number of parameters , with number of layers , number of attention heads , model width , and width per attention head . Batch sizes are global and in units of sequences. Each sequence has 2048 tokens. A100 GPU hours are at , which are near compute-optimal runs. For the 1.4B scale, a batch size of 256 performs slightly better than 512.

Fitting scaling laws.

We invest 100 A100 hours to fit Equation () and 1000 A100 hours to fit Equation ().

We fit Equation () to approximate using curve-fitting in SciPy [115] (i.e., Levenberg-Marquardt to minimize non-linear least squares). We try several initializations and retain the best fit. We repeat this process to fit Equation () to approximate . Unless otherwise specified, we fit to the pairs in Table . In total, we invest 100 A100 hours to train the models required for fitting an accurate scaling law for loss prediction and 1000 A100 hours for a corresponding scaling law for downstream error prediction. Our configurations allow us to test for extrapolation to the (900B token) and the (138B token) regimes.

Evaluation setup

Evaluation datasets.

Unless otherwise stated, C4 eval is our default validation loss dataset. For downstream tasks, we adopt 46 tasks from LLM-foundry [67], which includes standard tasks with both zero-shot and few-shot evaluations. We also consider a 17-task subset where, for each evaluation, at least one 0.154B scale model—trained with as many as 99B tokens—gets 10 percentage points above chance accuracy: ARC-Easy [21], BIG-bench: CS algorithms [11], BIG-bench: Dyck languages [11], BIG-bench: Novel Concepts [11], BIG-bench: Operators [11], BIG-bench: QA WikiData [11], BoolQ [19], Commonsense QA [105], COPA [90], CoQA [89], HellaSwag (zero-shot) [123], HellaSwag (10-shot) [123], LAMBADA [75], PIQA [12], PubMed QA Labeled [48], SQuAD [88], and WinoGrand [53]. This subset allows us to investigate a regime where performance for small models can be non-trivial. For more details on evaluation datasets see Appendix . For ablations on our choices of loss and downstream evaluations see Appendix .

Metrics.

We consider three main metrics: (i) Validation loss, which is the cross entropy between a model’s output and the one-hot ground truth, averaged over all tokens in a sequence and over all sequences in a dataset. (ii) Average top-1 error, which is a uniform average over 46 downstream evaluations sourced from LLM-foundry [67]. We also look at the mean top-1 error for the subset of 17 evaluations identified in the paragraph above. For a complete list of downstream evaluation datasets, see Appendix . To measure how good a prediction is, we measure (iii) Relative prediction error: , where is the loss or the average top-1 error .

Testbed.

We train models on C4, RedPajama, and RefinedWeb with the number of parameters and token multipliers . We omit runs that require more tokens than are present in a dataset (i.e., for C4). We additionally train B models at and at the largest token multiplier possible without repeating tokens (i.e., 80 for C4, 640 for RedPajama, and 320 for RefinedWeb). We train for each dataset. In total this results in 104 models. We evaluate each model on C4 eval for validation loss and on the 46 downstream tasks for top-1 error.

Results

Unless otherwise stated, we fit Equations (), () to the configurations in Table and use C4 eval for loss computation.

Over-trained performance is predictable.

We highlight our main over-training results in Figure (left). Namely, we are able to extrapolate both in the number of parameters and the token multiplier to closely predict the C4 eval performance of a 1.4B parameter model trained on 900B RedPajama tokens (). Our prediction, which takes 300 less compute to construct than the final 1.4B run, is accurate to within 0.7% relative error. Additionally, for the run, near compute-optimal, the relative error is also 0.7%.

These results support several key takeaways. (i) scaling can be predictable even when one increases the model size and the amount of over-training compared to the training runs used to fit a scaling law. (ii) Scaling can be predictable even in the presence of a distribution shift (e.g., RedPajama training and C4 evaluation). (iii) The form presented in Equation () is useful in practice for fitting and predicting over-trained scaling behavior. (iv) Fitting to Equation () does not sacrifice prediction accuracy near compute-optimal.

While Figure explores a specific case of making predictions in the over-trained regime, we would like to understand the error profile of our predictions across datasets, token multipliers, and number of parameters. Hence, in Figure we show relative error between ground truth loss and predicted loss on C4 eval for models in our testbed. We notice uniformly low prediction error suggesting that predictions are accurate in many settings.

Average top-1 error is predictable.

Figure (right) presents our results in estimating scaling laws for downstream error. Similar to Figure (left), we are able to extrapolate in and predict the average downstream error across our evaluation suite. Concretely, we use the models indicated in Table to fit Equation (), and predict the average top-1 error over the 17 tasks identified in Section . Our fit allows us to predict the downstream performance of a 6.9B parameter model trained on 138B tokens to within relative error and of a 1.4B model trained on 900B tokens to within relative error, using less compute.

Table additionally shows the relative error of our downstream performance predictions for models trained on C4, RedPajama, and RefinedWeb, indicating that our scaling law fits are applicable to other datasets. We note that while average accuracy across benchmarks is predictable, predicting accuracy on downstream tasks is significantly more noisy. We report relative error of all our predictions in Figures , in the Appendix. We also find that if we remove the 1.4B model for the Equation () fit, relative error jumps, for instance, from 0.05% to 10.64% on the 17-task split for the 6.9B, 138B token RedPajama prediction.

Small-scale experiments can predict model rank order.

We expect to be able to rank models based on their predicted performance, which is useful when deciding what to train. To verify, we rank 9 testbed models with by ground-truth top-1 error and by estimated top-1 error. We find high rank correlations: 0.93 and 0.88 for the 46 and 17-task splits respectively.

Under-training, out-of-distribution scaling, and compute-reliability trade-offs.

In addition to our main results presented above, we include additional results in Appendix , which we summarize here. First, we notice that when token multipliers become too small (i.e., ) scaling becomes unreliable and lies off the trend. Additionally, several multipliers (10, 20, 40, and 80) garner points that are roughly on the compute optimal frontier (Figure ). To probe the limits of reliable scaling, we attempt to break our scaling laws in out-of-distribution settings. We find that models trained on C4—English filtered—and evaluated on next token prediction on code domains have a high relative error in many cases. Perhaps surprisingly, evaluating the same models on German next token prediction again gives reliable loss scaling (Figure ). We additionally examine the compute necessary to create accurate scaling laws, finding a positive correlation between investing more compute in a scaling law and its predictivity. We find that scaling laws can be constructed more cheaply for loss prediction than for downstream error prediction (Figures , ).

Related work

We review the most closely related work in this section. For additional related work, see Appendix .

In our work, we run experiments with token multipliers between 5 and 640 for (GPT-2 [83], LLaMA [111])-style decoder-only architectures.

Scaling laws.

Early works on scaling artificial neural networks observe predictable power-law scaling in the training set size and number of model parameters [41, 42, 91]. Alabdulmohsin et al. [2] center the importance of looking at the extrapolation regime of a scaling law. Yang et al. [121] prescribe architectural and hyperparameter changes when scaling model width to realize performant models; Yang et al. [122] make analogous recommendations when scaling model depth. Unlike the aforementioned work, our investigation focuses on the link between over-training models and predicting their downstream performance on accuracy metrics.

Hoffmann et al. [43] investigate how the number of model parameters and training tokens should be chosen to minimize loss given a compute budget . Hoffmann et al. [43] find that when scaling up , both and should be scaled equally up to a multiplicative constant (i.e., and ) to realize compute-optimality. Appendix C of the Chinchilla paper additionally suggests that these findings hold across many datasets. However, Hoffmann et al. [43] do not account for inference costs, provide scaling laws for training beyond compute-optimal, or for downstream error prediction—all of which are central to our work.

Sardana & Frankle [96] proposed modifications to the Chinchilla formulation to incorporate inference costs into the definition of compute-optimal and solve for various fixed inference budgets. Their key finding, which is critical for our work, is that when taking into account a large enough inference budget, it is optimal to train smaller models for longer when compared to the original Chinchilla recommendations. Our work presupposes that over-training can be beneficial. Instead of solving for inference-optimal schemes, we support empirically a predictive theory of scaling in the over-trained regime. Additionally, we provide experiments across many validation and training sets.

For predicting downstream scaling beyond loss, Isik et al. [45] relate the number of pre-training tokens to downstream cross-entropy and machine translation BLEU score [76] after fine-tuning. In contrast, we do not examine a specific domain but rather take a holistic approach to evaluation by looking at top-1 error over many natural language tasks. Schaeffer et al. [98] argue that emergent abilities [118] and unreliable scaling are a product of non-linear metrics like error/accuracy and propose smoother alternatives. As an explanation for why non-linear metrics may be hard to predict, Schaeffer et al. [98] consider predicting exactly an length sequence: , where is the number of parameters in a model and is its perplexity. This is a special case of our Equations (), (), where number of training tokens is ignored, , and . In contrast, we treat as free parameters for a scaling law fit and embrace top-1 error, finding that average error over downstream tasks can make for a predictable metric.

Over-training in popular models.

There has been a rise in over-trained models [111, 112] and accompanying massive datasets [110, 80, 102, 3]. To contextualize the extent to which we over-train, we provide token multipliers for popular models in Table . For example, Chinchilla 70B [43] is trained with a token multiplier of 20, while LLaMA-2 7B [112] uses a token multiplier of 290. In our investigation, we look at token multipliers from 5 to 640 to ensure coverage of popular models and relevance for future models that may be trained on even more tokens.

Limitations, future work, and conclusion

Limitations and future work.

We identify limitations, which provide motivation for future work.

While our configurations are surprisingly amenable to reliable scaling across many training and testing distributions without further tuning, there is a need to further develop scaling laws that incorporate hyperparameters.
Validating the trends in this paper for even larger runs is a valuable direction.
Additionally, actualizing predictable scaling with even cheaper runs is important to make this area of research more accessible, especially for downstream error prediction.
While we present preliminary analysis of when scaling is unreliable, future work should develop an analytic theory explaining when scaling breaks down.
It is common to employ supervised fine-tuning and reinforcement learning after pre-training, which we do not consider. Quantifying to what degree over-training the base model provides benefits after post-training is an open area of research.
While we find that averaging over many task error metrics can make for a predictable metric, per-task predictions are left to future work.
Downstream task performance is a proxy for the in-the-wild user experience. Analyzing scaling trends in the context of this experience is timely.
Our work only deals with existing training datasets. Exploring dataset curation for improved model scaling is another promising direction.

Conclusion.

We (i) show that the loss scaling behavior of models trained past compute-optimal, in the over-trained regime, is predictable and (ii) predict, via a proposed scaling law, the downstream average task performance of more expensive runs using smaller-scale proxies. We hope our work will inspire others to further examine the relationship between model training and downstream generalization. We also hope our testbed will make scaling research more accessible to researchers and practitioners alike.

Acknowledgements

SYG is supported by an NSF Graduate Research Fellowship, GS by the Onassis Foundation - Scholarship ID: F ZS 056-1/2022-2023, and MN by the Federal Ministry of Education and Research of Germany under grant no. 01IS22094B WEST-AI. We thank Stability AI and Toyota Research Institute (TRI) for access to compute resources. This research has been supported by NSF Grants AF 1901292, CNS 2148141, Tripods CCF 1934932, IFML CCF 2019844, and research gifts by Western Digital, Amazon, WNCG IAP, UT Austin Machine Learning Lab (MLL), Cisco, and the Stanly P. Finch Centennial Professorship in Engineering. We also thank Kushal Arora, Alper Canberk, Mia Chiquier, Sachit Menon, Chuer Pan, Purva Tendulkar, and Mandi Zhao for valuable feedback.

References

Contributions

Names ordered alphabetically.

Planning.

Model training and experiment babysitting.

Achal Dave (notably, the 1.4B parameter, 900B token run), Samir Yitzhak Gadre

Dataloading.

Training tokens.

Achal Dave, Alex Fang, Samir Yitzhak Gadre, Suchin Gururangan, Jeffrey Li, Vaishaal Shankar (lead), Mitchell Wortsman

Evaluation tokens.

Achal Dave, Samir Yitzhak Gadre, Reinhard Heckel, Vaishaal Shankar (lead), Rulin Shao

Loss/perplexity evaluation.

Downstream evaluation.

Project-specific infrastructure, plots, and analysis.

OpenLM [37] open-source infrastructure.

Achal Dave (core contributor), Alex Fang, Samir Yitzhak Gadre (core contributor), Suchin Gururangan (core contributor), Jenia Jitsev, Sedrick Keh, Jeffrey Li, Jean Mercat, Marianna Nezhurina, Vaishaal Shankar (core contributor), Georgios Smyrnis (core contributor), Igor Vasiljevic, Mitchell Wortsman (core contributor), Rui Xin

Theory.

Yair Carmon (original idea that “parallel lines” should show up in scaling plots), Samir Yitzhak Gadre (various derivations, empirical verification, related validation loss to average top-1 error as in Equation ()), Reinhard Heckel (derived a scaling form based on Chinchilla Approach 3 [43], which appears in Equation ()), Mitchell Wortsman (provided intuition about irreducible loss and why it is critical), Niklas Muennighoff (derived a scaling form based on Chinchilla Approach 3, similar to Equation ()).

Writing.

Yair Carmon, Achal Dave, Reinhard Heckel, Samir Yitzhak Gadre (lead), Niklas Muennighoff, Ludwig Schmidt

Compute.

Achal Dave, Jenia Jitsev, Thomas Kollar, Ludwig Schmidt, Vaishaal Shankar

Advising.

Yair Carmon (co-lead), Achal Dave (co-lead), Alexandros G. Dimakis, Reinhard Heckel (co-lead), Gabriel Ilharco, Jenia Jitsev, Thomas Kollar, Niklas Muennighoff (co-lead), Ludwig Schmidt (co-lead), Shuran Song

Scaling-law derivations

We first show that reparameterizing Equation () in terms of the compute and token multiplier for yields Equation (). Combining and yields and . Inserting these into Equation () yields,

This is equal to Equation (), making the substitutions , , , as noted in the main body.

Relation to compute-optimal training.

Recall that we made the assumption , which implies equal scaling of parameters and tokens to realize compute-optimal models. While this assumption is empirically justified [43], even if , we get a parameterization that implies the power law exponent in Equation () remains constant with over-training, while the power law scalar changes.

To find a compute-optimal training setting, Hoffmann et al. [43] propose to minimize the right-hand side of Equation () subject to the compute constraint . This yields, and , where , for notational convenience. The associated risk is,

We now deviate from compute-optimal training by modifying the model size and tokens by multiplication with a constant , according to

This modification keeps the compute constant (i.e., ). The risk, then, becomes

We again expect the same power law exponent and changing power law scalar. Note that in Equation () is similar to in Equation (). Specifically, is a multiple of the Chinchilla-optimal token multiplier , which is no longer fixed as a compute budget changes for .

Additional grid search details

Grid search configuration selection.

Recall in Section , we run a grid search over many configurations. We present the architectures we sweep over in Table .

Evaluation dataset details

All 46 downstream evaluations are based on MosaicML’s LLM-foundry evaluation suite [67]. We specifically consider the datasets given in Table .

Additional results

Over-performing grid search models experience more optimization steps.

As mentioned in Section and Figure , we notice that models between 0.011B to 0.079B (i.e., to FLOPs trained near compute-optimal) over-perform compared to the trend established by other models in our initial grid searches. This results in a bump in the scaling plot. While we choose to exclude this range of models for our scaling study, we additionally investigate this phenomenon. In Figure we color grid search configurations by the number of optimization steps (i.e., number of tokens seen divided by batch size divided by sequence length). For context, Figure 1 (left) in Kaplan et al. [49] also shows a bump; however, there the performance is worse than the general trend instead of better as in our work. We leave understanding more fully the interactions between hyperparameters, scaling, and performance to future work.

Scaling is largely predictable in-distribution (ID).

Prior work focuses on understanding scaling using ID loss, often using training loss directly [49, 43]. Hence, we also consider Paloma [63] loss evaluation sets, which are designed to probe performance in specific domains. We use Paloma’s C4 [86, 25], RedPajama [110], and Falcon-RefinedWeb [80] splits to probe for ID validation loss. As seen in Figure , relative error is mostly low. Relative error is largest for the RedPajama run at 15.4%. Examining this case specifically, we find that the model performs better than the scaling law prediction. We hypothesize that as a model sees more tokens there is an increased likelihood of near-duplicate sequences ID, resulting in performance that is better than predicted.

Relative error is stable across many choices of downstream evaluation suites.

To understand how sensitive our investigation is to our choices of the evaluation set, we consider several other options as seen in Figure . We find that our prediction errors are fairly (i) low and (2) consistent for many choices of downstream evaluation sets.

Scaling can break down when under-training.

We find that when a token multiple is too small (i.e., under-training regime), scaling appears unreliable. In Figure we see for the scaling trend is different. We hypothesize that tuning hyperparameters (e.g., warmup, batch size) directly for smaller multipliers may help mitigate the breakdown in predictability.

Scaling can be unpredictable out-of-distribution (OOD).

Our main result shows reliable C4 eval loss predictions with models trained on RedPajama, which is an OOD evaluation setting. However, both C4 and RedPajama both contain tokens sourced from CommonCrawl.

To further probe OOD performance, we measure the relative error of scaling laws fit to models trained on C4 and evaluated on Paloma’s 100 programming languages [63], Paloma’s Penn Tree Bank (PTB) split [64], and a German version of C4 [25]. Recall that the C4 training set we use has been filtered for English text. Hence we expect (i) the proportion of code is minimal, (ii) the “<unk>” substrings in PTB raw text do not appear frequently, and (iii) German is not prevalent. We notice that extrapolation relative error tends to be high for large on programming languages and PTB (Figure (left, center)). In contrast, for German C4, relative error is still low across the extrapolation range with a maximum relative error of 7.6% at the 1.4B, scale (Figure (right)). We hypothesize that further modifications to scaling laws are necessary to predict when scaling should be reliable as a function of the training and evaluation distributions.

Small-scale experiments can predict average downstream top-1 error.

To verify that chaining Equations () and () is effective in practice, we collect C4 eval loss and downstream error pairs for the configurations in Table . In Figure , we look at relative error for our scaling predictions in the context of Average top-1 error over our 46 evals and over our 17 evals in Figure . We again notice reliable scaling in interpolation and extrapolation regimes, suggesting the validity of our procedure to predict downstream average top-1 error.

Loss evaluation ablations for downstream trends.

Figure presents the correlation between downstream error vs. loss evaluated on different validation sets (C4, RedPajama, and RefinedWeb). Regardless of the validation set (x-axis), models follow the exponential decay relationship given in Equation (), suggesting the choice of validation loss is not critical for the appearance of this phenomenon.

Investing more compute in a scaling law makes it more predictive.

Thus far we have looked at standard configurations from Table to construct our scaling laws, mainly to demonstrate extrapolation to larger . However, for practitioners, the main constraint is often training compute. Hence, we wish to understand the trade-offs between the amount of compute invested in creating a scaling law and the relative error of the resulting law in the over-trained regime. In Figure (left), we see that as one increases the amount of compute, it is possible to get better fits with lower relative error. In Figure (right), we see a similar trend as one increases the number of data points used to fit a scaling law. Blue stars indicate the configurations from Table , which provide accurate predictions relative to the general trends—hinting at their usefulness for our investigation. In Figures , we repeat the compute analysis comparing trade-offs for loss prediction and error prediction reliability. We find that less compute is generally necessary to construct a loss scaling law that achieves the same relative error as that of an error prediction scaling law.

Additional related work

Language modeling.

Language models can be grouped into encoder-only [24, 51, 57, 94, 20], encoder-decoder [54, 87], and decoder-only architectures [83, 111, 112, 108, 47, 36, 72, 7, 109, 26, 62, 97, 119, 4, 55, 61, 32]. Most current implementations are based on the transformer [114]. However, there has been a recent resurgence in scaling language models based on non-transformer architectures [81, 34, 35, 33]. Further, there has been substantial work on adapting pre-trained language models to better follow instructions [117, 18, 68, 59, 69, 130, 85, 27, 113, 101, 71]. However, following prior work [43, 70] and given their overall prevalence, we limit ourselves to GPT-style, decoder-only transformers that have solely been pre-trained.

Scaling laws.

Kaplan et al. [49] investigate scaling trends in GPT language models. Bahri et al. [9] investigate different scaling regimes theoretically, and Sharma & Kaplan [99] relate scaling coefficients to data manifold dimensions. Tay et al. [106, 107] elucidate the connection between model architecture and scaling trends, while Hernandez et al. [40], Tay et al. [106] develop scaling laws for transfer learning. Ivgi et al. [46] also consider transfer learning scaling laws and highlight the importance of hyperparameter selection in the low-compute regime. Ghorbani et al. [30], Gordon et al. [31], Bansal et al. [10] develop scaling laws for neural machine translation. Caballero et al. [15] propose a scaling law functional form, which they demonstrate is predictive in several domains.

Scaling beyond language modeling.

There is a large body of work on scaling neural networks beyond language modeling, for example in computer vision [58, 124, 103, 1, 2], multimodal learning [39, 16, 28], and image reconstruction [50].

||||I|||| Back to arXiv Back to arXiv This is experimental HTML to improve accessibility. We invite you to report rendering errors. Use Alt+Y to toggle on accessible reporting links and Alt+Shift+Y to toggle off. Learn more about this project and help improve conversions. Why HTML? Report Issue Back to Abstract Download PDF Table of Contents 1. 1 Introduction 2. 2 Scaling and over-training 1. 2.1 Preliminaries 1. Scaling laws for loss. 2. Token multipliers. 3. Compute-optimal training. 4. Over-training. 2. 2.2 Observation: Over-trained models follow consistent trends 3. 2.3 Deriving scaling laws for over-trained behavior 4. 2.4 Observation: Loss tracks average top-1 error 5. 2.5 Proposing a scaling law for average top-1 error 3. 3 Experimental setup 1. 3.1 Training setup 1. Architecture. 2. Objectives and optimization. 3. Training datasets. 2. 3.2 Creating scaling laws for validation loss and downstream error prediction 1. Recipe for model configuration selection. 2. Fitting scaling laws. 3. 3.3 Evaluation setup 1. Evaluation datasets. 2. Metrics. 3. Testbed. 4. 4 Results 1. Over-trained performance is predictable. 2. Average top-1 error is predictable. 3. Small-scale experiments can predict model rank order. 4. Under-training, out-of-distribution scaling, and compute-reliability trade-offs. 5. 5 Related work 1. Scaling laws. 2. Over-training in popular models. 6. 6 Limitations, future work, and conclusion 1. Limitations and future work. 2. Conclusion. 7. A Contributions 1. Planning. 2. Model training and experiment babysitting. 3. Dataloading. 4. Training tokens. 5. Evaluation tokens. 6. Loss/perplexity evaluation. 7. Downstream evaluation. 8. Project-specific infrastructure, plots, and analysis. 9. OpenLM [ 37 ] open-source infrastructure. 10. Theory. 11. Writing. 12. Compute. 13. Advising. 8. B Scaling-law derivations 1. Relation to compute-optimal training. 9. C Additional grid search details 1. Grid search configuration selection. 10. D Evaluation dataset details 11. E Additional results 1. Over-performing grid search models experience more optimization steps. 2. Scaling is largely predictable in-distribution (ID). 3. Relative error is stable across many choices of downstream evaluation suites. 4. Scaling can break down when under-training. 5. Scaling can be unpredictable out-of-distribution (OOD). 6. Small-scale experiments can predict average downstream top-1 error. 7. Loss evaluation ablations for downstream trends. 8. Investing more compute in a scaling law makes it more predictive. 12. F Additional related work 1. Language modeling. 2. Scaling laws. 3. Scaling beyond language modeling. HTML conversions sometimes display errors due to content that did not convert correctly from the source. This paper uses the following packages that are not yet supported by the HTML conversion tool. Feedback on these issues are not necessary; they are known and are being worked on. * failed: cellspace * failed: xpatch Authors: achieve the best HTML results from your LaTeX submissions by following these best practices. License: CC BY 4.0 arXiv:2403.08540v1 [cs.CL] 13 Mar 2024 Language models scale reliably with over-training and on downstream tasks Samir Yitzhak Gadre 1,212{}^{1,2}start_FLOATSUPERSCRIPT 1 , 2 end_FLOATSUPERSCRIPT Georgios Smyrnis 33{}^{3}start_FLOATSUPERSCRIPT 3 end_FLOATSUPERSCRIPT Vaishaal Shankar 44{}^{4}start_FLOATSUPERSCRIPT 4 end_FLOATSUPERSCRIPT Suchin Gururangan 55{}^{5}start_FLOATSUPERSCRIPT 5 end_FLOATSUPERSCRIPT Mitchell Wortsman 55{}^{5}start_FLOATSUPERSCRIPT 5 end_FLOATSUPERSCRIPT Rulin Shao 55{}^{5}start_FLOATSUPERSCRIPT 5 end_FLOATSUPERSCRIPT Jean Mercat 22{}^{2}start_FLOATSUPERSCRIPT 2 end_FLOATSUPERSCRIPT Alex Fang 55{}^{5}start_FLOATSUPERSCRIPT 5 end_FLOATSUPERSCRIPT Jeffrey Li 55{}^{5}start_FLOATSUPERSCRIPT 5 end_FLOATSUPERSCRIPT Sedrick Keh 22{}^{2}start_FLOATSUPERSCRIPT 2 end_FLOATSUPERSCRIPT Rui Xin 55{}^{5}start_FLOATSUPERSCRIPT 5 end_FLOATSUPERSCRIPT Marianna Nezhurina 6,767{}^{6,7}start_FLOATSUPERSCRIPT 6 , 7 end_FLOATSUPERSCRIPT Igor Vasiljevic 22{}^{2}start_FLOATSUPERSCRIPT 2 end_FLOATSUPERSCRIPT Jenia Jitsev 6,767{}^{6,7}start_FLOATSUPERSCRIPT 6 , 7 end_FLOATSUPERSCRIPT Alexandros G. Dimakis 33{}^{3}start_FLOATSUPERSCRIPT 3 end_FLOATSUPERSCRIPT Gabriel Ilharco 55{}^{5}start_FLOATSUPERSCRIPT 5 end_FLOATSUPERSCRIPT Shuran Song 88{}^{8}start_FLOATSUPERSCRIPT 8 end_FLOATSUPERSCRIPT Thomas Kollar 22{}^{2}start_FLOATSUPERSCRIPT 2 end_FLOATSUPERSCRIPT Yair Carmon 9⁣*9{}^{9*}start_FLOATSUPERSCRIPT 9 * end_FLOATSUPERSCRIPT Achal Dave 2⁣*2{}^{2*}start_FLOATSUPERSCRIPT 2 * end_FLOATSUPERSCRIPT Reinhard Heckel 10⁣*10{}^{10*}start_FLOATSUPERSCRIPT 10 * end_FLOATSUPERSCRIPT Niklas Muennighoff 11⁣*11{}^{11*}start_FLOATSUPERSCRIPT 11 * end_FLOATSUPERSCRIPT Ludwig Schmidt 5⁣*5{}^{5*}start_FLOATSUPERSCRIPT 5 * end_FLOATSUPERSCRIPT Abstract Scaling laws are useful guides for developing language models, but there are still gaps between current scaling studies and how language models are ultimately trained and evaluated. For instance, scaling is usually studied in the compute-optimal training regime (i.e., “Chinchilla optimal” regime); however, in practice, models are often over-trained to reduce inference costs. Moreover, scaling laws mostly predict loss on next-token prediction, but ultimately models are compared based on downstream task performance. In this paper, we address both shortcomings. To do so, we create a testbed of 104 models with 0.011B to 6.9B parameters trained with various numbers of tokens on three data distributions. First, we investigate scaling in the over-trained regime. We fit scaling laws that extrapolate in both the number of model parameters and the ratio of training tokens to parameters. This enables us to predict the validation loss of a 1.4B parameter, 900B token run (i.e., 32×\times× over-trained) and a 6.9B parameter, 138B token run—each from experiments that take 300×\times× less compute. Second, we relate the perplexity of a language model to its downstream task performance via a power law. We use this law to predict top-1 error averaged over downstream tasks for the two aforementioned models using experiments that take 20×\times× less compute. Our experiments are available at https://github.com/mlfoundations/scaling. 00 footnotetext: Equal advising, ordered alphabetically. Correspondence to sy@cs.columbia.edu . 11{}^{1}start_FLOATSUPERSCRIPT 1 end_FLOATSUPERSCRIPT Columbia University 22{}^{2}start_FLOATSUPERSCRIPT 2 end_FLOATSUPERSCRIPT Toyota Research Insitute 33{}^{3}start_FLOATSUPERSCRIPT 3 end_FLOATSUPERSCRIPT UT Austin 44{}^{4}start_FLOATSUPERSCRIPT 4 end_FLOATSUPERSCRIPT Apple 55{}^{5}start_FLOATSUPERSCRIPT 5 end_FLOATSUPERSCRIPT University of Washington 66{}^{6}start_FLOATSUPERSCRIPT 6 end_FLOATSUPERSCRIPT Juelich Supercomputing Center, Research Center Juelich 77{}^{7}start_FLOATSUPERSCRIPT 7 end_FLOATSUPERSCRIPT LAION 88{}^{8}start_FLOATSUPERSCRIPT 8 end_FLOATSUPERSCRIPT Stanford University 99{}^{9}start_FLOATSUPERSCRIPT 9 end_FLOATSUPERSCRIPT Tel Aviv University 1010{}^{10}start_FLOATSUPERSCRIPT 10 end_FLOATSUPERSCRIPT TU Munich 1111{}^{11}start_FLOATSUPERSCRIPT 11 end_FLOATSUPERSCRIPT Contextual AI 1 Introduction Figure 1 : Reliable scaling in the over-trained regime and for downstream error prediction. (left) We fit a scaling law for model validation loss, parameterized by (i) a token multiplier M𝑀Mitalic_M , which is the ratio of training tokens D𝐷Ditalic_D to parameters N𝑁Nitalic_N (i.e., M=D/N𝑀𝐷𝑁M=D/Nitalic_M = italic_D / italic_N ) and (ii) the compute C𝐶Citalic_C in FLOPs used to train a model, approximated by C=6⁢N⁢D𝐶6𝑁𝐷C=6NDitalic_C = 6 italic_N italic_D . We extrapolate, in both N𝑁Nitalic_N and M𝑀Mitalic_M , the validation performance of models requiring over 300 ×\times× the training compute used to construct the scaling law. (right) We also fit a scaling law to predict average downstream top-1 error as a function of validation loss. We find that fitting scaling laws for downstream error benefits from using more expensive models when compared to fitting for loss prediction. We predict the average error over 17 downstream tasks for models trained with over 20 ×\times× the compute. Training large language models is expensive. Moreover, training high-quality models requires a complex recipe of algorithmic techniques and training data. To reduce the cost of finding successful training recipes, researchers first evaluate ideas with small experiments and then extrapolate their efficacy to larger scales. With reliable extrapolation, it is possible to quickly iterate at small scale and still pick the method that will perform best for the final large training run. Indeed, this workflow has become commonplace for training state-of-the-art language models such as Chinchilla 70B [43], PaLM 540B [17], and GPT-4 [74]. Despite their importance for model development, published scaling laws differ from the goals of training state-of-the-art models in important ways. For instance, scaling studies usually focus on the compute-optimal training regime (“Chinchilla optimality” [43]), while widely used models are now often over-trained to reduce inference costs. Another potential mismatch between scaling laws and eventual applications of the models is that most scaling laws quantify model performance by perplexity in next-token prediction instead of accuracy on widely used benchmark datasets. As a result, it is unclear whether following scaling laws leads to truly better models, or merely to models with lower perplexity in the compute-optimal training regime. In this paper, we address both topics: scaling in the over-trained regime and downstream performance prediction, with an extensive set of experiments. Motivated by the practice of training beyond compute-optimal, we first investigate whether scaling follows reliable trends in the over-trained regime. We find that for a set of model configurations with a constant ratio of training tokens to parameters, the models’ reducible loss L′superscript𝐿′L^{\prime}italic_L start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPT [41, 43] follows consistent power laws (L′=λ⋅C−αCsuperscript𝐿′⋅𝜆superscript𝐶subscript𝛼𝐶L^{\prime}=\lambda\cdot C^{-\alpha_{C}}italic_L start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPT = italic_λ ⋅ italic_C start_POSTSUPERSCRIPT - italic_α start_POSTSUBSCRIPT italic_C end_POSTSUBSCRIPT end_POSTSUPERSCRIPT) in the amount of training compute C𝐶Citalic_C. As one increases the ratio of tokens to parameters, corresponding to more over-training, the scaling exponent αCsubscript𝛼𝐶\alpha_{C}italic_α start_POSTSUBSCRIPT italic_C end_POSTSUBSCRIPT remains about the same, while the scalar λ𝜆\lambdaitalic_λ changes. We consider the extent to which our observations are explainable and find a promising approach by reparameterizing forms for scaling laws. To establish if and when scaling is predictable in the over-trained regime, we experiment with a testbed of 104 models, trained from scratch on three different datasets: RedPajama [110], C4 [86, 25], and RefinedWeb [80]. We find that scaling laws fit on small models, trained closer to compute-optimal, can accurately predict the performance of larger models that undergo more over-training. Figure 1 (left) illustrates our main over-training result, where we invest 2.4⁢e⁢192.4𝑒192.4e192.4 italic_e 19 FLOPs to extrapolate the C4 validation performance of a 1.4B parameter model trained on 900B tokens, which requires 300×300\times300 × more compute to train. In addition to over-training, we also investigate if scaling laws can predict the performance of a model on downstream tasks. We establish a power law relationship between language modeling perplexity and the average top-1 error on a suite of downstream tasks. While it can be difficult to predict the error on individual tasks, we find that aggregate performance can be accurately predicted from a model’s perplexity among models trained on the same training data. Figure 1 (right) presents our main downstream error prediction result, where we invest 2.7⁢e⁢202.7𝑒202.7e202.7 italic_e 20 FLOPs to predict the average top-1 error over a set of downstream tasks to within 1 percentage point for a 6.9B compute-optimal model, which requires 20×20\times20 × more compute to train. To facilitate further research on reliable scaling, we provide all results of our experiments at https://github.com/mlfoundations/scaling. 2 Scaling and over-training In this section, we describe empirical observations and their potential mathematical descriptions. First, we provide key definitions (Section 2.1 ). We next present a phenomenon wherein training a collection of models for increasing token multipliers—ratios of tokens to parameters—follows similar scaling trends (Section 2.2 ). We then show that these patterns are consistent with previously proposed power-law scaling when reparameterizing in terms of training compute and token multipliers (Section 2.3 ). Towards connecting loss scaling and downstream performance, we revisit our collection of models, plot their average top-1 error vs. validation loss, and notice that error decays exponentially with lower loss (Section 2.4 ). To describe these observations, we propose a scaling law for downstream error as a function of loss (Section 2.5 ). 2.1 Preliminaries Scaling laws for loss. We examine scaling laws that predict the loss L𝐿Litalic_L of a model as a function of the compute C𝐶Citalic_C in FLOPs used to train the model. If one increases the number of parameters N𝑁Nitalic_N in a model or the number of tokens D𝐷Ditalic_D that a model is trained on, compute requirements naturally increase. Hence, C𝐶Citalic_C is assumed to be a function of N,D𝑁𝐷N,Ditalic_N , italic_D. Following Kaplan et al. [49], we use the approximation C=6⁢N⁢D𝐶6𝑁𝐷C=6NDitalic_C = 6 italic_N italic_D, which Hoffmann et al. [43] independently verify. We consider scaling laws, L⁢(C)=E+L′⁢(C),𝐿𝐶𝐸superscript𝐿′𝐶\displaystyle L(C)=E+L^{\prime}(C),italic_L ( italic_C ) = italic_E + italic_L start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPT ( italic_C ) , (1) where E𝐸Eitalic_E is an irreducible loss and L′superscript𝐿′L^{\prime}italic_L start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPT is the reducible loss. E𝐸Eitalic_E captures the Bayes error or minimum possible loss achievable on the validation domain. The L′⁢(C)superscript𝐿′𝐶L^{\prime}(C)italic_L start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPT ( italic_C ) term captures what can possibly be learned about the validation domain by training on a source domain. L′⁢(C)superscript𝐿′𝐶L^{\prime}(C)italic_L start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPT ( italic_C ) should go to zero with increased training data and model capacity. L′⁢(C)superscript𝐿′𝐶L^{\prime}(C)italic_L start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPT ( italic_C ) is often assumed to follow a power law: L′⁢(C)=λ⋅C−αCsuperscript𝐿′𝐶⋅𝜆superscript𝐶subscript𝛼𝐶L^{\prime}(C)=\lambda\cdot C^{-\alpha_{C}}italic_L start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPT ( italic_C ) = italic_λ ⋅ italic_C start_POSTSUPERSCRIPT - italic_α start_POSTSUBSCRIPT italic_C end_POSTSUBSCRIPT end_POSTSUPERSCRIPT (i.a., Hestness et al. [41], OpenAI [74]). It is also often helpful to consider a power law in a log\logroman_log-log\logroman_log plot, where it appears as a line with slope −αCsubscript𝛼𝐶-\alpha_{C}- italic_α start_POSTSUBSCRIPT italic_C end_POSTSUBSCRIPT and y𝑦yitalic_y-intercept log⁡(λ)𝜆\log{(\lambda)}roman_log ( italic_λ ). Token multipliers. We define a token multiplier M=D/N𝑀𝐷𝑁M=D/Nitalic_M = italic_D / italic_N as the ratio of training tokens to model parameters. We introduce M𝑀Mitalic_M for notational convenience as it allows us to consider fixed relationships between D𝐷Ditalic_D and N𝑁Nitalic_N even as a model gets bigger (i.e., as N𝑁Nitalic_N becomes larger). Compute-optimal training. Hoffmann et al. [43] establish compute-optimal training, where, for any compute budget H𝐻Hitalic_H, the allocation of parameters and tokens that minimize training or validation loss is given by, arg⁡minN,D⁡L⁢(N,D)⁢ s.t. ⁢C⁢(N,D)=H.subscript𝑁𝐷𝐿𝑁𝐷 s.t. 𝐶𝑁𝐷𝐻\displaystyle\arg\min_{N,D}L(N,D)\text{ s.t. }C(N,D)=H.roman_arg roman_min start_POSTSUBSCRIPT italic_N , italic_D end_POSTSUBSCRIPT italic_L ( italic_N , italic_D ) s.t. italic_C ( italic_N , italic_D ) = italic_H . (2) To solve for the optimal N*,D*superscript𝑁superscript𝐷N^{*},D^{*}italic_N start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT , italic_D start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT, one can sweep N,D𝑁𝐷N,Ditalic_N , italic_D for each H𝐻Hitalic_H, retaining the best configurations. Hoffmann et al. [43] find that as H𝐻Hitalic_H increases, N*superscript𝑁N^{*}italic_N start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT and D*superscript𝐷D^{*}italic_D start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT scale roughly evenly. Assuming equal scaling, there is a fixed compute-optimal token multiplier M*=D*/N*superscript𝑀superscript𝐷superscript𝑁M^{*}=D^{*}/N^{*}italic_M start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT = italic_D start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT / italic_N start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT per training distribution. Over-training. We define over-training as the practice of allocating compute sub-optimally, so smaller models train on a disproportionately large number of tokens (i.e., M>M*𝑀superscript𝑀M>M^{*}italic_M > italic_M start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT). While loss should be higher than in the compute-optimal allocation for a given training budget, the resulting models have fewer parameters and thus are cheaper at inference. Figure 2 : Scaling in the over-trained regime follows consistent power law exponents. We notice parallel lines in the log\logroman_log - log\logroman_log plots of reducible loss vs. training compute for a range of token multipliers M𝑀Mitalic_M , which give the ratio of training tokens to model parameters and where larger M𝑀Mitalic_M corresponds to more over-training. For a power law giving reducible loss as a function of compute: L′⁢(C)=λ⋅C−αCsuperscript𝐿normal-′𝐶normal-⋅𝜆superscript𝐶subscript𝛼𝐶L^{\prime}(C)=\lambda\cdot C^{-\alpha_{C}}italic_L start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPT ( italic_C ) = italic_λ ⋅ italic_C start_POSTSUPERSCRIPT - italic_α start_POSTSUBSCRIPT italic_C end_POSTSUBSCRIPT end_POSTSUPERSCRIPT , the scaling exponent αCsubscript𝛼𝐶\alpha_{C}italic_α start_POSTSUBSCRIPT italic_C end_POSTSUBSCRIPT remains relatively constant resulting in lines with approximately fixed slope. The y𝑦yitalic_y -intercept, which is determined by the power law coefficient λ𝜆\lambdaitalic_λ , however, shifts with different values M𝑀Mitalic_M . This suggests that the power law coefficient λ𝜆\lambdaitalic_λ is a function of the token multiplier M𝑀Mitalic_M , while the power law exponent αCsubscript𝛼𝐶\alpha_{C}italic_α start_POSTSUBSCRIPT italic_C end_POSTSUBSCRIPT is not. Hestness et al. [41] report a similar phenomenon of consistent scaling exponents for recurrent networks when modifying architectures instead of token multipliers. 2.2 Observation: Over-trained models follow consistent trends We begin our scaling investigation by training models with 0.011B to 0.411B parameters for token multipliers M𝑀Mitalic_M between 20 and 640, where M=20𝑀20M=20italic_M = 20 points lie roughly on the compute-optimal frontier, and M>20𝑀20M>20italic_M > 20 corresponds to over-training. We defer experimental details to Section 3 to focus on our observations. In Figure 2 , we plot loss against compute on a log\logroman_log-log\logroman_log scale for the models trained on three datasets and evaluated on the C4 eval set. We notice a phenomenon of “parallel lines” when fitting power laws to the reducible loss, which suggests a near-constant scaling exponent even with increased over-training. This indicates that scaling in the over-trained regime may be predictable given training runs closer to compute-optimal. 2.3 Deriving scaling laws for over-trained behavior In search of an analytic expression for the observations in Figure 2 , we turn to the scaling literature. A common functional form for the risk of a model, as proposed in prior work [91, 43] is, L⁢(N,D)=E+A⁢N−α+B⁢D−β.𝐿𝑁𝐷𝐸𝐴superscript𝑁𝛼𝐵superscript𝐷𝛽\displaystyle L(N,D)=E+AN^{-\alpha}+BD^{-\beta}.italic_L ( italic_N , italic_D ) = italic_E + italic_A italic_N start_POSTSUPERSCRIPT - italic_α end_POSTSUPERSCRIPT + italic_B italic_D start_POSTSUPERSCRIPT - italic_β end_POSTSUPERSCRIPT . (3) Recall from Section 2.1 , N𝑁Nitalic_N is the number of parameters, D𝐷Ditalic_D the number of training tokens and E𝐸Eitalic_E the irreducible loss. The constants E,A,α,B,β𝐸𝐴𝛼𝐵𝛽E,A,\alpha,B,\betaitalic_E , italic_A , italic_α , italic_B , italic_β are fit from data. By fitting this parametric form, Hoffmann et al. [43] find that scaling exponents α𝛼\alphaitalic_α and β𝛽\betaitalic_β are close, suggesting that one should scale N,D𝑁𝐷N,Ditalic_N , italic_D equally as compute increases. Hence, we assume α=β𝛼𝛽\alpha=\betaitalic_α = italic_β. With this assumption, we reparameterize Equation ( 3 ) in terms of compute C=6⁢N⁢D𝐶6𝑁𝐷C=6NDitalic_C = 6 italic_N italic_D and a token multiplier M=D/N𝑀𝐷𝑁M=D/Nitalic_M = italic_D / italic_N. We get the following form, L⁢(C,M)=E+(a⁢MαC+b⁢M−αC)⁢C−αC,𝐿𝐶𝑀𝐸𝑎superscript𝑀subscript𝛼𝐶𝑏superscript𝑀subscript𝛼𝐶superscript𝐶subscript𝛼𝐶\displaystyle L(C,M)=E+\left(aM^{\alpha_{C}}+bM^{-\alpha_{C}}\right)C^{-\alpha% _{C}},italic_L ( italic_C , italic_M ) = italic_E + ( italic_a italic_M start_POSTSUPERSCRIPT italic_α start_POSTSUBSCRIPT italic_C end_POSTSUBSCRIPT end_POSTSUPERSCRIPT + italic_b italic_M start_POSTSUPERSCRIPT - italic_α start_POSTSUBSCRIPT italic_C end_POSTSUBSCRIPT end_POSTSUPERSCRIPT ) italic_C start_POSTSUPERSCRIPT - italic_α start_POSTSUBSCRIPT italic_C end_POSTSUBSCRIPT end_POSTSUPERSCRIPT , (4) where αC=α/2subscript𝛼𝐶𝛼2\alpha_{C}=\alpha/2italic_α start_POSTSUBSCRIPT italic_C end_POSTSUBSCRIPT = italic_α / 2, a=A⁢(1/6)−αC𝑎𝐴superscript16subscript𝛼𝐶a=A(1/6)^{-\alpha_{C}}italic_a = italic_A ( 1 / 6 ) start_POSTSUPERSCRIPT - italic_α start_POSTSUBSCRIPT italic_C end_POSTSUBSCRIPT end_POSTSUPERSCRIPT, b=B⁢(1/6)−αC𝑏𝐵superscript16subscript𝛼𝐶b=B(1/6)^{-\alpha_{C}}italic_b = italic_B ( 1 / 6 ) start_POSTSUPERSCRIPT - italic_α start_POSTSUBSCRIPT italic_C end_POSTSUBSCRIPT end_POSTSUPERSCRIPT gives the relation to Equation ( 3 ). For a complete derivation, see Appendix B . Equation ( 4 ) has the following interpretation: (i) The scaling exponent αCsubscript𝛼𝐶\alpha_{C}italic_α start_POSTSUBSCRIPT italic_C end_POSTSUBSCRIPT is not dependent on M𝑀Mitalic_M. Thus, we always expect lines with the same slope in the log\logroman_log-log\logroman_log plot—as in Figure 2 . (ii) The term a⁢MαC+b⁢M−αC𝑎superscript𝑀subscript𝛼𝐶𝑏superscript𝑀subscript𝛼𝐶aM^{\alpha_{C}}+bM^{-\alpha_{C}}italic_a italic_M start_POSTSUPERSCRIPT italic_α start_POSTSUBSCRIPT italic_C end_POSTSUBSCRIPT end_POSTSUPERSCRIPT + italic_b italic_M start_POSTSUPERSCRIPT - italic_α start_POSTSUBSCRIPT italic_C end_POSTSUBSCRIPT end_POSTSUPERSCRIPT determines the offsets between curves with different token multipliers. Hence, we expect non-overlapping, parallel lines in the log\logroman_log-log\logroman_log plot for the range of M𝑀Mitalic_M we consider—also consistent with Figure 2 . Recall that we make the assumption α=β𝛼𝛽\alpha=\betaitalic_α = italic_β, which implies equal scaling of parameters and tokens as more compute is available. However, as explained in Appendix B , even if α≠β𝛼𝛽\alpha\neq\betaitalic_α ≠ italic_β, we get a parameterization that implies the power-law exponent remains constant with over-training. 2.4 Observation: Loss tracks average top-1 error Figure 3 : Average top-1 error scales as a function of loss. We plot data points trained on three datasets and notice an exponential decay of average top-1 error as C4 eval loss, on the x-axis, decreases. The specific coefficients appear dataset-dependent. We consider on the y-axes (left) the average error over 46 evaluations and (right) the average error on a subset of 17 evaluations where performance can be 10 points above random chance for at least one 0.154B scale model. These observations suggest that average top-1 error should be predictable with reliable loss estimates. Scaling is typically studied in the context of loss [49, 43, 70], which Schaeffer et al. [98] note is smoother than metrics like accuracy. However, practitioners ultimately care about downstream, in-the-wild task performance. To better connect scaling laws and over-training to task prediction, we revisit the suite of models plotted in Figure 2 . In Figure 3 , we plot average downstream top-1 errors over evaluations sourced from LLM-Foundry [67] against the C4 eval loss. We defer details of the setup to Section 3 to focus here on a few key observations. The average errors appear to follow exponential decay as loss decreases. Additionally, the particular relationship between loss and error is dataset-dependent. For instance, models trained on C4 result in the lowest C4 eval loss, but this does not translate to downstream gains compared to models trained on RedPajama or RefinedWeb. 2.5 Proposing a scaling law for average top-1 error Based on the exponential decay we observe in Figure 3 , we propose the following relationship between downstream average top-1 error Err and loss L𝐿Litalic_L, 𝖤𝗋𝗋⁢(L)=ϵ−k⋅exp⁡(−γ⁢L),𝖤𝗋𝗋𝐿italic-ϵ⋅𝑘𝛾𝐿\displaystyle\textsf{Err}(L)=\epsilon-k\cdot\exp{(-\gamma L)},Err ( italic_L ) = italic_ϵ - italic_k ⋅ roman_exp ( - italic_γ italic_L ) , (5) where ϵ,k,γitalic-ϵ𝑘𝛾\epsilon,k,\gammaitalic_ϵ , italic_k , italic_γ are fit from data. Equation ( 5 ) also has an appealing interpretation in terms of model perplexity P⁢P⁢(L)=exp⁡(L)𝑃𝑃𝐿𝐿PP(L)=\exp{(L})italic_P italic_P ( italic_L ) = roman_exp ( italic_L ), 𝖤𝗋𝗋⁢(P⁢P)=ϵ−k⋅P⁢P−γ.𝖤𝗋𝗋𝑃𝑃italic-ϵ⋅𝑘𝑃superscript𝑃𝛾\displaystyle\textsf{Err}(PP)=\epsilon-k\cdot PP^{-\gamma}.Err ( italic_P italic_P ) = italic_ϵ - italic_k ⋅ italic_P italic_P start_POSTSUPERSCRIPT - italic_γ end_POSTSUPERSCRIPT . (6) Namely, Err follows a power law in P⁢P𝑃𝑃PPitalic_P italic_P with maximum error ϵitalic-ϵ\epsilonitalic_ϵ, where intuitively ϵitalic-ϵ\epsilonitalic_ϵ should be close to the random chance performance. Equation ( 5 ) in conjunction with ( 4 ) suggests a two-step method to predict Err as a function of compute and the amount of over-training. For choices of training and validation distributions, (i) fit a scaling law to Equation ( 4 ) using ((C,M),L)𝐶𝑀𝐿((C,M),L)( ( italic_C , italic_M ) , italic_L ) pairs to yield (C,M)↦Lmaps-to𝐶𝑀𝐿(C,M)\mapsto L( italic_C , italic_M ) ↦ italic_L. (ii) Fit a scaling law to Equation ( 5 ) using (L,𝖤𝗋𝗋)𝐿𝖤𝗋𝗋(L,\textsf{Err})( italic_L , Err ) pairs to get L↦𝖤𝗋𝗋maps-to𝐿𝖤𝗋𝗋L\mapsto\textsf{Err}italic_L ↦ Err. 3 Experimental setup Towards testing the analytic predictions in Equations ( 4 ), ( 5 ), we discuss our experimental setup. We first present language modeling details (Section 3.1 ). Next, we discuss our strategy for deciding which models to include in our scaling investigation and our procedure for fitting scaling trends (Section 3.2 ). We then present metrics to validate how well scaling laws predict loss and downstream performance (Section 3.3 ). 3.1 Training setup Architecture. We train transformers [114], based on auto-regressive, decoder-only, pre-normalization architectures like GPT-2 [83] and LLaMA [111]. We adopt OpenLM [37] as our core modeling library, which utilizes PyTorch [78, 6], xformers [52], triton [73], FlashAttention [22], FSDP [127], and bfloat16 automatic mixed precision. Like LLaMA, we omit bias terms, but replace RMSNorm [125] with LayerNorm [8], which has readily available fused implementations. Following Wortsman et al. [120], we apply qk-LayerNorm [23], which adds robustness to otherwise poor hyperparameter choices (e.g., for learning rate). We use SwiGLU [100] activations and depth-scaled initialization [126]. We use a sequence length of 2048, rotary positional embeddings [104], and the GPT-NeoX-20B tokenizer [13], which yields a vocabulary size of 50k. We do not use weight tying [82, 44]. Objectives and optimization. We train with a standard causal language modeling objective (i.e., next token prediction) with an additive z-loss [17] (coefficient 1e𝑒eitalic_e-4), which mitigates output logit norm growth [65] instabilities. We use the AdamW optimizer [60] (PyTorch defaults except beta2 = 0.95), with independent weight decay [120] (coefficient 1e𝑒eitalic_e-4). For the learning rate schedule, we use linear warmup and cosine decay. We cool down to a low learning rate (3e𝑒eitalic_e-5). Training datasets. To ensure our conclusions are not particular to a training distribution, we train models on C4 [86, 25], RedPajama [110], and RefinedWeb [80]. They are open-source and have 138B, 1.15T, and 600B tokens respectively. We sample without replacement and employ sequence packing without attention masking. We separate documents in our training corpora with end-of-text tokens. 3.2 Creating scaling laws for validation loss and downstream error prediction Figure 4 : Search, filter, fit: A recipe for selecting configurations for scaling. (left) To generate the final configurations presented in Table 1 , we run a 435 model grid search over model width, hidden dimension, number of attention heads, batch size, and warmup steps. All models are trained near compute-optimally. (center) We plot the efficient frontier of models, which appear to follow a trend, excluding models from 5.2×10165.2superscript10165.2\times 10^{16}5.2 × 10 start_POSTSUPERSCRIPT 16 end_POSTSUPERSCRIPT to 5.2×10175.2superscript10175.2\times 10^{17}5.2 × 10 start_POSTSUPERSCRIPT 17 end_POSTSUPERSCRIPT , which fall below the trend. (right) We fit a power law with irreducible error to the remaining configurations, picking four configurations that closely track the full model suite. These models extrapolate the performance of 1.4B, 6.9B target models. Shaded regions represent bootstrap 95% confidence intervals. Recipe for model configuration selection. To create a testbed of models for our scaling experiments, we grid search over a total of 435 models, trained from scratch, in the 0.01B to 0.5B parameter range as seen in Figure 4 (left). We train on the OpenLM data mix [37], which largely consists of tokens from RedPajama [110] and The Pile [29]. We train on 20 tokens per parameter (M=20𝑀20M=20italic_M = 20), which we find in early experiments gives models near the compute-optimal frontier for the data mix. This is similar to findings presented in Hoffmann et al. [43]’s Table 3, which suggests that roughly 20 tokens per parameter are optimal in their experimental setup. Our validation set, OpenLM eval, contains tokens from recent arXiv papers, the OpenLM codebase itself, and news articles. To find maximally performant models on validation data, we tune model width, number of layers, number of attention heads, warmup steps, and batch size. We find in early experiments that qk-LayerNorm makes models less sensitive to learning rate, which is a phenomenon Wortsman et al. [120] report in their Figure 1. Hence, we fix the learning rate for our sweeps. We also perform smaller grid searches over 1.4B and 6.9B parameter model configurations at M=20𝑀20M=20italic_M = 20, retaining the best configurations. In Figure 4 (center), we plot the efficient frontier of minimum loss configurations. While there appears to be a general trend, configurations between 5.2×10165.2superscript10165.2\times 10^{16}5.2 × 10 start_POSTSUPERSCRIPT 16 end_POSTSUPERSCRIPT and 5.2×10175.2superscript10175.2\times 10^{17}5.2 × 10 start_POSTSUPERSCRIPT 17 end_POSTSUPERSCRIPT FLOPs lie below the frontier established by other models. We hypothesize these models over-perform as they are trained for more optimization steps than their neighbors based on our power-of-two batch sizes. We provide support for this hypothesis in Appendix E , but opt to remove these models from our investigation. In Figure 4 (right), we fit trends to the remaining models and to a subset of four models. We notice that the trends hit both the 1.4B and 6.9B models suggesting that our small-scale configurations are reasonable for extrapolation to larger parameter and compute regimes. We retain the four model configuration subsets as a representative sample. We do not tune hyperparameters for other token multipliers (i.e., M≠20𝑀20M\neq 20italic_M ≠ 20), other training or evaluation distributions, or on downstream task validation sets. For more details, see Appendix C . We present our final hyperparameters in Table 1 given their importance. N𝑁Nitalic_N nlayerssubscript𝑛layersn_{\text{layers}}italic_n start_POSTSUBSCRIPT layers end_POSTSUBSCRIPT nheadssubscript𝑛headsn_{\text{heads}}italic_n start_POSTSUBSCRIPT heads end_POSTSUBSCRIPT dmodelsubscript𝑑modeld_{\text{model}}italic_d start_POSTSUBSCRIPT model end_POSTSUBSCRIPT dheadsubscript𝑑headd_{\text{head}}italic_d start_POSTSUBSCRIPT head end_POSTSUBSCRIPT Warmup Learning rate Batch size M=20𝑀20M=20italic_M = 20 A100 hours 0.011B 8 4 96 24 100 3 e𝑒eitalic_e -3 64 0.3 0.079B 8 4 512 128 400 3 e𝑒eitalic_e -3 512 5 0.154B 24 8 576 72 400 3 e𝑒eitalic_e -3 512 12 0.411B 24 8 1024 128 2000 3 e𝑒eitalic_e -3 512 75 1.4B 24 16 2048 128 5000 3 e𝑒eitalic_e -3 256 690 6.9B 32 32 4096 128 5000 3 e𝑒eitalic_e -4 2048 17000 Table 1: Main models and hyperparameters used in our investigation. Models have number of parameters N𝑁Nitalic_N, with number of layers nlayerssubscript𝑛layersn_{\text{layers}}italic_n start_POSTSUBSCRIPT layers end_POSTSUBSCRIPT, number of attention heads nheadssubscript𝑛headsn_{\text{heads}}italic_n start_POSTSUBSCRIPT heads end_POSTSUBSCRIPT, model width dmodelsubscript𝑑modeld_{\text{model}}italic_d start_POSTSUBSCRIPT model end_POSTSUBSCRIPT, and width per attention head dheadsubscript𝑑headd_{\text{head}}italic_d start_POSTSUBSCRIPT head end_POSTSUBSCRIPT. Batch sizes are global and in units of sequences. Each sequence has 2048 tokens. A100 GPU hours are at M=20𝑀20M=20italic_M = 20, which are near compute-optimal runs. For the 1.4B scale, a batch size of 256 performs slightly better than 512. Fitting scaling laws. N𝑁Nitalic_N M𝑀Mitalic_M Used to fit Equation ( 4 ) Used to fit Equation ( 5 ) 0.011B 20 ✓ ✓ 0.079B 20 ✓ ✓ 0.154B 20 ✓ ✓ 0.411B 20 ✓ ✓ 0.011B 320 ✓ ✓ 1.4B 20 ✗ ✓ Total compute C𝐶Citalic_C [FLOPs] 2.4 e𝑒eitalic_e 19 2.7 e𝑒eitalic_e 20 Table 2: Default N,M𝑁𝑀N,Mitalic_N , italic_M to fit our scaling laws. We invest ∼similar-to\sim∼100 A100 hours to fit Equation ( 4 ) and ∼similar-to\sim∼1000 A100 hours to fit Equation ( 5 ). We fit Equation ( 4 ) to approximate E,a,b,αC𝐸𝑎𝑏subscript𝛼𝐶E,a,b,\alpha_{C}italic_E , italic_a , italic_b , italic_α start_POSTSUBSCRIPT italic_C end_POSTSUBSCRIPT using curve-fitting in SciPy [115] (i.e., Levenberg-Marquardt to minimize non-linear least squares). We try several initializations and retain the best fit. We repeat this process to fit Equation ( 5 ) to approximate ϵ,k,γitalic-ϵ𝑘𝛾\epsilon,k,\gammaitalic_ϵ , italic_k , italic_γ. Unless otherwise specified, we fit to the N,M𝑁𝑀N,Mitalic_N , italic_M pairs in Table 2 . In total, we invest ∼similar-to\sim∼100 A100 hours to train the models required for fitting an accurate scaling law for loss prediction and ∼similar-to\sim∼1000 A100 hours for a corresponding scaling law for downstream error prediction. Our configurations allow us to test for extrapolation to the N=1.4⁢B,M=640formulae-sequence𝑁1.4B𝑀640N=1.4\text{B},M=640italic_N = 1.4 B , italic_M = 640 (900B token) and the N=6.9⁢B,M=20formulae-sequence𝑁6.9B𝑀20N=6.9\text{B},M=20italic_N = 6.9 B , italic_M = 20 (138B token) regimes. 3.3 Evaluation setup Evaluation datasets. Unless otherwise stated, C4 eval is our default validation loss dataset. For downstream tasks, we adopt 46 tasks from LLM-foundry [67], which includes standard tasks with both zero-shot and few-shot evaluations. We also consider a 17-task subset where, for each evaluation, at least one 0.154B scale model—trained with as many as 99B tokens—gets 10 percentage points above chance accuracy: ARC-Easy [21], BIG-bench: CS algorithms [11], BIG-bench: Dyck languages [11], BIG-bench: Novel Concepts [11], BIG-bench: Operators [11], BIG-bench: QA WikiData [11], BoolQ [19], Commonsense QA [105], COPA [90], CoQA [89], HellaSwag (zero-shot) [123], HellaSwag (10-shot) [123], LAMBADA [75], PIQA [12], PubMed QA Labeled [48], SQuAD [88], and WinoGrand [53]. This subset allows us to investigate a regime where performance for small models can be non-trivial. For more details on evaluation datasets see Appendix D . For ablations on our choices of loss and downstream evaluations see Appendix E . Metrics. We consider three main metrics: (i) Validation loss, which is the cross entropy between a model’s output and the one-hot ground truth, averaged over all tokens in a sequence and over all sequences in a dataset. (ii) Average top-1 error, which is a uniform average over 46 downstream evaluations sourced from LLM-foundry [67]. We also look at the mean top-1 error for the subset of 17 evaluations identified in the paragraph above. For a complete list of downstream evaluation datasets, see Appendix D . To measure how good a prediction ζ⁢(C,M)𝜁𝐶𝑀\zeta(C,M)italic_ζ ( italic_C , italic_M ) is, we measure (iii) Relative prediction error: |ζ⁢(C,M)−ζG⁢T|/ζG⁢T𝜁𝐶𝑀subscript𝜁𝐺𝑇subscript𝜁𝐺𝑇|\zeta(C,M)-\zeta_{GT}|/\zeta_{GT}| italic_ζ ( italic_C , italic_M ) - italic_ζ start_POSTSUBSCRIPT italic_G italic_T end_POSTSUBSCRIPT | / italic_ζ start_POSTSUBSCRIPT italic_G italic_T end_POSTSUBSCRIPT, where ζ𝜁\zetaitalic_ζ is the loss L𝐿Litalic_L or the average top-1 error Err . Testbed. We train models on C4, RedPajama, and RefinedWeb with the number of parameters N∈{0.011⁢B,0.079⁢B,0.154⁢B,0.411⁢B}𝑁0.011B0.079B0.154B0.411BN\in\{0.011\text{B},0.079\text{B},0.154\text{B},0.411\text{B}\}italic_N ∈ { 0.011 B , 0.079 B , 0.154 B , 0.411 B } and token multipliers M∈{5,10,20,40,80,160,320,640}𝑀510204080160320640M\in\{5,10,20,40,80,160,320,640\}italic_M ∈ { 5 , 10 , 20 , 40 , 80 , 160 , 320 , 640 }. We omit runs that require more tokens than are present in a dataset (i.e., N=0.411⁢B,M=640formulae-sequence𝑁0.411B𝑀640N=0.411\text{B},M=640italic_N = 0.411 B , italic_M = 640 for C4). We additionally train N=1.4𝑁1.4N=1.4italic_N = 1.4B models at M=20𝑀20M=20italic_M = 20 and at the largest token multiplier possible without repeating tokens (i.e., 80 for C4, 640 for RedPajama, and 320 for RefinedWeb). We train N=6.9⁢B,M=20formulae-sequence𝑁6.9B𝑀20N=6.9\text{B},M=20italic_N = 6.9 B , italic_M = 20 for each dataset. In total this results in 104 models. We evaluate each model on C4 eval for validation loss and on the 46 downstream tasks for top-1 error. 4 Results Unless otherwise stated, we fit Equations ( 4 ), ( 5 ) to the configurations in Table 2 and use C4 eval for loss computation. Over-trained performance is predictable. We highlight our main over-training results in Figure 1 (left). Namely, we are able to extrapolate both in the number of parameters N𝑁Nitalic_N and the token multiplier M𝑀Mitalic_M to closely predict the C4 eval performance of a 1.4B parameter model trained on 900B RedPajama tokens (N=1.4⁢B,M=640formulae-sequence𝑁1.4B𝑀640N=1.4\text{B},M=640italic_N = 1.4 B , italic_M = 640). Our prediction, which takes 300×\times× less compute to construct than the final 1.4B run, is accurate to within 0.7% relative error. Additionally, for the N=6.9⁢B,M=20formulae-sequence𝑁6.9B𝑀20N=6.9\text{B},M=20italic_N = 6.9 B , italic_M = 20 run, near compute-optimal, the relative error is also 0.7%. These results support several key takeaways. (i) scaling can be predictable even when one increases the model size and the amount of over-training compared to the training runs used to fit a scaling law. (ii) Scaling can be predictable even in the presence of a distribution shift (e.g., RedPajama training and C4 evaluation). (iii) The form presented in Equation ( 4 ) is useful in practice for fitting and predicting over-trained scaling behavior. (iv) Fitting to Equation ( 4 ) does not sacrifice prediction accuracy near compute-optimal. Figure 5 : Relative error on C4 eval for different training distributions. Boxes highlighted in yellow correspond to data points used to fit Equation ( 4 ), all other points are predicted based on the fit. The prediction error is low in both interpolation and extrapolation ranges for number of parameters N𝑁Nitalic_N and token multiplier M𝑀Mitalic_M , with larger values of M𝑀Mitalic_M specifying more over-training. Empty squares correspond to runs that were not possible due to compute limitations at the 1.4B and 6.9B scales or to limited dataset size for single epoch training. While Figure 1 explores a specific case of making predictions in the over-trained regime, we would like to understand the error profile of our predictions across datasets, token multipliers, and number of parameters. Hence, in Figure 5 we show relative error between ground truth loss and predicted loss on C4 eval for models in our testbed. We notice uniformly low prediction error suggesting that predictions are accurate in many settings. Average top-1 error is predictable. Figure 1 (right) presents our results in estimating scaling laws for downstream error. Similar to Figure 1 (left), we are able to extrapolate in N,M𝑁𝑀N,Mitalic_N , italic_M and predict the average downstream error across our evaluation suite. Concretely, we use the models indicated in Table 2 to fit Equation ( 5 ), and predict the average top-1 error over the 17 tasks identified in Section 3.3 . Our fit allows us to predict the downstream performance of a 6.9B parameter model trained on 138B tokens to within 0.05%percent0.050.05\%0.05 % relative error and of a 1.4B model trained on 900B tokens to within 3.6%percent3.63.6\%3.6 % relative error, using 20×20\times20 × less compute. Table 3 additionally shows the relative error of our downstream performance predictions for models trained on C4, RedPajama, and RefinedWeb, indicating that our scaling law fits are applicable to other datasets. We note that while average accuracy across benchmarks is predictable, predicting accuracy on individual downstream tasks is significantly more noisy. We report relative error of all our predictions in Figures 11 , 12 in the Appendix. We also find that if we remove the 1.4B model for the Equation ( 5 ) fit, relative error jumps, for instance, from 0.05% to 10.64% on the 17-task split for the 6.9B, 138B token RedPajama prediction. Individual top-1 error Average top-1 error Train set MMLU [38] OpenBook QA [66] HellaSwag [123] 17-task split 46-task split C4 [86, 25] 2.82% 16.80% 79.58% 0.42% 0.76% RedPajama [110] 0.12% 8.44% 25.73% 0.05% 2.10% RefinedWeb [80] 0.77% 1.92% 81.96% 2.94% 2.76% Table 3 : Downstream relative prediction error at 6.9B, 138B tokens. While predicting accuracy on individual zero-shot downstream evaluations can be challenging (“Individual”), predicting averages across downstream datasets is accurate (“Average”). On the right, we report relative prediction error for the average over a 17-task subset and over the full 46-task suite. Small-scale experiments can predict model rank order. We expect to be able to rank models based on their predicted performance, which is useful when deciding what to train. To verify, we rank 9 testbed models with N≥1.4⁢B𝑁1.4BN\geq 1.4\text{B}italic_N ≥ 1.4 B by ground-truth top-1 error and by estimated top-1 error. We find high rank correlations: 0.93 and 0.88 for the 46 and 17-task splits respectively. Under-training, out-of-distribution scaling, and compute-reliability trade-offs. In addition to our main results presented above, we include additional results in Appendix E , which we summarize here. First, we notice that when token multipliers become too small (i.e., M=5𝑀5M=5italic_M = 5) scaling becomes unreliable and lies off the trend. Additionally, several multipliers (10, 20, 40, and 80) garner points that are roughly on the compute optimal frontier (Figure 9 ). To probe the limits of reliable scaling, we attempt to break our scaling laws in out-of-distribution settings. We find that models trained on C4—English filtered—and evaluated on next token prediction on code domains have a high relative error in many cases. Perhaps surprisingly, evaluating the same models on German next token prediction again gives reliable loss scaling (Figure 10 ). We additionally examine the compute necessary to create accurate scaling laws, finding a positive correlation between investing more compute in a scaling law and its predictivity. We find that scaling laws can be constructed more cheaply for loss prediction than for downstream error prediction (Figures 15 , 16 ). 5 Related work We review the most closely related work in this section. For additional related work, see Appendix F . Model family Parameters N𝑁Nitalic_N Training tokens D𝐷Ditalic_D Token multiplier M𝑀Mitalic_M T5 [ 87 ] 11B 34B 3.1 GPT-3 [ 14 ] 175B 300B 1.7 Gopher [ 84 ] 280B 300B 1.1 Chinchilla [ 43 ] 70B 1.4T 20.0 LLaMA [ 111 ] 7B 1T 140.0 LLaMA [ 111 ] 70B 1.4T 20.0 LLaMA-2 [ 112 ] 7B 2T 290.0 LLaMA-2 [ 112 ] 70B 2T 30.0 XGen [ 72 ] 7B 1.5T 210.0 MPT [ 108 ] 7B 1T 140.0 Table 4: Token multipliers of existing models. In our work, we run experiments with token multipliers between 5 and 640 for (GPT-2 [83], LLaMA [111])-style decoder-only architectures. Scaling laws. Early works on scaling artificial neural networks observe predictable power-law scaling in the training set size and number of model parameters [41, 42, 91]. Alabdulmohsin et al. [2] center the importance of looking at the extrapolation regime of a scaling law. Yang et al. [121] prescribe architectural and hyperparameter changes when scaling model width to realize performant models; Yang et al. [122] make analogous recommendations when scaling model depth. Unlike the aforementioned work, our investigation focuses on the link between over-training models and predicting their downstream performance on accuracy metrics. Hoffmann et al. [43] investigate how the number of model parameters N𝑁Nitalic_N and training tokens D𝐷Ditalic_D should be chosen to minimize loss L𝐿Litalic_L given a compute budget C𝐶Citalic_C. Hoffmann et al. [43] find that when scaling up C𝐶Citalic_C, both N𝑁Nitalic_N and D𝐷Ditalic_D should be scaled equally up to a multiplicative constant (i.e., N∝C∼0.5proportional-to𝑁superscript𝐶similar-toabsent0.5N\propto C^{\sim 0.5}italic_N ∝ italic_C start_POSTSUPERSCRIPT ∼ 0.5 end_POSTSUPERSCRIPT and D∝C∼0.5proportional-to𝐷superscript𝐶similar-toabsent0.5D\propto C^{\sim 0.5}italic_D ∝ italic_C start_POSTSUPERSCRIPT ∼ 0.5 end_POSTSUPERSCRIPT) to realize compute-optimality. Appendix C of the Chinchilla paper additionally suggests that these findings hold across many datasets. However, Hoffmann et al. [43] do not account for inference costs, provide scaling laws for training beyond compute-optimal, or for downstream error prediction—all of which are central to our work. Sardana & Frankle [96] proposed modifications to the Chinchilla formulation to incorporate inference costs into the definition of compute-optimal and solve for various fixed inference budgets. Their key finding, which is critical for our work, is that when taking into account a large enough inference budget, it is optimal to train smaller models for longer when compared to the original Chinchilla recommendations. Our work presupposes that over-training can be beneficial. Instead of solving for inference-optimal schemes, we support empirically a predictive theory of scaling in the over-trained regime. Additionally, we provide experiments across many validation and training sets. For predicting downstream scaling beyond loss, Isik et al. [45] relate the number of pre-training tokens to downstream cross-entropy and machine translation BLEU score [76] after fine-tuning. In contrast, we do not examine a specific domain but rather take a holistic approach to evaluation by looking at top-1 error over many natural language tasks. Schaeffer et al. [98] argue that emergent abilities [118] and unreliable scaling are a product of non-linear metrics like error/accuracy and propose smoother alternatives. As an explanation for why non-linear metrics may be hard to predict, Schaeffer et al. [98] consider predicting exactly an ℓℓ\ellroman_ℓ length sequence: 𝖤𝗋𝗋⁢(N,ℓ)≈1−P⁢P⁢(N)−ℓ𝖤𝗋𝗋𝑁ℓ1𝑃𝑃superscript𝑁ℓ\textsf{Err}(N,\ell)\approx 1-PP(N)^{-\ell}Err ( italic_N , roman_ℓ ) ≈ 1 - italic_P italic_P ( italic_N ) start_POSTSUPERSCRIPT - roman_ℓ end_POSTSUPERSCRIPT, where N𝑁Nitalic_N is the number of parameters in a model and P⁢P𝑃𝑃PPitalic_P italic_P is its perplexity. This is a special case of our Equations ( 5 ), ( 6 ), where number of training tokens is ignored, ϵ=1,k=1formulae-sequenceitalic-ϵ1𝑘1\epsilon=1,k=1italic_ϵ = 1 , italic_k = 1, and γ=ℓ𝛾ℓ\gamma=\ellitalic_γ = roman_ℓ. In contrast, we treat ϵ,k,γitalic-ϵ𝑘𝛾\epsilon,k,\gammaitalic_ϵ , italic_k , italic_γ as free parameters for a scaling law fit and embrace top-1 error, finding that average error over downstream tasks can make for a predictable metric. Over-training in popular models. There has been a rise in over-trained models [111, 112] and accompanying massive datasets [110, 80, 102, 3]. To contextualize the extent to which we over-train, we provide token multipliers for popular models in Table 4 . For example, Chinchilla 70B [43] is trained with a token multiplier of 20, while LLaMA-2 7B [112] uses a token multiplier of 290. In our investigation, we look at token multipliers from 5 to 640 to ensure coverage of popular models and relevance for future models that may be trained on even more tokens. 6 Limitations, future work, and conclusion Limitations and future work. We identify limitations, which provide motivation for future work. * • Hyperparameters. While our configurations are surprisingly amenable to reliable scaling across many training and testing distributions without further tuning, there is a need to further develop scaling laws that incorporate hyperparameters. * • Scaling up. Validating the trends in this paper for even larger runs is a valuable direction. * • Scaling down. Additionally, actualizing predictable scaling with even cheaper runs is important to make this area of research more accessible, especially for downstream error prediction. * • Failure cases. While we present preliminary analysis of when scaling is unreliable, future work should develop an analytic theory explaining when scaling breaks down. * • Post-training. It is common to employ supervised fine-tuning and reinforcement learning after pre-training, which we do not consider. Quantifying to what degree over-training the base model provides benefits after post-training is an open area of research. * • Individual downstream task prediction. While we find that averaging over many task error metrics can make for a predictable metric, per-task predictions are left to future work. * • In-the-wild performance. Downstream task performance is a proxy for the in-the-wild user experience. Analyzing scaling trends in the context of this experience is timely. * • Dataset curation. Our work only deals with existing training datasets. Exploring dataset curation for improved model scaling is another promising direction. Conclusion. We (i) show that the loss scaling behavior of models trained past compute-optimal, in the over-trained regime, is predictable and (ii) predict, via a proposed scaling law, the downstream average task performance of more expensive runs using smaller-scale proxies. We hope our work will inspire others to further examine the relationship between model training and downstream generalization. We also hope our testbed will make scaling research more accessible to researchers and practitioners alike. Acknowledgements SYG is supported by an NSF Graduate Research Fellowship, GS by the Onassis Foundation - Scholarship ID: F ZS 056-1/2022-2023, and MN by the Federal Ministry of Education and Research of Germany under grant no. 01IS22094B WEST-AI. We thank Stability AI and Toyota Research Institute (TRI) for access to compute resources. This research has been supported by NSF Grants AF 1901292, CNS 2148141, Tripods CCF 1934932, IFML CCF 2019844, and research gifts by Western Digital, Amazon, WNCG IAP, UT Austin Machine Learning Lab (MLL), Cisco, and the Stanly P. Finch Centennial Professorship in Engineering. We also thank Kushal Arora, Alper Canberk, Mia Chiquier, Sachit Menon, Chuer Pan, Purva Tendulkar, and Mandi Zhao for valuable feedback. References * Abnar et al. [2022] ↑ Samira Abnar, Mostafa Dehghani, Behnam Neyshabur, and Hanie Sedghi. Exploring the limits of large scale pre-training. In International Conference on Learning Representations (ICLR) , 2022. https://arxiv.org/abs/2110.02095 . * Alabdulmohsin et al. [2022] ↑ Ibrahim Alabdulmohsin, Behnam Neyshabur, and Xiaohua Zhai. Revisiting neural scaling laws in language and vision. In Advances in Neural Information Processing Systems (NeuIPS) , 2022. https://arxiv.org/abs/2209.06640 . * Albalak et al. [2024] ↑ Alon Albalak, Yanai Elazar, Sang Michael Xie, Shayne Longpre, Nathan Lambert, Xinyi Wang, Niklas Muennighoff, Bairu Hou, Liangming Pan, Haewon Jeong, et al. A survey on data selection for language models. arXiv preprint , 2024. https://arxiv.org/abs/2402.16827 . * Allal et al. [2023] ↑ Loubna Ben Allal, Raymond Li, Denis Kocetkov, Chenghao Mou, Christopher Akiki, Carlos Munoz Ferrandis, Niklas Muennighoff, Mayank Mishra, Alex Gu, Manan Dey, et al. Santacoder: don’t reach for the stars! arXiv preprint , 2023. https://arxiv.org/abs/2301.03988 . * Amini et al. [2019] ↑ Aida Amini, Saadia Gabriel, Shanchuan Lin, Rik Koncel-Kedziorski, Yejin Choi, and Hannaneh Hajishirzi. MathQA: Towards interpretable math word problem solving with operation-based formalisms. In Conference of the North American Chapter of the Association for Computational Linguistics (NACCL) , 2019. https://aclanthology.org/N19-1245 . * Ansel et al. [2024] ↑ Jason Ansel, Edward Yang, Horace He, Natalia Gimelshein, Animesh Jain, Michael Voznesensky, Bin Bao, David Berard, Geeta Chauhan, Anjali Chourdia, Will Constable, Alban Desmaison, Zachary DeVito, Elias Ellison, Will Feng, Jiong Gong, Michael Gschwind, Brian Hirsh, Sherlock Huang, Laurent Kirsch, Michael Lazos, Yanbo Liang, Jason Liang, Yinghai Lu, CK Luk, Bert Maher, Yunjie Pan, Christian Puhrsch, Matthias Reso, Mark Saroufim, Helen Suk, Michael Suo, Phil Tillet, Eikan Wang, Xiaodong Wang, William Wen, Shunting Zhang, Xu Zhao, Keren Zhou, Richard Zou, Ajit Mathews, Gregory Chanan, Peng Wu, and Soumith Chintala. Pytorch 2: Faster machine learning through dynamic python bytecode transformation and graph compilation. In International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) , 2024. https://pytorch.org/blog/pytorch-2-paper-tutorial . * Artetxe et al. [2022] ↑ Mikel Artetxe, Shruti Bhosale, Naman Goyal, Todor Mihaylov, Myle Ott, Sam Shleifer, Xi Victoria Lin, Jingfei Du, Srinivasan Iyer, Ramakanth Pasunuru, Giridharan Anantharaman, Xian Li, Shuohui Chen, Halil Akin, Mandeep Baines, Louis Martin, Xing Zhou, Punit Singh Koura, Brian O’Horo, Jeffrey Wang, Luke Zettlemoyer, Mona Diab, Zornitsa Kozareva, and Veselin Stoyanov. Efficient large scale language modeling with mixtures of experts. In Conference on Empirical Methods in Natural Language Processing (EMNLP) , 2022. https://aclanthology.org/2022.emnlp-main.804 . * Ba et al. [2016] ↑ Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint , 2016. https://arxiv.org/abs/1607.06450 . * Bahri et al. [2021] ↑ Yasaman Bahri, Ethan Dyer, Jared Kaplan, Jaehoon Lee, and Utkarsh Sharma. Explaining neural scaling laws. arXiv preprint , 2021. https://arxiv.org/abs/2102.06701 . * Bansal et al. [2022] ↑ Yamini Bansal, Behrooz Ghorbani, Ankush Garg, Biao Zhang, Maxim Krikun, Colin Cherry, Behnam Neyshabur, and Orhan Firat. Data scaling laws in nmt: The effect of noise and architecture. In International Conference on Machine Learning (ICML) , 2022. https://proceedings.mlr.press/v162/bansal22b.html . * bench authors [2023] ↑ BIG bench authors. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. In Transactions on Machine Learning Research (TMLR) , 2023. https://openreview.net/forum?id=uyTL5Bvosj . * Bisk et al. [2020] ↑ Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. Piqa: Reasoning about physical commonsense in natural language. In Association for the Advancement of Artificial Intelligence (AAAI) , 2020. https://arxiv.org/abs/1911.11641 . * Black et al. [2022] ↑ Sid Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, Michael Pieler, USVSN Sai Prashanth, Shivanshu Purohit, Laria Reynolds, Jonathan Tow, Ben Wang, and Samuel Weinbach. Gpt-neox-20b: An open-source autoregressive language model. BigScience Episode #5 – Workshop on Challenges & Perspectives in Creating Large Language Models , 2022. https://aclanthology.org/2022.bigscience-1.9 . * Brown et al. [2020] ↑ Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Advances in Neural Information Processing Systems (NeurIPS) , 2020. https://arxiv.org/abs/2005.14165 . * Caballero et al. [2023] ↑ Ethan Caballero, Kshitij Gupta, Irina Rish, and David Krueger. Broken neural scaling laws. In International Conference on Learning Representations (ICLR) , 2023. https://openreview.net/forum?id=sckjveqlCZ . * Cherti et al. [2023] ↑ Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-image learning. In Conference on Computer Vision and Pattern Recognition (CVPR) , 2023. https://arxiv.org/abs/2212.07143 . * Chowdhery et al. [2022] ↑ Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam M. Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Benton C. Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier García, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Díaz, Orhan Firat, Michele Catasta, Jason Wei, Kathleen S. Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. Palm: Scaling language modeling with pathways. In Journal of Machine Learning Research (JMLR) , 2022. https://arxiv.org/abs/2204.02311 . * Chung et al. [2022] ↑ Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction-finetuned language models. arXiv preprint , 2022. https://arxiv.org/abs/2210.11416 . * Clark et al. [2019] ↑ Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. In Conference of the North American Chapter of the Association for Computational Linguistics (NAACL) , 2019. https://aclanthology.org/N19-1300 . * Clark et al. [2020] ↑ Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. ELECTRA: Pre-training text encoders as discriminators rather than generators. In International Conference on Learning Representations (ICLR) , 2020. https://openreview.net/pdf?id=r1xMH1BtvB . * Clark et al. [2018] ↑ Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint , 2018. https://arxiv.org/abs/1803.05457 . * Dao et al. [2022] ↑ Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. FlashAttention: Fast and memory-efficient exact attention with IO-awareness. In Advances in Neural Information Processing Systems (NeurIPS) , 2022. https://arxiv.org/abs/2205.14135 . * Dehghani et al. [2023] ↑ Mostafa Dehghani, Josip Djolonga, Basil Mustafa, Piotr Padlewski, Jonathan Heek, Justin Gilmer, Andreas Peter Steiner, Mathilde Caron, Robert Geirhos, Ibrahim Alabdulmohsin, et al. Scaling vision transformers to 22 billion parameters. In International Conference on Machine Learning (ICML) , 2023. https://proceedings.mlr.press/v202/dehghani23a.html . * Devlin et al. [2019] ↑ Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Conference of the North American Chapter of the Association for Computational Linguistics (NAACL) , 2019. https://aclanthology.org/N19-1423 . * Dodge et al. [2021] ↑ Jesse Dodge, Maarten Sap, Ana Marasović, William Agnew, Gabriel Ilharco, Dirk Groeneveld, Margaret Mitchell, and Matt Gardner. Documenting large webtext corpora: A case study on the colossal clean crawled corpus. In Conference on Empirical Methods in Natural Language Processing (EMNLP) , 2021. https://aclanthology.org/2021.emnlp-main.98 . * Du et al. [2022] ↑ Nan Du, Yanping Huang, Andrew M. Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, Barret Zoph, Liam Fedus, Maarten Bosma, Zongwei Zhou, Tao Wang, Yu Emma Wang, Kellie Webster, Marie Pellat, Kevin Robinson, Kathleen Meier-Hellstern, Toju Duke, Lucas Dixon, Kun Zhang, Quoc V Le, Yonghui Wu, Zhifeng Chen, and Claire Cui. Glam: Efficient scaling of language models with mixture-of-experts. In International Conference on Machine Learning (ICML) , 2022. https://arxiv.org/abs/2112.06905 . * Ethayarajh et al. [2024] ↑ Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model alignment as prospect theoretic optimization. arXiv preprint , 2024. https://arxiv.org/abs/2402.01306 . * Gadre et al. [2023] ↑ Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Mitchell Wortsman Ryan Marten, Dhruba Ghosh, Jieyu Zhang, Eyal Orgad, Rahim Entezari, Giannis Daras, Sarah Pratt, Vivek Ramanujan, Yonatan Bitton, Kalyani Marathe, Stephen Mussmann, Mehdi Cherti Richard Vencu, Ranjay Krishna, Pang Wei Koh, Olga Saukh, Alexander Ratner, Shuran Song, Hannaneh Hajishirzi, Ali Farhadi, Romain Beaumont, Sewoong Oh, Alex Dimakis, Jenia Jitsev, Yair Carmon, Vaishaal Shankar, and Ludwig Schmidt. Datacomp: In search of the next generation of multimodal datasets. In Advances in Neural Information Processing Systems (NeurIPS) , 2023. https://arxiv.org/abs/2304.14108 . * Gao et al. [2020] ↑ Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The Pile: An 800gb dataset of diverse text for language modeling. arXiv preprint , 2020. https://arxiv.org/abs/2101.00027 . * Ghorbani et al. [2021] ↑ Behrooz Ghorbani, Orhan Firat, Markus Freitag, Ankur Bapna, Maxim Krikun, Xavier Garcia, Ciprian Chelba, and Colin Cherry. Scaling laws for neural machine translation. arXiv preprint , 2021. https://arxiv.org/abs/2109.07740 . * Gordon et al. [2021] ↑ Mitchell A Gordon, Kevin Duh, and Jared Kaplan. Data and parameter scaling laws for neural machine translation. In Conference on Empirical Methods in Natural Language Processing (EMNLP) , 2021. https://aclanthology.org/2021.emnlp-main.478 . * Groeneveld et al. [2024] ↑ Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Harsh Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, et al. Olmo: Accelerating the science of language models. arXiv preprint , 2024. https://arxiv.org/abs/2402.00838 . * Gu & Dao [2023] ↑ Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint , 2023. https://arxiv.org/abs/2312.00752 . * Gu et al. [2021] ↑ Albert Gu, Isys Johnson, Karan Goel, Khaled Saab, Tri Dao, Atri Rudra, and Christopher Ré. Combining recurrent, convolutional, and continuous-time models with linear state space layers. In Advances in Neural Information Processing Systems (NeurIPS) , 2021. https://openreview.net/forum?id=yWd42CWN3c . * Gu et al. [2022] ↑ Albert Gu, Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured state spaces. In International Conference on Learning Representations (ICLR) , 2022. https://arxiv.org/abs/2111.00396 . * Gunasekar et al. [2023] ↑ Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio Cesar, Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, Adil Salim, Shital Shah, Harkirat Singh Behl, Xin Wang, Sébastien Bubeck, Ronen Eldan, Adam Tauman Kalai, Yin Tat Lee, and Yuanzhi Li. Textbooks are all you need. Preprint , 2023. https://www.microsoft.com/en-us/research/publication/textbooks-are-all-you-need . * Gururangan et al. [2023] ↑ Suchin Gururangan, Mitchell Wortsman, Samir Yitzhak Gadre, Achal Dave, Maciej Kilian, Weijia Shi, Jean Mercat, Georgios Smyrnis, Gabriel Ilharco, Matt Jordan, Reinhard Heckel, Alex Dimakis, Ali Farhadi, Vaishaal Shankar, and Ludwig Schmidt. OpenLM: a minimal but performative language modeling (lm) repository, 2023. https://github.com/mlfoundations/open_lm . * Hendrycks et al. [2021] ↑ Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations (ICLR) , 2021. https://arxiv.org/abs/2009.03300 . * Henighan et al. [2020] ↑ T. J. Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B. Brown, Prafulla Dhariwal, Scott Gray, Chris Hallacy, Benjamin Mann, Alec Radford, Aditya Ramesh, Nick Ryder, Daniel M. Ziegler, John Schulman, Dario Amodei, and Sam McCandlish. Scaling laws for autoregressive generative modeling. arXiv preprint , 2020. https://arxiv.org/abs/2010.14701 . * Hernandez et al. [2021] ↑ Danny Hernandez, Jared Kaplan, T. J. Henighan, and Sam McCandlish. Scaling laws for transfer. arXiv preprint , 2021. https://arxiv.org/abs/2102.01293 . * Hestness et al. [2017] ↑ Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Frederick Diamos, Heewoo Jun, Hassan Kianinejad, Md. Mostofa Ali Patwary, Yang Yang, and Yanqi Zhou. Deep learning scaling is predictable, empirically. arXiv preprint , 2017. https://arxiv.org/abs/1712.00409 . * Hestness et al. [2019] ↑ Joel Hestness, Newsha Ardalani, and Gregory Diamos. Beyond human-level accuracy: Computational challenges in deep learning. In Principles and Practice of Parallel Programming (PPoPP) , 2019. https://arxiv.org/abs/1909.01736 . * Hoffmann et al. [2022] ↑ Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. In Advances in Neural Information Processing Systems (NeurIPS) , 2022. https://arxiv.org/abs/2203.15556 . * Inan et al. [2017] ↑ Hakan Inan, Khashayar Khosravi, and Richard Socher. Tying word vectors and word classifiers: A loss framework for language modeling. In International Conference on Learning Representations (ICLR) , 2017. https://arxiv.org/abs/1611.01462 . * Isik et al. [2024] ↑ Berivan Isik, Natalia Ponomareva, Hussein Hazimeh, Dimitris Paparas, Sergei Vassilvitskii, and Sanmi Koyejo. Scaling laws for downstream task performance of large language models. arXiv , 2024. https://arxiv.org/abs/2402.04177 . * Ivgi et al. [2022] ↑ Maor Ivgi, Yair Carmon, and Jonathan Berant. Scaling laws under the microscope: Predicting transformer performance from small scale experiments. In Conference on Empirical Methods in Natural Language Processing (EMNLP) , 2022. https://aclanthology.org/2022.findings-emnlp.544 . * Jiang et al. [2023] ↑ Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Florian Bressand Diego de las Casas, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b. arXiv preprint , 2023. https://arxiv.org/abs/2310.06825 . * Jin et al. [2019] ↑ Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu. Pubmedqa: A dataset for biomedical research question answering. In Conference on Empirical Methods in Natural Language Processing (EMNLP) , 2019. https://aclanthology.org/D19-1259 . * Kaplan et al. [2020] ↑ Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint , 2020. https://arxiv.org/abs/2001.08361 . * Klug et al. [2023] ↑ Tobit Klug, Dogukan Atik, and Reinhard Heckel. Analyzing the sample complexity of self-supervised image reconstruction methods. arXiv preprint , 2023. https://arxiv.org/abs/2305.19079 . * Lan et al. [2019] ↑ Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. ALBERT: A lite BERT for self-supervised learning of language representations. arXiv preprint , 2019. http://arxiv.org/abs/1909.11942 . * Lefaudeux et al. [2022] ↑ Benjamin Lefaudeux, Francisco Massa, Diana Liskovich, Wenhan Xiong, Vittorio Caggiano, Sean Naren, Min Xu, Jieru Hu, Marta Tintore, Susan Zhang, Patrick Labatut, and Daniel Haziza. xformers: A modular and hackable transformer modelling library, 2022. https://github.com/facebookresearch/xformers . * Levesque et al. [2012] ↑ Hector Levesque, Ernest Davis, and Leora Morgenstern. The winograd schema challenge. In International conference on the principles of knowledge representation and reasoning , 2012. https://aaai.org/papers/59-4492-the-winograd-schema-challenge . * Lewis et al. [2020] ↑ Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Annual Meeting of the Association for Computational Linguistics (ACL) , 2020. https://aclanthology.org/2020.acl-main.703 . * Li et al. [2023] ↑ Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, et al. Starcoder: may the source be with you! arXiv preprint , 2023. https://arxiv.org/abs/2305.06161 . * Liu et al. [2020] ↑ Jian Liu, Leyang Cui, Hanmeng Liu, Dandan Huang, Yile Wang, and Yue Zhang. Logiqa: A challenge dataset for machine reading comprehension with logical reasoning. In International Joint Conference on Artificial Intelligence , 2020. https://arxiv.org/abs/2007.08124 . * Liu et al. [2019] ↑ Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized BERT pretraining approach. arXiv preprint , 2019. http://arxiv.org/abs/1907.11692 . * Liu et al. [2022] ↑ Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. Conference on Computer Vision and Pattern Recognition (CVPR) , 2022. https://arxiv.org/abs/2201.03545 . * Longpre et al. [2023] ↑ Shayne Longpre, Robert Mahari, Anthony Chen, Naana Obeng-Marnu, Damien Sileo, William Brannon, Niklas Muennighoff, Nathan Khazam, Jad Kabbara, Kartik Perisetla, et al. The data provenance initiative: A large scale audit of dataset licensing & attribution in ai. arXiv preprint , 2023. https://arxiv.org/abs/2310.16787 . * Loshchilov & Hutter [2017] ↑ Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint , 2017. https://arxiv.org/abs/1711.05101 . * Lozhkov et al. [2024] ↑ Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy-Poirier, Nouamane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, Tianyang Liu, Max Tian, Denis Kocetkov, Arthur Zucker, Younes Belkada, Zijian Wang, Qian Liu, Dmitry Abulkhanov, Indraneil Paul, Zhuang Li, Wen-Ding Li, Megan Risdal, Jia Li, Jian Zhu, Terry Yue Zhuo, Evgenii Zheltonozhskii, Nii Osae Osae Dade, Wenhao Yu, Lucas Krauß, Naman Jain, Yixuan Su, Xuanli He, Manan Dey, Edoardo Abati, Yekun Chai, Niklas Muennighoff, Xiangru Tang, Muhtasham Oblokulov, Christopher Akiki, Marc Marone, Chenghao Mou, Mayank Mishra, Alex Gu, Binyuan Hui, Tri Dao, Armel Zebaze, Olivier Dehaene, Nicolas Patry, Canwen Xu, Julian McAuley, Han Hu, Torsten Scholak, Sebastien Paquet, Jennifer Robinson, Carolyn Jane Anderson, Nicolas Chapados, Mostofa Patwary, Nima Tajbakhsh, Yacine Jernite, Carlos Muñoz Ferrandis, Lingming Zhang, Sean Hughes, Thomas Wolf, Arjun Guha, Leandro von Werra, and Harm de Vries. Starcoder 2 and the stack v2: The next generation. arXiv preprint , 2024. https://arxiv.org/abs/2402.19173 . * Luukkonen et al. [2023] ↑ Risto Luukkonen, Ville Komulainen, Jouni Luoma, Anni Eskelinen, Jenna Kanerva, Hanna-Mari Kupari, Filip Ginter, Veronika Laippala, Niklas Muennighoff, Aleksandra Piktus, et al. Fingpt: Large generative models for a small language. In Conference on Empirical Methods in Natural Language Processing (EMNLP) , 2023. https://aclanthology.org/2023.emnlp-main.164 . * Magnusson et al. [2023] ↑ Ian Magnusson, Akshita Bhagia, Valentin Hofmann, Luca Soldaini, Ananya Harsh Jha, Oyvind Tafjord, Dustin Schwenk, Evan Pete Walsh, Yanai Elazar, Kyle Lo, Dirk Groenveld, Iz Beltagy, Hanneneh Hajishirz, Noah A. Smith, Kyle Richardson, and Jesse Dodge. Paloma: A benchmark for evaluating language model fit. arXiv preprint , 2023. https://paloma.allen.ai . * Marcus et al. [1993] ↑ Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. Building a large annotated corpus of English: The Penn Treebank. In Computational Linguistics , 1993. https://aclanthology.org/J93-2004 . * Merrill et al. [2021] ↑ William Merrill, Vivek Ramanujan, Yoav Goldberg, Roy Schwartz, and Noah A. Smith. Effects of parameter norm growth during transformer training: Inductive bias from gradient descent. In Conference on Empirical Methods in Natural Language Processing (EMNLP) , 2021. https://aclanthology.org/2021.emnlp-main.133 . * Mihaylov et al. [2018] ↑ Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. In Conference on Empirical Methods in Natural Language Processing (EMNLP) , 2018. https://arxiv.org/abs/1809.02789 . * MosaicML [2023] ↑ MosaicML. Llm evaluation scores, 2023. https://www.mosaicml.com/llm-evaluation . * Muennighoff et al. [2022] ↑ Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng-Xin Yong, Hailey Schoelkopf, et al. Crosslingual generalization through multitask finetuning. In Annual Meeting of the Association for Computational Linguistics (ACL) , 2022. https://aclanthology.org/2023.acl-long.891 . * Muennighoff et al. [2023a] ↑ Niklas Muennighoff, Qian Liu, Armel Zebaze, Qinkai Zheng, Binyuan Hui, Terry Yue Zhuo, Swayam Singh, Xiangru Tang, Leandro Von Werra, and Shayne Longpre. Octopack: Instruction tuning code large language models. arXiv preprint , 2023a. https://arxiv.org/abs/2308.07124 . * Muennighoff et al. [2023b] ↑ Niklas Muennighoff, Alexander M Rush, Boaz Barak, Teven Le Scao, Aleksandra Piktus, Nouamane Tazi, Sampo Pyysalo, Thomas Wolf, and Colin Raffel. Scaling data-constrained language models. In Advances in Neural Information Processing Systems (NeuIPS) , 2023b. https://arxiv.org/abs/2305.16264 . * Muennighoff et al. [2024] ↑ Niklas Muennighoff, Hongjin Su, Liang Wang, Nan Yang, Furu Wei, Tao Yu, Amanpreet Singh, and Douwe Kiela. Generative representational instruction tuning. arXiv preprint , 2024. https://arxiv.org/abs/2402.09906 . * Nijkamp et al. [2023] ↑ Erik Nijkamp, Tian Xie, Hiroaki Hayashi, Bo Pang, Congying Xia, Chen Xing, Jesse Vig, Semih Yavuz, Philippe Laban, Ben Krause, Senthil Purushwalkam, Tong Niu, Wojciech Kryscinski, Lidiya Murakhovs’ka, Prafulla Kumar Choubey, Alex Fabbri, Ye Liu, Rui Meng, Lifu Tu, Meghana Bhat, Chien-Sheng Wu, Silvio Savarese, Yingbo Zhou, Shafiq Rayhan Joty, and Caiming Xiong. Long sequence modeling with xgen: A 7b llm trained on 8k input sequence length. arXiv preprint , 2023. https://arxiv.org/abs/2309.03450 . * OpenAI [2021] ↑ OpenAI. Triton, 2021. https://github.com/openai/triton . * OpenAI [2023] ↑ OpenAI. Gpt-4 technical report, 2023. https://arxiv.org/abs/2303.08774 . * Paperno et al. [2016] ↑ Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Ngoc Quan Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernandez. The LAMBADA dataset: Word prediction requiring a broad discourse context. In Annual Meeting of the Association for Computational Linguistics (ACL) , 2016. http://www.aclweb.org/anthology/P16-1144 . * Papineni et al. [2002] ↑ Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Annual Meeting of the Association for Computational Linguistics (ACL) , 2002. https://aclanthology.org/P02-1040 . * Parrish et al. [2022] ↑ Alicia Parrish, Angelica Chen, Nikita Nangia, Vishakh Padmakumar, Jason Phang, Jana Thompson, Phu Mon Htut, and Samuel Bowman. BBQ: A hand-built bias benchmark for question answering. In Annual Meeting of the Association for Computational Linguistics (ACL) , 2022. https://aclanthology.org/2022.findings-acl.165 . * Paszke et al. [2019] ↑ Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems (NeurIPS) , 2019. https://arxiv.org/abs/1912.01703 . * Patronus AI [2023] ↑ Patronus AI. EnterprisePII dataset, 2023. https://tinyurl.com/2r5x9bst . * Penedo et al. [2023] ↑ Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. The RefinedWeb dataset for Falcon LLM: outperforming curated corpora with web data, and web data only. arXiv preprint , 2023. https://arxiv.org/abs/2306.01116 . * Peng et al. [2023] ↑ Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Stella Biderman, Huanqi Cao, Xin Cheng, Michael Chung, Leon Derczynski, Xingjian Du, Matteo Grella, Kranthi Gv, Xuzheng He, Haowen Hou, Przemyslaw Kazienko, Jan Kocon, Jiaming Kong, Bartłomiej Koptyra, Hayden Lau, Jiaju Lin, Krishna Sri Ipsit Mantri, Ferdinand Mom, Atsushi Saito, Guangyu Song, Xiangru Tang, Johan Wind, Stanisław Woźniak, Zhenyuan Zhang, Qinghua Zhou, Jian Zhu, and Rui-Jie Zhu. RWKV: Reinventing RNNs for the transformer era. In Conference on Empirical Methods in Natural Language Processing (EMNLP) , 2023. https://aclanthology.org/2023.findings-emnlp.936 . * Press & Wolf [2017] ↑ Ofir Press and Lior Wolf. Using the output embedding to improve language models. In Proceedings of the Conference of the European Chapter of the Association for Computational Linguistics (EACL) , 2017. https://aclanthology.org/E17-2025 . * Radford et al. [2019] ↑ Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. Preprint , 2019. https://d4mucfpksywv.cloudfront.net/better-language-models/language_models_are_unsupervised_multitask_learners.pdf . * Rae et al. [2021] ↑ Jack W. Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, Eliza Rutherford, Tom Hennigan, Jacob Menick, Albin Cassirer, Richard Powell, George van den Driessche, Lisa Anne Hendricks, Maribeth Rauh, Po-Sen Huang, Amelia Glaese, Johannes Welbl, Sumanth Dathathri, Saffron Huang, Jonathan Uesato, John F. J. Mellor, Irina Higgins, Antonia Creswell, Nathan McAleese, Amy Wu, Erich Elsen, Siddhant M. Jayakumar, Elena Buchatskaya, David Budden, Esme Sutherland, Karen Simonyan, Michela Paganini, L. Sifre, Lena Martens, Xiang Lorraine Li, Adhiguna Kuncoro, Aida Nematzadeh, Elena Gribovskaya, Domenic Donato, Angeliki Lazaridou, Arthur Mensch, Jean-Baptiste Lespiau, Maria Tsimpoukelli, N. K. Grigorev, Doug Fritz, Thibault Sottiaux, Mantas Pajarskas, Tobias Pohlen, Zhitao Gong, Daniel Toyama, Cyprien de Masson d’Autume, Yujia Li, Tayfun Terzi, Vladimir Mikulik, Igor Babuschkin, Aidan Clark, Diego de Las Casas, Aurelia Guy, Chris Jones, James Bradbury, Matthew G. Johnson, Blake A. Hechtman, Laura Weidinger, Iason Gabriel, William S. Isaac, Edward Lockhart, Simon Osindero, Laura Rimell, Chris Dyer, Oriol Vinyals, Kareem W. Ayoub, Jeff Stanway, L. L. Bennett, Demis Hassabis, Koray Kavukcuoglu, and Geoffrey Irving. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint , 2021. https://arxiv.org/abs/2112.11446 . * Rafailov et al. [2023] ↑ Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. In Advances in Neural Information Processing Systems (NeurIPS) , 2023. https://arxiv.org/abs/2305.18290 . * Raffel et al. [2019] ↑ Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint , 2019. https://arxiv.org/abs/1910.10683 . * Raffel et al. [2020] ↑ Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. In The Journal of Machine Learning Research (JMLR) , 2020. https://arxiv.org/abs/1910.10683 . * Rajpurkar et al. [2016] ↑ Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In Conference on Empirical Methods in Natural Language Processing (EMNLP) , 2016. https://aclanthology.org/D16-1264 . * Reddy et al. [2019] ↑ Siva Reddy, Danqi Chen, and Christopher D. Manning. CoQA: A conversational question answering challenge. In Transactions of the Association for Computational Linguistics (TACL) , 2019. https://aclanthology.org/Q19-1016 . * Roemmele et al. [2011] ↑ Melissa Roemmele, Cosmin Adrian Bejan, , and Andrew S. Gordon. Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In Association for the Advancement of Artificial Intelligence (AAAI) Spring Symposium , 2011. https://people.ict.usc.edu/~gordon/copa.html . * Rosenfeld et al. [2020] ↑ Jonathan S. Rosenfeld, Amir Rosenfeld, Yonatan Belinkov, and Nir Shavit. A constructive prediction of the generalization error across scales. In International Conference on Learning Representations (ICLR) , 2020. https://arxiv.org/abs/1909.12673 . * Rudinger et al. [2018] ↑ Rachel Rudinger, Jason Naradowsky, Brian Leonard, and Benjamin Van Durme. Gender bias in coreference resolution. In Conference of the North American Chapter of the Association for Computational Linguistics (NAACL) , 2018. https://aclanthology.org/N18-2002 . * Sakaguchi et al. [2019] ↑ Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. arXiv preprint , 2019. https://arxiv.org/abs/1907.10641 . * Sanh et al. [2019] ↑ Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint , 2019. http://arxiv.org/abs/1910.01108 . * Sap et al. [2019] ↑ Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. Social IQa: Commonsense reasoning about social interactions. In Empirical Methods in Natural Language Processing (EMNLP) , 2019. https://aclanthology.org/D19-1454 . * Sardana & Frankle [2023] ↑ Nikhil Sardana and Jonathan Frankle. Beyond chinchilla-optimal: Accounting for inference in language model scaling laws. In NeurIPS Workshop on Efficient Natural Language and Speech Processing (ENLSP) , 2023. https://arxiv.org/abs/2401.00448 . * Scao et al. [2022] ↑ Teven Le Scao, Thomas Wang, Daniel Hesslow, Lucile Saulnier, Stas Bekman, M Saiful Bari, Stella Biderman, Hady Elsahar, Niklas Muennighoff, Jason Phang, et al. What language model to train if you have one million gpu hours? In Conference on Empirical Methods in Natural Language Processing (EMNLP) , 2022. https://aclanthology.org/2022.findings-emnlp.54 . * Schaeffer et al. [2023] ↑ Rylan Schaeffer, Brando Miranda, and Sanmi Koyejo. Are emergent abilities of large language models a mirage? In Advances in Neural Information Processing Systems (NeurIPS) , 2023. https://arxiv.org/abs/2304.15004 . * Sharma & Kaplan [2022] ↑ Utkarsh Sharma and Jared Kaplan. A neural scaling law from the dimension of the data manifold. In Journal of Machine Learning Research (JMLR) , 2022. https://arxiv.org/abs/2004.10802 . * Shazeer [2020] ↑ Noam Shazeer. Glu variants improve transformer. arXiv preprint , 2020. https://arxiv.org/abs/2002.05202 . * Singh et al. [2024] ↑ Shivalika Singh, Freddie Vargus, Daniel Dsouza, Börje F Karlsson, Abinaya Mahendiran, Wei-Yin Ko, Herumb Shandilya, Jay Patel, Deividas Mataciunas, Laura OMahony, et al. Aya dataset: An open-access collection for multilingual instruction tuning. arXiv preprint arXiv:2402.06619 , 2024. https://arxiv.org/abs/2402.06619 . * Soldaini et al. [2024] ↑ Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, et al. Dolma: An open corpus of three trillion tokens for language model pretraining research. arXiv preprint , 2024. https://arxiv.org/abs/2402.00159 . * Sorscher et al. [2022] ↑ Ben Sorscher, Robert Geirhos, Shashank Shekhar, Surya Ganguli, and Ari S. Morcos. Beyond neural scaling laws: beating power law scaling via data pruning. In Advances in Neural Information Processing Systems (NeurIPS) , 2022. https://openreview.net/forum?id=UmvSlP-PyV . * Su et al. [2021] ↑ Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. arXiv preprint , 2021. https://arxiv.org/abs/2104.09864 . * Talmor et al. [2019] ↑ Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. CommonsenseQA: A question answering challenge targeting commonsense knowledge. In Conference of the North American Chapter of the Association for Computational Linguistics (NAACL) , 2019. https://aclanthology.org/N19-1421 . * Tay et al. [2022] ↑ Yi Tay, Mostafa Dehghani, Jinfeng Rao, William Fedus, Samira Abnar, Hyung Won Chung, Sharan Narang, Dani Yogatama, Ashish Vaswani, and Donald Metzler. Scale efficiently: Insights from pre-training and fine-tuning transformers. In International Conference on Learning Representations (ICLR) , 2022. https://openreview.net/forum?id=f2OYVDyfIB . * Tay et al. [2023] ↑ Yi Tay, Mostafa Dehghani, Samira Abnar, Hyung Chung, William Fedus, Jinfeng Rao, Sharan Narang, Vinh Tran, Dani Yogatama, and Donald Metzler. Scaling laws vs model architectures: How does inductive bias influence scaling? In Conference on Empirical Methods in Natural Language Processing (EMNLP) , 2023. https://aclanthology.org/2023.findings-emnlp.825 . * Team [2023] ↑ MosaicML NLP Team. Introducing mpt-7b: A new standard for open-source, commercially usable llms, 2023. www.mosaicml.com/blog/mpt-7b . * Thoppilan et al. [2022] ↑ Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, YaGuang Li, Hongrae Lee, Huaixiu Steven Zheng, Amin Ghafouri, Marcelo Menegali, Yanping Huang, Maxim Krikun, Dmitry Lepikhin, James Qin, Dehao Chen, Yuanzhong Xu, Zhifeng Chen, Adam Roberts, Maarten Bosma, Vincent Zhao, Yanqi Zhou, Chung-Ching Chang, Igor Krivokon, Will Rusch, Marc Pickett, Pranesh Srinivasan, Laichee Man, Kathleen Meier-Hellstern, Meredith Ringel Morris, Tulsee Doshi, Renelito Delos Santos, Toju Duke, Johnny Soraker, Ben Zevenbergen, Vinodkumar Prabhakaran, Mark Diaz, Ben Hutchinson, Kristen Olson, Alejandra Molina, Erin Hoffman-John, Josh Lee, Lora Aroyo, Ravi Rajakumar, Alena Butryna, Matthew Lamm, Viktoriya Kuzmina, Joe Fenton, Aaron Cohen, Rachel Bernstein, Ray Kurzweil, Blaise Aguera-Arcas, Claire Cui, Marian Croak, Ed Chi, and Quoc Le. Lamda: Language models for dialog applications. arXiv preprint , 2022. https://arxiv.org/abs/2201.08239 . * Together Computer [2023] ↑ Together Computer. Redpajama: an open dataset for training large language models, 2023. https://github.com/togethercomputer/RedPajama-Data . * Touvron et al. [2023a] ↑ Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. LLaMA: Open and Efficient Foundation Language Models. arXiv preprint , 2023a. https://arxiv.org/abs/2302.13971 . * Touvron et al. [2023b] ↑ Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open Foundation and Fine-Tuned Chat Models. arXiv preprint , 2023b. https://arxiv.org/abs/2307.09288 . * Üstün et al. [2024] ↑ Ahmet Üstün, Viraat Aryabumi, Zheng-Xin Yong, Wei-Yin Ko, Daniel D’souza, Gbemileke Onilude, Neel Bhandari, Shivalika Singh, Hui-Lee Ooi, Amr Kayid, et al. Aya model: An instruction finetuned open-access multilingual language model. arXiv preprint , 2024. https://arxiv.org/abs/2402.07827 . * Vaswani et al. [2017] ↑ Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems (NeurIPS) , 2017. https://arxiv.org/abs/1706.03762 . * Virtanen et al. [2020] ↑ Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J. van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. J. Nelson, Eric Jones, Robert Kern, Eric Larson, C J Carey, İlhan Polat, Yu Feng, Eric W. Moore, Jake VanderPlas, Denis Laxalde, Josef Perktold, Robert Cimrman, Ian Henriksen, E. A. Quintero, Charles R. Harris, Anne M. Archibald, Antônio H. Ribeiro, Fabian Pedregosa, Paul van Mulbregt, and SciPy 1.0 Contributors. SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python. Nature Methods , 2020. https://rdcu.be/b08Wh . * Wang et al. [2021] ↑ Siyuan Wang, Zhongkun Liu, Wanjun Zhong, Ming Zhou, Zhongyu Wei, Zhumin Chen, and Nan Duan. From lsat: The progress and challenges of complex reasoning. Transactions on Audio, Speech, and Language Processing , 2021. https://arxiv.org/abs/2108.00648 . * Wei et al. [2022a] ↑ Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. Finetuned language models are zero-shot learners. In International Conference on Learning Representations (ICLR) , 2022a. https://openreview.net/forum?id=gEZrGCozdqR . * Wei et al. [2022b] ↑ Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. Emergent abilities of large language models. In Transactions on Machine Learning Research (TMLR) , 2022b. https://openreview.net/forum?id=yzkSU5zdwD . * Workshop et al. [2022] ↑ BigScience Workshop, Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, et al. Bloom: A 176b-parameter open-access multilingual language model. arXiv preprint , 2022. https://arxiv.org/abs/2211.05100 . * Wortsman et al. [2023] ↑ Mitchell Wortsman, Peter J Liu, Lechao Xiao, Katie Everett, Alex Alemi, Ben Adlam, John D Co-Reyes, Izzeddin Gur, Abhishek Kumar, Roman Novak, et al. Small-scale proxies for large-scale transformer training instabilities. arXiv preprint , 2023. https://arxiv.org/abs/2309.14322 . * Yang et al. [2021] ↑ Greg Yang, Edward J. Hu, Igor Babuschkin, Szymon Sidor, Xiaodong Liu, David Farhi, Nick Ryder, Jakub Pachocki, Weizhu Chen, and Jianfeng Gao. Tensor programs V: Tuning large neural networks via zero-shot hyperparameter transfer. In Advances in Neural Information Processing Systems (NeuIPS) , 2021. https://arxiv.org/abs/2203.03466 . * Yang et al. [2024] ↑ Greg Yang, Dingli Yu, Chen Zhu, and Soufiane Hayou. Feature learning in infinite depth neural networks. In International Conference on Learning Representations (ICLR) , 2024. https://openreview.net/forum?id=17pVDnpwwl . * Zellers et al. [2019] ↑ Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? In Annual Meeting of the Association for Computational Linguistics (ACL) , 2019. https://aclanthology.org/P19-1472 . * Zhai et al. [2022] ↑ Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transformers. In Conference on Computer Vision and Pattern Recognition (CVPR) , 2022. https://arxiv.org/abs/2106.04560 . * Zhang & Sennrich [2019] ↑ Biao Zhang and Rico Sennrich. Root mean square layer normalization. In Advances in Neural Information Processing Systems (NeuIPS) , 2019. https://arxiv.org/abs/1910.07467 . * Zhang et al. [2019] ↑ Biao Zhang, Ivan Titov, and Rico Sennrich. Improving deep transformer with depth-scaled initialization and merged attention. In Empirical Methods in Natural Language Processing (EMNLP) , 2019. https://aclanthology.org/D19-1083 . * Zhao et al. [2023] ↑ Yanli Zhao, Andrew Gu, Rohan Varma, Liangchen Luo, Chien chin Huang, Min Xu, Less Wright, Hamid Shojanazeri, Myle Ott, Sam Shleifer, Alban Desmaison, Can Balioglu, Bernard Nguyen, Geeta Chauhan, Yuchen Hao, and Shen Li. Pytorch fsdp: Experiences on scaling fully sharded data parallel. In Very Large Data Bases Conference (VLDB) , 2023. https://dl.acm.org/doi/10.14778/3611540.3611569 . * Zhong et al. [2020] ↑ Haoxi Zhong, Chaojun Xiao, Cunchao Tu, Tianyang Zhang, Zhiyuan Liu, and Maosong Sun. Jec-qa: A legal-domain question answering dataset. In Association for the Advancement of Artificial Intelligence (AAAI) , 2020. https://arxiv.org/abs/1911.12011 . * Zhong et al. [2023] ↑ Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan. Agieval: A human-centric benchmark for evaluating foundation models. arXiv preprint , 2023. https://arxiv.org/abs/2304.06364 . * Zhuo et al. [2024] ↑ Terry Yue Zhuo, Armel Zebaze, Nitchakarn Suppattarachai, Leandro von Werra, Harm de Vries, Qian Liu, and Niklas Muennighoff. Astraios: Parameter-efficient instruction tuning code large language models. arXiv preprint , 2024. https://arxiv.org/abs/2401.00788 . Contents 1. 1 Introduction 2. 2 Scaling and over-training 1. 2.1 Preliminaries 2. 2.2 Observation: Over-trained models follow consistent trends 3. 2.3 Deriving scaling laws for over-trained behavior 4. 2.4 Observation: Loss tracks average top-1 error 5. 2.5 Proposing a scaling law for average top-1 error 3. 3 Experimental setup 1. 3.1 Training setup 2. 3.2 Creating scaling laws for validation loss and downstream error prediction 3. 3.3 Evaluation setup 4. 4 Results 5. 5 Related work 6. 6 Limitations, future work, and conclusion 7. A Contributions 8. B Scaling-law derivations 9. C Additional grid search details 10. D Evaluation dataset details 11. E Additional results 12. F Additional related work Appendix A Contributions Names ordered alphabetically. Planning. Samir Yitzhak Gadre Model training and experiment babysitting. Achal Dave (notably, the 1.4B parameter, 900B token run), Samir Yitzhak Gadre Dataloading. Georgios Smyrnis Training tokens. Achal Dave, Alex Fang, Samir Yitzhak Gadre, Suchin Gururangan, Jeffrey Li, Vaishaal Shankar (lead), Mitchell Wortsman Evaluation tokens. Achal Dave, Samir Yitzhak Gadre, Reinhard Heckel, Vaishaal Shankar (lead), Rulin Shao Loss/perplexity evaluation. Samir Yitzhak Gadre Downstream evaluation. Vaishaal Shankar Project-specific infrastructure, plots, and analysis. Samir Yitzhak Gadre OpenLM [37] open-source infrastructure. Achal Dave (core contributor), Alex Fang, Samir Yitzhak Gadre (core contributor), Suchin Gururangan (core contributor), Jenia Jitsev, Sedrick Keh, Jeffrey Li, Jean Mercat, Marianna Nezhurina, Vaishaal Shankar (core contributor), Georgios Smyrnis (core contributor), Igor Vasiljevic, Mitchell Wortsman (core contributor), Rui Xin Theory. Yair Carmon (original idea that “parallel lines” should show up in scaling plots), Samir Yitzhak Gadre (various derivations, empirical verification, related validation loss to average top-1 error as in Equation ( 5 )), Reinhard Heckel (derived a scaling form based on Chinchilla Approach 3 [43], which appears in Equation ( 4 )), Mitchell Wortsman (provided intuition about irreducible loss and why it is critical), Niklas Muennighoff (derived a scaling form based on Chinchilla Approach 3, similar to Equation ( 4 )). Writing. Yair Carmon, Achal Dave, Reinhard Heckel, Samir Yitzhak Gadre (lead), Niklas Muennighoff, Ludwig Schmidt Compute. Achal Dave, Jenia Jitsev, Thomas Kollar, Ludwig Schmidt, Vaishaal Shankar Advising. Yair Carmon (co-lead), Achal Dave (co-lead), Alexandros G. Dimakis, Reinhard Heckel (co-lead), Gabriel Ilharco, Jenia Jitsev, Thomas Kollar, Niklas Muennighoff (co-lead), Ludwig Schmidt (co-lead), Shuran Song Appendix B Scaling-law derivations We first show that reparameterizing Equation ( 3 ) in terms of the compute C𝐶Citalic_C and token multiplier M𝑀Mitalic_M for α=β𝛼𝛽\alpha=\betaitalic_α = italic_β yields Equation ( 4 ). Combining C=6⁢N⁢D𝐶6𝑁𝐷C=6NDitalic_C = 6 italic_N italic_D and M=D/N𝑀𝐷𝑁M=D/Nitalic_M = italic_D / italic_N yields N=C/(6⁢M)𝑁𝐶6𝑀N=\sqrt{C/(6M)}italic_N = square-root start_ARG italic_C / ( 6 italic_M ) end_ARG and D=C⁢M/6𝐷𝐶𝑀6D=\sqrt{CM/6}italic_D = square-root start_ARG italic_C italic_M / 6 end_ARG. Inserting these into Equation ( 3 ) yields, L⁢(C,M)𝐿𝐶𝑀\displaystyle L(C,M)italic_L ( italic_C , italic_M ) =E+A⁢(C6⁢M)−α2+B⁢(C⁢M6)−α2,absent𝐸𝐴superscript𝐶6𝑀𝛼2𝐵superscript𝐶𝑀6𝛼2\displaystyle=E+A\left(\frac{C}{6M}\right)^{-\frac{\alpha}{2}}+B\left(\frac{CM% }{6}\right)^{-\frac{\alpha}{2}},= italic_E + italic_A ( divide start_ARG italic_C end_ARG start_ARG 6 italic_M end_ARG ) start_POSTSUPERSCRIPT - divide start_ARG italic_α end_ARG start_ARG 2 end_ARG end_POSTSUPERSCRIPT + italic_B ( divide start_ARG italic_C italic_M end_ARG start_ARG 6 end_ARG ) start_POSTSUPERSCRIPT - divide start_ARG italic_α end_ARG start_ARG 2 end_ARG end_POSTSUPERSCRIPT , =E+(A⁢(16)−α2⁢Mα2+B⁢(16)−α2⁢M−α2)⁢C−α2.absent𝐸𝐴superscript16𝛼2superscript𝑀𝛼2𝐵superscript16𝛼2superscript𝑀𝛼2superscript𝐶𝛼2\displaystyle=E+\left(A\left(\frac{1}{6}\right)^{-\frac{\alpha}{2}}M^{\frac{% \alpha}{2}}+B\left(\frac{1}{6}\right)^{-\frac{\alpha}{2}}M^{-\frac{\alpha}{2}}% \right)C^{-\frac{\alpha}{2}}.= italic_E + ( italic_A ( divide start_ARG 1 end_ARG start_ARG 6 end_ARG ) start_POSTSUPERSCRIPT - divide start_ARG italic_α end_ARG start_ARG 2 end_ARG end_POSTSUPERSCRIPT italic_M start_POSTSUPERSCRIPT divide start_ARG italic_α end_ARG start_ARG 2 end_ARG end_POSTSUPERSCRIPT + italic_B ( divide start_ARG 1 end_ARG start_ARG 6 end_ARG ) start_POSTSUPERSCRIPT - divide start_ARG italic_α end_ARG start_ARG 2 end_ARG end_POSTSUPERSCRIPT italic_M start_POSTSUPERSCRIPT - divide start_ARG italic_α end_ARG start_ARG 2 end_ARG end_POSTSUPERSCRIPT ) italic_C start_POSTSUPERSCRIPT - divide start_ARG italic_α end_ARG start_ARG 2 end_ARG end_POSTSUPERSCRIPT . This is equal to Equation ( 4 ), making the substitutions αC=α/2subscript𝛼𝐶𝛼2\alpha_{C}=\alpha/2italic_α start_POSTSUBSCRIPT italic_C end_POSTSUBSCRIPT = italic_α / 2, a=A⁢(1/6)−αC𝑎𝐴superscript16subscript𝛼𝐶a=A(1/6)^{-\alpha_{C}}italic_a = italic_A ( 1 / 6 ) start_POSTSUPERSCRIPT - italic_α start_POSTSUBSCRIPT italic_C end_POSTSUBSCRIPT end_POSTSUPERSCRIPT, b=B⁢(1/6)−αC𝑏𝐵superscript16subscript𝛼𝐶b=B(1/6)^{-\alpha_{C}}italic_b = italic_B ( 1 / 6 ) start_POSTSUPERSCRIPT - italic_α start_POSTSUBSCRIPT italic_C end_POSTSUBSCRIPT end_POSTSUPERSCRIPT, as noted in the main body. Relation to compute-optimal training. Recall that we made the assumption α=β𝛼𝛽\alpha=\betaitalic_α = italic_β, which implies equal scaling of parameters and tokens to realize compute-optimal models. While this assumption is empirically justified [43], even if α≠β𝛼𝛽\alpha\neq\betaitalic_α ≠ italic_β, we get a parameterization that implies the power law exponent in Equation ( 4 ) remains constant with over-training, while the power law scalar changes. To find a compute-optimal training setting, Hoffmann et al. [43] propose to minimize the right-hand side of Equation ( 3 ) subject to the compute constraint C=6⁢N⁢D𝐶6𝑁𝐷C=6NDitalic_C = 6 italic_N italic_D. This yields, N∗=γ1α+β⁢(C/6)βα+βsuperscript𝑁∗superscript𝛾1𝛼𝛽superscript𝐶6𝛽𝛼𝛽N^{\ast}=\gamma^{\frac{1}{\alpha+\beta}}(C/6)^{\frac{\beta}{\alpha+\beta}}italic_N start_POSTSUPERSCRIPT ∗ end_POSTSUPERSCRIPT = italic_γ start_POSTSUPERSCRIPT divide start_ARG 1 end_ARG start_ARG italic_α + italic_β end_ARG end_POSTSUPERSCRIPT ( italic_C / 6 ) start_POSTSUPERSCRIPT divide start_ARG italic_β end_ARG start_ARG italic_α + italic_β end_ARG end_POSTSUPERSCRIPT and D∗=γ−1α+β⁢(C/6)αα+βsuperscript𝐷∗superscript𝛾1𝛼𝛽superscript𝐶6𝛼𝛼𝛽D^{\ast}=\gamma^{-\frac{1}{\alpha+\beta}}(C/6)^{\frac{\alpha}{\alpha+\beta}}italic_D start_POSTSUPERSCRIPT ∗ end_POSTSUPERSCRIPT = italic_γ start_POSTSUPERSCRIPT - divide start_ARG 1 end_ARG start_ARG italic_α + italic_β end_ARG end_POSTSUPERSCRIPT ( italic_C / 6 ) start_POSTSUPERSCRIPT divide start_ARG italic_α end_ARG start_ARG italic_α + italic_β end_ARG end_POSTSUPERSCRIPT, where γ=α⁢Aβ⁢B𝛾𝛼𝐴𝛽𝐵\gamma=\frac{\alpha A}{\beta B}italic_γ = divide start_ARG italic_α italic_A end_ARG start_ARG italic_β italic_B end_ARG, for notational convenience. The associated risk is, L⁢(N∗,D∗)=E+(A⁢γ−αβ+α+B⁢γββ+α)⁢(C6)−α⁢βα+β.𝐿superscript𝑁∗superscript𝐷∗𝐸𝐴superscript𝛾𝛼𝛽𝛼𝐵superscript𝛾𝛽𝛽𝛼superscript𝐶6𝛼𝛽𝛼𝛽\displaystyle L(N^{\ast},D^{\ast})=E+\left(A\gamma^{\frac{-\alpha}{\beta+% \alpha}}+B\gamma^{\frac{\beta}{\beta+\alpha}}\right)\left(\frac{C}{6}\right)^{% -\frac{\alpha\beta}{\alpha+\beta}}.italic_L ( italic_N start_POSTSUPERSCRIPT ∗ end_POSTSUPERSCRIPT , italic_D start_POSTSUPERSCRIPT ∗ end_POSTSUPERSCRIPT ) = italic_E + ( italic_A italic_γ start_POSTSUPERSCRIPT divide start_ARG - italic_α end_ARG start_ARG italic_β + italic_α end_ARG end_POSTSUPERSCRIPT + italic_B italic_γ start_POSTSUPERSCRIPT divide start_ARG italic_β end_ARG start_ARG italic_β + italic_α end_ARG end_POSTSUPERSCRIPT ) ( divide start_ARG italic_C end_ARG start_ARG 6 end_ARG ) start_POSTSUPERSCRIPT - divide start_ARG italic_α italic_β end_ARG start_ARG italic_α + italic_β end_ARG end_POSTSUPERSCRIPT . We now deviate from compute-optimal training by modifying the model size and tokens by multiplication with a constant m𝑚\sqrt{m}square-root start_ARG italic_m end_ARG, according to Nm=1m⁢N∗,Dm=m⁢D∗.formulae-sequencesubscript𝑁𝑚1𝑚superscript𝑁∗subscript𝐷𝑚𝑚superscript𝐷∗\displaystyle N_{m}=\frac{1}{\sqrt{m}}N^{\ast},\quad D_{m}=\sqrt{m}D^{\ast}.italic_N start_POSTSUBSCRIPT italic_m end_POSTSUBSCRIPT = divide start_ARG 1 end_ARG start_ARG square-root start_ARG italic_m end_ARG end_ARG italic_N start_POSTSUPERSCRIPT ∗ end_POSTSUPERSCRIPT , italic_D start_POSTSUBSCRIPT italic_m end_POSTSUBSCRIPT = square-root start_ARG italic_m end_ARG italic_D start_POSTSUPERSCRIPT ∗ end_POSTSUPERSCRIPT . (7) This modification keeps the compute constant (i.e., 6⁢Nm⁢Dm=6⁢N∗⁢D∗6subscript𝑁𝑚subscript𝐷𝑚6superscript𝑁∗superscript𝐷∗6N_{m}D_{m}=6N^{\ast}D^{\ast}6 italic_N start_POSTSUBSCRIPT italic_m end_POSTSUBSCRIPT italic_D start_POSTSUBSCRIPT italic_m end_POSTSUBSCRIPT = 6 italic_N start_POSTSUPERSCRIPT ∗ end_POSTSUPERSCRIPT italic_D start_POSTSUPERSCRIPT ∗ end_POSTSUPERSCRIPT). The risk, then, becomes L⁢(fNm,Dm)=E+(mα2⁢A⁢γ−αβ+α+m−β2⁢B⁢γββ+α)⁢C−α⁢βα+β.𝐿subscript𝑓subscript𝑁𝑚subscript𝐷𝑚𝐸superscript𝑚𝛼2𝐴superscript𝛾𝛼𝛽𝛼superscript𝑚𝛽2𝐵superscript𝛾𝛽𝛽𝛼superscript𝐶𝛼𝛽𝛼𝛽\displaystyle L(f_{N_{m},D_{m}})=E+\left(m^{\frac{\alpha}{2}}A\gamma^{\frac{-% \alpha}{\beta+\alpha}}+m^{-\frac{\beta}{2}}B\gamma^{\frac{\beta}{\beta+\alpha}% }\right)C^{-\frac{\alpha\beta}{\alpha+\beta}}.italic_L ( italic_f start_POSTSUBSCRIPT italic_N start_POSTSUBSCRIPT italic_m end_POSTSUBSCRIPT , italic_D start_POSTSUBSCRIPT italic_m end_POSTSUBSCRIPT end_POSTSUBSCRIPT ) = italic_E + ( italic_m start_POSTSUPERSCRIPT divide start_ARG italic_α end_ARG start_ARG 2 end_ARG end_POSTSUPERSCRIPT italic_A italic_γ start_POSTSUPERSCRIPT divide start_ARG - italic_α end_ARG start_ARG italic_β + italic_α end_ARG end_POSTSUPERSCRIPT + italic_m start_POSTSUPERSCRIPT - divide start_ARG italic_β end_ARG start_ARG 2 end_ARG end_POSTSUPERSCRIPT italic_B italic_γ start_POSTSUPERSCRIPT divide start_ARG italic_β end_ARG start_ARG italic_β + italic_α end_ARG end_POSTSUPERSCRIPT ) italic_C start_POSTSUPERSCRIPT - divide start_ARG italic_α italic_β end_ARG start_ARG italic_α + italic_β end_ARG end_POSTSUPERSCRIPT . (8) We again expect the same power law exponent and changing power law scalar. Note that m𝑚mitalic_m in Equation ( 8 ) is similar to M𝑀Mitalic_M in Equation ( 4 ). Specifically, m𝑚mitalic_m is a multiple of the Chinchilla-optimal token multiplier M∗=D∗/N∗superscript𝑀∗superscript𝐷∗superscript𝑁∗M^{\ast}=D^{\ast}/N^{\ast}italic_M start_POSTSUPERSCRIPT ∗ end_POSTSUPERSCRIPT = italic_D start_POSTSUPERSCRIPT ∗ end_POSTSUPERSCRIPT / italic_N start_POSTSUPERSCRIPT ∗ end_POSTSUPERSCRIPT, which is no longer fixed as a compute budget changes for α≠β𝛼𝛽\alpha\neq\betaitalic_α ≠ italic_β. Appendix C Additional grid search details Grid search configuration selection. Recall in Section 3.2 , we run a grid search over many configurations. We present the architectures we sweep over in Table 5 . Table 5 : Topologies for our grid searches. We consider 130 architectures for our grid search. After sweeping over batch size and warmup, we get a total of 435 configurations. For a complete list of hyperparameter configurations, please see: https://github.com/mlfoundations/scaling nl⁢a⁢y⁢e⁢r⁢ssubscript𝑛𝑙𝑎𝑦𝑒𝑟𝑠n_{layers}italic_n start_POSTSUBSCRIPT italic_l italic_a italic_y italic_e italic_r italic_s end_POSTSUBSCRIPT nh⁢e⁢a⁢d⁢ssubscript𝑛ℎ𝑒𝑎𝑑𝑠n_{heads}italic_n start_POSTSUBSCRIPT italic_h italic_e italic_a italic_d italic_s end_POSTSUBSCRIPT dm⁢o⁢d⁢e⁢lsubscript𝑑𝑚𝑜𝑑𝑒𝑙d_{model}italic_d start_POSTSUBSCRIPT italic_m italic_o italic_d italic_e italic_l end_POSTSUBSCRIPT Number of parameters [B] 4 4 96 0.010 4 12 96 0.010 12 12 96 0.011 12 4 96 0.011 8 4 96 0.011 16 4 96 0.011 16 12 96 0.011 8 12 96 0.011 24 4 96 0.012 24 12 96 0.012 4 4 192 0.021 4 8 192 0.021 4 12 192 0.021 8 8 192 0.023 8 4 192 0.023 8 12 192 0.023 12 4 192 0.025 12 8 192 0.025 12 12 192 0.025 16 4 192 0.026 16 8 192 0.026 16 12 192 0.026 24 8 192 0.030 24 4 192 0.030 24 12 192 0.030 4 12 288 0.033 4 4 288 0.033 8 12 288 0.037 8 4 288 0.037 4 4 320 0.038 4 8 320 0.038 12 12 288 0.041 12 4 288 0.041 8 8 320 0.043 8 4 320 0.043 16 4 288 0.045 16 12 288 0.045 12 4 320 0.049 12 8 320 0.049 24 4 288 0.053 24 12 288 0.053 16 8 320 0.055 16 4 320 0.055 4 12 488 0.062 4 4 512 0.065 4 16 512 0.065 4 8 512 0.065 24 8 320 0.066 24 4 320 0.066 4 4 576 0.074 4 8 576 0.074 4 12 576 0.074 8 12 488 0.075 8 4 512 0.079 8 8 512 0.079 8 16 512 0.079 4 4 640 0.085 4 16 640 0.085 4 8 640 0.085 12 12 488 0.087 8 4 576 0.090 8 12 576 0.090 8 8 576 0.090 12 16 512 0.093 12 8 512 0.093 nl⁢a⁢y⁢e⁢r⁢ssubscript𝑛𝑙𝑎𝑦𝑒𝑟𝑠n_{layers}italic_n start_POSTSUBSCRIPT italic_l italic_a italic_y italic_e italic_r italic_s end_POSTSUBSCRIPT nh⁢e⁢a⁢d⁢ssubscript𝑛ℎ𝑒𝑎𝑑𝑠n_{heads}italic_n start_POSTSUBSCRIPT italic_h italic_e italic_a italic_d italic_s end_POSTSUBSCRIPT dm⁢o⁢d⁢e⁢lsubscript𝑑𝑚𝑜𝑑𝑒𝑙d_{model}italic_d start_POSTSUBSCRIPT italic_m italic_o italic_d italic_e italic_l end_POSTSUBSCRIPT Number of parameters [B] 12 4 512 0.093 16 12 488 0.100 8 16 640 0.105 8 4 640 0.105 8 8 640 0.105 12 8 576 0.106 16 16 512 0.106 4 4 768 0.106 12 12 576 0.106 16 8 512 0.106 4 8 768 0.106 12 4 576 0.106 4 16 768 0.106 16 4 512 0.106 4 12 768 0.106 16 12 576 0.122 16 4 576 0.122 16 8 576 0.122 12 4 640 0.126 24 12 488 0.126 12 16 640 0.126 12 8 640 0.126 24 8 512 0.133 24 4 512 0.133 24 16 512 0.133 8 8 768 0.134 8 16 768 0.134 8 4 768 0.134 8 12 768 0.134 16 16 640 0.146 16 8 640 0.146 16 4 640 0.146 24 8 576 0.154 24 4 576 0.154 24 12 576 0.154 4 8 1024 0.155 4 16 1024 0.155 4 4 1024 0.155 12 8 768 0.162 12 4 768 0.162 12 12 768 0.162 12 16 768 0.162 24 16 640 0.186 24 8 640 0.186 24 4 640 0.186 16 16 768 0.191 16 4 768 0.191 16 8 768 0.191 16 12 768 0.191 8 8 1024 0.206 8 4 1024 0.206 8 16 1024 0.206 24 8 768 0.247 24 12 768 0.247 24 4 768 0.247 24 16 768 0.247 12 8 1024 0.257 12 4 1024 0.257 12 16 1024 0.257 16 8 1024 0.309 16 4 1024 0.309 16 16 1024 0.309 24 16 1024 0.412 24 8 1024 0.412 24 4 1024 0.412 . Table 5 : Topologies for our grid searches. We consider 130 architectures for our grid search. After sweeping over batch size and warmup, we get a total of 435 configurations. For a complete list of hyperparameter configurations, please see: https://github.com/mlfoundations/scaling Appendix D Evaluation dataset details All 46 downstream evaluations are based on MosaicML’s LLM-foundry evaluation suite [67]. We specifically consider the datasets given in Table 6 . Downstream task LLM-foundry category Evaluation type Shots Samples Chance accuracy AGIEval LSAT AR [ 129, 128, 116 ] symbolic problem solving multiple choice 3 230 0.25 AGIEval LSAT LR [ 129, 128, 116 ] reading comprehension multiple choice 3 510 0.25 AGIEval LSAT RC [ 129, 128, 116 ] reading comprehension multiple choice 3 268 0.25 AGIEval SAT English [ 129 ] reading comprehension multiple choice 3 206 0.25 ARC-Challenge [ 21 ] world knowledge multiple choice 10 2376 0.25 ARC-Easy [ 21 ] world knowledge multiple choice 10 2376 0.25 BBQ [ 77 ] safety multiple choice 3 58492 0.50 BIG-bench: CS algorithms [ 11 ] symbolic problem solving language modeling 10 1320 0.00 BIG-bench: Conceptual combinations [ 11 ] language understanding multiple choice 10 103 0.25 BIG-bench: Conlang translation [ 11 ] language understanding language modeling 0 164 0.00 BIG-bench: Dyck languages [ 11 ] symbolic problem solving language modeling 10 1000 0.00 BIG-bench: Elementary math QA [ 11 ] symbolic problem solving multiple choice 10 38160 0.25 BIG-bench: Language identification [ 11 ] language understanding multiple choice 10 10000 0.25 BIG-bench: Logical deduction [ 11 ] symbolic problem solving multiple choice 10 1500 0.25 BIG-bench: Misconceptions [ 11 ] world knowledge multiple choice 10 219 0.50 BIG-bench: Novel Concepts [ 11 ] commonsense reasoning multiple choice 10 32 0.25 BIG-bench: Operators [ 11 ] symbolic problem solving language modeling 10 210 0.00 BIG-bench: QA WikiData [ 11 ] world knowledge language modeling 10 20321 0.00 BIG-bench: Repeat copy logic [ 11 ] symbolic problem solving language modeling 10 32 0.00 BIG-bench: Strange stories [ 11 ] commonsense reasoning multiple choice 10 174 0.50 BIG-bench: Strategy QA [ 11 ] commonsense reasoning multiple choice 10 2289 0.50 BIG-bench: Understanding fables [ 11 ] reading comprehension multiple choice 10 189 0.25 BoolQ [ 19 ] reading comprehension multiple choice 10 3270 0.50 COPA [ 90 ] commonsense reasoning multiple choice 0 100 0.50 CoQA [ 89 ] reading comprehension language modeling 0 7983 0.00 Commonsense QA [ 105 ] commonsense reasoning multiple choice 10 1221 0.25 Enterprise PII classification [ 79 ] safety multiple choice 10 3395 0.50 HellaSwag (10-shot) [ 123 ] language understanding multiple choice 10 10042 0.25 HellaSwag (zero-shot) [ 123 ] language understanding multiple choice 0 10042 0.25 Jeopardy [ 67 ] world knowledge language modeling 10 2117 0.00 LAMBADA [ 75 ] language understanding language modeling 0 5153 0.00 LogiQA [ 56 ] symbolic problem solving multiple choice 10 651 0.25 MMLU (5-shot) [ 38 ] world knowledge multiple choice 5 14042 0.25 MMLU (zero-shot) [ 38 ] world knowledge multiple choice 0 14042 0.25 MathQA [ 5 ] symbolic problem solving multiple choice 10 2983 0.25 OpenBook QA [ 66 ] commonsense reasoning multiple choice 0 500 0.25 PIQA [ 12 ] commonsense reasoning multiple choice 10 1838 0.50 PubMed QA Labeled [ 48 ] reading comprehension language modeling 10 1000 0.00 SIQA [ 95 ] commonsense reasoning multiple choice 10 1954 0.50 SQuAD [ 88 ] reading comprehension language modeling 10 10570 0.00 Simple Arithmetic: NoSpaces [ 67 ] symbolic problem solving language modeling 10 1000 0.00 Simple Arithmetic: WithSpaces [ 67 ] symbolic problem solving language modeling 10 1000 0.00 WinoGender MC: Female [ 92 ] safety multiple choice 10 60 0.50 WinoGender MC: Male [ 92 ] safety multiple choice 10 60 0.50 WinoGrande [ 93 ] language understanding schema 0 1267 0.50 WinoGrand [ 53 ] language understanding schema 0 273 0.50 Table 6 : 46 downstream tasks. All downstream tasks considered in this work, evaluated via LLM-foundry [67] . For more information on each dataset and specifics about the LLM-foundry category and evaluation type, please see: https://www.mosaicml.com/llm-evaluation . Appendix E Additional results Over-performing grid search models experience more optimization steps. As mentioned in Section 3.2 and Figure 4 , we notice that models between 0.011B to 0.079B (i.e., 5.2×10165.2superscript10165.2\times 10^{16}5.2 × 10 start_POSTSUPERSCRIPT 16 end_POSTSUPERSCRIPT to 5.2×10175.2superscript10175.2\times 10^{17}5.2 × 10 start_POSTSUPERSCRIPT 17 end_POSTSUPERSCRIPT FLOPs trained near compute-optimal) over-perform compared to the trend established by other models in our initial grid searches. This results in a bump in the scaling plot. While we choose to exclude this range of models for our scaling study, we additionally investigate this phenomenon. In Figure 6 we color grid search configurations by the number of optimization steps (i.e., number of tokens seen divided by batch size divided by sequence length). For context, Figure 1 (left) in Kaplan et al. [49] also shows a bump; however, there the performance is worse than the general trend instead of better as in our work. We leave understanding more fully the interactions between hyperparameters, scaling, and performance to future work. Figure 6 : Understanding over-performing models in our grid search. (left) Models trained with 5.2×10165.2superscript10165.2\times 10^{16}5.2 × 10 start_POSTSUPERSCRIPT 16 end_POSTSUPERSCRIPT to 5.2×10175.2superscript10175.2\times 10^{17}5.2 × 10 start_POSTSUPERSCRIPT 17 end_POSTSUPERSCRIPT FLOPs over-perform relative to their neighbors. In looking at the number of optimization steps, we notice that the over-performing models experience more optimization steps than their x-axis neighbors. We hypothesize that the number of optimization steps is important, especially for smaller models, when trying to find models that lie along a trend. (right) A view of the same phenomenon, specifically on the efficient frontier. Scaling is largely predictable in-distribution (ID). Prior work focuses on understanding scaling using ID loss, often using training loss directly [49, 43]. Hence, we also consider Paloma [63] loss evaluation sets, which are designed to probe performance in specific domains. We use Paloma’s C4 [86, 25], RedPajama [110], and Falcon-RefinedWeb [80] splits to probe for ID validation loss. As seen in Figure 7 , relative error is mostly low. Relative error is largest for the N=1.4⁢B,M=640formulae-sequence𝑁1.4B𝑀640N=1.4\text{B},M=640italic_N = 1.4 B , italic_M = 640 RedPajama run at 15.4%. Examining this case specifically, we find that the model performs better than the scaling law prediction. We hypothesize that as a model sees more tokens there is an increased likelihood of near-duplicate sequences ID, resulting in performance that is better than predicted. Figure 7 : In-distribution (ID) settings. Boxes highlighted in yellow correspond to data points used to fit Equation ( 4 ). Relative error is generally low across interpolation and extrapolation regimes. Relative error is largest for the RedPajama N=1.4⁢B,M=640formulae-sequence𝑁1.4B𝑀640N=1.4\text{B},M=640italic_N = 1.4 B , italic_M = 640 prediction at 15.4%. In this case, we find that our scaling law predicts the model should perform worse than it does in practice. Relative error is stable across many choices of downstream evaluation suites. To understand how sensitive our investigation is to our choices of the evaluation set, we consider several other options as seen in Figure 8 . We find that our prediction errors are fairly (i) low and (2) consistent for many choices of downstream evaluation sets. Figure 8 : Downstream evaluation set ablation for 6.9B parameter, 138B token runs. Recall that in addition to the 46 task evaluation suite, we consider a 17 task evaluation suite created by including only test sets where any 0.154B model we trained (for any token multiplier and training dataset) gets t=10𝑡10t=10italic_t = 10 percentage points above random chance. Here we wish to understand how sensitive our results are to this choice t𝑡titalic_t . (left) We see that relative prediction error is fairly low before a threshold of t=35𝑡35t=35italic_t = 35 . When too many tasks are excluded (i.e., t>40𝑡40t>40italic_t > 40 relative error spikes. (right) A parallel view, showing how many tasks are removed as t𝑡titalic_t increases. 40 out of the 46 tasks can be removed and relative error is still fairly stable. Scaling law fit Train set MMLU [ 38 ] OpenBook QA [ 66 ] HellaSwag [ 123 ] 17 eval 46 eval Table 2 C4 [ 86, 25 ] 2.82% 16.80% 79.58% 0.14% 0.07% Table 2 w/o 1.4B C4 [ 86, 25 ] 1.86% 96.16% 61.79% 0.42% 0.76% Table 2 RedPajama [ 110 ] 0.12% 8.44% 25.73% 0.05% 2.10% Table 2 w/o 1.4B RedPajama [ 110 ] 1.07% 7.56% 30.98% 10.64% 7.68% Table 2 RefinedWeb [ 80 ] 0.77% 1.92% 81.96% 2.94% 2.76% Table 2 w/o 1.4B RefinedWeb [ 80 ] 2.29% 6.79% 6.52% 15.79% 8.57% Table 7 : Downstream relative prediction error at 6.9B, 138B tokens, with and without the 1.4B data point. Recall in Table 2 , we introduce a N=1.4𝑁1.4N=1.4italic_N = 1.4 B, M=20𝑀20M=20italic_M = 20 run to get better downstream error predictions. Here we compare, prediction errors with and without this model for fitting the scaling law. Note that without the model (i.e., rows with “w/o 1.4B”) average top-1 predictions over a 17 task subset and over the full 46 task suite are less accurate. Scaling can break down when under-training. Figure 9 : Scaling with small token multipliers. For smaller multipliers (e.g., M=5𝑀5M=5italic_M = 5 in cyan), scaling does not follow the same trend as that of larger multipliers. Additionally, many token multiplies (e.g., M∈{10,20,40,80}𝑀10204080M\in\{10,20,40,80\}italic_M ∈ { 10 , 20 , 40 , 80 } ) garner points close to the compute-optimal frontier. We find that when a token multiple is too small (i.e., under-training regime), scaling appears unreliable. In Figure 9 we see for M=5𝑀5M=5italic_M = 5 the scaling trend is different. We hypothesize that tuning hyperparameters (e.g., warmup, batch size) directly for smaller multipliers may help mitigate the breakdown in predictability. Scaling can be unpredictable out-of-distribution (OOD). Our main result shows reliable C4 eval loss predictions with models trained on RedPajama, which is an OOD evaluation setting. However, both C4 and RedPajama both contain tokens sourced from CommonCrawl. To further probe OOD performance, we measure the relative error of scaling laws fit to models trained on C4 and evaluated on Paloma’s 100 programming languages [63], Paloma’s Penn Tree Bank (PTB) split [64], and a German version of C4 [25]. Recall that the C4 training set we use has been filtered for English text. Hence we expect (i) the proportion of code is minimal, (ii) the “” substrings in PTB raw text do not appear frequently, and (iii) German is not prevalent. We notice that extrapolation relative error tends to be high for large M,N𝑀𝑁M,Nitalic_M , italic_N on programming languages and PTB (Figure 10 (left, center)). In contrast, for German C4, relative error is still low across the extrapolation range with a maximum relative error of 7.6% at the N=𝑁absentN=italic_N =1.4B, M=80𝑀80M=80italic_M = 80 scale (Figure 10 (right)). We hypothesize that further modifications to scaling laws are necessary to predict when scaling should be reliable as a function of the training and evaluation distributions. Figure 10 : Out-of-distribution (OOD) settings. Boxes highlighted in yellow correspond to data points used to fit Equation ( 4 ). Recall that the C4 training set is English-filtered. Relative error can spike, suggesting unreliable scaling, for (left) programming languages and (center) Penn Tree Bank, which contains many frequently occurring, uncommon substrings. However, scaling is relatively reliable when evaluating on (right) German. These results motivate future studies of OOD conditions that affect scaling in the over-trained regime. Small-scale experiments can predict average downstream top-1 error. Figure 11 : Relative error on average top-1 predictions (46 task split). Boxes highlighted in yellow correspond to data points used to fit Equation ( 5 ). Using our fits, we accurately predict downstream average top-1 error across interpolation and extrapolation regimes. This result supports that (i) chaining a scaling law and our proposed exponential decay function is a valid procedure and (ii) average top-1 error can be highly predictable. To verify that chaining Equations ( 4 ) and ( 5 ) is effective in practice, we collect C4 eval loss and downstream error pairs for the configurations in Table 2 . In Figure 11 , we look at relative error for our scaling predictions in the context of Average top-1 error over our 46 evals and over our 17 evals in Figure 12 . We again notice reliable scaling in interpolation and extrapolation regimes, suggesting the validity of our procedure to predict downstream average top-1 error. Figure 12 : Relative error on average top-1 predictions (17 task split). Boxes highlighted in yellow correspond to data points used to fit Equation ( 5 ). Using our fits, we accurately predict downstream average top-1 error across interpolation and extrapolation regimes. This result supports that (i) chaining a scaling law and our proposed exponential decay function is a valid procedure and (ii) average top-1 error can be highly predictable. Loss evaluation ablations for downstream trends. Figure 13 presents the correlation between downstream error vs. loss evaluated on different validation sets (C4, RedPajama, and RefinedWeb). Regardless of the validation set (x-axis), models follow the exponential decay relationship given in Equation ( 5 ), suggesting the choice of validation loss is not critical for the appearance of this phenomenon. Figure 13 : Correlation between average top-1 error and evaluation loss. We observe that regardless of evaluation loss distribution (x-axis), models tend to follow Equation ( 5 ). This suggests that there can be several reasonable choices for the validation loss distribution. Additionally, ID models trained on C4 and evaluated on a C4 validation set, perform best in terms of loss, but these gains don’t necessarily translate to lower error downstream (e.g., (left column) ). This suggests the need to fit Equation ( 5 ) per dataset and also suggests comparing models trained on different data distributions with a single loss evaluation can be misleading in terms of downstream performance. Investing more compute in a scaling law makes it more predictive. Figure 14 : Trade-offs between scaling law for loss fitting considerations and reliability. Each red circle represents a scaling law fit to Equation ( 4 ) with as many as 29 models trained on RedPajama. Specifically, a grid formed by N∈{0.011⁢B,0.079⁢B,0.154⁢B,0.411⁢B},M∈{5,10,20,40,80,160,320}formulae-sequence𝑁0.011B0.079B0.154B0.411B𝑀510204080160320N\in\{0.011\text{B},0.079\text{B},0.154\text{B},0.411\text{B}\},M\in\{5,10,20,% 40,80,160,320\}italic_N ∈ { 0.011 B , 0.079 B , 0.154 B , 0.411 B } , italic_M ∈ { 5 , 10 , 20 , 40 , 80 , 160 , 320 } gives 28 models and a N=1.4⁢B,M=20formulae-sequence𝑁1.4𝐵𝑀20N=1.4B,M=20italic_N = 1.4 italic_B , italic_M = 20 run gives the last model. We sort models by training FLOPs in increasing order and sample models uniformly from index windows [1,2,…,n]12normal-…𝑛[1,2,...,n][ 1 , 2 , … , italic_n ] for n∈[5,6,..,29]n\in[5,6,..,29]italic_n ∈ [ 5 , 6 , . . , 29 ] to fit Equation ( 4 ). The blue star represents the default configuration presented in Table 2 . The prediction target is a N=1.4⁢B,M=640⁢ ⁢(D=900⁢B)formulae-sequence𝑁1.4𝐵𝑀640 𝐷900BN=1.4B,M=640\text{ }(D=900\text{B})italic_N = 1.4 italic_B , italic_M = 640 ( italic_D = 900 B ) model. As the amount of compute (left) and the number of points (right) used to fit the scaling law increases, relative error trends downwards. Our default configuration keeps compute and number of points low, while still providing low prediction error compared to the trend. Figure 15 : Compute vs. relative error for the 1.4B, 900B token RedPajama run. (left) The compute necessary to accurately predict loss is less than that needed to accurately predict (right) average downstream error. This claim is supported by the fact that the slope of the trend for loss is steeper than for top-1 error. These findings corroborate Figure 16 . Figure 16 : Compute vs. relative error for the 6.9B, 138B token RedPajama run. (left) The compute necessary to accurately predict loss is less than that needed to accurately predict (right) average downstream error. This claim is supported by the fact that the slope of the trend for loss is steeper than for top-1 error. These findings corroborate Figure 15 . Thus far we have looked at standard configurations from Table 2 to construct our scaling laws, mainly to demonstrate extrapolation to larger N,M𝑁𝑀N,Mitalic_N , italic_M. However, for practitioners, the main constraint is often training compute. Hence, we wish to understand the trade-offs between the amount of compute invested in creating a scaling law and the relative error of the resulting law in the over-trained regime. In Figure 14 (left), we see that as one increases the amount of compute, it is possible to get better fits with lower relative error. In Figure 14 (right), we see a similar trend as one increases the number of data points used to fit a scaling law. Blue stars indicate the configurations from Table 2 , which provide accurate predictions relative to the general trends—hinting at their usefulness for our investigation. In Figures 15 , 16 we repeat the compute analysis comparing trade-offs for loss prediction and error prediction reliability. We find that less compute is generally necessary to construct a loss scaling law that achieves the same relative error as that of an error prediction scaling law. Figure 17 : Scaling exponent vs. token multiplier. In Figure 2 we notice roughly parallel lines (i.e., roughly constant scaling exponent αCsubscript𝛼𝐶\alpha_{C}italic_α start_POSTSUBSCRIPT italic_C end_POSTSUBSCRIPT ) in the log\logroman_log - log\logroman_log plot of loss vs. compute, even as the token multiplier M𝑀Mitalic_M changes. Here we plot αCsubscript𝛼𝐶\alpha_{C}italic_α start_POSTSUBSCRIPT italic_C end_POSTSUBSCRIPT vs. M𝑀Mitalic_M directly, where the shaded region gives a 95% bootstrap confidence interval for the trend. This view supports that αCsubscript𝛼𝐶\alpha_{C}italic_α start_POSTSUBSCRIPT italic_C end_POSTSUBSCRIPT is relatively constant. Figure 18 : Downstream top-1 error vs. C4 eval loss for each of the 46 downstream evals. Here we plot models from our testbed for each scatter plot. We see that some individual evaluations, like ARC-Easy, follow exponential decay. Others, like BIG-bench: CS algorithms, show step function behavior. Still others, like MathQA, hover around random chance. Appendix F Additional related work Language modeling. Language models can be grouped into encoder-only [24, 51, 57, 94, 20], encoder-decoder [54, 87], and decoder-only architectures [83, 111, 112, 108, 47, 36, 72, 7, 109, 26, 62, 97, 119, 4, 55, 61, 32]. Most current implementations are based on the transformer [114]. However, there has been a recent resurgence in scaling language models based on non-transformer architectures [81, 34, 35, 33]. Further, there has been substantial work on adapting pre-trained language models to better follow instructions [117, 18, 68, 59, 69, 130, 85, 27, 113, 101, 71]. However, following prior work [43, 70] and given their overall prevalence, we limit ourselves to GPT-style, decoder-only transformers that have solely been pre-trained. Scaling laws. Kaplan et al. [49] investigate scaling trends in GPT language models. Bahri et al. [9] investigate different scaling regimes theoretically, and Sharma & Kaplan [99] relate scaling coefficients to data manifold dimensions. Tay et al. [106, 107] elucidate the connection between model architecture and scaling trends, while Hernandez et al. [40], Tay et al. [106] develop scaling laws for transfer learning. Ivgi et al. [46] also consider transfer learning scaling laws and highlight the importance of hyperparameter selection in the low-compute regime. Ghorbani et al. [30], Gordon et al. [31], Bansal et al. [10] develop scaling laws for neural machine translation. Caballero et al. [15] propose a scaling law functional form, which they demonstrate is predictive in several domains. Scaling beyond language modeling. There is a large body of work on scaling neural networks beyond language modeling, for example in computer vision [58, 124, 103, 1, 2], multimodal learning [39, 16, 28], and image reconstruction [50]. Generated on Wed Dec 14 18:01:44 2022 by L A T E xml Instructions for reporting errors We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below: * Click the "Report Issue" button. * Open a report feedback form via keyboard, use "Ctrl + ?". * Make a text selection and click the "Report Issue for Selection" button near your cursor. * You can use Alt+Y to toggle on and Alt+Shift+Y to toggle off accessible reporting links at each section. Our team has already identified the following issues. We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all. Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion, and welcome developer contributions. Report Issue Report Github Issue Title:Description: Submit without GithubSubmit in Github Report Issue for Selection

Search across your past Websets

Research Papers: scaling laws, training compute, audio models

Search Criteria

Enrichments

Find more results