What is the minimum sample size needed for Bayesian optimization?

There is no fixed minimum sample size, but a common rule of thumb is to start with at least 5-10 initial random samples to build a reliable surrogate model. The exact number depends on the problem's dimensionality and complexity, with higher-dimensional spaces often requiring more initial points.

Can Bayesian optimization work with only 2 or 3 data points?

With only 2-3 data points, Bayesian optimization can still run but the surrogate model will be highly uncertain and the acquisition function may lead to poor exploration. In practice, such small samples are only viable for extremely simple, low-noise problems.

How does the number of dimensions affect the minimum sample size?

Higher-dimensional search spaces require more initial samples to adequately cover the space and build a meaningful model. A common heuristic is to use at least 10 times the number of dimensions as the initial sample size, though this can be relaxed with informative priors.

Is there a formula to calculate the minimum sample size for Bayesian optimization?

No universal formula exists, but practitioners often use rules like 5-10 initial points plus 1-2 per dimension. The actual minimum is influenced by the noise level, the smoothness of the objective function, and the chosen surrogate model (e.g., Gaussian process).

What happens if I use too few initial samples in Bayesian optimization?

Using too few initial samples can cause the surrogate model to be inaccurate, leading to poor acquisition function decisions and premature convergence to a suboptimal solution. The optimization may also fail to explore the search space adequately, especially in high-dimensional or noisy problems.

Minimum Sample Size for Bayesian Optimization

If you are running Bayesian optimization, you can often get useful results with as few as 10 to 20 initial samples, depending on the noise in your objective function and the number of parameters you are tuning. There is no single magic number, but a practical minimum sample size for Bayesian optimization is roughly 2 to 5 times the number of hyperparameters you are optimizing. This guide explains how to determine that number for your own project and why starting too small can waste time.

The Role of the Surrogate Model

Bayesian optimization works by building a probabilistic model of your objective function, then using that model to decide where to sample next. The most common surrogate model is a Gaussian process, which needs enough data points to estimate a meaningful correlation between inputs and outputs Scikit-learn. If you give it too few points, the model has high uncertainty everywhere, and the acquisition function (the rule that picks the next point) behaves almost randomly.

A Simple Rule of Thumb

For a low-dimensional problem (1 to 3 parameters), a minimum of 10 initial random samples is a safe starting point. For each additional parameter, add roughly 5 more samples. So for 5 parameters, you would want at least 20 initial evaluations. This rule assumes your objective function is relatively smooth and deterministic. If your function is noisy (for example, measuring model accuracy with a small validation set), you should increase the initial sample size by 50% to 100%.

What Happens Below the Minimum

If you start with only 3 or 4 samples for a 2-parameter problem, the Gaussian process will often fit a model that is too simple. It might miss the true shape of the function entirely. The acquisition function then directs sampling to areas that look promising based on a bad model, and you waste evaluations exploring regions that are actually poor. In practice, you can end up needing more total evaluations than if you had just run a grid search.

How Noise Affects Your Minimum

The minimum sample size for Bayesian optimization changes significantly when your objective function has measurement noise. Noise is common in real-world tasks like cross-validation accuracy, A/B test metrics, or hardware measurements. A Gaussian process can handle noise, but it needs more data to separate signal from randomness.

Practical Noise Guidelines

Low noise (standard deviation less than 5% of the signal range): Use the standard 2 to 5 times parameter count rule.
Medium noise (5% to 15%): Double the initial sample size. For 3 parameters, start with 20 points instead of 10.
High noise (above 15%): Bayesian optimization may still work, but you should consider averaging multiple evaluations per point or switching to a different optimizer. The minimum sample size can easily be 50 or more.

A good check is to run a few initial points in duplicate. If the results vary by more than 10%, you are in the medium-to-high noise regime.

Practical Examples and Software Defaults

Most Bayesian optimization libraries have sensible defaults for the initial sample size. The popular Python package BayesianOptimization uses 5 initial random points by default Bayesian Optimization in Python. That is fine for 1 or 2 parameters with low noise, but it is too low for 4 or more parameters.

Worked Example: Tuning a Machine Learning Model

Suppose you are tuning 4 hyperparameters (learning rate, batch size, dropout rate, and number of layers) for a neural network. Your objective is validation accuracy, which has moderate noise (around 10% variation between runs with the same settings). Using the rule of thumb:

Base minimum: 4 parameters x 3 = 12 samples.
Noise adjustment: multiply by 1.5 = 18 samples.
Round up to 20 initial random evaluations.

After those 20 points, run 30 to 50 Bayesian iterations. This setup usually finds a good configuration in fewer total evaluations than a random search of 100 points.

When You Can Use Fewer Samples

There are two cases where you can safely reduce the minimum sample size:

You have prior knowledge. If you already know the approximate location of the optimum, you can initialize the Gaussian process with points clustered near that region. The model needs fewer points to fit a local surface.
You are using a different surrogate model. Tree-based models like random forests or gradient boosting can work with fewer initial points than Gaussian processes, especially in high dimensions. The trade-off is that they may not model uncertainty as well.

How Dimensionality Changes the Minimum

The number of parameters you optimize has the largest effect on the minimum sample size for Bayesian optimization. As dimensionality increases, the search space grows exponentially, and the surrogate model needs more points to form a reasonable picture of the function.

Low Dimensions (1 to 3 Parameters)

For one parameter, you can get away with as few as 3 to 5 initial samples. The Gaussian process can interpolate smoothly between those points, and the acquisition function will quickly guide you toward promising regions. Two parameters typically need 8 to 12 samples. Three parameters need 12 to 15. These numbers assume you are using a standard Matern kernel, which is a good default for most smooth functions.

Medium Dimensions (4 to 6 Parameters)

At 4 parameters, the minimum jumps to about 15 to 20 initial points. At 5 parameters, you need 20 to 25. At 6 parameters, 25 to 30. Beyond 6 parameters, Bayesian optimization with a Gaussian process becomes less effective unless you have a very large evaluation budget. The reason is that the kernel’s length-scale parameters become harder to estimate, and the model needs exponentially more data to avoid overfitting.

High Dimensions (7+ Parameters)

For 10 or more parameters, the minimum sample size for Bayesian optimization often exceeds 50 initial points. At that point, random search or a tree-structured Parzen estimator (TPE) may outperform Gaussian process-based methods. Some practitioners use dimensionality reduction techniques like random embeddings to make the problem tractable, but that adds complexity. If you must use Bayesian optimization in high dimensions, plan for at least 100 initial evaluations and expect the optimization to require several hundred total evaluations.

Key Takeaways

Start with at least 2 to 5 times the number of parameters you are optimizing as initial random samples.
Increase the sample size by 50% to 100% if your objective function has moderate noise.
Library defaults like 5 initial points are only safe for problems with 1 or 2 parameters.
If you have prior knowledge about the optimum, you can reduce the initial sample size.
For high-noise problems, consider averaging multiple measurements per point before starting Bayesian optimization.

Minimum Sample Size for Bayesian Optimization: A Practical Guide