Size your sample count

SmcConfig.samples defaults to 10,000. But what if you want to be within some tolerance of the true probability at 95 percent confidence? How many samples do you need? SENTIL has two functions to size the count and we look at them in detail on this page.

The Chernoff-Hoeffding budget

The Chernoff-Hoeffding inequality bounds how far the empirical estimate can stray from the truth for a given sample count, with no assumption about the underlying distribution beyond independent bounded draws. Inverted, it says how many samples guarantee the estimate lands within $\varepsilon$ of the true probability with confidence at least $1 - \delta$:

chernoff_hoeffding_samples(epsilon, delta) returns the ceiling of that. Within 0.1 at 95 percent confidence takes 185 samples:

use sentil::stats::chernoff_hoeffding_samples;

assert_eq!(chernoff_hoeffding_samples(0.1, 0.05)?, 185);

The bound is distribution-free and this means that the function above gives a guarantee that holds for any noise distribution. In exchange, it sizes for the worst case, so the true accuracy at the count it returns is usually better than $\varepsilon$.

The Wilson half-width alternative

If you only need the reported interval to be narrow, and not a distribution-free guarantee on the estimate itself, size for the Wilson interval width directly. wilson_samples(epsilon, level) returns the count that holds the half-width to $\varepsilon$ at the worst-case probability of one half, using $N \geq z^2 / (4\varepsilon^2)$ with $z = \Phi^{-1}(1 - \alpha/2)$. Pinning the half-width to 0.01 at 95 percent takes 9,604 samples:

use sentil::stats::wilson_samples;

assert_eq!(wilson_samples(0.01, 0.95)?, 9604);

For the same tolerance and confidence, Wilson asks for half the Chernoff-Hoeffding budget.

A budget table

At 95 percent confidence, halving the tolerance quadruples the count, the signature of the $1/\sqrt{N}$ convergence rate.

Set the budget

Once you have the number of samples needed, feed the count into SmcConfig.samples.

use sentil::{SmcConfig, stats::chernoff_hoeffding_samples};

let config = SmcConfig {
    samples: chernoff_hoeffding_samples(0.01, 0.05)?,
    ..SmcConfig::default()
};

import sentil
from sentil import SmcConfig

config = SmcConfig(samples=sentil.stats.chernoff_hoeffding_samples(0.01, 0.05))

When the run finishes, the confidence interval reports the accuracy you actually achieved.