Before you start your experiment, you calculate how many samples you need based on the estimated effect size you're looking for and how small you want your confidence interval to be.
In the physical sciences you can often estimate the noise level in a null measurement -- or even measure it. You often do this just to get your setup working before doing something like wasting a precious specimen on a "this time for real" measurement.
> Running experiments until you get a hit
But if I'm running an experiment how do I know how many time to run it.