Tjyylli

I have 2 samples, one with sample size of 30,000 customers and the other with 150,000. I have to perform a 2 sample t test(on conversion rates of the 2 groups). My question is, will t test in this case be biased towards the smaller sample? If yes, what is the correct approach to perform a test?

I can hardly imagine any worthwhile effect size that requires such a large sample size to be decently powered. There's no "bias" of having unequal sample sizes$^1$. The only disadvantage is that the power of the test tends to be somewhat limited by the smaller group. For even very small effects, 30,000 observations may confer quite a powerful test.

$^1$ except if you inappropriately use the "equal variance" assumption, in which case the "pooled variance" estimate is more heavily weighted toward the larger group (not toward the smaller as you suggested).

Maybe a couple of examples will help to illustrate some of the issues.

Suppose the two populations are $X sim mathsf{Norm}(mu = 500, sigma =30)$
and $Y sim mathsf{Norm}(mu = 501, sigma = 20.)$

If both sample sizes are $150,000,$ then there is sufficient power to detect
the small difference in means.

set.seed(422)

x = rnorm(150000, 500, 30)

y = rnorm(150000, 501, 20)

t.test(x, y)



        Welch Two Sample t-test



data:  x and y

t = -10.983, df = 261530, p-value < 2.2e-16

alternative hypothesis: true difference in means is not equal to 0

95 percent confidence interval:

 -1.2042715 -0.8395487

sample estimates:

mean of x mean of y 

 499.9804  501.0023 

If we use only the first 30,000 values in the first sample, results are
very nearly the same for most practical purposes.

t.test(x[1:30000], y)



        Welch Two Sample t-test



data:  x[1:30000] and y

t = -6.3728, df = 35463, p-value = 1.879e-10

alternative hypothesis: true difference in means is not equal to 0

95 percent confidence interval:

 -1.5126269 -0.8010336

sample estimates:

mean of x mean of y 

 499.8455  501.0023 

Here is a boxplot of the data used in the second t test (the wider box indicates a larger sample):

Issues of minimal concern:

• Even though labeled as 'Welch t tests', sample sizes are sufficiently large
  that these are essentially t tests. Unless the data are very far from normal,
  we would still detect the small difference in means.




• The power of the test is heavily dependent on the smaller sample size. But
  power is not a concern here.

Issues warranting attention:

• With such large samples
  in the real world (not the simulation world),
  one is entitled to wonder whether data are truly simple random samples from
  their respective populations. Could smaller, more carefully collected samples provide better information?




• Although we did not do a formal test to confirm that variances differ, it seems clear from the boxplot that they do. In the Welch test,
  it is OK for variances to differ. But would different variances have important practical implications?




• Although the null hypothesis that the two population means are equal is soundly rejected with minuscule P-values, it is important to realize that "statistically significant" differences (by whatever definition) are not necessarily differences of practical importance or interest. For what purpose are you
  taking the effort of check whether means are different? And what do the results
  of the t test actually contribute to that purpose?