Java applets for power and sample size

Note: These require a web browser capable of running Java applets (version 1.3 or higher). If you do not see a selection list above, chances are that you either have disabled Java, or you have an outdated  implementation of Java. In the latter case, you need to download and install the JRE plug-in from www.oracle.com/technetwork/java/. 
Due to a compatibility bug, many plug-ins size the applet window before allowing for an additional strip with a security warning.; to compensate, drag the bottom of the window downward a bit.

Please read this comment

If you're blocked by a security setting

Questions/Discussion

Citing this software

Download to run locally

Frequently Asked Questions

• What formula(s) do you use in these calculations?  
  In most cases, power is an exact calculation based on the distributional situation in question.  Typically it is a probability associated with a non-central distribution.  In a few cases, an approximation is used, and is labeled as such.  Sample sizes are calculated using root-finding methods in conjunction with power calculations.  There are usually not nice neat formulas.  That's why we need this software.
  
  
• Help on using a particular applet  
  I am willing to provide minimal support if you truly don't understand what inputs are required.  However, each applet has a help menu, and I do request that you carefully read that before you e-mail me with questions.
  
  
• I need consulting help  
  I am providing this software for free, but I do not have time to also answer substantive questions on power/sample size for your research project.  If you need statistical advice on your research problem, you should contact a statistical consultant; and if you want expert advice, you should expect to pay for it.  Most universities with statistics departments or statistics programs also offer a consulting service.  If you think your research is important, then it is also important to get good advice on the statistical design  and analysis (do this before you start collecting data).
  
  
• How to do... Retrospective power ... Cohen's effect sizes  
  I recommend against these (see Advice section below).  I have been asked why the Options menu in every single applet has links for retrospective power and Cohen effect sizes.  It seems to some to be placing undue emphasis on methods I don't like. The technical answer to the question is that these menu items are inherited from a base class, along with some other things (e.g., the graphics capabilities).  The other answer is that people ask me about this all the time, in spite of everything I say on this site.   If you follow those menu links, you get explanations of why not to do it.  I'm especially proud of the dialog for retrospective power.
  
  
• Disclaimer  
  This software is made available as-is, with no guarantees; use it at your own risk.  I welcome comments on bugs, additional capabilities you'd like to see, etc. 
  
  
• Other questions  
  If you have carefully read the above FAQs, and still find it appropriate to contact me, my e-mail address is russell-lenth@uiowa.edu.

Advice

• Retrospective power (a.k.a. observed power, post hoc power).  You've got the data, did the analysis, and did not achieve "significance."  So you compute power retrospectively to see if the test was powerful enough or not.  This is an empty question.  Of course it wasn't powerful enough -- that's why the result isn't significant.  Power calculations are useful for design, not analysis.
  (Note: These comments refer to power computed based on the observed effect size and sample size.  Considering a different sample size is obviously prospective in nature.  Considering a different effect size might make sense, but probably what you really need to do instead is an equivalence test; see Hoenig and Heisey, 2001.)
  
  
• Specify T-shirt effect sizes ("small", "medium", and "large").  This is an elaborate way to arrive at the same sample size that has been used in past social science studies of large, medium, and small size (respectively).  The method uses a standardized effect size as the goal.  Think about it: for a "medium" effect size, you'll choose the same n regardless of the accuracy or reliability of your instrument, or the narrowness or diversity of your subjects.  Clearly, important considerations are being ignored here.  "Medium" is definitely not the message!
  

• Use power prospectively for planning future studies.  Software such as is provided on this website is useful for determining an appropriate sample size, or for evaluating a planned study to see if it is likely to yield useful information.
  
  
• Put science before statistics.  It is easy to get caught up in statistical significance and such; but studies should be designed to meet scientific goals, and you need to keep those in sight at all times (in planning and analysis).  The appropriate inputs to power/sample-size calculations are effect sizes that are deemed clinically important, based on careful considerations of the underlying scientific (not statistical) goals of the study.  Statistical considerations are used to identify a plan that is effective in meeting scientific goals -- not the other way around.
  
  
• Do pilot studies.  Investigators tend to try to answer all the world's questions with one study.  However, you usually cannot do a definitive study in one step.  It is far better to work incrementally.  A pilot study helps you establish procedures, understand and protect against things that can go wrong, and obtain variance estimates needed in determining sample size.  A pilot study with 20-30 degrees of freedom for error is generally quite adequate for obtaining reasonably reliable sample-size estimates. 
  

• Lenth, R. V. (2001), ``Some Practical Guidelines for Effective Sample Size Determination,'' The American Statistician, 55, 187-193.
  
  
• Hoenig, John M. and Heisey, Dennis M. (2001), ``The Abuse of Power: The Pervasive Fallacy of Power Calculations for Data Analysis,'' The American Statistician, 55, 19-24.
  
  
• An earlier draft of the Lenth reference above is _here_,  and a shorter summary of some comments I made in a panel discussion at the 2000 Joint Statistical Meetings in Indianapolis is _here_.
  
  
• Additional brief comments, prepared as a handout for my poster presentation at the 2001 Joint Statistical Meetings in Atlanta, are _here_.
  


• For your amusement (or despair), here is a video I found that shows how not to ask a statistician about sample-size:
  
  (Thanks to Susan Geyer, Morsani College of Medicine, Health Informatics Institute University of South Florida -- aka JavaMama926. The URL is http://www.youtube.com/watch?v=PbODigCZqL8. Dr. Geyer created this video for use in a workshop she teaches at the American Society of Hematology's Clinical Research Training Institute.)

---

Accuracy

• Formula accuracy
Most computations are ``exact'' in the sense that they are based on exact formulas for sample size, power, etc.  The exception is Satterthwaite approximations; see below.


• Machine accuracy
Even with exact formulas, computed values are inexact, as are all double-precision floating-point computations.   Many computations (especially noncentral distributions) require summing one or more series, and there is a serious tradeoff between speed and accuracy.  The error bound set for cdfs is 1E-8 or smaller, and for quantiles the bound is 1E-6.  Actual errors can be much larger due to accumulated errors or other reasons.  Quantiles, for example, are computed by numerically solving an equation involving the cdf; thus, in extreme cases, a small error in the cdf can create a large error in the quantile.
A warning (typically, ``too many iterations'') is generated when an error bound is not detected to have been achieved.  However, in the case of quantile computations, no warning message is generated for extreme quantiles.  If you want a power of .9999 at alpha=.0001, you can expect the computed sample size to not be accurate to the nearest integer!  If you specify reasonable criteria, the answers will be pretty reliable.


• Satterthwaite approximations
Some of the dialogs (two-sample t, mixed ANOVA) implement Satterthwaite approximations when certain combinations of inputs require an error term to be constructed.  These are of course not exact, even in their formulation.  Moreover, the Satterthwaite degrees of freedom is used as-is in computing power from a noncentral t or noncentral F distribution, and this introduces further errors that could be large in some cases.  In the two-sample t setting, I'd expect the worst errors to exist when there is a huge imbalance in sample sizes and/or variances.  In the dialogs for mixed ANOVA models (either F tests or multiple comparisons/contrasts), I expect these errors to get worse as more variance components are involved, especially when one or more of them is given negative weight.

View My Stats