Inspired by a post on Andrew Gelman’s blog, The harm done by tests of signficance:

The first four steps to understanding cause and effect:

1. Formulate your signal hypotheses, where i = 1, …, n.
2. Fit your signal models to your data, .  Obtain model parameter values, , under each model.
3. Reality-check your fit results. Does at least one of the fit models do a decent job of fitting the data?  (If is crazy low for all of your signal hypotheses then either you’ve got a highly anomalous observation on your hands or your signal hypotheses do not include the one which gave rise to the data.)
4. Compute posterior probabilities:

Get through those steps and you may have a story to tell. Continue reading →

I list Tibshirani’s paper describing his Least Absolute Shrinkage and Selection Operator (LASSO) method on my Primary Sources page.  Full disclosure:  I’ve never had occasion to use it myself but I’ve seen it used to productive effect.  Sparsity-promoting ridge regression is just a really smart and (in many circumstances) useful concept.   Not too many know about it currently but in 100 years (probably sooner) it will be standard content in stat textbooks.  Because it’s not yet widely known it’s worth calling attention both to the method and to how it’s implemented.  Tibshirani published his original paper in 1996.  The big deal is that he and several colleagues have developed a significance test for predictor variables.  From Andrew Gelman’s blog:

 I received the following email from Rob Tibshirani:

Over the past few months I [Tibshirani] have been consumed with a new piece of work [with Richard Lockhart, Jonathan Taylor, and Ryan Tibshirani] that’s finally done. Here’s the paper, the slides (easier to read), and the R package.

I’m very excited about this. We have discovered a test statistic for the lasso that has a very simple Exp(1) asymptotic distribution, accounting for the adaptive fitting. In a sense, it’s the natural analogue of the drop in RSS chi-squared (or F) statistic for adaptive regression!

It also could help bring the lasso into the mainstream. It shows how a basic adaptive (frequentist) inference—difficult to do in standard least squares regression, falls out naturally in the lasso paradigm.

This is a big deal.  If you do regression analyses in your work then check it out.  (If you’re not familiar with ridge regression, a.k.a. regularized regression, then best to check that out first.)

[UPDATE 3/16/2013:  I’ve revised this post multiple times over the past ~12 hours and will likely continue to do so over the next few days.]

There’s a lively discussion over at Andrew Gelman’s blog following his post, Misunderstanding the p-value.  (No, it has nothing to do with urine tests.)  The Wikipedia definition is actually spot on:  “In statistical hypothesis testing the p-value is the probability of obtaining a test statistic at least as extreme as the one that was actually observed, assuming that the null hypothesis is true.”   The way to describe it formally (and a poster on Gelman’s blog did) is , where t is the calculated test statistic, T is the user-specified critical value of the test statistic, and means  conditional on the null hypothesis, , being true.  (If you fed your detector a continuous stream of data where was true then the p-value would be your Type I error rate – or vice versa, depending upon how you prefer to look at it.)

The thing which has made the discussion on Gelman’s blog lively is that not all concur with that definition.  Specifically, some object to calling the p-value a conditional probability.  Continue reading →

AC-130 gunship video.

For non-statisticians in the audience, Type I error is incorrect rejection of the null hypothesis (false positive) and Type II error is failure to reject a false null hypothesis (false negative).

The Fire Control Officer has rejected the null hypothesis.

Continue reading →

[UPDATE 2/12/2013:  Given a few more weeks to think about it, this post hardly seems worth the electrons.  The example I use to justify usage of “accept H0” doesn’t involve distinct H0 and H1 hypotheses:  H0 = ~H1.  If the only options are A and ~A then, sure, A = ~~A but that’s not exactly a revelation.  The real question is “Is it legitimate to say you “accept H0″ when there are distinct H0 and H1 hypotheses?”  For that case, I believe “reject H1” is appropriate but “accept H0” is not, i.e., I accept conventional usage.  Anyhow, read on if you’re hurting for entertainment.]

Hypothesis testing is a basic problem in statistical analysis.  You consider a null hypothesis (H0) in the context of an alternative hypothesis (H1).  If your experiment is testing for an effect then H0 is ‘no effect’ and H1 is ‘effect is present’.   In statistics parlance, one may “reject H0” or “accept H1” but one never “accepts” H0.  You can “reject H1” but you don’t “accept H0”.  On the one hand, that makes sense.  You could establish with a high degree of confidence that your data isn’t consistent with H1 and therefore reject it but that doesn’t mean that H0 is true.   (H1 could be true and you just happened to collect funky data which led you to reject it.  Type II error.  Oops.)

But on the other hand, maybe it doesn’t make so much sense.  Continue reading →

http://xkcd.com/892/