A blog on statistics, methods, and open science. Understanding 20% of statistics will improve 80% of your inferences.

Tuesday, November 18, 2014

Evaluating Estimation Accuracy With the Award Winning V-Statistic


Clintin Davis-Stober recently pointed me to his blog about the v-statistic. The v-statistic (nothing Greek, just the letter v) is a measure of how accurately your data is estimating the corresponding parameter values in the population. The paper he wrote with Jason Dana won the Clifford T. Morgan award (awarded to the best paper in each Psychonomic Society journal, this being from Behavior Research Methods). This makes v an award winning new statistic. I’m really happy this paper won an award for a number of reasons.

First of all, the v-statistic is not based on Frequentist of Bayesian statistics. It introduces a third perspective on accuracy. This is great, because I greatly dislike any type of polarized discussion, and especially the one between Frequentists and Bayesians. With a new kid on the block, perhaps people will start to acknowledge the value of multiple perspectives on statistical inferences.

Second, v is determined by the number of parameters you examine (p), the effect size R-squared (Rsq), and the sample size. To increase accuracy, you need to increase the sample size. But where other approaches, such as those based on the width of a confidence interval, lack a clear minimal value researchers should aim for, the v-statistics has a clear lower boundary to beat: 50% guessing average. You want a v>.5. It’s great to say people should think for themselves, and not blindly use numbers (significance levels of 0.05, 80% power, medium effect sizes of .5, Bayes Factors > 10) but let’s be honest: That’s not what the majority of researchers want. And whereas under certain circumstances the use of a p = .05 is rather silly, you can’t go wrong with using v > .5 as a minimum. Everyone is happy.

Third, Ellen Evers and I wrote about the v-statistic in our 2014 paper on improving the informational value of studies (Lakens & Evers, 2014), way before v won an award. It’s like discovering a really great band before it becomes popular. 

Fourth, mathematically v is the volume of a hypersphere. How cool is that? It’s like it’s from an X-men comic!

I also have a weakness for v because calculating it required R, which I had never used before I wanted to be able to calculate v, and so v was the reason I started using R. When re-reading the paper by Clintin & Jason, I felt the graphs they present (for studies estimating 3 to 18 parameters, and sample sizes from 0 to 600) did not directly correspond to my typical studies. So, it being the 1.5 year anniversary of R and me, I thought I’d plot v as a function of R-squared for some more typical numbers of parameters  (2, 3, 4, and 6), effect sizes (R-squared of 0.01 - 0.25), and sample sizes in psychology (30-300).






A quick R-squared to R conversion table for those who need it, and remember Cohen’s guidelines suggest an R = .1 is small, R = .3 = medium, and R = .5 is large.  

R-squared       0.05     0.10     0.15     0.20     0.25
R                    0.22     0.32     0.39     0.44     0.50

As we see, v depends on the sample size, number of parameters, and the effect size. For 2, 3, and 4 parameters, the effect sizes at which v > .5 doesn’t change substantially, but with more parameters being estimated (e.g., > 6) accuracy decreases substantially, which means you need substantially larger samples. For example, when estimating 2 parameters, a sample size of 50 requires an effect size larger than R-squared = 0.115 (R = .34) to have a v >.5.

When planning sample sizes, the v-stat can be one criterion you can use to decide which sample size you will plan for. You can also use v to evaluate the accuracy in published studies (see Lakens & Evers, 2014 for two examples). The R script to create these curves for different numbers of parameters, sample sizes, and effect sizes is available below.


Tuesday, November 4, 2014

Negotiations between Elsevier and Dutch Universities break down: Time for change

Excellent news this morning: The negotiations between Dutch universities and Elsevier about new contracts for access to the scientific literature have broken down (report in Dutch by VSNU and De Volkskrant). This means universities are finally taking a stand: We want scientific knowledge to be freely available as open access to anyone who wants to read it, and we believe publishers are making more than enough money as it is. Moving towards fully open access should not cost Dutch tax payers anything extra.

There might be some small inconveniences along the way for us scientists. Just as we needed to mail authors of articles in the 70's and 80's, we might need to start e-mailing authors for their papers in 2015. Researchers who don't know what a #Icanhazpdf hashtag is might need some Twitter education. Sometimes, you might not get an article as quickly as you were used to. We might experience the consequences of not having access to literature ourselves.

I think that is good. One of my favorite quotes from Zen and the Art of Motorcycle Maintenance reads as follows: "Stuckness shouldn't be avoided. It's the psychic predecessor of all real understanding." It's excellent that the Dutch universities and Elsevier are stuck. If this stuckness lasts, it's excellent that a researcher gets stuck when searching the literature. We need this to understand how untenable the status quo is.

There are very cheap and easy solutions. I've talked to some people very close to these negotiations, and there is a plan B. There is even a more interesting plan C, where the Dutch universities examine the possibility of making large deals with Plos One and The Peer J to pay a lump sum to provide all researchers opportunity to publish in these journals, paid by the government. If we would compare that to what we currently pay Elsevier alone, that would probably be a sweet deal.

The major thing standing in our way is our own ego. Elsevier has journals with a 'reputation', and many researchers are willing to screw over Dutch tax payers just to be able to publish in a journal like Nature or Science. Their work will be just as good when it is published in Peer J. However, these researchers are not making a rational choice, but an emotional choice, driven by self-interest. Short term self-interest will always trump long term public benefit. You can imagine the wide grins on Elsevier's faces whenever this topic comes up. They know scientists are often little ego-factories, and they are more than happy to cash in.

That's why I think it's good that we are stuck. Someone needs to put their foot down. Open Access publishing is the light bulb of the scientific world. Regulations were needed to make sure the public would switch from the low priced but energy inefficient light bulb to the energy saving light bulbs that were slightly more expensive to buy, but better for everyone in the long term. Researchers similarly need a push to move away from publishing options that thrive on short term benefit, but screw Dutch tax payers in the long term. I hope we stay stuck until we all reach some real understanding.