Diet Studies Suck

This article from US News and World Report is a great example of why we really don’t know much about what diets work, which don’t, and which are harmful. The problem is outlined here quite nicely and explains why any study you read should be carefully reviewed to find out:

• Who sponsored it – where the money comes from can alter the science (can you believe it?)
• Methodology – how was it done? Was it a study where people self-reported information – like they kept a diary – or was there some way to objectively measure the data? Self-reported data is so horrible – people lie, or are just plain wrong, or answer questions to try to fit to the reality they would like to believe rather than the reality that is.
• How big was the study? If it had a total of 8 people, maybe that’s too small to make any real conclusion.
• Was it a clinical study or an epidemiological study? The first studies the people directly. The second studies populations. Lots of assumptions are made in population studies. Say that the heart attack rate is lower in Japan than other countries. Then compare the diets and you find they eat more fish. You might conclude that eating fish reduces heart attack. Not so fast. there are a 1000 factors – known and unknown – that might contribute to this. You can try to factor out the knowns using statistics, but you can’t factor out the unknowns – you don’t know what they are.
• Was the study on animals or people? Animal studies can sometimes be better because you can control what a rat eats – they aren’t keeping a diary – you can measure it. You can also cut them up and take a real hard look at their arteries – gross, but that is what they do. But people are not rats, and while we are genetically close – we are also genetically close to that crazy uncle of ours with the giant ball of tinfoil in the garage who eats Vaseline because he thinks it’s good for his health. So take this rat, and feed him the equivalent of 400 diet sodas a day to compensate for the fact that a rat only lives a few years, watch him get cancer, and make the leap that the stuff will cause cancer in humans. Possible? Yes. A fact? No.

It’s hard to really really know anything, so my advice is to act arrogant and self-righteous when studies come out that support your world-view, and question the science of any and all studies that don’t support your world-view.