Larry Husten at CardioBrief asked me for a comment on the American Heart Association’s Presidential Advisory on Dietary Fats and Cardiovascular Disease, which was published yesterday. I got carried away in my response and Larry asked if he could publish it as a guest post, which he did after some revisions. Here’s that post. As usual, I would have rather spent a few days on this, but that wasn’t in the cards.

The human understanding, once it has adopted opinions, either because they were already accepted and believed, or because it likes them, draws everything else to support and agree with them. And though it may meet a greater number and weight of contrary instances, it will, with great and harmful prejudice, ignore or condemn or exclude them by introducing some distinction, in order that the authority of those earlier assumptions may remain intact and unharmed.
     –Francis Bacon, Novum Organum, 1620

Four hundred years ago, give or take a couple of years, Francis Bacon pioneered thinking about the scientific method by noting that humans are programmed to pay more attention to evidence that agrees with their preconceptions and to reject evidence that doesn’t, and that this thinking leads to very effective religious dogma but not to reliable knowledge of the universe. Hence what was needed was a new technology of reasoning –Novum Organum, per the title of his book in Latin – that would minimize these tendencies, although he recognized that getting rid of them entirely was not an option. Humans didn’t function that way.

In good science, this kind of cognitive bias is addressed, among other methodologies, by deciding in advance of looking at the evidence (or doing a trial) what criteria will be used to judge the worth of the evidence (the results of the trial) without knowledge of whether that evidence supports our hypotheses. This is one reason why clinical trials are done double blind, and the data analyzed by researchers who are blinded to whether the subjects were interventions or controls, such that the biases of the investigators (or even the subjects) don’t bias the interpretation of the results.

For whatever reason, when it comes to heart disease and dietary fat, the investigators whom the American Heart Association chooses to determine what we should or should not eat have never been believers in this kind of, well, scientific methodology. This was the general conclusion of my first investigation into the dietary fat story going on 20 years ago for the journal Science. I’d like to say the situation has improved, but clearly it hasn’t. The latest Presidential Advisory from the AHA on saturated fat is the AHA’s expert authorities – what Inspector Renaud in Casablanca would have called “the usual suspects” – reiterating that they were right fifty years ago, and they were right 20 years ago, and they’re still right. And the techniques they used to come to those conclusions can be used again and again until someone stops them. Which is unlikely to happen.

A Scottish cardiologist/epidemiologist described this pseudoscientific methodology to me as “Bing Crosby epidemiology” – i.e., “accentuate the positive and eliminate the negative.” In short, it’s cherry picking, and it’s how a lawyer builds an argument but not how a scientist works to establish reliable knowledge, which is the goal of the enterprise. Not winning per se, but being right. It’s why I wrote in the epilogue of my first book on nutrition, Good Calories, Bad Calories, that I didn’t consider these people doing research in the nexus of diet, obesity and disease to be real scientists. They don’t want to know the truth; they only wanted to convince maybe themselves and certainly the rest of us that they already do and have all along. While all good science requires making judgments about what evidence is reliable and what isn’t, scientists have to do this keeping in mind that the first principle of good science, now quoting Feynman, “is that you must not fool yourself and you’re the easiest person to fool.” The history of science is littered with failed hypotheses based on selective interpretation of the evidence. Regrettably the AHA experts simply don’t believe that what’s true of far better scientists then themselves, could possibly be true of them as well.

Today’s Presidential Advisory, written by a dozen esteemed experts led by Harvard’s Frank Sacks, may be the most egregious example of Bing Crosby epidemiology that I’ve ever seen. It’s particularly interesting because four years ago the AHA released a report claiming to be evidence-based medicine, co-authored by an intersecting set of these usual suspects, that also claimed that the strongest possible evidence existed to restrict the saturated fat (SFA) in our diet and replace it with polyunsaturates (PUFAs). It was fascinating because multiple other meta-analyses, co-authored by independent researchers, had found the evidence to be weak or lacking. So how could it be the strongest possible? Surely there was room for improvement. That 2013 AHA document, though, made it exceedingly difficult to duplicate the analysis of the AHA experts and establish how they had come to such a paradoxical conclusion. This latest document in effect tells us what they did then and are still doing – i.e., what they’ve been doing all along.

Whether consciously or unconsciously, they assume that what they think is true surely is, and then they methodically eliminate the negative and accentuate the positive until they can make the case that they are surely, clearly and unequivocally right. And they might be, just as a lawyer arguing a case to a jury might always be right, but you can never know it from the lawyer’s argument alone. You have to hear the counter as well and then maybe you can decide.

So let’s look at this process of eliminating the negative: the AHA concludes that only four clinical trials have ever been done with sufficiently reliable methodology to allow them to assess the value of replacing SFAs with PUFAs (in practice replacing animal fats by vegetable oils) and concludes that this replacement will reduce heart attacks by 30 percent. In the history of this debate, this is a huge, if not unprecedented number. These four trials are the ones that are left after the AHA experts have systematically picked through the others and found reasons to reject all that didn’t find such a large positive effect, including a significant number that happened to suggest the opposite. For these trials they carefully identify why these trials were critically if not fatally flawed, and so why their results cannot and should not be used in any reasonable assessment. As Bacon might have said, “with great and harmful prejudice [the AHA experts] ignore or condemn or exclude them by introducing some distinction, in order that the authority of [their] earlier assumptions may remain intact and unharmed.”

They do this for every trial but the four, including among the rejections the largest trials ever done: the Minnesota Coronary Survey, the Sydney Heart Study, and, most notably, the Women’s Health Initiative, which was the single largest and most expensive clinical trial ever done. All of these resulted in evidence that refuted the hypothesis. All are rejected from the analysis. And the AHA experts have good reasons for all of these decisions, but when other organizations – most notably the Cochrane Collaboration – did this exercise correctly, deciding on a strict methodology in advance that would determine which studies to use and which not, without knowing the results, these trials were typically included.

What the AHA experts don’t do (perhaps because they are convinced they can’t possibly be fooling themselves) is make the same effort with the trials that do support their hypothesis and assumptions. If they did, they make little indication of it. Of the four studies that support the 30 percent reduction, all are ancient by the standards of nutrition science. All date to the 1960s. One of them, for instance, is the Oslo Diet-Heart Study. This trial reported a significant reduction in CVD events, in line with the beliefs of the AHA authors, and so it’s included among the four trials considered worthy of making the cut. The Oslo trial was indeed typical for the era, which means very primitive by today’s standards. A single investigator, Paul Leren, has local physicians recommend to him for inclusion in the study patients who are at high risk of heart disease or have already had heart attacks. He randomizes half of these patients, now subjects, to eat a low-SFA, high PUFA diet and then gives them intensive counseling for years (“continuous instruction and supervision,” as Leren puts it), and he compares them to a control group that gets no counseling and eats the standard Norwegian diet.

So one group gets a “healthy” diet and intensive counseling for years; the other group gets nothing. Nada. This is technically called performance bias and it’s the equivalent of doing an unblinded drug trial without a placebo. It is literally an uncontrolled trial, despite the randomization. (In this case, as Leren explains, all the physicians involved also knew whether their patients were assigned to the intervention group or the control, which makes investigator bias all that much more likely.) We would never accept such a trial as a valid test of a drug. Why do it for diet? Well, maybe because it can be used to support our preconceptions, but that’s not really a good answer. I’m guessing that the AHA experts made no attempt to find out if this trial was worthy of rejection because they liked the result If I’m wrong, I apologize and I hope one of them will write to tell me.

Why do I know this about Oslo? Because I was curious, always a good thing, and, of course, because it disagrees with my preconceptions and my biases. Still, my curiosity could not be satisfied by reading the published literature because Leren didn’t give the necessary details in the published studies. He probably didn’t have the space. He did in a monograph he published in 1966. I bought a copy a few weeks ago. That’s how curious I was. It’s in this monograph that Leren assesses the state of the science, just as our AHA experts do now, fifty-one years later, and he then describes in pretty good detail what he actually did in the trial. He also discusses the dietary changes achieved in his intervention group, and here’s where the performance bias, rather than the PUFA/SFA shift, may have determined the study outcome.

Leren mentions in passing that sugar consumption in his intervention group was very low, about 50 grams a day, which is 40 pounds a year and is probably less than half of the per capita consumption in Norway in that era. (I’m extrapolating back from this data — i.e., guessing.) So this is a critical problem with performance bias in a diet study, any diet study. As we’re taught in eight grade science classes, good scientific experiments change a single variable with an intervention such that we can see the effect of that change. In this trial, the variable that’s supposed to be different is the SFA/PUFA ratio, but the performance bias introduces another one. One group gets continuous counseling to eat healthy, one group doesn’t. Now how can that continuous counseling influence health status? One way is that apparently the group that got it decided to eat a hell of lot less sugar. This unintended consequence now gives another possible explanation for why these folks had so many fewer heart attacks. I don’t know if this is true. The point is neither did Leren. And neither do our AHA authorities. Although we can speculate that had they decided in advance what criteria they would use to reject studies and then have the studies assessed blindly, such that the individuals making the choice had no knowledge of the results of the study, they would have rejected this one, too. And the others, as well. All of the four studies used to support the 30 percent number had significant flaws, often this very same performance bias. Reason to reject them.

The PrediMed trial is another good example of the AHA’s Bing Crosby epidemiology. The AHA authorities, as they say in passing, would like us to eat a Mediterranean diet, and so they conclude the evidence from PrediMed supports this advice. PrediMed may be the most influential clinical trial of the last decade, but it, too, was critically flawed. No, fatally flawed. You had to read the supplemental data in this case to find out. The researchers randomized subjects to three arms, one of which got nuts (Mediterranean) and regular counseling; one got olive oil (Mediterranean) and counseling; one (non-Mediterranean) got bupkus and no counseling. Hence, significant performance bias. Midway through the trial, the researchers actually realize that this was a problem and decide to address it. Here’s how they describe this revelation on page 10 of the supplemental material:

The initial dietary protocol for the Control group started with the delivery of a leaflet summarizing the recommendations to follow a low-fat diet (Table S2-S3) and scheduled one yearly visit. In October 2006, 3 years into the trial, we realized that such a low-grade intervention might potentially represent a weakness of the trial and amended the protocol to include quarterly individual and group sessions with delivery of food descriptions [my italics] shopping lists, meal plans and recipes (adapted to the low-fat diet) in such a way that the intensity of the intervention was similar to that of the Mediterranean diet groups, except for the provision of supplemental foods for free. This amendment of the protocol in no way meant a change in the quality and specific goals of the recommendations to the control group; it was only an enhancement in the eagerness of the intervention to make it similar to that delivered to participants in the Mediterranean diet groups.

Sound of throat clearing… Imagine a drug trial, in which “three years into the trial” the investigators realize that it might be a problem that they neglected to give the control group a placebo. Oops. Would editors of a prestigious journal buy the idea that “such a low-grade intervention might [might!!] potentially represent a weakness of the trial?” Would such a trial get published in any respectable journal? In nutrition, and because the cognoscenti in the nutrition community like the results, it’s published in the New England Journal of Medicine, the most prestigious medical journal in the world, and makes it to the front page of the New York Times. And the admission of this potential weakness is only made in the supplemental material. Not in the paper itself. Imagine had the study found that the Mediterranean diet was actually harmful. That giving nuts and olive oil increased the risk of death. Do you think the assembled experts of the AHA would have included it in this assessment, or would they have found this performance bias problem and rejected it on that basis? I’m voting for the latter, but we’ll never know.

Ultimately this AHA document is a recapitulation of what the AHA experts have been arguing for decades. The only reason to publish it is because it’s been taken heat lately from folks like me and Nina Teicholz and a host of others who point out that we’re dealing with a pseudoscience here and the public deserves far better. Those of us who have become critics may indeed be biased about what we believe now – I certainly am — but ultimately we’re arguing for better science. This kind of post-hoc analyses of clinical trials, whether subgroup analysis or otherwise, can only be hypothesis generating. That’s basic logic. We don’t have to take a vote. Just open a basic science or biostatistics textbook. What the AHA experts are doing here is saying that their assessment of the data leads to what they consider a compelling hypothesis: replacing SFA with PUFA should reduce heart disease by 30 percent. But that’s all they can say. By deciding what data to include and what not based on their preconceptions of what’s true and what’s not, they cannot say this is a fact, as they claim, only that it’s still a reasonable hypothesis and has yet to be refuted.

This leads to three further critical points.

1. One reason why the AHA’s four favored trials were done in the 1960s was not just to see if exchanging SFA for PUFA reduced heart disease risk, but to see if it reduced mortality. Like any drug, it’s not enough to show that an intervention has positive effects, benefits, you have to demonstrate that those benefits counterbalance the negatives, the risks. In the 1960s, the researchers and the public health authorities understood that and so most of these trials looked at total mortality as an endpoint. Only the Finnish Mental Hospital study showed a benefit of the diet on longevity and that was only in men. Not in women. (In fact, all the trials used to establish the 30 percent reduction number were done only in men. The Women’s Health Initiative was done in large part to see if what might be true for men would also be true for women, but the AHA doesn’t like this study, so we’re stuck with all men.)

Recently the epidemiologists discussing dietary fats and disease have taken again to focusing only on CHD, but they don’t say why. Even the Cochrane meta-analyses focus only on heart disease. My guess is they do this because the clinical trials showed no benefit for total mortality (they were mostly underpowered) and total mortality is hopelessly confounded in the observational studies. Personally, I’d rather die of heart disease than cancer or Alzheimer’s, but that’s may be because my familial experience has been with cancer and Alzheimer’s and it wasn’t pretty. Either way, if I’m going to change my diet and start consuming vegetable oils I want to know if I’m going to live longer. The AHA doesn’t even address that question. The first rule of medicine, preventive or otherwise, is still do no harm, and they’re making no attempt to assess harm. You can argue that they’re the AHA so what they care about is heart disease. But it’s not good enough. It’s never been good enough. And this leads to the second point.

2. The AHA experts do acknowledge that they’re discussing the same decades-old trials that we’ve been arguing about for, well, decades, and they do acknowledge implicitly that these trials cannot resolve this controversy, and then they state explicitly what would be necessary to do so:

The core trials reviewed in this section were started in the late 1950s and early 1960s. Readers may wonder why at least 1 definitive clinical trial has not been completed since then. Reasons include the high cost of a trial having upward of 20 000 to 30 000 participants needed to achieve satisfactory statistical power, the feasibility of delivering the dietary intervention to such a large study population, technical difficulties in establishing food distribution centers necessary to maintain high adherence for at least 5 years, and declining CVD incidence rates caused by improved lifestyle and better medical treatment [my italics]. These linked issues, which must be managed to obtain a definitive result, remain the central considerations for dietary trials on CVD and indeed are the overarching reason why few of these trials have ever been done. Finally, by the 1980s, with rising rates of breast and colon cancer, the US government committed to conducting the WHI (Women’s Health Initiative), a trial that studied a diet aimed at decreasing total fat in the diet to 20% with the expectation that saturated fat would likewise be substantially decreased. Consequently, carbohydrates were increased in the diet. Details are discussed subsequently.

So a rigorous test probably can’t be done. And, more importantly, if this is what it takes to rigorously test the hypothesis—“20,000 to 30,000 participants needed to achieve satisfactory statistical power”— then why are we even discussing these other trials with nothing like that number? (And, of course, that’s why they had to dismiss the WHI as meaningful because that trial does have this kind of statistical power.) They’ve effectively eviscerated their own case. If this was a legal case, the judge would now throw it out and we’d all be having coffee in the lobby (with or without cream) discussing how this fiasco played out and why it ever got to court to begin with. And this leads to the third point.

3. Did I say that the first rule of medicine, as Hippocrates pointed out, is do no harm? I believe I did. Back in 1981, Geoffrey Rose, a pioneer thinker in the field of preventive medicine, wrote an article in the BMJ on the strategy of preventive medicine, and he pointed out the same problem about vegetable oils that confronts us today. Again history keeps repeating itself in this world, in part because these researchers and authorities don’t think we have to do the experiments necessary to resolve this controversy and find out if the AHA’s hypothesis is indeed true. They’re too hard. (Imagine if physicists took this tact with their science. Why bother raising ten billion dollars to build a single accelerator so technology challenging that we have to work out the technological details as we go along, just because that’s what’s necessary to answer the next question they want to see answered? Too hard. They’ll never do it. Let’s not try. We can speculate and pretend it’s fact. Sigh.) As Rose observed, it’s one thing to tell people not to eat something because we evolved to eat very little of it and there’s good evidence that eating less of it will reduce chronic disease risk. This is what Rose called removing an “unnatural factor and the restoration of `biological normality’—that is, of the conditions to which presumably we are genetically adapted.” As Rose put it, “Such normalizing measures [for instance, telling people not to smoke] may be presumed to be safe, and therefore we should be prepared to advocate them on the basis of a reasonable presumption of benefit.”

But telling people to eat something new to the environment — an unnatural factor, à la virtually any vegetable oil (other than olive oil if your ancestor happen to come from the Mediterranean or mid-East), which was what concerned Rose and concerns us today — is an entirely different proposition. Now you’re assuming that this unnatural factor is protective, just like we assume a drug can be protective say by lowering our blood pressure or cholesterol. And so the situation is little different than it would be if these AHA authorities were concluding that we should all take statins prophylactically or beta blockers. The point is that no one would ever accept such a proposal for a drug without large-scale clinical trials demonstrating that the benefits far outweigh the risks. So even if the AHA hypothesis is as reasonable and compelling as the AHA authors clearly believe it is, it has to be tested. They are literally saying (not figuratively, literally) that vegetable oils — soy, canola, etc — are as beneficial as statins and so we should all consume them. Maybe so, but before we do (or at least before I do), they have a moral and ethical obligation to rigorously test that hypothesis, just as they would if they were advising us all to take a drug. And then, well, they should probably do it twice, since a fundamental tenet of good science is also independent replication. And what we need here is good science