How Much Evidence is Enough?

Nov• 14•2010
Been thinking back again to something I said a couple posts ago, where after reviewing the recent literature I declared that it was time for nephrologists and internists to alter their treatment regimes and begin using sodium bicarbonate in the treatment of chronic kidney disease.  Well is such a radical statement really valid?  Why don’t we start by first pretending that sodium bicarbonate were a patentable substance and consider the degree of evidence needed for FDA approval. 


Since the Kefauver-Harris amendment of 1962 of the original Federal Food Drug and Cosmetic Act, there are two requirements for FDA approval, that is that a drug be demonstrated to be both safe and effective.  Prior to 1962 it was only necessary to show safety, which in my opinion, ironically enough, would lead to more efficacious treatments on the market.  There is an involved process of pharmacokinetic, pharmacodynamic, toxicology, animal studies and phase I, II and III human studies employed for “new molecular entities” to begin to demonstrate safety and efficacy.  Though safety really only becomes clear, as has been demonstrated with horrific mishaps of approved drugs as in the case of Resulin, Vioxx and too many others to enumerate, in phase IV post marketing studies, though these studies are rarely rigorously followed.

In the case of a dietary supplement such as sodium bicarbonate, safety is less of an issue and has been defined and documented through extensive previous use.   Baking soda (sodium bicarbonate) is historically known to be “safe” as compared with most prescribed and over the counter drugs and there is an extensive history of clinical use.

Leaving aside safety then, we come to efficacy.  Now the FDA wants, “substantial evidence of effectiveness” for the drug in the particular disease for which it is to be marketed.  This evidence, per Title 21 section 314 of the Federal Code of Regulations, consists of “adequate and well-controlled investigations …”  Of note, again per regulations, this does not necessarily mean a placebo controlled trial, it may mean,

  • 1) a placebo control
  • 2) a dose comparison control
  • 3) a no treatment control
  • 4) an active treatment control or
  • 5) a historical control (i.e what the disease would have been expected to do absent treatment.

Of course after Dr. Beatrice Golomb’s recent article noting that over 90% of placebos reported in major medical journal studies are left undefined, the whole gold standard of placebo controlled trials is looking more than a mite bit adulterated at the moment.  Be that as it may, the FDA standard does not call for placebo controlled trials only “well-controlled” trials, perhaps even with historical controls.

In addition, especially in the case of many “blockbuster” drugs, the trials themselves are not tied to any clinical endpoint, such as “how did the disease respond to this treatment?” but rather to surrogate end-points.  Surrogate end-points in turn are a nebulous assortment of “markers” for disease which may be closely or loosely tied to disease.

In the case of say red blood cell count as an indicator of clinical anemia one might argue the lab value is the best indicator of the disease state, likewise creatine clearance would be a good indication of kidney function in acute kidney failure.  However, especially in long-term chronic diseases, the modification of surrogate markers such as cholesterol level or blood pressure level may not tie so closely to the disease for which the drug is indicated.  Nonetheless, the various statin drugs for heart disease have been approved, efficacy wise, solely on whether they decrease cholesterol.  Similarly, the “flu” vaccine has been approved based on whether an immune response to injected foreign protein is mounted, a surrogate marker, not on any evidence concerning actual disease rates and leaving aside post marketing questions of safety for statins or flu vaccine.  Additionally, chemotherapeutic drugs have been approved based from a modest decrease in tumor size even when it has been documented that patients lived no longer in the treatment versus the control arm of the study, and this when the chemotherapeutic is itself horrifically toxic.

The “adequate and well-controlled investigations” phrase is generally interpreted as at least two studies, however even this is not always the case.  Especially in rare diseases one sees exceptions to this rule. So for example the approval of the Crofab(TM) rattlesnake antivenom was based entirely from a single open label study of, from memory, 39 patients.  This drug, and thankfully so, is likely stocked in every large emergency room where rattlesnake bite is a potential threat.  To become even more extreme, the very first recombinant enzyme replacement therapy to market, pegademase, was for an extremely rare disease, Severe Combined Immunodeficiency Syndrome (SKIDS) of the adenosine deaminase deficiency subtype.  This product was approved to market based from a single open label trial comprising a grand total of 10, yes ten, patients.

The 1997 Food and Drug Administration Modernization Act did include an update where “substantial evidence of effectiveness” now included a single adequate well controlled trial and “confirmatory evidence,” however, I think it should be clear by now that obtaining market approval from the FDA is not always iron clad, overwhelming evidence of effectiveness, again leaving the safety issue aside for now.

So in light of these regulations, what do we have in terms of evidence with sodium bicarbonate in chronic kidney disease.  Well in this case we actually have three “well-controlled investigations.”  I will only go off the information available in the abstracts of these studies as this is all that is publicly available, someone investigating this line of reasoning further should of course read all the studies with a fine tooth comb.

So the first study is, Oral Sodium Bicarbonate Improves Thyroid Function in Predialysis Chronic Kidney Disease.  This study then was a “no treatment control” prospective, controlled human trial which demonstrated that sodium bicarbonte preserved decline in kidney function.  Judging from the abstract however, kidney function was not the primary end-point of this study, so let’s be very conservative and class this study as confirmatory evidence.  Per regulations, though at this point we really only need one controlled trial to go along with this confirmatory evidence.

Speaking of confirmatory evidence we may also throw in, Higher serum bicarbonate levels within the normal range are associated with better survival and renal outcomes in African Americans. and Serum bicarbonate and long-term outcomes in CKD, both studies having been discussed in a previous post on the topic.

Next, Daily oral sodium bicarbonate preserves glomerular filtration rate by slowing its decline in early hypertensive nephropathy.  As described by the authors, this was a “5-year, prospective, randomized, placebo-controlled, and blinded interventional study.”  To reiterate, this was a blinded, placebo controlled trial, the gold standard, that is assuming they mentioned what the placebo was!  As most people don’t have access to the full text I will only discuss what’s in the abstract, so for instance we don’t know how many patients were involved, the methodology, the degree of difference between the two arms, etc, that led the author’s to conclude, “daily sodium bicarbonate is an effective kidney protective adjunct.”  However, this type of studied, if carried out honestly and correctly is the best type of evidence we have to judge clinical effectiveness.  Let me repeat that, this study design is the best we have for judging clinical effectiveness.

That’s one controlled trial, here is a second, “Bicarbonate supplementation slows progression of CKD and improves nutritional status.” The full text of this article is also freely available on line so we can say a bit more about it.  It was a “no treatment” control, prospective controlled trial carried out in 134 adult patients with low bicarbonate ion blood levels and Chronic Kidney Disease (CKD).  There were three primary endpoints related to kidney function 1) rate of Creatine Clearance (CrCl) decline 2) rapid decline of CrCl and 3) development of End Stage Renal Disease (ESRD) defined as CrCl less than 10 ml/min.  ESRD generally means one must go on kidney dialysis.  All three endpoints were statistically significant to the p <0.001 level.  Let’s just focus on the last endpoint, development of ESRD.  The authors note that “fewer patients supplemented with bicarbonate developed ESRD (6.5 versus 33%; relative risk 0.13; 95% confidence interval 0.04 to 0.40; P < 0.001).”  Well, I might argue that the more meaningful way to express the relative risk is from the untreated vantage point, so using the ole 1/x button on the calculator (0.13/1), we can also say that if left untreated the risk of kidney failure is nearly eight times as great!  Really you can also just look at the raw numbers 4 treated patients went on to kidney failure, 22 untreated patients went on to kidney failure, pretty simple.

Now the last argument, might be something along the lines of well, these are not “phase III” studies, as though phase III is defined by having 100s if not thousands of patients.  To which I would point out a few things, first, phase II studies are generally performed to try and optimize end-points, patient number and other aspects of study design for the larger more expensive phase III studies.  These pivotal phase III studies then are often “powered” by choosing a particular number of patients that can demonstrate a statistically significant difference for a certain “percentage” of true difference between the two groups.  That sounds a bit confusing but all it really says is that as you use more and more patients, smaller and smaller differences can be seen as statistically significant, I mean if you only have two patients they’re either all in one camp or the other or split dead even.  The drawback, here is that with larger and larger numbers there can be situations like 900 untreated patients became ill and 810 treated patients became ill and the difference was shown to be statistically significant.  While such a result may be valid, for logistical and other reasons, such a statistically significant finding may never have “clinical significance.”  Again 4 treated patients got ill 22 untreated got ill, sounds pretty clinically significant to me.

Secondly, it is wonderful to have large numbers when one is uncertain of the safety of the treatment, in the case of sodium bicarbonate this is far, far less of an issue than say with a new molecular entity.  Lastly, I would note that the study on sodium bicarbonate that we just got through discussing had over 13 times more patients then the single study used to get pegademase on the market for SCIDS, not to mention all the other bicarbonate studies.

So yes, I stand by my statement that it is time for these findings to be put into practice by nephrologists.  You know there is a bit of biological plausibility or “common sense” as it were involved here, many chronic kidney disease patients have low bicarbonate ion levels, this can be corrected, safely and cheaply and there is strong evidence it is helpful.

As physicians, wouldn’t it be nice to get away from practicing the “cookbook” type medicine so often being promulgated by the agencies which determine reimbursements, or worse yet, writing endless scripts for some medicine engraved on the side of a gift novelty pen.  Of course, if a patient with CKD comes in who has a high bicarbonate ion level such a treatment would not likely be appropriate.  I am not a nephrologist, perhaps there are other reasons to avoid such a treatment.  But wouldn’t it be nice to put all those long years of study, training and clinical experience into actual practice as opposed to practicing medicine based on clinical guidelines put together by a physician’s assistant working part time for a Health Maintenance Organization?

Lastly, I made this rather radical statement, that it is time for treatment of chronic kidney disease to change, because very few will hear this and of those who do probably only some will bother to look into the studies.   Perhaps a bold statement will at least get some discussion going.  If the evidence from these trials holds up, chronic kidney disease can almost be stopped dead in its tracks.  No one will get rich doing so, in fact those involved in dialysis or other expensive interventions will lose money, though the community will be wealthier, individuals and nations will save money, and many people might be spared a lingering death from kidney failure.

As I mentioned in an earlier post, there have been some 17 randomized trials looking at sodium bicarbonate in IV contrast induced nephropathy (i.e. to decrease the risk of $1,000+ dollar contrast-CT scans and the like).  I would fear that even if there were 20 positive studies of sodium bicarbonate in CKD, things still might not change without the drug reps coming by to promote it, or the headline in Time magazine or the industry sponsored symposium, and those things aint gonna happen.

While it would be better for such treatment to occur under the supervision of a physician, if evidence is ignored, some bright patients might just put two and two together and notice, gee, they used two or three grams of baking soda given orally in these studies.  That’s about a half teaspoon of baking soda, mix it up in a few ounces of water and take it once a day and that’s about all they did in those studies.  If they get good results, they will, rightfully so, no longer look at physicians as healers, just crass business men with a dangerous agenda, and once that trust is lost, as is already happening, it aint coming back any day soon.  To take care of people when they are ill and/or vulnerable, has for millenia been viewed a sacred trust, are we as physicians living up to that trust? Are we showing patients that no matter the situation or the pressures involved at the very minimum one will first do no harm, or is that just treated these days as a quaint joke?

So, to take a page from the PR firms, “If you or someone you love suffers from chronic kidney disease, ask your doctor about baking soda, the cheap white powder in the bright yellow box.”

You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

3 Comments

  1. Neonomide says:

    An incredibly important topic! I put this on blogroll already, thank you so much for information. Just shocking!

  2. Neonomide says:

    Just to add, it seem every month some fellow IBD patient writes me how his/her bloody stools and crapms dramatically improve, often within days, via Vitamin D3 supplementation. What's more, many of them are Colitis patients, that are not supposed to benefit epidemiologically from D3 supplementation. By now, actually even more Colitis than Crohn's patients have reported the change and no acute reports so far. D3 simply seems to work. We need TRIALS !

    I got the usual response from my gastroenterologist when I told him that I could eat foods I couldn't eat in 10+ years without any symptoms and it was clear cut. He simply started to speak loudly about the lack of intervention studies – even if I told about the extreme lack of toxicity in 1999 & 2006 reviews.

    I got an impression that as a patient, I have not a right to read and comment about the studies my disease on my own.

    Toxicity matters were of course very important to me and we agreed that my calcium levels would be monitored bimonthly. There actually is one case of hypercalcemia with Crohn's patient, so the monitoring made me feel more confident.

    The thing is, we still may need our doctor even if we can drop meds, get better etc.

    I really dig the Track Your Plaque approach, because people may track their own health markers and inform their doctors subsequently. More responsibility, less excess work and resourses wasted.

    I love your vitamin C series too, by the way.

  3. PDM says:

    Thx Neonomide,

    And am happy to hear that you are getting benefit from vitamin D supplements. You are right we need trials and honest trials. Will have to wait and see, maybe this time will be different.

    It is unfortunate that, especially for health educated patients, the interaction with the physician they've hired can be awkward to almost adversarial at times. As you point out, there is a lot that the physician brings to the table, but too often, I suspect, a visit does not seem so much like a consultation as a lecture.

    Will try and wrap up the C series before to long!