Would Our Treatment Decisions Be Better Justified in the Absence of Observational Data?

Published online before print September 22, 2011, doi: 10.3174/ajnr.A2755
AJNR 2011 32: E180

A. Laaksoa, M. Niemeläa and J. Hernesniemia
aDepartment of Neurosurgery
Helsinki University Central Hospital
Helsinki, Finland

In the May 2011 issue of the American Journal of Neuroradiology, Raymond et al1scrutinized at length the methodologic weaknesses of 2 recent articles about the natural history and risk of hemorrhage in brain arteriovenous malformations (AVMs), one from our department2 and the other from Toronto Western Hospital.3 While part of their criticism is undeniable—such as inevitable patient selection bias in any observational study concerning a life-threatening disease or the effect of the choice of variables and statistical methods on the observed outcome—the main conclusion of their denunciatory analysis leaves us perplexed. The authors state that studies like this should not be used to inform clinical decisions and that relying on prognostic estimates based on these studies can be dangerous.

The question that remains unanswered, then, is, “What should we base our treatment decisions on at present?” As of today, no results from a randomized clinical trial concerning the treatment of brain AVMs exist. A Randomized Trial of Unruptured Brain AVMs is a laudable effort to shed more light on the issue and is underway, but we will probably have to wait for another decade before having conclusive results from that at our disposal. It will also tell us nothing about the behavior of ruptured AVMs, and it is doubtful that a randomized trial on ruptured AVMs will ever be conceived. Moreover, while the value of randomized controlled trials (RCTs) as the criterion standard to prove or disprove the effectiveness of a treatment cannot be disputed, they are unfortunately not immune to many of the same pitfalls the authors blame on observational studies, especially in cases of complex, rare, dangerous, and invasively treated diseases. Considerable subject selection bias makes them poorly generalizable to all patients, the cohort sizes and follow-up times are not likely to be more extensive than those in observational studies, results vary from center to center, disease progression may lead to crossing over to another treatment arm or termination of the study for many patients, and the choice of variables and statistical methods is based on educated guesses just like in retrospective studies. We are not saying that RCTs are unnecessary; we are just wondering whether our treatment decisions would, at present, really be less “dangerous” if we did not have even the observational data from historical cohorts to inform us?

While the authors are entitled to their opinion, there were, however, some factual errors in their analysis that need to be rectified.

• 1) The authors state that the overall hemorrhage rates differ by a factor of 2 in these 2 reports.2,3 A closer look reveals that the average hemorrhage rate in our study for the first 5 years of observation was 4.7% per year (Table 22), very similar to 4.6% per year in the Toronto study3 with a mean follow-up period of 2.9 years. This comparison is more meaningful than using the annual hemorrhage rate of 2.4% derived from the whole follow-up period of our study (with a mean duration of 13.5 years), especially because both studies observed and reported a decline in the hemorrhage risk with time.
• 2) Concerning the beginning of the follow-up, the authors claim that “In both studies, it is unclear when the clock started (first admission, diagnosis, referral?).” In our report, it is stated explicitly in the “Methods” that “follow-up data were collected starting from the admission to a neurosurgical referral center.”2
• 3) The authors claim that “Neither article presented any confidence intervals around their estimates of risk [of hemorrhage].” We have reported 95% confidence intervals both for all cumulative hemorrhage risk estimates derived from Kaplan-Meier life-table analyses (Table 2) and for relative risk ratios based on Cox proportional hazards uni- and multivariate models (Table 32).
• 4) The authors state that “Furthermore, extrapolation of risks observed during a relatively small number of years to lifetime risks by multiplying the observed rate by the number of years the patient is expected to live is, to say the least, uncertain.” While we neither suggested nor performed such an arithmetic exercise (the cumulative rupture rates in our report are real observational data subjected to life-table analyses), we would still like to remind the authors that the annual probability of a certain outcome (eg, hemorrhagic stroke) should never be multiplied by years at risk; the proper formula to estimate the cumulative probability of the outcome is 1-(1-p)t, where p = the annual probability of the outcome and t = time at risk in years, given that the risk remains constant with time (which does not seem to be the case for AVMs).

Despite the obvious shortcomings of observational analyses of historical cohorts and the complexity of AVMs as a disease, we still honestly believe that the results from our study, as well as from similar ones performed by others, will lead to better informed treatment decisions than complete lack of knowledge. It is true that the data from AVM natural history studies are not unequivocal in terms of all hemorrhage risk factors or for annual risk rates; thus, independent replications in different cohorts are important. Moreover, perhaps depending on the eye of the beholder, the results from different cohorts for many risk factors are not that divergent, after all, and eventually common trends will emerge (eg, our recent review4).

References

1. Raymond J, Naggara O, Guilbert F, et al. Assessing prognosis from nonrandomized studies: an example from brain arteriovenous malformations. AJNR Am J Neuroradiol 2011;32:809–12 » Abstract/FREE Full Text
2. Hernesniemi JA, Dashti R, Juvela S, et al. Natural history of brain arteriovenous malformations: a long-term follow-up study of risk of hemorrhage in 238 patients. Neurosurgery 2008;63:823–29, discussion 829–31 » CrossRef » Medline
3. da Costa L, Wallace MC, Ter Brugge KG, et al. The natural history and predictive features of hemorrhage from brain arteriovenous malformations. Stroke 2009;40:100–05 » Abstract/FREE Full Text
4. Laakso A, Dashti R, Juvela S, et al. Natural history of arteriovenous malformations: presentation, risk of hemorrhage and mortality. Acta Neurochir Suppl 2010;107:65–69 » CrossRef » Medline

Reply

Published online before print September 22, 2011, doi: 10.3174/ajnr.A2803
AJNR 2011 32: E181

J. Raymonda, T.E. Darsauta and F. Guilberta
aDepartment of Radiology
Interventional Neuroradiology Research Unit
International Consortium of Neuroendovascular Centres
Centre Hospitalier de l’Université de Montréal
Notre-Dame Hospital
Montreal, Quebec, Canada

O. Naggarab
bDepartment of Neuroradiology
Paris-Descartes University
Centre Hospitalier Sainte-Anne
Paris, France

D.G. Altmanc
cCentre for Statistics in Medicine
University of Oxford
United Kingdom

In a previous article in the American Journal of Neuroradiology, we reviewed numerous methodologic difficulties encountered in observational studies on the natural history of cerebral AVMs. We concluded: “The estimates of risk of rupture per year are uncertain. Multiplying those uncertain numbers by the life expectancy of individuals can inflate error beyond control. Hence relying on these estimates to make clinical decisions may be dangerous.”1

In their letter to the editor, Laakso et al asked a most important question regarding the management of AVMs, “What should we base our treatment decisions on, at present?” Before we turn our attention to this crucial issue, we must first clarify a few points that Laakso et al called “factual errors in our analysis”:

• 1) As much as we would like to reconcile estimates from different studies, no matter how diverging the results, we must be careful not to create, out of diversity, selection, and statistical models, a “natural history of AVMs” that will make us believe we know more than we actually do. If similarities between the annual rupture rates calculated in Toronto (4.6% per year) and the ones in Helsinki (2.4%) are so desperately needed, why publish the 2.4% hemorrhage rate in the conclusion of the abstract in the first place or why use patient-years or report the entire follow-up period?2,3
• 2) We questioned when the clock should start counting the length of the follow-up period. The question regards the biases introduced by the fact that some lesions can be revealed by symptoms such as seizures, while others can only present with hemorrhages. Presumably, all lesions are present from birth. Hence the length of the follow-up period used in the denominators of these calculations is, we suspect, more a reflection of detection bias than a witness of various specific risk factors for ruptures.
• 3) It is true that the authors of the Helsinki study provided confidence intervals for the cumulative rupture rate for the first 5 years. Unfortunately confidence intervals were not given for the annual rupture rates (ie, the 4.7% or 1.6% or 2.4% we are supposed to compare with the 4.6% from Toronto). Confidence intervals only reflect sample variation. They take no account of other sources of uncertainty, such as loss to follow-up or censuring because of treatments.
• 4) If the rupture rates decrease so markedly with time, it does not matter whether clinicians use p or 1- (1-p)t. The warning we are trying to give is that these numbers are not only too uncertain to be multiplied, but they probably should not be used at all to justify risky preventive interventions.

We must then apologize if the authors thought they were unfairly quoted. They did their best to provide the community with estimates of the risks of hemorrhage of brain AVMs, but as they now concede, these types of observational studies are fraught with insurmountable problems. When what is needed is to acknowledge the uncertainty, studies that add pseudoprecision to the analysis of unreliable data do not help. It is just another way of evading the problem raised in the title of their letter, “Would Our Treatment Decisions Be Better Justified in the Absence of Observational Data?”

How, we ask, are we supposed to use observational data to make treatment decisions? More important, how could these data justify our risky interventions? This, then, is the crux of the matter: In the absence of reliable evidence, should we make clinical decisions at all, pretending we know, and then attempt to do the biased research without the consent of participants, to justify, after the fact, what we have already done? The authors mentioned the ARUBA (A Randomized Trial of Unruptured Brain AVMs) study, dedicated to unruptured AVMs but could not wait for the results of the trial. For unruptured AVMs, the Helsinki study reported 17 events that occurred in 99 patients during more than 60 years between 1940 and 2005. Rare events are further split into size, location, venous drainage, and so forth. How much knowledge can one extract from so little data, and how many clinical decisions are to be guided by this meager experience? Clinicians who cannot wait for the results of the trial should simply recruit patients into the trial. Trials are designed for that very purpose.4

Observational studies of patients selected for conservative management were never meant to replace trials to guide clinical interventions. They cannot test nor show the benefit or harm of therapy.5 We look up to the leaders of this field, such as the Helsinki group, to guide us into the future, a future where treatment decisions are made on reliable evidence that they do more good than harm. Rather than attempt to justify our decisions after the fact, we must squarely confront the uncertainty, reveal it in a transparent manner to our patients, and design trials that will help us act in a scientifically sound and ethically right manner.

References

1. Raymond J, Naggara O, Guilbert F, et al. Assessing prognosis from nonrandomized studies: an example from brain arteriovenous malformations. AJNR Am J Neuroradiol. 2011;32:809–12 » Abstract/FREE Full Text
2. Hernesniemi JA, Dashti R, Juvela S, et al. Natural history of brain arteriovenous malformations: a long-term follow-up study of risk of hemorrhage in 238 patients. Neurosurgery 2008;63:823–29, discussion 829–31 » CrossRef » Medline
3. da Costa L, Wallace MC, Ter Brugge KG, et al. The natural history and predictive features of hemorrhage from brain arteriovenous malformations. Stroke 2009;40:100–05 » Abstract/FREE Full Text
4. Pelz M, Milde M. Letter to the Editor – Re: Raymond J, Mohr JP, The TEAM-ARUBA Collaborative Groups: prevention of hemorrhagic stroke—a review of the rational and ethical principles of clinical trials on unruptured intracranial aneurysms and arteriovenous malformations. Interv Neuroradiol 2009;15:369–71. Epub 2009 Nov 4 » Medline
5. Byar DP. Problems with using observational databases to compare treatments. Stat Med1991;10:663–66 » Medline

http://www.ajnrblog.org/2011/10/07/would-our-treatment-decisions-be-better-justified-in-the-absence-of-observational-data/

Sent with MobileRSS for iPhone

Júlio Leonardo B. Pereira
http://lattes.cnpq.br/7687651239699170
https://neurocirurgiabr.comhttp://www.radiocirurgia.org