Posts
Test Drive This! Be an FDA Reviewer!
Below part of an assignment I am giving my students. It would be great to hear the thoughts of people here, both about what is and what should be....
***********************************************************
You are an FDA reviewer and need to balance the benefits and risks of a potential new drug.. What are the most important criteria or data that you think you need in order to make an informed, correct decision?
The circumstances listed below are just to help spark (but not limit) your thinking. Circle and comment upon those you think morst important.
--Your sister-in-law works for the company, and her job is uncertain.
--The company has always had an excellent track record, especially for drugs of this kind.
--The two major phase 3 trials (A and B) submitted for NDA approval were: A with 400 subjects enrolled over six months, and B with 400 S's . enrolled for one year. It is anticipated that many thousands, perhaps millions, of people will eventually use the drug for a chronic condition (thus, over many years).
--The studies submitted within the proposal showed a 37% efficacy (mild to moderate improvement in the condition) relative to 29% for the placebo (sugar pill). Adverse events (potential side effects) were about equal. Thus 63% of subjects showed no improvement (and no evident harm) in the timeframe of the submitted studies.
--The drug is primarily to be used for osteoarthritis.
--The drug is primarily to be used for breast cancer.
--Several large studies show that a competing drug (made by another company) in the same class, and for the same condition, is twice as effective as the drug submitted.
--In one of the trials submitted, the data suggested the drug might increase the risk for heart attacks and strokes, but the “signal” is just under being statistically significant within the trials submitted.
--Same situation as above, and you know that other drugs already approved in the same have shown statistically significant increased risk of heart attacks and strokes.
--Same situation as above two, and your supervisor says you have 48 hours to complete your review because of the PDUFA deadline. So far, most of the other reviewers have supported approving the NDA.. Either point relevant? Either point likely to make a difference in the "real world"?
Sunday, March 1, 2009
Subscribe to:
Post Comments (Atom)
Justice, One important thing to add to the discussion would be the oath that an FDA Reviewer takes. I haven't been able to find it but I understand that this oath can or should be a guiding tenant for decisions that need to be made.
ReplyDeletePerhaps someone can refer us to a copy of the oath...
ReplyDeleteHistorically, and currently, the role of the FDA reviewer is to assess whether or not the treatment is appropriate for use -- that is, does the treatment appear to be safe and effective based upon the results of the research to date. There are certainly a lot of confounding issues that exist in the real world, but they shouldn't affect the existing goals.
ReplyDeleteNow, the assessment of efficacy and (perhaps to a lesser degree) safety are subjective. The statistics can point to a difference in active vs. placebo, but if the measures are flawed, or inappropriate, the statistics are meaningless.
In today's world, the science is often complex. In tomorrow's world, the politics will be. The new stimulus bill calls for funding comparator trials to assess how, perhaps, new drug A performs against already approved drug B. When the findings are published, you're going to see manufacturers appealing, litigating and positioning to keep/get their drug at the top of the list. It won't be pretty.
The way I'd like to see this play out, is that our scientific understanding continues to improve to the point that FDA can say "Drug A works best for patients like this", and "Drug B works best for patients like that". At that point, everyone wins -- patients, manufacturers and payers.
Just my 2 cents...
thanks for your thoughtful comments, dhandelsman. It would certainly be great if we got to the point of your last paragraph.
ReplyDeleteWhen I heard David Graham speak once, he made reference to .05 stat significance being important both for confirming benefit and risk. Of course, the actual balancing and outcome is more complex than that, and--in my view--ideally includes consideration of what the indications are, whether there are reasons to "fast track," etc..
Knowing there is no "formula," can anyone else speak to the question of the use of statistical floors?
Having been an FDA reviewer I can tell you that the only relevant criteria is that the drug showed some degree of efficacy over placebo. NOTHING ELSE MATTERS! If you say anything to the contrary you will be retaliated against. If it happens a second time you will be retaliatated against even worse. Most people leave the FDA at this point. If it happens a third time. Kiss your career and livelihood goodbye.
ReplyDeleteAs for the oath it is the virtually the same as the president takes as laid out in the constitution.
Anon.- I was hoping you'd see this, and thanks for responding.
ReplyDeleteSo the thing I thought I remembered from David Graham about .05 doesn't fit anywhere? As I recall, he was talking about the way risk is assessed....
Just as an interested reader, I've gone through pieces of some NDA's (via the CDER site) and seen reviewer's comments. Certainly, there is comment on AEs, etc., and stat signif. And there is the famous Vioxx reviewer who raised a question. But...
But you are speaking about the practice, as opposed to the theory? Or some combo?
First, let me state that I'm not a statistician. That being said, a 'p-value' is the criteria often used to determine the statistical signficance of some difference in outcomes. A p-value of 0.05 is often cited as the indicator of statistical significance, but it need not be 0.05. (In my own poor-man's interpretation of this, it would mean that there is 95% confidence that the proposed hypothesis is true). I can almost guarantee that 0.05 isn't legislated anywhere, as it's simply one way to determine significance.
ReplyDeleteBack to the idea of an oath. The oath of office for goverment officials is basically the oath to protect and defend the Constitution of the US. That can mean a lot of different things but generally I would say that it means - I owe my allegiance to the citizens of the US for which the constitution was written.
ReplyDeleteIt certainly does not mean that I will protect and defend selfish interests of corporations, my boss, or anything that goes against the common citizen's life, liberty and pursuit of happiness protected by the constitution.
This general requirement defines a number of scenario that were stated above.
Justice, in answer to your questions:
ReplyDelete--Your sister-in-law works for the company, and her job is uncertain.
I would withdraw from the review to avoid the appearance of impropriety if the product was approved. I would be concerned about retaliation if the product was not approved.
--The company has always had an excellent track record, especially for drugs of this kind.
Would still need to see all of the data.
--The two major phase 3 trials (A and B) submitted for NDA approval were: A with 400 subjects enrolled over six months, and B with 400 S's . enrolled for one year. It is anticipated that many thousands, perhaps millions, of people will eventually use the drug for a chronic condition (thus, over many years).
I would want to know how it compares to other drugs for the same purpose. If it appears to be better than others I would give it a one year conditional approval while continued testing on larger groups is being done. I would want regular reports on safety and effectiveness as long as the drug is being marketed. If there is an uptrend in negative effects, the drug would be pulled from the market.
--The studies submitted within the proposal showed a 37% efficacy (mild to moderate improvement in the condition) relative to 29% for the placebo (sugar pill). Adverse events (potential side effects) were about equal. Thus 63% of subjects showed no improvement (and no evident harm) in the timeframe of the submitted studies.
If 63% of subjects showed no improvement, and the efficacy is only marginally better than a placebo, I’d suggest the drug company go back to the drawing board.
--The drug is primarily to be used for osteoarthritis.
--The drug is primarily to be used for breast cancer.
My response would be the same for either condition.
--Several large studies show that a competing drug (made by another company) in the same class, and for the same condition, is twice as effective as the drug submitted.
--In one of the trials submitted, the data suggested the drug might increase the risk for heart attacks and strokes, but the “signal” is just under being statistically significant within the trials submitted.
The competing drug is superior, even though it has some risks. Those risks, although low, should be fully disclosed to the doctor and pt, who can then decide on the benefit/risk factor.
--Same situation as above, and you know that other drugs already approved in the same have shown statistically significant increased risk of heart attacks and strokes.
Drugs that have shown a significantly higher risk of heart attacks and strokes should be withdrawn from the market. Rule # 1: Do No Harm.
--Same situation as above two, and your supervisor says you have 48 hours to complete your review because of the PDUFA deadline. So far, most of the other reviewers have supported approving the NDA.. Either point relevant? Either point likely to make a difference in the "real world"?
No, my job is to protect the public. The new drug didn’t make the grade and there is no reason to approve it.
In the real world it will be released as the newest and best product available. The ad will show a beautiful woman on a swing hanging from an apple tree abundant with blossoms, being pushed by Mr. America. In the background there is a lovely Cape Cod overlooking a blue ocean, made brilliant by radiant rays of sunshine, soft music is playing…
Everyone is gonna want this pill
Dhandelsman
ReplyDeleteThe following is an excerpt from Wikipedia on the work "confidence" as defined in statistics:
"Meaning of the term confidence
There is a difference in meaning between the common usage of the word 'confidence' and its statistical usage, which is often confusing to the layman. In common usage, a claim to 95% confidence in something is normally taken as indicating virtual certainty. In statistics, a claim to 95% confidence simply means that the researcher has seen something occur that only happens one time in twenty or less." If one were to roll two dice and get double six, few would claim this as proof that the dice were fixed, although statistically speaking one could have 97% confidence that they were. Similarly, the finding of a statistical link at 95% confidence is not proof, nor even very good evidence, that there is any real connection between the things linked.
When a study involves multiple statistical tests, some laymen assume that the confidence associated with individual tests is the confidence one should have in the results of the study itself. In fact, the results of all the statistical tests conducted during a study must be judged as a whole in determining what confidence one may place in the positive links it produces. If a study involving 40 statistical tests at 95% confidence was performed, about two of the tests can be expected to return false positives. If 3 links are found, the confidence associated with those links 'as the result of the survey' is actually about 32%; it's what should be expected two-thirds of the time."
JimK
I too, do not understand statistics and have provided the following scenario:
ReplyDeleteI have a study broken down into three groups:
Those taking the new product;
Those taking an existing therapy;
Those taking a placebo.
The endpoint of the study is 52 weeeks and the results are as follows:
The New Treatment shows a 23% improvement;
The Existing Therapy shows a 15% improvement;
The placebo group shows a 10% improvement.
Based upon what I have read, my understanding of Confidence Intervals for this study should be .1 or 10% as demonstrated by the placebo group. Therefore any improvement over the existing therapy of less then 10% is statistically insignificant.
Is this an incorrect assumption?