How not to predict the outcome of clinical trials

Yesterday, I came across a concept that I had never considered before (or even heard of before). In fact, it’s a concept that took me by surprise. Basically, it’s an application of a concept to a problem that I never considered applying the concept to before–probably with very good reason. Basically, it’s a guy named Michael Slattery writing about The Wisdom Of The Crowds And Sangamo’s Phase II B Clinical Trial Results.

I’ve never been convinced that there is a such thing as the “wisdom of crowds.” Certainly, the numerous examples throughout history of mob behavior and downright idiotic actions of crowds don’t help. Neither does the example of Wikipedia, which is supposed to be based on the wisdom of crowds but whose editors and managers have found in recent years that perhaps crowds are not so wise at all, forcing the to lock articles to prevent tampering. Never mind the fact that in Wikipedia and the wisdom of crowds, the opinion of an expert in the field being examined and a complete and utter crank can be of equal value if the rest of the participants can’t perceive the difference. So, no, I’m not a big believer in the wisdom of crowds; personally I tend to think that the stupidity that people are capable of is only magnified in crowds, except under situations that are very artificial indeed. Indeed, one of the best brief summation of the so-called “wisdom of crowds” that I’ve seen:

Collecting people in crowds increases exactly three things with any certainty, and none of them are wisdom:

  • It increases the likelihood of a single individual member of the crowd surviving against predators.
  • It increases the effective conservatism of decision-making, not only when consensus (or even majority) decisions are made, but often even individual decisions.
  • It increases the tendency of individuals within the crowd to excuse their own actions and justify them regardless of how reprehensible those actions may be when contemplated in solitude.

Be that as it may, Mr. Slattery proposes a particularly bizarre use of the “wisdom of crowds”:

Would you like to be able to accurately predict the outcome of Sangamo’s (SGMO) Diabetic neuropathy, Phase II clinical trial of SB-509? Given the certainty that the stock will more than double if successful and will jump again shortly thereafter when a partnership deal is announced, I will assume that you, like me, would like this information well in advance of Sangamo’s announcement. No single individual can predict the outcome of a double blinded human clinical trial. But working as a crowd, we can get very close to knowing with a high probability of success, what the results of this trial will be, well in advance of any announcement.

Huh? Actually, Slattery is quite wrong. I’m not at all familiar with SB-509 is or what the evidence is that it will alleviate the symptoms of diabetic neuropathy, nor do I need to know anything about it for purposes of this discussion. The reason that I know that Slattery is quite wrong is easy. I know that a single person can estimate the probability that a clinical trial will be positive if that person has a good set of studies and other data upon which to base an estimate of prior probability using Bayesian principles. True, it might not be so precise as to be within ten percentage points of a specific percent probability, but it can certainly be high, medium, or low, which is about all the precision that most investigators need. At least, there’s no evidence that any crowd can produce a better estimate of prior probability than a statistician or scientist applying the proper type of analysis.

But let’s look at it another way. Even if the “wisdom of crowds” was an actual phenomenon (and there is not really a lot of good evidence that it is, I’d be willing to bet that even James Surowiecki would cringe at this proposed use of the concept:

By soliciting a large number of individual opinions (Yours!) we can aggregate the contributed opinions and analyze them in a manner that will return to the group, the wisdom of the Seeking Alpha readership. In this case, the probability that Sangamo’s study of Diabetic Neuropathy will be successful.

Slattery explains why he thinks this will work thusly:

The reason Wisdom of Crowds (WOC) works so well is not completely known. The author of the book Wisdom of Crowds believes the following explains the basics of this amazing process. As the frequency of the selections decline, those selections will more often fall into one of two categories, negative error and positive error, eliminating them from the accurate information we are seeking. When positive error cancels out negative error you are left with the information that will most likely benefit you goals and in this case that will be the most likely probability for the success of Sangamos Phase II B Diabetic Neuropathy clinical trial.

Does anyone see the problem with this line of reasoning?

First off, it is not “information” that Slattery is seeking. At least, he’s not seeking any sort of information that is verifiable; that is, unless the “wisdom of crowds” concludes that the probability of success of the trial is either 0% or 100%, in which case its conclusion is falsifiable. After all, if the crowd predicts a 0% probability of success, and the trial is successful, that would pretty much bury the crowd’s prediction, and vice versa.

Anything else is useless and unfalsifiable, however. Take the example of the crowd coming up with a consensus that the probability of the trial’s success is 65%. If the trial is, in fact, successful, it proves nothing about the accuracy of the estimate. Neither does it prove anything if the trial turns out to be unsuccessful, because a 65% chance of succeeding still leaves a 35% chance of failure, which is more than 1 in 3 and therefore sufficiently common that it’s not so surprising if there’s a negative result even if the pre-trial probability was 65% In fact, even if the crowd estimated that the pre-trial probability of a positive result was 99%, a negative result would still not prove them wrong. Yes, if the crowd suggested that there was a 99% pre-trial probability that the clinical trial would be a success and it is a success, it would suggest that they got it right, but there would still be uncertainty.

Even leaving that aside, the way Slattery decided to ask for the crowd’s input violates even the very highly artificial conditions that Surowiecki declares as prerequisites for this whole “wisdom of crowds” thing works:

These elements are: 1) The crowd must be as diverse as possible. Each person should have private information even if it’s just an eccentric interpretation of the known facts. 2) The crowd must be large enough to guarantee this diversity. 3) Each contributor’s opinion must be completely independent. People’s opinions aren’t determined by the opinions of those around them. 4) Decentralization; People are able to specialize and draw on local knowledge. 5) Aggregation; Some mechanism exists for turning private judgments into a collective decision.

So, even if you accept the validity of the “wisdom of crowds,” I’d say that at the very least condition #3 is violated. People are posting estimates in the comments, and that’s bound to influence people who encounter the article later and decide to try their hand at coming up with an estimate. Also, I have no idea how large the blog’s readership is, but I’m betting that it’s not nearly as diverse as necessary to meet conditions #1 and #2.

In the end, I must admit that this particular post amused me. (If it didn’t, I wouldn’t have bothered to devote a post to it.) I had, however, hoped for more when I read the title in that I figured that maybe Slattery could educate me about how one might try to use the “wisdom of crowds” to predict clinical trial outcomes. Alas, it was not to be, and I was disappointed. I didn’t really learn anything, and all I encountered was an utterly useless attempt to apply “wisdom of crowds” thinking to a problem quite unsuited for it. After all, it’s not as if such an exercise can in any way bypass clinical trials.

I just hope that no investors are planning on basing their decision to invest or not to invest in Sangamo based on this.