The Expert: The Statistical Analyst In Litigation
Why some people should read The Expert:
At over 900 pages, this book is not for everyone. It is written for any attorney who proposes to engage a statistical expert, and anyone who proposes to be a statistical expert.
Why most people have not read The Expert:
Because it has not been reviewed. Why? Because there are few rewards for reviewing a book, especially a book as long and difficult as this. And, finally, because few people—including those who should—would understand it.
Is it that difficult?
No. However, I was shocked by a note I received from a law professor who should have known the material in Chapter 5, “The Composition of The Jury.” Most people who have analyzed jury composition have based their work on Michael Finkelstein’s ““The Application of Statistical Decision Theory to the Jury Discrimination Cases,” 80 Harvard Law Review 338 (1966). The U.S. Supreme Court based its decision in Castaneda v. Partida, 430 U.S. 482 (1977), on Finkelstein’s article. Almost all analyses since Castaneda have followed that template. But, as I demonstrate in Chapter 5, they are all wrong.
The Supreme court as much as said so only two years later, in Duren v. Missouri, 439 U.S. 357 (1979). The Supreme court, in a history that has never been written, came to the realization that “discrimination” must refer to actions, not outcomes, to events, not situations. That decision, a remarkable intellectual breakthrough, was correct.
But a professor with limited time, faced with such a detailed book, skimmed the text, concentrating on the footnotes. He penned a note to me saying that I had covered all the major cases, apparently never realizing that my conclusions from those cases were the opposite of what he teaches. Had he studied my book, or even just read that chapter carefully, he might have come out the other end a very different person. He assumed that I was just rehashing what he teaches, how Finkelstein set the standard for ever more. Why would I bother?
So The Expert is radical?
To the extent that “radical” means going back to the roots, then perhaps it is. In this case, however—given Duren—it should not be seen as radical, but as understanding what intellectual gyrations the Court went though after Castaneda. That decision must have made them uncomfortable, and they figured out why. The decision in Castaneda assumed that if juries did not look like the population, they must have been created with bias. They may well have been—those juries were created under the “key man” system of recruitment. But they may not have been. No one investigated what these “key men” did, only what the result was. In Duren the Court told complainants to specify the actions made by the jury administration that were discriminatory.
Why do courts, post Duren, permit Castaneda-type analyses?
When I am the opposing expert, they do not. Courts get it—the shift from Castaneda to Duren—if they are told about it. And they especially get it if they are presented with a non-bias explanation of how the outcome, about which complainants are complaining, came about. That is, if the statistical analyst studies events he may well come to a very different understanding of jury selection than the expert who posits a model, and finds that the real world does not look that way. But as attorneys, just like this professor, do not put in the effort to understand why Castaneda-type analyses are inappropriate, they do not have the tools necessary to defend jury selection systems that probably have no discriminatory elements.
And, needless to say, those attorneys engage the wrong experts, who also use the wrong tools.
So, this is all in your book, The Expert?
I have been discussing just one chapter. What follows here is an excerpt from the Introduction, describing the book (footnotes omitted):
Chapter 1 contains a history of my profession. How did the introduction of “expertise” into a trial come about? Who does it? I will note some trends other than in the law that have contributed to the explosive growth in the use of statistical analysis as expertise in litigation.
The next two chapters appear to be organized by analysis method: selection and then regression. That is not the only way to see it. One point of Chapter 2 is to show that experts traditionally utilized incorrect statistical methods to solve a particular kind of selection problem. The major point, however, is that judges, although not having the tools to develop a correct analysis, should have been able to reject faulty analyses. This chapter begins the de-mystification of statistical expertise in the courtroom. Where experts fail is in the relationship of the analysis to the law and institutions at issue.
In Chapter 3 I discuss how “experts” have bedazzled attorneys and judges alike with words that bear no relationship to their actual analyses, and “analyses” that are incapable of shedding light on the questions before the court. Regression is a fine statistical tool, but I show many instances in which neither experts nor their clients, and certainly not judges, understood it—not “understood” in the mathematical sense, but “understood” in the sense of realizing that the expert’s results simply did not mean what he said they meant. I do not intend to condemn all experts, and certainly not all regression analyses. What Lizzy Borden did to her parents says nothing about the functionality of an axe. I do mean to demonstrate that there has been a lot less expertise than there pretends to be, and that the way to discern helpful from awful expert work is not through statistics, but through intelligent reasoning in law.
The story of Chapter 4 is how social scientists failed to grasp the moment in their harsh criticism of the Court’s decisions on the size of the jury. Chapters 2 and 3 concern the failures and successes of social science in the courtroom. Chapter 4 concerns social scientists’ failure in criticizing the lack of social science in the courtroom. I show that criticisms of a set of Supreme Court decisions on the size of the jury are themselves failed social science, because of a conceptually inappropriate approach to the question.
Increased knowledge brings complexity rather than simplicity, uncertainty rather than certainty, frequently blurring distinctions rather than clarifying them. This knowledge thus becomes less helpful to the all‑or‑none, two-valued decision‑making process of the law. [Bernard Diamond in U. of Pa. Law Review, 1974]
The right approach might—just might—justify the Court’s decisions on jury size. My work opens that possibility, creating doubt where self‑satisfied social scientists expressed certainty. But, as I will express in several places, it is not the job of social science to draw lines for judges, only to provide some measures to inform judges what their line‑drawing means.
Selection issues return in Chapter 5. Here I take on both in‑court analysts and academic commentators. The subject is the composition of persons recruited to become jurors, with respect to constitutional provisions that jury‑gathering be impartial. The Supreme Court (and federal law) thinks that means the outcome should be “representative” of some “community.” When the Court substitutes “representative” for “impartial,” it substitutes an outcome for what surely was originally seen as a procedural constraint on juror selection. When it substitutes “population” for “community,” it also substitutes an outcome for a procedure in which who is in the “community” is determined within the jury‑selection process. What is needed is to understand “impartial” and “community” as procedural terms.
One influential law‑review article appeared to supply rigor to the Supreme Court’s outcome‑oriented direction. In the more than thirty‑five years since its publication, neither the Court nor any academic in this field has seen its essential fallacy, or that of the entire line of jury‑representation analyses and cases. Analysts posit a statistical model that cannot be estimated. Pretending to estimate it, they often obtain the wrong answer. I suggest an entirely different way to analyze jury representation issues, one that has infuriated opposing experts, but been accepted by every judge to whom it has been put forward.
Chapter 5 also forms the foundation for my assertion that experts should be more familiar with the laws under which they work. They should not accept the role of legal eunuch, the role that current rules and practices assign to them. I do not expand on that point until Chapter 8, however.
Here is another take on Chapters 2 through 5. Chapters 2 and 3 are about the failure of experts to understand their tools, using the wrong ones or using them improperly. They are about statistical methodology, but (it is my intention) framed in a way that attorneys and judges should understand. Chapters 4 and 5 take a larger view. They are about paradigms, approaches to legal questions. They are part of the meta‑debate, how we go about finding relevant facts in law in a broad sense. Chapters 2 and 3 are as much about the analyst failing to answer well‑framed questions, as their being the wrong questions. They can be read as about technical or conceptual error. Chapters 4 and 5 are more clearly about the framing itself. The analysts’ answers may be correct, but their questions were not.
With these four chapters I intend to establish that many analyses considered good, even expert, are failures because the expert lacks the understanding necessary to produce relevant work in the litigation context. Attorneys want experts who seem to support them, even if they do so irrelevantly. Judges often decide the questions posed by the attorneys, rather than the questions posed by the law.
I will demonstrate that bringing academics into the system is no answer, because the best of academics are often the worst of experts. Chapters 6 through 8 use my experiences, the examples provided in Chapters 2 through 5, and research material to discuss attorneys as clients and adversaries, judges as determiners of good science, and experts as all too often not up to the task they have taken on.
In Chapter 9 I follow academic experts back to their home base, the university, wondering how they can get away with such bad work. One answer appears to be that they do not bring litigation back to school, at least not to educate. This is a loss to them and their students. I also illustrate just how badly some “experts” mangle their own work, and how they, their professional organizations, and publishers suppress exposure of it. Where Chapter 9 most fits in the puzzle, however, is in my suggestion that academia was noticeably misdirected in the 1960s through 1980s towards technology, away from good thinking. This is a problem that suggests its own solution.
Chapter 10 is about the rules of expert‑witness work in relation to attorneys and courts. I contribute to the discussion about procedures for dealing with “science” experts, an active subject in the law‑review literature, and almost completely devoid of input from experts themselves. In previous chapters I have argued that the system is seriously flawed: Fact is hard to pull out of expert presentations, experts are not there to help, and judges are ill‑equipped to function without the help the experts should—but fail to—provide. In Chapter 10 I try to develop a theory of how this dilemma is evolving, and suggest who can arrest it and how.
This book is not a cry for help. It is an offer of help. Chapter 10 is an attempt to move forward, draw conclusions, set some standards, and at least understand, perhaps improve, the system of fact finding in the courts. The most important part of Chapter 10 is the development of a framework for judicial assessment of expert presentations. Legal scholars have been asking how judges can assess experts under Daubert (1993).My answer is that there is no single way. I define four kinds of expert, which are distinguishable by and large easily and cleanly. This classification leads to different methods of assessing expert work.
I also suggest changes to procedures with the goal of improving discernment among judges, so that attorneys have more incentives to bring better expert work to trial. I offer no magic bullet, no single key to the king’s gold. Chapter 10 is long, difficult, and, I think, rewarding. Then again, I would characterize the entire book the same way.