One of the most interesting products of the first wave of wrongful convictions exposed by DNA has been a vigorous debate over potential changes in the design and execution of the lineups and photographic arrays, familiar to every television viewer, that police rely on to probe memory in eyewitness cases, the category that dominates the exoneration lists. All of the current proposals for change in investigative practice derive from extensive laboratory inquiry, and they have at their cores the novel “double-blind, sequential” technique for conducting eyewitness identification procedures. In this technique the law enforcement personnel conducting an identification procedure are “blind” concerning which person in the lineup or photo array is the police suspect, and they present the “fillers” and the suspect to the witness individually (“sequentially”) rather than in a group (“simultaneously”), as in the traditional practice. The changes from current procedure are designed to ensure that witnesses discern no inadvertent cues as to which individual they should or should not identify, to encourage witnesses to compare each individual they see to the remembered image of the criminal (rather than to make a relative, “looks-most-like,” judgment comparing the individuals displayed to each other), and to eliminate unnecessary “feedback” to witnesses who have made a selection and might look to the lineup administrator for confirmation or contradiction. 2006 2006 Unfortunately for criminal justice practitioners who must decide whether procedures should be changed, the early scientific commentaries on the Mecklenburg Report generally aligned with the views on the potential of these particular procedural innovations that the commentators had announced throughout their long careers of involvement with the issue of eyewitness memory. Seizing on this, partisans on both sides of the debate over procedures have unfairly dismissed some criticism and praise of the Mecklenburg Report as reflecting nothing more than the scientific commentators’ stubborn loyalty to their own pre-existing beliefs. A standoff has arisen. Although everyone agrees that further field studies are required, practitioners considering future field studies have been left to wonder whether they should simply repeat the Illinois Study described in the Mecklenburg Report, or attempt to find a new design. 2006 2006 2006 2006 It is easy to understand the sentiment expressed by Mecklenburg in her Addendum that not all variables can be controlled in a field study such as the one she designed and describes in the Report. Confounds can occur in laboratory studies as well as field studies. The issue that always arises in such cases concerns the implications of the confound: Is it critically related to interpreting the major outcome of the study? Or is the confound incidental to the main conclusion, such that even though the confound is acknowledged, the major results of the study are still interpretable? The Mecklenburg Report asserts that “The Illinois Pilot Study was properly designed to answer the question: how do the current procedures compare with the proposed procedures, both in terms of identification rates and implementation?” From this perspective, the confound between blind/non-blind and sequential/simultaneous would not be critical, because non-blind simultaneous reflects the current procedure to which the blind/sequential procedure is compared. Unfortunately, this perspective seems seriously problematic. Our reading of the materials forces us to conclude that the confound has devastating consequences for assessing the real-world implications of this particular study. If it is the case that the better outcome from the non-blind/simultaneous procedure is partly or entirely attributable to subtle, unintentional cues provided by the administrator, then the Illinois results may simply underscore that the present procedure produces a biased outcome that may ultimately result in the increased conviction of innocent individuals. Stated slightly differently, it is critical to determine whether the seemingly better result from the simultaneous procedure is attributable to properties of the simultaneous procedure itself, or to the influence of the non-blind administrator. We should note that under these testing conditions, if the results had shown the sequential lineup to be superior, one would not know whether it was really the use of the sequential lineup or the use of a blind investigator conducting the lineup that produced the result. Of course, any difference between conditions could be due to some combination of the factors. Even if no difference in outcome occurred between the procedures, one could not safely conclude there is no difference between them if the detectives were informed in one condition and not in the other. Thus, although the conditions used in the study made some sense from a practical standpoint, the design guaranteed that most outcomes would be difficult or impossible to interpret. The only way to sort this out is by conducting further studies including, at a minimum, a blind/simultaneous condition (it would also be desirable to include a non-blind/sequential condition to fill out the design, but it is not an absolutely necessary condition for the present purposes). In the materials we have reviewed, the Mecklenburg Report’s detractors (including Wells) and its advocates (including Mecklenburg herself) disagree on whether the misidentification rates in two of the three participating jurisdictions (a zero rate of “filler” identifications) are suspiciously low. However, Wells cites enough evidence that they may be low to justify the concern that administrator bias is operating, either consciously or unconsciously; either by failing to count tentative “filler” choices, or in steering witnesses away from fillers, or toward suspects. The problem is that we cannot know on the basis of the Mecklenburg study whether such bias is operating, even though the entire interpretation of the significance of the study for real-world practices hinges on this issue. Mecklenburg states in her Addendum that the question of how blind administrators affect simultaneous lineups is one of several questions to be addressed in future studies. We certainly hope so. But the statement that follows is problematic: “However, the Illinois Pilot Program was not intended to answer those questions and any attempt to discredit the Illinois study on that basis is misguided.” If the Illinois study was not designed to address the question of what happens in a blind/simultaneous line-up, given its centrality to the issue, then our assessment is that the Illinois study addressed a question (comparing blind/sequential and non-blind/simultaneous) that is not worth addressing, because the results do not inform everyday practice in a useful manner. No single field study can produce a final blueprint for procedural reform; we will need many. The design of these studies, however, will be crucial. A well-designed field study that avoids the flaw built into the Illinois effort, can be an important first step toward learning what we need to know about the best practices in identification procedures.