This Sidley Update addresses the following recent developments and court decisions involving e-discovery issues:
- a decision from the U.S. District Court for the District of New Jersey resolving several disputes regarding a protocol for the use of technology-assisted review (TAR), including that the Defendant could not limit validation testing of its TAR model by excluding documents that had already gone through a quality-control review process
- an order from the U.S. District Court for the Northern District of Texas finding that Slack messages produced in PDF format were produced in a “reasonably usable format”
- a ruling from the U.S. District Court for the Northern District of California declining to issue an order requiring the Defendant to preserve relevant electronically stored information (ESI) because the Plaintiff did not show that relevant evidence was or would be spoliated
- an opinion from the U.S. District Court for the Northern District of Illinois denying a motion to enforce a confidentiality order against nonparty subpoena recipients who objected to provisions of the order, including to allowing the parties’ in-house counsel to review confidential documents
This litigation involved an alleged scheme between insulin manufacturers and pharmacy benefit managers to inflate the price of insulin and other diabetes medications. At the outset of discovery, the parties disputed how one Defendant proposed to implement and validate its TAR model. Id. at *1. Defendant proposed to use a continuous active learning model (CAL), Reveal’s Brainspace CCML Predictive Coding Software, in its document review process and agreed to disclose to Plaintiffs certain metrics from its validation process.
But Plaintiffs argued that Defendant’s proposed TAR workflow was “inadequate,” including its methodology for training the model, the lack of a preset criterion for stopping TAR review and beginning validation, the use of elusion sampling from the null set of documents for the validation process, the scope of any disclosure to Plaintiffs as to documents that may be produced from the validation samples, and the lack of a preset validation recall target. Plaintiffs filed a motion seeking to have the court enter a TAR protocol order incorporating their proposed methodology, while Defendant opposed and argued that its proposed methodology was reasonable and provided sufficient transparency and validation.
Magistrate Judge Singh began her analysis by explaining that “[r]esponding parties are best situated to evaluate the procedures, methodologies, and technologies appropriate for preserving and producing their own” ESI. Id. (quoting The Sedona Conference, The Sedona Principles, Third Edition: Best Practices, Recommendations & Principles for Addressing Electronic Document Production, 19 SEDONA CONF. J. 1, 52, 118 (2018)). But she noted that courts have considered disputes over TAR methodology in the context of the “reasonable inquiry” standard under Rule 26(g) and considerations of proportionality under Rule 26(b)(1).
Because “transparency and cooperation among counsel” are important considerations when using TAR, Magistrate Judge Singh observed that “courts have mandated some level of transparency and validation of TAR methodologies.” Id. at *2. But she cautioned that “while cooperation and disclosures remain important, there may be other ways to ensure a producing party has appropriately trained and implemented TAR through statistically sound validation methods.” She noted that this was especially true “[w]ith the evolution of technology from TAR 1.0 to TAR 2.0, which uses CAL instead of simple active learning (SAL).” Finally, she explained that “commentary has recommended that any TAR protocol should allow sufficient flexibility for revisions to be considered if unforeseen problems arise.”
Considering the need for transparency, cooperation, and flexibility as well as “a responding party’s ability to make reasonable decisions about their ESI review and production methodology, and the extent of court oversight into the intricacies of TAR implementation,” Magistrate Judge Singh addressed each of the deficiencies raised by Plaintiffs with respect to Defendant’s proposed TAR methodology. Id. at *3.
Magistrate Judge Singh first discussed the specifics for training Defendant’s TAR model. Plaintiff argued that Defendant should “iteratively train” its TAR model and use “[a]ll relevance-based coding decisions” to “retrain the TAR system on a regular basis.” Defendant countered that it wanted “flexibility for it to use only QC’d (quality controlled) determinations at the tail end of the review to train the model,” which it claimed was “a proven method frequently implemented to improve model accuracy.” In support of their position, Plaintiffs proffered a declaration from Maura Grossman, a Research Professor in the School of Computer Science at the University of Waterloo, in which she stated that “continuously re-train[ing] the TAR model using all attorney relevance-based coding decisions” is “the better method.” She further claimed that “cherry-picking” training examples after the initial seed set “is a questionable practice due to its unpredictable impact on the learning algorithm.”
Considering the need for “[f]lexibility in any review process, whether TAR or otherwise,” Magistrate Judge Singh held that Defendant could proceed as it proposed with the caveat that Defendant provide advance notice to Plaintiffs and meet and confer “when it decides to limit training to only those documents reviewed for quality control.” Id. at *4.
Magistrate Judge Singh next discussed the stopping criterion that Defendant would use. Plaintiffs proposed that Defendants pause review and turn to validation when “two or more consecutive review batches sequentially populated by the highest-ranking uncoded documents remaining in the project in order from highest to lowest scores and containing a total of at least 1,000 documents are found to contain 10% or fewer documents marked responsive.” Plaintiff argued that their stopping criterion “would result in a marginal precision of 10%, which is objective, reasonable, and technologically sound.”
Defendant argued that “the decision of when to stop belongs to it” and there was no need to predetermine a stopping point because it had “agreed to make certain validation disclosures.”
Magistrate Judge Singh observed that prior cases “reflect that a reasonable and proportional stopping point is not solely quantitatively based, but also qualitative.” Id. at *5. Accordingly, she held that “reasonableness and proportionality factors will determine the appropriate stopping point” and declined to adopt Plaintiffs’ proposed stopping point.
Magistrate Judge Singh next addressed the scope of the dataset that Defendant would use to validate its TAR modal. The parties disagreed over whether Defendant would be required to rely “on samples drawn from the entire universe of documents subject to TAR” or would be permitted to use an “elusion set” that would exclude documents that had gone through the quality-control review process. Magistrate Judge Singh explained that elusion is “the percentage of documents of a search’s null set that were missed by the search, usually determined with review of a random sample of the null set.”
With respect to validation, Defendant argued that it planned to “reasonably QC and correct human error before validation.” Id. (emphasis in original). As a result, Defendant refused to include in the validation metrics any documents that had gone though “a reviewer QC validation process.”
Magistrate Judge Singh surveyed prior court decisions addressing validation methods, noting that there was some disagreement as to whether “evaluating the totality of the evidence derived from multiple sources” was required or whether “sampling of the null set” alone was sufficient. Id. at *6. She focused in particular on the decision in In re Diisocyanates Antitrust Litig., No. 18-1001, 2021 WL 4295729, at *6 (W.D. Pa. Aug. 23, 2021) (Report and Recommendation), adopted by 2021 WL 4295719 (Sept. 21, 2021). In that case, the court had held that “incorporating an analysis of the accuracy of the determinations made by human reviewers” into the recall estimation formulas was an “overreach” and refused to compel the defendants to use the plaintiffs’ proposed validation process. But Magistrate Judge Singh also noted that significant disputes had arisen after the court’s ruling in Diisocyanates, and the defendants ultimately calculated and disclosed an “end-to-end” recall rate for the entire review.
Taking these considerations into account, Magistrate Judge Singh held that Defendant would be required to include a sampling from the full TAR document universe as part of its validation process. Id. at *6-7. She expressed reluctance “to force a responding party to adopt validation metrics imposed by a requesting party.” But she found that Defendant’s proposal to conduct elusion testing only on a null set was not sufficiently reasonable in accordance with Rule 26(g) because “the weight of the available authority raises concerns that adopting [Defendant’s] proposed methodology would result in opaque and potentially unreliable recall calculations.”
Magistrate Judge Singh next addressed whether Defendant would be required to disclose to Plaintiffs “information concerning the stratum” from which any document produced out of a review from the validation samples was drawn. Id. at *7. Plaintiffs argued that this information was necessary to assess the “quality of the relevant documents that were not identified during validation.” Magistrate Judge Singh declined to impose this requirement on Defendant, finding that it was premature in light of the fact that Defendant had not even begun to implement its TAR model.
Finally, Magistrate Judge Singh addressed a dispute regarding the validation recall target. Plaintiffs proposed a target recall rate between 70% and 80% or more, but Defendant would agree only to “a reasonable proportional recall rate of 70 or higher” unless Plaintiff would agree that the ultimate determinant of the validation process would be “reasonable and proportional.” Here, Magistrate Judge Singh agreed with Defendant that “a reasonable and proportional recall rate of 70% or higher” was “sufficiently reasonable at this stage of the process.” She also noted that the parties had agreed to meet and confer following the validation disclosures and could at that point discuss “the adequacy of the recall measures in the context of quantitative and qualitative factors.”
This case arose out of an alleged misrepresentation about whether a 2009 Stephen Curry trading card was physically altered at the time Defendant authenticated and graded it. Among other discovery requests, Defendant requested that Plaintiff produce “communications and correspondence with any person concerning the Curry Card.” Id. at *4. In response to this request, Plaintiff produced certain Slack communications in PDF form. Defendant objected on the grounds that PDF was not an “appropriate format” for the Slack messages and requested that the messages be produced in their native format.
In particular, Defendant requested that Plaintiff re-produce the Slack files in a “.json” file format, which Plaintiff agreed to do. However, Defendant found that format unusable once it received the files. Plaintiff objected to re-producing the Slack messages in any other format, arguing that it would require “retaining an e-discovery vendor at cost.” Defendant filed a motion to compel Plaintiff to re-produce the Slack messages in their native format, arguing that the PDF versions sometimes included “pop ups or bubbles obscuring the conversation” and did not include dates for the Slack conversations.
Chief Judge Godbey quickly determined that the PDF copies of the Slack messages originally produced by Plaintiff were produced in a “reasonably usable format.” He explained that Rule 34 provides that “[i]f a request does not specify a form for producing electronically stored information, a party must produce it in a form or forms in which it is ordinarily maintained or in a reasonably usable form or forms; and (iii) a party need not produce the same electronically stored information in more than one form.” Id. (quoting Rule 34(b)(2)(E)(ii)–(iii)). Chief Judge Godbey noted that Defendant’s request did not require production in any specific form, so Plaintiff was permitted to produce Slack messages in the form they were “ordinarily maintained or in a reasonably usable form.”
With respect to Defendant’s argument that the PDF versions sometimes included “pop ups or bubbles obscuring the conversation” and did not include dates for the Slack conversations, Chief Judge Godbey noted that these issues did not require Plaintiff to re-produce the messages. He noted that Defendant raised no other issues indicating that PDF format was not reasonably usable and that other “courts have found PDF to be a reasonably usable format for electronic communications.”
Chief Judge Godbey found that the lack of dates for the Slack communications did not require reproduction because Defendant’s requests for production did not require the production of date metadata. And while the “pop ups or bubbles obscuring” communications did not require wholesale reproduction, Chief Judge Godbey ordered the parties to meet and confer regarding Plaintiff supplementing its production to resolve issues with any specific Slack messages that were obstructed. Beyond that, he declined to require Plaintiff to produce the Slack messages in any alternate format.
This action involved claims against the Plaintiff’s employer alleging employment discrimination for failure to hire on the basis of race, color, religion, national origin, and age, in violation of Title VII of the Civil Rights Act of 1964 and the Age Discrimination in Employment Act of 1967.
On July 8, 2024, Plaintiff sent Defendant a letter requesting that Defendant “preserve all evidence related to this case.” Id. at *1. In the letter, Plaintiff also reminded Defendant’s counsel that the “the duty to preserve evidence arises when litigation is reasonably anticipated” and requested that Defendant’s counsel acknowledge receipt of the letter and confirm in writing counsel’s intent to comply with the preservation demand.
Plaintiff had not received a response to this letter by October 2024, causing Plaintiff to file a motion to compel Defendant to “preserve all relevant evidence, including [ESI], communications, documents, and any other material related to this action.” Plaintiff’s motion also sought an order appointing a forensic expert “to ensure the integrity of the discovery process and to prevent the loss or destruction of vital [ESI]” and to expedite a ruling on the preservation motion “due to the imminent risk of spoliation and potential irreparable harm that could result from the destruction or alteration” of evidence.
Plaintiff argued that Defendant’s failure to respond to the preservation letter gave rise to an imminent threat of spoliation and an “elevated risk that relevant documents and communications may be inadvertently lost or altered.” Plaintiff also argued that “[w]ithout timely preservation Plaintiff will suffer prejudice.”
After Plaintiff filed its motion, Defendant’s counsel sent Plaintiff a letter providing Plaintiff with “formal written confirmation” that Defendant “has preserved and will continue to preserve evidence, including ESI related to the litigation.” Defendant also offered to meet and confer with Plaintiff, but Plaintiff did not agree to meet with Defendant or withdraw his motion.
Turning to the merits of Plaintiff’s motion, Magistrate Judge Jones explained that “[a] common-law duty arises, even before the commencement of litigation, for a party to preserve relevant information, including ESI, when litigation is reasonably foreseeable.” Id. at *2 (citing Rule 37(e)). He noted that Defendant was therefore “already under an obligation to preserve relevant evidence and has been since litigation was reasonably foreseeable.”
In light of Defendant’s common law duty, Magistrate Judge Jones found that Plaintiff had “presented no evidence to support the likelihood that relevant evidence is being or will be spoliated absent court intervention.” He further found that Plaintiff had failed to show any prejudice from the fact that Plaintiff did not receive the acknowledgment he requested from Defendant, because “Defendant was not required to respond to Plaintiff's request.”
Magistrate Judge Jones also faulted Plaintiff for failing to meet and confer with Plaintiff before filing his motion. Plaintiff argued that his letter to Defendant satisfied the requirement in the court’s local rules to make a “good faith effort to resolve discovery disputes prior to the filing of any discovery motions.” Magistrate Judge Jones noted that Plaintiff’s failure to withdraw his motion after Defendant represented that evidence was being preserved “strikes the court as something other than acting in good faith.”
Accordingly, because Plaintiff had failed to “demonstrate any evidence of spoliation beyond mere speculation and has failed to demonstrate prejudice,” Magistrate Judge Jones denied Plaintiff’s motion to compel preservation and to appoint a forensic expert. Id. at *3.
In this putative antitrust class action litigation stemming from a proposed telecom merger, the parties entered into a confidentiality order governing discovery in the case. But after the parties issued subpoenas to numerous nonparties, a dispute with the nonparties arose regarding the provisions of the confidentiality order. Id. at *1. Although the parties and the nonparty subpoena recipients met and conferred and “nearly reached agreement for amending the Confidentiality Order,” they could not agree on certain topics, including whether Defendant’s in-house counsel would be permitted to review highly confidential information from the nonparties.
Magistrate Judge Cole first addressed the fact that nonparties have “vastly different expectations regarding the confidentiality of their information.” He explained that although “parties to a lawsuit must accept the invasive nature of discovery, nonparties are just that, not parties to the lawsuit, and they generally do not have anywhere near the same skin in the game.” He therefore concluded that “a non-party is entitled to greater protection in the discovery process than parties in the litigation.”
Magistrate Judge Cole also found that there was “good cause” for some modification to the parties’ confidentiality order to protect information from nonparties that had no part in the negotiation of that order. He disagreed with Defendant’s position that nonparties should be bound by the confidentiality order because the court was “well-aware of the nonparty discovery that Plaintiffs’ claims would entail.” He noted that the terms the parties agreed to specifically stated that the confidentiality order would apply only to “any named Party to this action … and to Non-Parties who agree to be bound by this Order.” Id. (emphasis in original).
Magistrate Judge Cole pointed to Heraeus Kulzer, GmbH v. Biomet, Inc., 881 F.3d 550 (7th Cir. 2018), in which the court identified certain factors that should be considered when the modification of a confidentiality agreement is sought. Id. at *2. He found that the first factor, the nature of the order, favored modification “because the non-parties did not agree to it.” He concluded that the second factor, the foreseeability that modification would become necessary, also favored the nonparties because the parties’ agreement “left open the very real possibility that non-parties — competitors with one of the parties — would disagree with it.” Finally, he found that the third factor, the parties’ reliance on the order, “is really neither here nor there” because they knew “non-parties they planned on subpoenaing would understandably balk.”
Regarding the substance of the nonparties’ objection to the parties’ confidentiality order, Magistrate Judge Cole noted that “[t]he non-parties have some very real concerns about in-house counsel for a competitor pouring [sic] over their documents” and that “it is no small matter for in-house counsel to compartmentalize information learned in discovery.”
But Magistrate Judge Cole also found that the nonparties’ proposal of requiring Defendant to “litigate each particular request” was “unworkable or, at least, incredibly unwieldy.” He explained that this kind of approach would result in “extreme burdens” for Defendant and would “needlessly strain judicial resources.” He also noted that the court would likely not be in a position to “undertake some sort of meaningful in camera review of documents each time there is a dispute.”
Magistrate Judge Cole urged the parties and nonparties alike to “take a critical look at their current positions and make another attempt to come up with something workable” and to “think creatively with an eye toward what is truly and not merely academically meaningful.” Id. at *3. He also suggested that the parties and nonparties “consider the employment of a Special Master for their many, inevitable third-party discovery disputes.” He noted that the parties and nonparties “might want to get ahead of that possibility and put together some sort of mutually acceptable agreement rather than have something perhaps imposed on them down the road.”
Ultimately, Magistrate Judge Cole denied the motion to amend the confidentiality order and ordered the parties to meet and confer “with the forgoing considerations and weaknesses in their positions in mind.”
Sidley Austin LLPはクライアントおよびその他関係者へのサービスの一環として本情報を教育上の目的に限定して提供します。本情報をリーガルアドバイスとして解釈または依拠したり、弁護士・顧客間の関係を結ぶために使用することはできません。
弁護士広告 - ニューヨーク州弁護士会規則の遵守のための当法律事務所の本店所在地は、Sidley Austin LLP ニューヨーク：787 Seventh Avenue, New York, NY 10019 (+212 839 5300)、シカゴ：One South Dearborn, Chicago, IL 60603、(+312 853 7000)、ワシントン：1501 K Street, N.W., Washington, D.C. 20005 (+202 736 8000)です。
