Following up on last year’s Corporate Legal Ops survey, we have again teamed up with Ari Kaplan, this time to survey a group of 25 AmLaw eDiscovery partners on over 40 topics. We also laid some of that data over last year’s CLO responses, which generated some pretty interesting insights on topics like data security, reporting, outsourcing, and Predictive Coding. Read on to see how your practice stacks up.
Metrics, Metrics, and Metrics
Business intelligence has taken on heightened importance for corporations and firms with mature eDiscovery programs. Last year, there was a clear theme that corporate respondents were not getting the kind of reports and transparency from their outside counsel that they wanted.
We struggle with outside counsel; they are very good about submitting invoices, but not on case status reports.
Perhaps in response to pressure from corporate clients, the firm leaders we surveyed this year had dramatically expanded reporting programs. And we found that what they were tracking was surprisingly different.
The CLOs we surveyed were tracking the total volume of data managed for their eDiscovery projects and very few were tracking review efficiency. In contrast, this year’s law firm respondents indicated review efficiency as the most tracked metric. It’s not entirely surprising; most respondents worked on the billable hour after all.
Audits are Still Largely Aspirational
Despite a near-universal concern over security, the responses from both surveys indicate that very few technical audits are actually occurring. In fact, the data showed an exact match in this realm: only 16% of CLO respondents audited the technical competency of their outside counsel and only 16% of law firm partners reported that they were audited. And there were many comments by the law firm partners indicating that they welcomed an audit, because it would differentiate their practice.
I wish they did [audit us] because people would appreciate our skill more.
However, when we asked about security practices, only about 10% of law firm respondents could articulate specific protocols like pen testing, anti-phishing, ISO certification, or encryption at-rest and in-motion. 20% of the law firm partners responded that they didn’t know what security protocols were employed. The remainder provided generic responses like “industry standard protocols” or “advanced protocols.”
Firms Innovate with Predictive Coding, but Misunderstandings Persist
It’s no surprise that machine learning legaltech is on the rise; analyst groups like Forrester and industry think tanks like CTRL consistently report as much. What’s more interesting is the qualitative responses we got around Predictive Coding, as 68% rated their experience as a 6 or more (out of 10). One respondent indicated that they trusted the AI more than their associates.
But misunderstandings still abound as the qualitative responses showed a great deal of confusion over different methodologies. 13% of respondents indicated a negative experience with Predictive Coding (defined as anything <5). After reviewing the comments, I suspect that this minority group used a tool that didn’t match their needs. For instance, there were a few complaints that Predictive Coding, while beneficial, required extensive negotiation, seed set development, and SME involvement.
These complaints and issues are typically rooted in older tools (aka TAR 1.0)—myths that we’ve worked hard to dispel: we hosted a 2-hour webinar on this very topic and published blogs. Obviously, there’s still more that can be done to educate the market in this space (for instance, CTRL’s recently updated Model Stipulation & Protocol has several paragraphs delineating TAR 1.0 from TAR 2.0 methods).
Outsourcing, Billing, The Cloud, and Everything Else
We found that 88% of law firm respondents outsourced some aspect of the eDiscovery process, generally dependent on the size of the project. For the work that is conducted in-house—primarily review—the vast majority of firms still bill by the hour. Only one respondent indicated “almost all alternative fee arrangements.” And despite responses indicating a growing acceptance of cloud-based tools, law firms still generally lag behind their corporate legal counterparts although there were a few comments from firms that had embraced the cloud.