Feb 10, 2024 14:00:00

What was the result of a legal contract review showdown between AI and a lawyer?

ChatGPT, which can perform sophisticated sentence generation and

summarization, will help law school students pass exams , and there are plans to have AI lawyers based on the large-scale language model (LLM) GPT-3 make legal arguments. has been announced, and it has been pointed out that generative AI may be able to demonstrate its power in the judicial field. Onit , which provides business legal services in New Zealand, has published a paper that compares LLMs and traditional reviewers in terms of reviews, including identifying problems and pointing out mistakes in legal contracts.

[2401.16212] Better Call GPT, Comparing Large Language Models Against Lawyers
https://arxiv.org/abs/2401.16212

The paper examines the task of reviewing legal contracts, which is often outsourced to inexperienced lawyers, and asks, ``Are LLMs or junior lawyers better at determining and identifying legal issues in contracts? '', ``Who can complete a contract review faster, an LLM or a junior lawyer?'', and ``Which can review a contract more cheaply, an LLM or a junior lawyer?''

The research began by preparing 10 'procurement contracts' based on actual legal agreements, anonymized to maintain confidentiality. Procurement contracts were chosen because they primarily involve bidding and are often considered by legal practitioners. Non-disclosure agreements (NDAs), which are often treated similarly, are intentionally excluded from the analysis due to their generally brief form.

In addition, the scope of contract jurisdiction is balanced between the United States, which is based on a combination of statutory law and common law, and New Zealand, which is based on common law. This is an approach to ensure that findings are relevant across different legal systems, increasing the usefulness and applicability of research.

The results of having a lawyer and LLM review the contract were compared with the correct data set by a senior lawyer, and the degree of agreement was evaluated using the F value .

The results regarding 'judgment ability on legal issues' are shown in the table below, and the higher the 'F-score' (F value) shown in the red frame in the table, the higher the accuracy of the review. The most accurate is LPO (Legal Outsourcing), which generally handles contract reviews as one of its specialized services, making it possible to perform highly stable reviews. LLM's 'GPT4-1106,' which was announced in November 2023, recorded a high score almost on par with LPO, and it was successfully reviewed with higher accuracy than 'Junior (Young Lawyer).'

Furthermore, the table below shows the performance regarding 'Identification of legal issues', where the LLM 'GPT4-32k (32K version)' released in September 2023 after the top LPO recorded a high score. Next was 'Claude 2.1' from OpenAI's competitor Anthropic, followed by GPT4-1106, which scored high in judgment, followed by Young Lawyers in fifth place.

The time taken to review legal contracts was then compared. The paper found that the average time it takes for a senior lawyer to complete a single contract was approximately 43 minutes for senior lawyers, approximately 56 minutes for junior lawyers, and approximately 201 minutes for LPOs. On the other hand, the paper concludes that LLM is significantly superior in terms of time efficiency, with GPT4-1106 taking 4.7 minutes, GPT4-32k taking 2.11 minutes, GPT3.5 taking 1.44 minutes, and Claude 2.1 taking about 2.05 minutes. Masu.

Finally, below is a table showing the average cost per contract. If you rely on a lawyer, it costs about $75 (about 10,000 yen), and LPO costs about $36.85 (about 5,400 yen), but in the case of LLM, it costs a lot more than $1, except for the 32K version of GPT4 which costs $1.24 (about 180 yen) has fallen below. When it comes to cost efficiency, LLM is overwhelmingly lower when compared simply.

The paper found that LLMs are significantly faster and cheaper than lawyers and LPOs, but can provide comparable performance. However, the ability of LLMs to identify problems in contracts is dependent on the model, so at the time of conducting the study, the process of ``selecting the appropriate model for the legal task'' was of great significance. , people with a certain degree of specialized knowledge need to handle LLM.

“The implications of our findings for the legal industry are deep and multifaceted,” the paper states. LLMs may replace junior lawyers and LPOs, at least when it comes to contract review, and may allow promising lawyers to advance into more complex work at an early stage. It also concludes that because LLM offers significant efficiency gains and cost savings, early adopters could 'gain a significant competitive advantage and perhaps spark an arms race in the legal field.'