{"id":13636,"date":"2025-03-17T16:44:29","date_gmt":"2025-03-17T16:44:29","guid":{"rendered":"https:\/\/prizmlaw.com\/site\/?p=13636"},"modified":"2025-04-18T17:37:40","modified_gmt":"2025-04-18T17:37:40","slug":"llm-benchmarking","status":"publish","type":"post","link":"https:\/\/prizmlaw.com\/site\/2025\/03\/17\/llm-benchmarking\/","title":{"rendered":"Evaluating A.I. Language Model Performance for Legal Reasoning"},"content":{"rendered":"\t\t
\n\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t
<\/h6>

My Work Evaluating A.I. Language Model Performance for Legal Reasoning<\/h3><\/div><\/div>\"\"<\/a><\/div><\/div>\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t

TLDR: Quick Summary <\/h2>\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t
<\/g><\/svg><\/div>Language models power A.I. tools that can substantially boost attorney productivity such as ChatGPT, Anthropic\u2019s Claude, and Google\u2019s NotebookLM.[1]<\/span><\/div>
<\/g><\/svg><\/div>The MMLU benchmark is a standard test for evaluating language models. It consists of thousands of multiple-choice questions. Language model scores are the percentage of questions answered correctly.<\/span><\/div>
<\/g><\/svg><\/div>The published public MMLU scores are not very useful for evaluating which A.I. language models to use for legal reasoning tasks.<\/span><\/div>
<\/g><\/svg><\/div>Here I privately test several language models on the \u2018professional law\u2019 subset of MMLU questions.<\/span><\/div>
<\/g><\/svg><\/div>These private test scores show that the tested models perform between 5% - 21% worse on legal reasoning than the published public score.<\/span><\/div>
<\/g><\/svg><\/div>OpenAI\u2019s \u201co1\u201d models performed the best and provided the correct answer 80% of the time.<\/span><\/div><\/div><\/div>\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t

Background<\/h2>\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t

Recently A.I. systems have gained the ability to process natural language in a way that can substantially increase attorney productivity. At the core of these systems are language models.[1]<\/a> Several new language models are released every week. Companies usually evaluate the quality of these models using standard benchmark tests. There are hundreds of these tests, but this brief focuses on just one of them – the MMLU (Massive Multitask Language Understanding) benchmark.<\/p>

Developed in 2021, MMLU tests models with multiple-choice questions spanning 57 subjects\u2014from elementary mathematics to professional fields like law and medicine. While newer benchmarks now better assess complex reasoning capabilities, MMLU remains valuable for the breadth subjects it tests and its specific legal subcategory which consists of over 1,700 questions. For legal professionals, MMLU’s professional law subset offers insights into how well these models understand legal concepts and reasoning.<\/p>

Because I regularly use private versions of these language models to assist in some of my legal work, I needed a better understanding of how these models are evaluated. So, I began to write my own benchmarking scripts to test available models for legal reasoning ability.<\/p>

[1]<\/a> Often called large language models (LLMs), though this term is becoming outdated as some effective models are smaller and many now handle multiple types of data beyond text.<\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t

Private Benchmark Test Results<\/h2>\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t

(MMLU Professional Law Question Subset)<\/h4>\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t

Below are both the publicly published MMLU benchmark scores and the scores I got from my private tests. Because of time and budget constraints, I could not test each model against the entire 1,700 set of questions in the professional law dataset. My private results are just average results from testing models on 25 questions at a time.<\/p>

The table below illustrates the significant discrepancy between published scores and actual performance on legal questions:<\/p>\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t

\n\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t