Artificial intelligence isn’t coming for a defense lawyer’s job anytime soon, but a new study suggests that these models can grade law school exams quite similarly to a real law school professor.
The study examined how large language model (LLM) AI programs graded law school exams and compared the results to the grades given when a real law school professor reviewed the test. The main conclusion from the study was that “when supplied with grading rubrics, [AI] produced scores that closely tracked human graders with correlations up to 0.93”
“Our analysis suggests that LLMs have the capacity to roughly approximate the grading of a law professor, with the best results coming from prompts that incorporate detailed rubrics.”
AI Law Exam Grading
The study was conducted by researchers right here in our own backyard, with Daniel Schwarcz, a professor at the University of Minnesota, helping lead the charge. He believes that while these models may not fully replace human grading, they certainly could make it easier for prospective students to receive valid feedback without having to have their practice exams reviewed by a real person.
“Even if AI never replaces human grading, these tools already appear capable of offering meaningful feedback on practice exams, midterms, draft memos, and other legal writing exercises that students often struggle to get reviewed,” said Schwarcz. “For young associates, the same techniques point toward new ways to refine written advocacy before it ever reaches a partner’s desk.”
Not surprisingly, researchers said the AI models provided more similar grading feedback when they were given more detailed grading rubrics to use for their analysis. The more information and specificity that can be given to these models, the more accurate the results they tend to produce.
Researchers believe that LLMs have the ability to give law school students quicker feedback on practice exams and prompts, while they also believe that law school professors may find the tool helpful for reviewing their own grading for errors or any potential bias.
AI has certainly entered the academic ranks, but it won’t replace the need for lawyers who truly understand the law and how to practice it in the courtroom. Students and professors may want to lean on these models in some situations, but you can’t turn to AI in the courtroom for answers, so at the end of the day you need to ensure you truly understand the material. AI can help you better understand, but students and professors should be wary of becoming over reliant on the technology, because if all you know how to do is put information into AI and regurgitate its results, you’re replaceable.
If you want a real person to take your case and fight for your rights in court, connect with Avery and the team at Appelman Law Firm today at (952) 224-2277.
