Research shows that AI can answer students' questions better than law professors, and in blind tests, AI was judged to be 'less likely to cause misunderstandings.'
A study led by a law professor at Stanford Law School showed that AI could answer student questions better than human law professors. Furthermore, the study reported that the AI's answers were judged to be misleading or harmful at a lower rate than those of human professors.
Law Professors Prefer AI Over Peer Answers - Journal Article - Stanford Law School
AI Outperforms Law Professors in Stanford Law Study - SLS News and Announcements - Stanford Law School
https://law.stanford.edu/press/ai-outperforms-law-professors-in-stanford-law-study/
Julian Nyarco , a law professor at Stanford Law School and head of the Legal Innovation and Frontier Technology Lab (liftlab), along with colleagues from other top universities such as Yale and New York University, investigated how well AI can answer legal questions from students.
The experiment conducted by Nyarko et al. involved 16 law professors from American law schools. The professors created 40 typical questions that students might ask during or after lectures on contract law , and each wrote down their own answers to those questions. The research team also had an AI generate answers to the same questions, and the professors were asked to evaluate the content of the answers without knowing which were human and which were AI-generated.
The research team adjusted the AI-generated responses to match the length and structure of human responses in order to ensure the validity of the experimental results. Nyarko stated, 'This research is extremely important, so we designed it to be as rigorous as possible.'
Previous research on AI has mainly focused on issues where the right or wrong answer is clear. However, in legal reasoning, it is necessary to carefully analyze conflicting arguments and draw reasonable conclusions. Nyarko explains, 'We focused on law because it requires not only the memorization of facts, but also judgment, nuanced reasoning, and the ability to overcome ambiguity.'
After professors evaluated a total of 2,918 responses, they found that they rated AI-generated responses significantly higher than those created by human law professors. AI-generated responses achieved an approximately 75% win rate in direct comparisons with human professors.
One of the most noteworthy points raised is the percentage of responses that were deemed 'educationally harmful.' While approximately 12% of responses from human professors were flagged as harmful, this figure dropped to a mere 3.5% for AI-generated responses.
Nyarko stated, 'These weren't all simple questions with obvious answers. Many required integrating complex content, applying it to new situations, and explaining legal concepts in a way that would allow students to develop their analytical skills. This research challenges important assumptions about the role of AI in legal education.'
These research findings offer important insights into the use of 'AI tutors' in legal education. While American law schools are grappling with the challenge of integrating AI tools into legal education, some institutions are cautious about introducing AI tools due to concerns about potential risks such as hallucinations, excessive reliance on AI, and a decline in critical thinking skills.
Alejandro Salinas, a researcher at liftlab, commented, 'Our research focuses on how AI-powered personalized tutoring can contribute to learning in judgment-based fields such as law. Evaluations by legal educators have shown that AI tutors can provide high-quality on-demand support that complements classroom instruction and have the potential to expand access to expert guidance.'
Nyarko stated, 'Legal education is about training future lawyers to think critically, develop persuasive arguments, and deal with ethically complex issues. This research is an important step in determining whether AI can help with that mission. We are not advocating for the full-scale implementation of AI tutors. However, our data suggests that being completely skeptical may be equally unfair. The focus of the discussion should shift from 'Can AI provide accurate and high-quality answers?' to 'How can AI be used responsibly for the benefit of students?''
Related Posts:
in AI, Posted by log1h_ik