Oct 04, 2023 21:00:00

GPT-4 and Claude 2 can translate up to ``Latin magic books written in the 16th century,'' and scholars say they are ``comparable to human professionals.''

AI support is about to bring about major changes in academic research, as some researchers say,

``ChatGPT has reduced the time it takes to read papers by 99% .'' Benjamin Breen, a historian at the University of California, Santa Cruz, uses OpenAI's large-scale language model (LLM) GPT-4 and Anthropic's Claude 2 to translate a 16th century Latin text. , reported that results with surprisingly high accuracy were obtained.

Translating Latin demonology manuals with GPT-4 and Claude
https://resobscura.substack.com/p/translating-latin-demonology-manuals

In an attempt to utilize LLM's support for research, Mr. Breen focused on the following three functions of GPT-4 and Claude 2.

・The ability to infer the content of incomplete original texts, such as sentences with garbled characters obtained by OCR of pre-modern books.
・Unlike dedicated translation machines like Google Translate, it has a wide range of learning data, including the historical background in which the book was written.
-Ability to summarize and analyze.

In particular, historians like Mr. Breen have to look through history books in various languages, so the ability of AI to extract items related to the research topic from a huge amount of material is extremely important. It will be something.

◆Test 1: Latin translation
In order to test the abilities of AI, Mr. Breen decided to have GPT-4 and Claude 2 translate the demonology book ``Magical Investigations (original title: Disquisitionum Magicarum Libri Sex)'' published in 1599. The author of this book, Martín Del Rio, was a Jesuit monk with deep knowledge of witchcraft and demonic summoning, and wrote ``Magical Investigations'' as a guide to countering these black magics. That's what he said.

For the test, Mr. Breen input the original text, which had been converted into text with omissions and garbled characters using Google Books' OCR function, without any modification, into the AI, and instructed it to translate the content in as much detail as possible. The prompts Breen used can be found at

this link .

As a result, GPT-4, which was used through Claude 2 and Bing Chat's creative mode, was an inferior resource for Mr. Breen's understanding of the '

Noonday Demon' mentioned in Psalm 91:6 in the Bible. We were able to show you the most accurate translation results ever.

Although the results of GPT-4 had some difficulty in writing about Hebrew, Mr. Breen praised the results of Claude 2, saying, ``It seems to be the best translation of all.''

◆Test 2: Latin Summary
Below is the result of asking Claude 2 to tabulate the demon types, characteristics, and pages mentioned. Claude 2, which supports a context window of 100,000 tokens, also showed excellent results in summarization.

When Mr. Breen checked the original text, he found a proper description of the devil on the page written on the table. There were some pages that were incorrect, but Mr. Breen believes that this is a mistake caused by smudged page number printing or typographical errors.

Based on these results, Breen said, ``I think charts and summaries like this will be a game changer for people conducting research in multiple languages.This does not mean that AI can replace researchers. By having them play a role similar to language research assistants, they can gain a starting point for research.'

◆Test 3: Translation of Portuguese medical books
Tired of stories about the devil, Breen decided to focus on the books of João Curvo Semedo, a Portuguese physician active in the 17th and 18th centuries. When Curvo Semedo translated a sentence from a medical book on case studies written in 1707 into GPT-4, it was discovered that Curvo Semedo correctly recognized the toxicity of mercury, which was unusual among medical practitioners at the time. did. The prompts I used can be found at

this link .

Although it is widely known in modern times that mercury is harmful, until the 20th century it was used as a treatment for syphilis and depression, and it is famous that Abraham Lincoln also took mercury as a medicine. I am. Breen chose Curvo Semedo's book without much thought, but thanks to GPT-4 he was able to make a historically important discovery.

As mentioned above, GPT-4 was a little inaccurate at translating Latin, but Breen said it was fluent in early modern Portuguese and could rival human experts, especially when it came to translating medical texts. I am evaluating it. GPT-4 was also able to list the therapeutic drugs mentioned by Curvo Semedo.

Commenting on the results of the three tests, Breen said, ``We found that the translation and analysis of primary sources using LLM can be a very useful tool for historical researchers and translators. It is not a substitute for.”