What is revolutionary about the interactive AI ``ChatGPT'' that allows conversation with human-level accuracy?
by
OpenAI's interactive AI, ChatGPT, is based on a derivative of the large-scale language model GPT-3.5, and it is possible to interact with natural sentences as if they were written by humans. John Stokes explains what makes ChatGPT groundbreaking.
ChatGPT Explained: A Normie's Guide To How It Works
https://www.jonstokes.com/p/chatgpt-explained-a-guide-for-normies
There are two types of processing processes: ``determinism'', which always produces the same output when a specific input is given, and ``probability theory,'' where the output changes according to probability when a specific input is given. According to Mr. Stokes, the epoch-making part of ChatGPT is that all the parts that were previously deterministic are now probabilistic.
For example, if you put 100 yen into a capsule toy machine and turn the handle, a capsule will always come out. Capsule toy machines can be said to be deterministic in the sense that an output of 'capsule' can be obtained for an input of '100 yen'.
by
However, the contents of the essential capsule are not the same. If you have a capsule toy with 10 types of figures, assuming that the same number of each type is included, the probability of getting the figure you want is 1/10. In this case, we can interpret the capsule toy machine as probabilistic, considering that there is a possibility that the output of 'the contents of the capsule' may differ from the constant input of '100 yen'.
by
ChatGPT is the same as this capsule toy, selecting and outputting words based on probability processing for the input text (prompt).
Stokes compares ChatGPT to the orbital of electrons in atoms. The figure below visualizes the electrons of a hydrogen atom, indicating that the darker the color, the higher the probability that electrons are present. In junior high school science, we are taught the atomic model that ``electrons are spinning concentrically around the nucleus. I can only explain that it exists. In other words, this electron trajectory diagram only shows the probability distribution.
Stokes said that ChatGPT is like a ``probability distribution of words'' like this electron trajectory. What is ChatGPT's language model? A set of characters called 'tokens' exists stochastically, from a list of characters that do not make sense to passages of Shakespeare. Then, when you enter a word in the input field of ChatGPT, sentences are assembled by ranking based on the relevance and probability of tokens.
Technically what kind of probability processing is being done is summarized in the following article.
What does ChatGPT do and why does it work? A theoretical physicist explains - GIGAZINE
For example, Stable Diffusion, an image generation AI, generates the same image every time if you enter the same prompt and the same seed value and generate with the same settings. Given a specific prompt and seed value to Stable Diffusion, it picks up a specific point in the 'latent space' where the learned features are distributed and produces the same output.
On the other hand, ChatGPT creates sentences by probabilistic processing. Normally, it is set to select the 'highest ranked' token, but ChatGPT is daringly set to select tokens with low rankings. Furthermore, according to Mr. Stokes, it seems that the seed value in Stable Diffusion is forcibly changed each time the prompt is entered. Therefore, randomness is born in the output result.
Also, since ChatGPT is an AI that performs 'conversation', it is designed to acquire and use not only the prompt that was entered immediately before, but also the entire conversation history up to that point. Therefore, the more you talk with ChatGPT, the larger the input prompt, the more output, and the more accurate the answer.
In addition, ChatGPT does not understand the language like humans do, but it creates sentences by stochastically arranging characters. Therefore, even if the sentence output by ChatGPT after probabilistic processing looks natural as if it was written by a human, the content may be messy or contradictory. At such times, ChatGPT is described as 'hallucinating'.
Related Posts:
in Software, Posted by log1i_yk