Nov 12, 2022 10:00:00

``Tokenizer'' that shows what kind of tokens prompts and spells for image generation AI are actually transmitted as tokens

In recent years, there has been growing interest

in AI that generates images simply by entering text (prompts), and an environment is being developed in which anyone can easily use image generation AI. But it's hard to understand how the AI actually loads the prompts. NovelAI , which provides an image generation AI service, has released a web application `` Tokenizer '' that allows you to see how prompts are entered into AI.

Tokenizer - Novel AI
https://novelai.net/tokenizer

Natural Language Processing and Transformers Learned in the Hugging Face Course [part5]
https://zenn.dev/ttya16/articles/0e3e1bff645f161fb4d7

Natural language processing (NLP) is required to input prompts to image generation AI. Natural language processing is a technology that extracts content by processing natural language used by humans on a computer. In order to perform natural language processing, it is necessary to first decompose a sentence into words called ' tokens ', perform ' tokenize ' by assigning an ID to each token, and convert it into an input format that can be processed by a computer. The program that visualizes it is 'Tokenizer'.

When you access 'Tokenizer', it looks like this. First, select the natural language processing model you want to use from the pull-down tab. I chose the default ' GPT-2 Tokenizer' this time.

Enter the text you want to output in the 'Text Input' field in English. This time, try entering the text 'I have a pen'.

Then, 'I have a pen' color-coded in the 'Text' column below was output.

Next, when I selected the 'Token IDs' tab, four numbers [40, 423, 257, 3112] were displayed. A token assigned an ID in this process is output. This is the “tokenize” process. By performing this step, you can add tokens to the input text that provide the information needed to input the model.

Next, enter the number output earlier in the 'ID Input' tab next to 'Text Input' above.

And by selecting the 'Text' tab below, you can output the original 'I have a pen'.

Natural language processing is used in all aspects of daily life, and representative examples include search engines such as Google and AI assistants such as Siri and Alexa. These perform searches by natural language processing of input sentences and voices.