Jan 31, 2025 10:40:00

Japan-based Sakana AI releases 'TinySwallow-1.5B', a Japanese AI model that also works on iPhones, distilling knowledge from large-scale language models to create small, high-performance AI models

Sakana AI, a Japan-based AI company, announced the small-scale Japanese language model ' TinySwallow-1.5B ' on Thursday, January 30, 2025. TinySwallow-1.5B was developed using the knowledge distillation method '

Temporarily Adaptive Interpolated Distillation (TAID) ' to realize small-scale, high-performance AI models, and can also be run locally on the iPhone 14.

Release of 'TinySwallow-1.5B,' a small-scale Japanese language model using the new 'TAID' method
https://sakana.ai/taid-jp/

TinySwallow-1.5B is a small-scale language model with 1.5 billion parameters developed by knowledge distillation from a large-scale language model with 32 billion parameters. A newly developed method called TAID is used for knowledge distillation, and in benchmark tests evaluating Japanese performance, it recorded the highest performance among language models of the same size.

TinySwallow-1.5B is said to be lightweight enough to run smoothly on PCs and smartphones. Sakana AI has actually released a video of TinySwallow-1.5B running on the iPhone 14.

Sakana AI's small-scale Japanese language model 'TinySwallow-1.5B' running locally on iPhone 14 - YouTube


Sakana AI has also released a web app called 'TinySwallow-1.5B Chat' that allows users to download and run TinySwallow-1.5B on their PC.

TinySwallow-1.5B Chat
https://pub.sakana.ai/tinyswallow/

The results of actually running TinySwallow-1.5B Chat on a laptop are below. In response to the question 'Tell me how to eat tuna other than sashimi,' it suggested 'sushi,' 'teriyaki,' 'fried food,' and other dishes.

On the other hand, it also output names of dishes that seem unlikely to exist, such as 'Saba Saba Tataki' (seared saba) and 'Nekori Cutlets.'

The model data for TinySwallow-1.5B is available at the following link.

TinySwallow - a SakanaAI Collection
https://huggingface.co/collections/SakanaAI/tinyswallow-676cf5e57fff9075b5ddb7ec

You can also find instructions on how to run TinySwallow-1.5B on an iPhone at the link below.

Using TinySwallow on iPhone · SakanaAI/TinySwallow-ChatUI · GitHub
https://github.com/SakanaAI/TinySwallow-ChatUI/blob/main/docs/iphone.md

The TAID distillation method used in the development of TinySwallow-1.5B adopts an approach of 'gradually changing the teacher model according to the learning progress of the student model.' Sakana AI plans to continue verifying TAID under various models and environments.