The reason why 'Litestream' was born, which simplifies 'too complicated server configuration' to the origin



In modern systems, it has become commonplace to separate services across multiple servers from the perspective of redundancy and load distribution. However, system configurations can be unnecessarily complex, such as cache and queue abuse. To simplify the 'complex server configuration', ' Litestream ' was created by

Ben Johnson , the developer of Bolt , known as Go's local database.

Why I Built Litestream --Litestream
https://litestream.io/blog/why-i-built-litestream/

Decades ago, when Johnson said, 'It's a good old era, but it's actually the worst,' it works for any development site with a single programming language and knowledge of SQL, and all websites are basic. It was an era when it was created with typical HTML technology. In the latter half of the 1990s, the so-called ' LAMP ' configuration, which combines 'Linux', 'Apache', 'MySQL', and 'PHP', appeared. LAMP became an iron plate configuration in website development, but for the CPU at that time, the LAMP configuration was a little heavy load. In addition, PHP and Ruby, which were the mainstream languages in web development, were slow to execute, and database servers such as MySQL and PostgreSQL tended to be complicated to manage, so all systems were integrated into one server. There was a limit to how to pack.



For this reason, Johnson explains that a configuration that physically separates the database server from the application server has become mainstream. However, the results began to take more than one server configuration, in-memory cache of

Redis and the connection pool PgBouncer , distributed message queue Kafka , such as the server is born 'additional layers' of because there are more than one, configuration However, it became more and more complicated.

Developed by Google, Kubernetes enables automatic construction of complex systems, zero downtime service updates, etc., and is used in many services . However, Johnson points out that the 'complexity of Kubernetes' itself impairs availability. Kubernetes is a great system, but there are other indications that deploying it is not always the optimal solution.

Kubernetes, quit | Gaido Father's Takumi
http://blog.father.gedow.net/2020/06/03/goodbye-kubernetes/

Against this background, Johnson turned to SQLite to simplify application development. SQLite is a lightweight database management system that does not function as a server like MySQL and PostgreSQL, but is characterized by being embedded in an application. It is often thought that 'SQLite is not used in the production environment', but Johnson exemplifies that SQLite has the ability to process a large number of queries and that there are products that use SQLite, and its high functionality I'm evaluating.

However, the main reason why SQLite is not used in production is that it has no replication function. Replication is possible using rqlite , but you need to prepare three servers. ' Litestream ' was developed by Mr. Johnson with the desire to 'replicate SQLite more easily'.

Litestream is software that can replicate SQLite to Amazon S3 , and it is very easy to set up and can be used at low cost. Litestream itself is free open source software, so the only cost required is an S3 usage fee of about $ 1 (about 100 yen) per month. In addition, modern cloud providers have multiple layers of redundancy, so if you use the virtual machines of those providers as servers, you can obtain extremely high availability. The replication destination S3 also claims 99.99% availability and 99.999999999% durability.



Regarding the scaling of the configuration using Litestream, it is said that the first

scale-up is to improve the processing capacity of the server itself, such as replacing it with a high-performance CPU and increasing RAM. Even if you no longer can handle in just one server, for SQLite and Litestream is easy to deploy, data sharding to increase the server scale-out carried out, and it is possible to improve the processing capacity of the entire system Johnson explains

Mr. Johnson plans to add a read replica function to Litestream in the future. The specific installation method of Litestream is published on the following page.

Getting Started --Litestream
https://litestream.io/getting-started/

in Software, Posted by darkhorse_log