What kind of server configuration does Dragon Quest X make to achieve "one world"?

Square Enix's popular RPG "Dragon QuestThe latest work "Dragon Quest X (Dragon Quest 10)" in the series became the first online work of the series, but what about the behind the scenes? Tomohiro Moriyama, senior programmer, Square Enix Development Department, talks about how the server system that supports the world view of the game is configured and how the structure and functions of Dragon Quest X are to develop difficulties .

Title | CEDEC 2012 | Computer Entertainment Development Developers Conference
http://cedec.cesa.or.jp/2012/program/NW/C12_P0040.html

Tomoki Moriyama:
Everyone, thank you very much for your time today. I will be in charge of this session,square EnixI am Tomohiro Moriyama of the development department. Thank you very much. todayDragon Quest XAlthough it is the title of saying, although the theme is rather server-side, maniac theme if it is rather theme, I assumed about 30 people as image training intention, but what about what people do gather and this I do not mind, but I will do my best so I will do my best.


Well today's session, I will proceed with this flow. First of all, it is about the outline of this session. Although it is said that behind the scenes of Dragon Quest X, after explaining briefly what kind of content it is specifically, how the server's design and development actually progressed I would like to introduce you to that. Also, since the service has already begun anyhow, we are introducing to what extent the currently running server is in a situation not to get angry, let's go ahead with a summary I would like to receive it.


Before entering the main subject, I will introduce myself only briefly. I worked at SQUARE ENIX in 2006 and was assigned to the Mobile Division at that time, after developing a mobile related web as a server programmer, experienced a wide range of billing systems and game servers , I join the development team of Dragon Quest X now. I first found a job after graduating from collegeJapan OracleIt is also a database company that says that it has been involved in the design and construction of large and small databases so far, and currently Square Enix is ​​also responsible for the work around the database at present is. It is occasionally our different title in that trendFINAL FANTASY XIVI am also doing a little help, but I am in charge of backend development of Dragon Quest X.


Well, I'd like to move on to an overview of the session today. I used to use the word "back end" casually, but in reality, I think that there are also some people who do not seem to know the image of what it is still a bit of a thing alone, so first of all Dragon Quest X After introducing the basic server configuration, I would like to explain in detail what is the backend that is the theme of today.


There are servers in Dragon Quest X, can be roughly divided into 4 groups. There are various ways to divide this wayMMOThere are, however, there is not much difference, so some people may be beaten as "it's normal whatsoever". I will explain in order, first of all it is written as No. 1Lobby serveris not it. Here, it is a server that performs authentication processing according to its name, although it may do various ways of saying "authentication server" or "login server" if it is another MMO. In the case of Dragon Quest X, it is where we create and delete what we call adventure books, and manage such accounts. Next is game number 2. It is exactly the server that moves monsters, controls events, and is in charge of the game itself. Actually, there are several kinds depending on the role of the game server, but since it is out of the subject of this session, here we are making it simple with the game server. Next, it's number three, behind the lobby or game serverBackendis not it. Today we are calling the server group consisting of database and cache as backend for convenience. I will introduce this in detail later on here. At the end is No. 4, which is written as Web. It is in Dragon Quest X, which is linked with data in the game "Wake up and adventurer's squareThere is a website called "Website" which refers to game data of other players by himself or to communicate with other players. We also implement various functions via the back end.


In the server configuration of Dragon Quest X, I mentioned that such classification itself is not special, but in reality the backend configuration has one big feature. I will introduce it from now on, it will become the main subject of today. Including other titles of our company, not limited to MMO, the configuration of a server which is often taken in many online games, we prepare multiple sets of independent servers in this way.


The game is called differently, but in many cases it is said to be a world or a server, and many servers and worlds are operated separately. As the work hits and a lot of customers came and increasing this server set more and more, I can expand the system with relatively low risk. For many people who are here today I think that it is a novelty story, for example, as recently as Amazon's net service, so-calledPublic cloudIt is very good compatibility with a service called so, I think in recent years it may be safe to say that it is a royal road as a server configuration of online games. Speaking of low risk and high scalability, it is a very excellent method. On the other hand, it is in Dragon Quest X, if you draw it with a single picture, it is structured like this.


You can increase the number of game servers according to the number of people, but there is actually only one database. All data is managed in one place in one place. The composition here introduced on the previous page is, there are many merits, but there is one disadvantage. Communication with one player's players in other world is very limited. Of course, in most cases, since that data is in a different database, that kind of limit will arise absolutely. However, in the Dragon Quest X, there was an idea from the beginning that "you can not play together the same game while playing the same game", it was said from the beginning that you make it with such a composition . If you say, "Everyone can play together," it sounds good to hear it, but in reality it is a recent trendScalabilityAndload distributionIt is a direction that goes 180 degrees opposite, so it was pretty fun actually, but it was that it was essential for servicing Dragon Quest X, so how did you realize this difficulty Whether it was the subject of this session. By the way it is a bit of a digression, but when I was told that this database is done with one piece, I was saying "It is serious, earnest" just after entering the development team of Dragon Quest X is. In fact, I knew that I would make that dragon 's online actually, but I do not know that it is a numbering title, "I wonder how long people will come, I wonder what I'm gonna do" I know that it is actually a numbering title just after that, there are memories of being shocked. At that time it was a situation I could not catch on drawing anymore, so design and development started in the situation that I have to do something.

The concept of "the world one" the previous time is easy to understand from the stage of thinking, but from the viewpoint of a programmer, it is equal to not having decided anything in fact that only a lot of load comes. With this alone we can not design or develop anything. Also a design drawingBlueprintSince it is said that it can not draw anything, I decided from what I could decide after hearing various things.

First of all I decided, but considering the ease of handling of the large amount of data handledRDBMSIt was decided from the place to store it.


For a while this RDBMS will be abbreviated as DB because it is long. The ease of handling data is not merely to save and load it. For example, say "I want to make sure the value here is unique all over the world", or when aggregating data or doing various investigations, say somethingSQLIt's very easy to use, user-friendly. Well, I chose to use the database because it seems that there is no choice of DB, considering the ease of backup operation as well. Considering our database development and operation know-how at that timeOracleIs adopted. Later, since the database has only one after allCluster configurationI decided to make it a bit more extensible, but it was still uneasy by itself, so it was decided until somehow how dangerous it would be if you do not use load balancing caching servers well It was.

This time it is a little more designed-oriented story, actually it was decided from the beginning that there are several programs that actually access this database.


What will be the mainC ++It is a game related program written in, but it is also the other web, this is written in Java, and in fact other tools like games and scripts, in fact, various games connect to the database. It is very verbose to implement some processing in all languages ​​at the same time at this time, so it was written in the SQL language on the Oracle sideAPITo implement the same process from any language, even if you do not know the implementation at all, you can call the same process. Especially when I was helping with FINAL FANTASY XIV, FINAL FANTASY XIVThe LodestoneThere is an interlocking website like this, and the creator here is sitting in the middle right now, but he was struggling a lot to access data in various games, so even with Dragon Quest X When doing something like this, we wanted to make it possible for various programs to easily access the data in the game as much as possible, so we advanced with such a design.

Another one. Actually, I have decided on an architectural point. Although this is said to be a database relay server, it is literally a server process for relaying between the main game server and the database. Why is this necessary? Actually, since the number of server processes of Dragon Quest X is several thousand, the number reaching 4 digits already moves, we will connect this all to Oracle at once I knew that Oracle's memory was gone by itself. So, the number of servers called database relay servers will be around 300 to 400, but it was decided to avoid having Oracle's memory exhausted by collecting database sessions. Actually, the database relay server will take on various other roles later, but I will explain it later. This picture is drawn with the game server viewpoint. There are a lot of DB relay servers, but the game server needs to select an appropriate DB relay server and connect it, even if one DB relay server falls down, since the game server gives up and connects to another place, This is supposed not to be noticed by the players just by dropping one or two.


Later, this is a rather functional story of the game, but as you progress through the hearing, the database is not just save and load, you have to do a lot of other things I understand to some extent what I can not do.


What I introduce here is the function to promote the interaction among various players implemented by Dragon Quest X. However, it is important to share these things, for example, by sharing data on the entire system of the world Or, because it is necessary to exchange data with the logged-out user, it was realistic to install the logic on the database side as well, so the database of Dragon Quest X is saved · Besides loading it contains various functions.

Well, until now, do you hear a variety of hearings and call it a rough backend, it was decided to be the specification that the DB should satisfy. I would like to focus on more concrete implementation from here. In the past we saved data or saved data in a single word, but in fact the saved data of the player is "What's going on?" It is said that it is not too much to tell this, but I will briefly introduce it. The actual state of saved data of Dragon Quest X is a structure consisting of a set of C ++ structures. If you save and load these structures in the database table, OK is OK, but in the case of Dragon Quest X, the size of this saved data structure is very large , Perhaps perhaps to hundreds of kilobytes. In addition, it is necessary to divide one player data into tens of tables on the table itself and save it. So if you say "You can not just map and save in the database", it is right, but if you implement this process straightforwardly it will be very heavy. Saving or loading such data is done not only at the beginning or the end of the game, but it is actually done quite frequently even during the game. For example, every time a player enters the town or enters the cave, or when moving the area, you save and load hundreds of kilobytes of data. If this place is heavy, the whole game becomes very heavy. Therefore, how to make the whole database work comfortably a bit more comfortably, I am trying to make the database easier to go a little more via the database relay server earlier. Although it is very simple as a way of thinking, it is often said that it is simple, save data, the structure of hundreds of kilobytes just before, can be serialized as it is and stored in the cache server as it is . I think that there are also many people who know the cache server used hereKyoto Tycoon, For serializing dataMessagePackIt is very fast data save and loading by using. With this rule, if you increase the number of Kyoto Tycoon more and more, you can save the load on the system as a whole, because it can make it clearer and more effective.


By delegating the majority of save and load operations to the cache, the database will be easier, but it will not be enough, and eventually you will need to save it to Oracle somewhere. The main timing written here is written when logging out, when data moves between players, and on a regular basis, but the most important thing is second.


When an item moves from one person to another, if only one person's data is saved correctly and one side rewinds, it is not very good that the item disappears or duplicates Because it becomes a situation, it is necessary to save the database without fail at such timing. Well, the first logout and the third regularly, so that there is no data that simply saved in the cache but not stored in the database. We save the database at such timing. At this point, I think many people think that "what you are doing is actually quite simple", you are right. However, in the case of Dragon Quest X, it is a point that it was good, I will explain easily from now on, but it is tuning. There is something that the tuning phase after this went very well if we talk about the backend.

Well, if you say what you were doing about tuning, here is it.


It is the verification client dedicated to load test here, which is the subject of backend tuning. In the development room "Botto" "Botto you" is called. This is a very high performance load verification client, from where you log in to the game and make a new character out of a character, you can control it with various scripts, move it, use items, look for an enemy You can simulate the movement of the player with considerable accuracy, by fighting. Since this is also a Linux program, we loaded loads of servers in huge numbers and hundreds of thousands of clients were put into the server, so we steadily began load verification. Actually, the server got filled up just by hundreds of entries at the beginning, but finding the cause, testing again, fixing again, testing and repeating it all at once By doing so, the number of people using the system increased to 1,000, 2,000, 5,000, and 10,000. Ultimately, at the present time, the backend has come to the point where hundreds of thousands of connections come to be judged without problems.

Since time has gone, I push it a little. In the load verification client, we will briefly introduce the place where we put the load verification client and tune it. If you are experienced in server development, I think that it is relatively general, but the first one is API call times. To put it another way, it means that you could discover early that why so many such APIs that you should not have been called so much are true.


Actually, I was quite awake by saying that the specifications were unchanged indeed. The second one is, is it also saying that it is a royal road a certain meaning in the tuning of the database, I think that it is sure to do. We were able to find things that are hard to find at the unit test level before catastrophe by doing massive load checks on a large scale. The end is, in fact, this was very important though. When saving data to the cache server, it says where the limit comes first when the number of connections increases, in fact the bandwidth of the network between the game server and the database relay server is lost first, It turned out to be exhausted. This is because it was caused by sending hundreds of kilobytes of data as it was, so unless it is not compressed, using the actual saved dataLZ4WhetherGoogle SnappyAs a result of various verification of compression libraries such as LZ4 which was compatible with save data of Dragon Quest X was adopted. By compressing it in this way, the saved data which was hundreds of kilobytes actually has been compressed from 1% to 5% so that you can efficiently use the network or cache. By the way, it seems likely to receive Tsukkomi why I did not do it from the beginning, but at that time the game server side was still full of performance. There was also that I could not afford to do extra processing, and I was postponed for a while. It is a place where it was better to have done earlier if it is over. And after doing such a load test, we started the service on August 2, saying "I can manage somehow."

Finally, probably everyone here comes to me because I think I'm a little interested in "What is actually going on now?"


First of all, although it is a database, the database of Dragon Quest X is now, tens of thousands of queries per second are flying, but it is the current situation that it is quietly discounting without problems. Although the response time of the query is almost ten minutes at the target value of 10 milliseconds or less, as the simulation of the load test showed a tendency of load, it seems that it was good to spare time on the load test It is place. Just a few, still unexpected things are also present. I briefly introduced "Travelers bazaarIt is trading of items called 'I'm sorry. As for the implementation of this search processing, it is only tuning with simple select processing and index, but since this calling line is called several dozens of expected times, this performance will be expanded further in the future I think that it is better to devise another idea because it is required.

Later, I think that it is not interesting just by saying "it is going well", so I will introduce that there was such a problem that this is not getting angry, too.


It is one piece, this is an example I got forced to do,DeadlockI have caused a database NG one time. When implementing the database, it is very carefully tuned that data in character units will not cause deadlock, but Bazaar's processing is based on the type of item, various kinds of data for each item I think that it is really less than a thousandth of a second that the process of selling an item and the process of buying that item by another person are truly less than a thousandth of a second, but there is a timing to become NG if they overlap . I could not detect this with a slight load test, but after the service started, it was August 7, I woke up once. It is now solved, but since performance problems have almost never come out when talking about the backend, this was a very frustrating incident for me personally.

Finally, I would like to briefly talk about the point that I felt important about the development of this Dragon Quest X that I met. The first one seems to be natural, but it was that "the load test is really important". Unfortunately, some troubles have already appeared as Dragon Quest X as a whole since we started the service, but again after looking at the cause, it was difficult to test the load, It is almost a place that has leaked out of the scenario. So, the load test is amazing. Perhaps, in my projects I have experienced up to now, this project was the first in a load test, but it was the first project, but there was one thing that did not match the outcome. And the other thing I'm afraid is that among the various members who were involved in the development, the goals and the vision sharing were clearly made. What does this mean? Dragon Quest X is online. Although I had anticipated that there were many negative reactions at the beginning, though I believe that the members involved in the above consideration will still be a game that many people can play with That means that. Even if it is time to decide on a scene, for example, even when thinking about the composition of a server, it is based on the fact that it is really okay to hundreds of thousands of people, I decided things As a result, I think that a relatively stable backend, server could be built.


Well, it's been about a long time, so the story from me will almost end. When I joined the project, the server architecture was decided and only the back end was left blank and it was a start from the state where both hands were tied up, but I really let you do it freely, very It was worthwhile. If you have the opportunity to play Dragon Quest X in the future, I would be pleased if you remind me of this session for a moment. Thank you for your kind attention today.