Summary of what we learned from analyzing 150 successful machine learning models, Booking.com
by
The booking site Booking.com incorporates machine learning into the platform and conducts a number of A / B tests every day to find the best options and display methods. Booking.com, who has experienced both failure and success many times in machine learning, published a paper about what they learned from 150 success models. Adrian Colyer , a Venture Partner of Accel after serving as CTO of VMware , explains this paper, which describes how machine learning affects the performance of real companies and platforms. .
150 Successful Machine Learning Models: 6 Lessons Learned at Booking.com
(PDF file) http://delivery.acm.org/10.1145/3340000/3330744/p1743-bernardi.pdf
150 successful machine learning models: 6 lessons learned at Booking.com – the morning paper
https://blog.acolyer.org/2019/10/07/150-successful-machine-learning-models/
It's not a big deal if “Recommendation” doesn't meet the user ’s hobbies if it ’s a platform that provides music, but in the field of overnight stays where the price is never cheap, the accuracy of “Recommendation” Will be very important. So, after many successes and failures, Booking.com analyzed 150 customer-facing apps that were successfully developed and used, and published what they learned from it.
Booking.com doesn't have a bulleted list of '6 lessons' in the dissertation, but Colyer summarizes the following six.
1: Projects with machine learning models bring significant business value
2: Machine learning model performance is not the same as business performance
3: It is important to clarify the problem to be solved
4: Prediction introduces delay
5: Get quick feedback on model quality
6: Measure the business impact of using a model using a randomized controlled trial
The machine learning models used by Booking.com can be broadly divided into the following six categories.
A model that predicts a wide range of traveler preferences
・ A model to predict the context of travel
・ Model that tracks user's view from history
・ Model to optimize UI
・ A model that collects content including reviews and decides which to display
・ A model for computing price and option trends
The lessons that Booking.com has learned from these 6 patterns of machine learning models are as follows.
1: Projects with machine learning models bring significant business value
According to Booking.com, “successful projects that used machine learning” tended to generate higher profits than “successful projects that did not use machine learning”. Also, it seems that machine learning-based projects were often the basis for subsequent product development.
2: Machine learning model performance is not the same as business performance
Booking.com uses a randomized controlled trial to analyze the value of what is provided by a machine learning model. As a result, we found that improving model performance does not necessarily increase business value. There are various reasons for this, including the fact that the value of the business is already saturated, the number of users who are provided in the first place is few, and the
In the image below, a Booking.com user asked, “I just entered Salzburg and London, but why does Booking.com know that I ’m going to Vienna?” You can see that This is an example of an eerie valley phenomenon.
3: It is important to clarify the problem to be solved
When creating a machine learning model, it is very important to clarify the problem. It seems that the performance improvement can be brought about not by improving the model but by changing the setting itself.
4: Prediction introduces delay
Resources are needed when machine learning makes predictions. An experiment introducing 'artificial latency' at Booking.com showed that if the latency increased by 30%, the conversion rate dropped by 0.5%. Booking.com viewed this as a related cost , but minimized the length of the latency by using a model with fewer parameters and an in-house linear prediction engine.
5: Get quick feedback on model quality
by Nick Fewings
Whether the machine learning model is responding correctly to the request should be verified by monitoring the quality of the output. But for Booking.com, this is very difficult. This is because in the model that predicts whether a user will make a “special request”, the prediction is made when the user is looking for an inn, but the actual “special request” is made This is at the time of reservation and there is a time gap. Similarly, for models that predict whether a user will review, the predictions and answers may take days or weeks.
In this regard, Booking.com observes the distribution of responses output by the model in binomial classification and determines that a distribution with one stable fixed point and two elevations is a sign of a model that distinguishes two classes well. I was doing it. “Response distribution analysis is a useful tool for quickly detecting model defects,” Booking.com says.
6: Measure the business impact of using a model using a randomized controlled trial
Many of the successful machine learning models were based on advanced experimental design, says Booking.com.
The paper includes proposals for several types of randomized controlled trials. In a randomized controlled trial, the control group has access to the unchanged platform, and only the experimental group has access to the modified platform. For example, if it is not appropriate to expose changes to all users, create a control group and an experimental group from eligible subsets.
There are also a control group where no model is assigned, an experiment group 1 that changes when the model is assigned and meets certain criteria, and an experiment group 2 that changes when the model is assigned but does not meet the criteria and does not change .
When comparing two models, the experiment is divided into three groups: Control Group, Experiment Group 1, and Experiment Group 2.
Many of the papers on machine learning are mathematical and algorithmic, and few discuss the impact on commercial use. The book on Booking.com is very valuable in that it describes the impact of machine learning models on platforms and industrial environments, Adrian Colyer said.
Related Posts:
in Web Service, Posted by darkhorse_log