ML Pipelines in Production

Google's Play store recommends apps with models built on TFX, the production machine learning platform Google described at KDD in 2017. One TFX component computes summary statistics for every feature, both over the training data and over the logs of requests the serving system actually received. When the Play team compared the two for the same day, they found a few features that were present in every training example and missing from every serving request. The model had learned to rely on information that production never gave it. Removing the discrepancy raised the app install rate on the store's main landing page by 2% in an online A/B experiment.