Think back to the last time an algorithm recommended something to you. Was it on social media? Streaming entertainment? Online checkout? Was the recommendation interesting to you? That last question is hard for engineers and researchers to answer, and yet it is essential for us in order to deliver products you enjoy. Fortunately, Adobe Analytics engineers have created a new tool to help us figure out how relevant our model predictions are in the wild.

Machine learning model development is an inherently iterative process. The researcher typically builds a model, then tests it, then refines it, then tests it again. It often takes years of development before a model is ready for production, and only then would the customers see the results on their own data and be able to provide relevant feedback. A few years ago, we saw a way to get better models out faster: let the users in on what we were working on early. We built Analytics Labs to do just that.

What is Analytics Labs?

Analytics Labs provides a common pattern for researchers or engineers to deploy models they are working on to our customers. It provides an opinionated way of model deployment with built-in guardrails to ensure high-quality experience for all our users.

Analytics Labs has five main components:

– Labs Portal is an internal service that allows a researcher to navigate a simple-to-use UI to get started on “productizing” their model. One of the artifacts the portal produces is an intuitive template code in an internal GitHub repository.

– Templates come in several languages. These are self-contained Docker containers missing only the model that the researcher wants to try out. The template comes with a library capable of accessing various data sources within Adobe. All that is left to do is implement a few simple functions like validate, train, and score. (NOTE: most of our models to date have been trained offline).

– Labs Service is a Java DropWizard microservice that takes care of tasks common to all models (authentication, permissions checks, container management, etc.).

– Worker Container is generated from the filled-out template produced by Labs Portal through our Jenkins deployment pipeline.

– Clickable Prototypes is a fork of our production UI with the visualization for the feature we are trying to test. Note that our user selects only one model at a time. Unlike some implementations by other companies, we felt strongly that the best feedback comes from testing one thing at a time.

When the user leaves the prototype, he is prompted to give freeform feedback and to answer some questions curated by product management.

All the usage data is collected by Adobe Analytics and is made available to researchers, engineers, and product managers using industry-leading tools.

We have seen the time between model inception and first contact with users shrink from months to weeks. Several Analytics Labs models are now in full production in Adobe Analytics, but more importantly, we have been able to get enough data to kill some models early before investing a lot of time. Our customers tell us they love it, both because they see the latest and greatest and because it helps us stay more aligned with their needs.

The back end of Analytics Labs uses Adobe Ethos, which provides us access to a Kubernetes cluster while greatly reducing the headaches of maintaining one. Labs Service is a Kubernetes service that talks to the UI through a standard REST protocol.

Initially, Worker Container was designed as a Kubernetes service to support real-time scoring. A worker receives the job payload over REST from Labs Service. Why REST and not something like gRPC? We found that it made it much simpler for researchers to test locally. Today most of our models do not fit the real-time scoring paradigm, so we have created another version of Worker Container that runs as a Kubernetes job. Upon creation, the job receives some information (including job id) from Labs Service. It then retrieves the payload from MongoDB and writes the results back to MongoDB when done. The schema flexibility and scalability of MongoDB make it the perfect fit for this use case.

The Clickable Prototypes component is a static React app served from CDN. Internally, Labs Portal is also a React App primarily responsible for CRUD operations on metadata as well as template creation.