# What is CaraML? CaraML is a **Machine Learning Operations (MLOps) platform** that helps data scientists focus on data science by abstracting the engineering complexities of building and running large scale Machine Learning applications. To access this documentation in GitBook, please follow this URL [CaraML Docs](https://caraml.gitbook.io/caraml-doc/) ## **Main components for CaraML:** ### **Models** Models component is a framework for serving machine learning models. The project was born of the belief that model deployment should be: * Easy and self-serve: Humans should not become the bottleneck for deploying models into production. * Scalable: Any model deployed should be able to handle large scale traffic * Fast: The framework should be able to let users iterate quickly. * Cost Efficient: It should provide all benefits above in a cost efficient manner. CaraML Models solves this by: * **Abstracting Infrastructure:** Models uses familiar concepts such as Project, Model, and Version as its core component and abstracts away complexity of deploying services from our users. * **Auto Scaling:** Models component is built on top KNative and KFServing to provide a production ready serverless solution which can automatically scale up and down to handle dynamic demand. CaraML Models project code name is Merlin, which may show in code, SDK, API documentations ### **Feature Store** CaraML Feature Store is an operational data system for managing and serving machine learning features to models in production. CaraML's Feature store is forked from the open source feature store [Feast](https://feast.dev/), and customised to be geared towards more production ready use cases. ### **Routers** CaraML routers is a fast, scalable and extensible system that can be used to design, deploy and evaluate ML experiments in production. Broadly, its capabilities can be divided into the following two areas that may be utilised in conjunction or separately: * **Experimentation** - Routers component supports designing and managing experiment configurations and running them, through its in-built experiment engine. * **Orchestration** - Routers supports deploying experiment workflows (through composable 'routers'). It is designed to work with pluggable pre- and post-processors and is backed by existing systems like CaraML Models for model endpoints. Routers takes care of all of the core Engineering aspects such as traffic routing, autoscaling, outcome logging, system monitoring and alerting. CaraML Models project code name is Turing, which may show in code, SDK, API documentations ### **Experiments** CaraML Experiments supports designing and managing experiment configurations in a safe and holistic manner. At run time, these configurations can be used (within the Router, or externally) to run the experiments and generate treatments. The experiments can be run either deterministically (A/B Experiments) or as a function of time (Switchback Experiments), or a combination of both (Randomized Switchbacks). ### **Pipelines** CaraML Pipelines are a set of solutions to build data application systems like ETL processes and ML pipelines. CaraML Pipelines is powered by [Flyte](https://docs.flyte.org/en/latest/), an open-source workflow automation platform to create concurrent, scalable, and maintainable workflows for machine learning and data processing. *** ### Guides: Jump right in Follow our handy guides to get started on the basics as quickly as possible: {% content-ref url="/pages/5530OXKdMs5hZ7oAnFiP" %} [Projects](/user-guides/projects) {% endcontent-ref %} {% content-ref url="/pages/bsCO0N1OLCU7oak7cECf" %} [Models](/user-guides/01_getting_started) {% endcontent-ref %} {% content-ref url="/pages/VUrly4nb2Wzc9vZeRYNl" %} [Feature Store](/user-guides/feature-store) {% endcontent-ref %} {% content-ref url="/pages/PXWG4hzrcaY4ZFW8z9bk" %} [Routers](/user-guides/router) {% endcontent-ref %} {% content-ref url="/pages/Bx57xydI8O2j3ABv0jQx" %} [Experiments](/user-guides/01_getting_started-1) {% endcontent-ref %} {% content-ref url="/pages/Lr8S13pJUkQTjjluEBfM" %} [Pipelines](/user-guides/pipelines) {% endcontent-ref %} ## Deploying CaraML in your infrastructure? Please refer to our deployment guide below to deploy CaraML into your own infrastructure. {% content-ref url="/pages/0hkFJ6SOMfdlBgLNX60Z" %} [Deploying CaraML](/deployment-guide/deploying-caraml) {% endcontent-ref %} # Architecture Please refer to the below diagram for a high level architecture of CaraML. You may find the component specific architecture in each of the subpages. ![CaraML Architecture](/files/Rb2Xd94COEZ2NVvOlYQk) {% content-ref url="/pages/BnWIVGHqK9issHcG0fIh" %} [Feature Store Architecture](/introduction/architecture/feature-store-architecture) {% endcontent-ref %} {% content-ref url="/pages/YQSz0MfwIMdDX9SUiQnv" %} [Models Architecture](/introduction/architecture/architecture) {% endcontent-ref %} {% content-ref url="/pages/nUzKnVkQYZ4Bwx3yFbEu" %} [Routers Architecture](/introduction/architecture/routers-architecture) {% endcontent-ref %} {% content-ref url="/pages/vkKxCLy6RF0K8yPqgFfe" %} [Experiments Architecture](/introduction/architecture/experiments-architecture) {% endcontent-ref %} {% content-ref url="/pages/Vq2eEp21NgoRRzrriErP" %} [Pipelines Architecture](/introduction/architecture/pipelines-architecture) {% endcontent-ref %} # Feature Store Architecture The overall view of the system architecture of CaraML Feature Store can be illustrated in the diagram below: ![Feature Store architecture](/files/no7HPD2xF4t7LmYGrMFR) You may refer to more details in the [repository](https://github.com/caraml-dev/caraml-store) over here. # Models Architecture The overall view of the system architecture of Merlin can be illustrated in the diagram below: ![architecture](/files/PO3nJcMVNuxAXivRYhlQ) ## Interacting with Merlin A few components in the design allow for users to interact with Merlin via different methods, namely: 1. Merlin API - This is the orchestrator for deploying and serving models. It is usually not accessed directly by users. 2. Merlin UI - This is the GUI layer working on top of Merlin API for interacting with Merlin graphically. 3. Merlin SDK - This is the python interface which provide users with all the functionalities they can perform on Merlin. ### Merlin Merlin API is the central component of the deployment and serving component of Machine Learning Platform. It plays the role of the orchestrator and integrates with 3rd-party components (MLflow Tracking, Kaniko, Istio, and KFServing). #### Merlin API Merlin API can be accessed via REST API. The most recent API methods and request/response schemas are exposed via Swagger UI. #### Merlin UI Merlin UI is a React application that acts as interface for users to interact with Merlin ecosystem graphically. However, logging of model is not possible via the UI. Users can, however, deploy and serve their models via the UI. The UI allows users to check on the health and logs generated by the model deployment conveniently and also offers a convenient way to generate and test configurations for standard transformers. #### Database Merlin API uses PostgreSQL as an underlying persistence layer for all the metadata regarding user's models, versions, deployed endpoints, etc. **DB Migrations** Merlin uses [golang-migrate/migrate](https://github.com/golang-migrate/migrate) to apply incremental migrations to the Merlin database. The big advantage of a golang-migrate is that it can read migration files from the remote sources (GCS, S3, Gitlab, and Github repositories etc), that simplifies a process of continuous delivery. ### Merlin SDK [Merlin SDK](https://pypi.org/project/merlin-sdk/) is a python library for interacting with Merlin. Data scientist can install merlin-sdk from Pypi and import it into their Python project or Jupyter notebook. It provides all the functionalites that users are allowed to perform in Merlin. Models can only be logged via the SDK. ### CaraML MLP The [CaraML MLP](https://github.com/caraml-dev/mlp) provides UI for the end-user and REST API for Merlin. It exposes a shared concepts such as Project, Secrets, and User Roles. ## Control Cluster The control cluster of Merlin is a kubernetes cluster managing the deployment and serving of models supplied by users. It contains pods of the Merlin UI, Merlin API, databases, kubernetes API server and Kaniko ### Control flow The following scenarios illustrates how the various components in the control cluster works together in different situations: #### Creating a new model version When user creates a new model version via the SDK, the model artifacts are managed by MLflow and stored in Google Cloud Storage. #### Deploying model version for standard model To deploy a model version of a standard model, merlin-api will send a request to the model cluster to create a new model deployment (i.e. a new KFService) #### Deploying model version for PyFunc model To deploy a model version of a PyFunc model, a custom docker image must be built and pushed before merlin-api send a KFService creation request. In such a case, merlin-api delegate the image building and pushing to Kaniko, which will build and push the image for the first time to Google Container Registry (GCR). The model artifacts are downloaded by the Docker image in order to be used. #### Deploying model version for custom model (e.g. GoFunc, JavaFunc) For deploying a model version of a custom model, user are expected to build their own Docker image and published it in their preferred container registry. Merlin will then pull the associated Docker image to run in the model cluster as a KFService. #### Serving a model To serve a model, merlin-api will send an update to the model cluster to update the route of the model endpoint. ### MLflow Tracking Merlin uses bundled [MLflow Tracking](https://www.mlflow.org/docs/latest/tracking.html) server for tracking the evolution of user's models, logging the parameters and metrics of trained models and storing the artifacts of the model training pipelines. However, Merlin is using a different form of MLflow terminology to describe the system's entities: | Merlin | MLflow | | ------- | ---------- | | project | -- | | model | experiment | | version | run | Since MLflow doesn't support a project-level aggregation of experiments, we use the project's name as a part of MLflow experiment's name: \/\. i.e., a model with a name `driver-allocation-1` from the project `driver-allocator` would correspond to the experiment `driver-allocator/driver-allocation-1` in MLflow. ## Model Cluster Model cluster is a target Kubernetes cluster where a model and batch prediction job will be deployed to. Model cluster will be different for each region and/or environment, and each model cluster will and must have Istio, Knative, KFServing, and Spark Operator installed in it. ### KFServing, Knative & Istio KFServing enables serverless inferencing on Kubernetes and delivers high performance and abstraction interfaces for machine learning frameworks like TensorFlow, XGBoost, scikit-learn, PyTorch out of the box. Model versions deployed are managed as a KFService in the architecture. #### Knative & Istio Knative and Istio provides out of the box performance metrics that include HTTP requests, resource usage (for nodes, pods, and deployments) when used with KFServing. Knative is also used to manage the routing of the traffics to the model versions deployed, while istio is used as a load balance and ingress gateway to handle the varied incoming request loads. ### Spark Operator The spark operator manages spark clusters for batch prediction jobs. ## Technical Stack The techical stacks used by Merlin includes the following: * golang * [gorilla/mux](https://github.com/gorilla/mux) - HTTP router * [jinzhu/gorm](http://github.com/jinzhu/gorm) - ORM / queries DSL for accessing data from the persistence layer * [go-playground/validator](https://github.com/go-playground/validator) - basic validation of the client inputs * [GoogleContainerTools/kaniko](https://github.com/GoogleContainerTools/kaniko) - build container images in Kubernetes * [k8s.io](http://k8s.io/api) - Kubernetes API and Golang Client * [kubeflow/kfserving](http://github.com/kubeflow/kfserving) - deploy ML models to Kubernetes * [GoogleCloudPlatform/spark-on-k8s-operator](https://github.com/caraml-dev/docs/blob/main/module/model/developer/github.com/GoogleCloudPlatform/spark-on-k8s-operator/README.md) - start Spark Application on Kubernetes # Routers Architecture The overall view of the system architecture of CaraML Traffic Routers can be illustrated in the diagram below: ![Routers architecture](/files/pU3qoOchEQXZa9b8cUZ8) You may refer to more details in the [repository](https://github.com/caraml-dev/turing) over here. # Experiments Architecture The overall view of the system architecture of CaraML Experiment engine can be illustrated in the diagram below: ![Experiments architecture](/files/Nekv7s7NaDyk0SRXVLnH) You may refer to more details in the [repository](https://github.com/caraml-dev/xp) over here. # Pipelines Architecture The overall view of the system architecture of CaraML Pipelines can be illustrated in the diagram below: ![Pipelines Architecture](/files/O8nxHyTNF2CdMjlbJD8C) # Core Concepts ## Project Project represents a namespace for a collection of CaraML resources, that belong to a specific team such as service accounts, Models, Routers, Pipelines etc. Project is one of the main building blocks for access control in CaraML. For creating a project, please refer to [Create a project](https://github.com/caraml-dev/docs/blob/main/module/router/how-to/create-project.md) ## Model This section describes the main concepts related to CaraML Models component. {% content-ref url="/pages/G0iX4liCkIkw1PibhF8c" %} [Models Concepts](/introduction/core-concepts/00_introduction) {% endcontent-ref %} ## Feature Store This section describes the main concepts related to CaraML Feature Store component. ### **Entities:** Entities are the objects in an organization on which features occur. They map to your business domain (users, products, transactions, locations). ### **Feature Tables:** Defines a group of features that occur on a specific entity. ### **Features:** Individual features within a feature table. ## Router This section describes the main concepts related to CaraML Routers component. {% content-ref url="/pages/6S81dR2g12BcHtlFNHdz" %} [Router Concepts](/introduction/core-concepts/concepts) {% endcontent-ref %} ## Experiment This section describes the main concepts related to CaraML Experiments component. {% content-ref url="/pages/gSWUgOwVHj2WCTc0RN4J" %} [Experiment Concepts](/introduction/core-concepts/concepts-1) {% endcontent-ref %} ## Pipelines This section describes the main concepts related to CaraML Pipelines component. ### Tasks You may refer to the definition of tasks in Flyte's documentation [here](https://docs.flyte.org/projects/cookbook/en/latest/auto/core/flyte_basics/task.html#sphx-glr-auto-core-flyte-basics-task-py). ### Workflows You may refer to the definition of workflows in Flyte's documentation [here](https://docs.flyte.org/projects/cookbook/en/latest/auto/core/flyte_basics/basic_workflow.html#sphx-glr-auto-core-flyte-basics-basic-workflow-py). ### Launch Plans You may refer to the definition of launch plans in Flyte's documentation [here](https://docs.flyte.org/projects/cookbook/en/latest/auto/core/flyte_basics/lp.html#sphx-glr-auto-core-flyte-basics-lp-py). # Models Concepts After you have built a model with high-quality training data and the perfect algorithm, it’s time to apply it to make predictions and serve the outcome for future decision making. For many data scientists, model training can be done easily within their Jupyter notebook. However, things become trickier when it comes to productionizing the model to serve real traffic, which is engineering intensive. There are many tools available, but learning when and how to use them requires a lot of exploration, which can be a headache. ## What is Merlin Merlin is a platform designed to help users productionize their models quickly without deep knowledge on MLOps. Users only need to deploy their model into Merlin, and it will take care of the traffic routing and resources scaling in the background, saving lots of engineering hours and expertise required otherwise. ## User Flows Productionizing a model with Merlin can be easily done in 3 steps, as detailed in the diagram below: ![User Flow](/files/NxXY3Ldf9r8ULTx6JUTT) 1. **Deploy a model** We want to make the deployment experience as seamless as possible, directly from Jupyter notebook. With the Merlin SDK, we can now upload the model and trigger the deployment pipeline, by simply calling a few functions in the notebook. Alternatively, Merlin UI supports the same, with just 1 click. 2. **Setup serving endpoint** Once the model is deployed with an auto-generated HTTP endpoint, you can then specify the serving model version in the console. Give it a minute and your model will automagically be able to serve prediction. 3. **Evaluate and iterate** The Merlin UI allows you to deploy and track different model versions and tag any version to run experiment easily. All model artifacts are synchronized into MLflow Tracking, which can be used to track and compare the model performance. ## Key Concepts of Merlin The design of Merlin uses a few key concepts below, you should familiarize yourself with: **Project**: Project represents a namespace for a collection of model. For example, a project could be food Recommendations, driver allocation, ride pricing, etc. **Model**: Every model is associated with one (and only one) project and model endpoint. Model also can have zero or more model versions. In the entities' hierarchy of MLflow, a model corresponds to an MLflow experiment. **Model Version**: The model version represents an iteration within a model. A model version is associated with a run within MLflow. A Model Version can be deployed as a service, there can be multiple deployments of model version with different endpoint each. **Model Endpoint**: Every model has its own endpoint that contains routing rule(s) to an active model version endpoint (serving mode). This endpoint is usually used to serve traffic in production. The model version it is routed to changes in the background when a serving model version is changed. Hence there is no need to change the endpoint used to serve traffics when the serving model version is changed. **Model Version Endpoint**: A model version endpoint is a way to obtain model inference results in real-time, over the network (HTTP). This endpoint is unique to each model version. Model endpoint will route to the model version endpoint in the background, when the associated model version is set to serving. **Environment**: The environment’s name is a user-facing property that will be used to determine the target Kubernetes cluster where a model will be deployed to. The environment has two important properties, name and Kubernetes cluster. # Router Concepts **Project**: Holds all [MLP](https://github.com/caraml-dev/mlp) resources that belong to a specific team such as service accounts, Merlin models, etc. **Router**: The router is the nucleus of the Turing system. It is responsible for coordinating the traffic routing to multiple model endpoints, invoking the pre and post processors, incorporating the response from the Experiment engine and logging of these responses. **Request**: Incoming message from the client to the Turing system. **Response**: The Turing workflow involves the pre-processor (Enricher), the model endpoints, the Experiment engine and the post-processor (Ensembler), some of which are optional. Each component creates a response which becomes the request to the next component in the workflow. In general, the Response refers to the final response from the Turing system, after passing through all stages. **Route**: Model endpoint which may be a Merlin model or any arbitrary URL that can be reached from the Turing infrastructure. **Experiment**: An application of rules, filters and configurations that determine how the traffic is routed and responses are combined to create the final response to the Turing request and enables evaluation of different models and parameters. **Treatment**: The set of configurations and actions to be applied to the current request which results in an outcome that can be evaluated. **Unit**: Smallest entity that can receive different treatments. **Rule**: Conditions determining which treatment to apply to a specific unit. **Enricher**: An optional service to perform arbitrary transformations on the incoming request or supplementing the request with data from external sources. **Ensembler**: An optional external service that accepts responses from the model endpoints altogether with the experiment configuration and responds back to the Turing router with a final response. Exploration policies or combining responses from multiple models into one can be implemented here. # Experiment Concepts This section describes the main concepts related to XP. ### Project Projects are the fundamental structure in the MLP ecosystem. In terms of experimentation, a project is a service intending to run experiments for a specific use case. Eg: Driver Matching, Trip Duration Estimation, etc. All experiments defined in XP are grouped by the project. ### Variables Experiment Variables are input values that have an impact on the treatment generated by XP. These are retrieved from the incoming request and applied when running the experiment. #### Segmenters A segmenter is an attribute of the population considered for the experiment. XP supports the following segmenters: * S2 IDs * Days of the Week * Hours of the Day #### Segment A combination of one or more segmenters with their specific values makes up a segment. Experiments are defined over segments and the experiment applicable to a given treatment request is determined by matching the segment. For example, consider the following experiments. | Experiment Name | Segment | | --------------- | --------------------------------------- | | exp\_1 | country=\[ID], service=\[ride] | | exp\_2 | country=\[ID], service=\[package, food] | The parameters in the incoming treatment request must match each segmenter (AND) and one of the values in each values list (IN): * A request containing `country=ID` and `service=ride` would match the first experiment. Similarly, A request containing `country=ID` and `service=package` (or `country=ID` and `service=food`) would match the second experiment. * A request containing `country=ID` and `service=car` does not match any experiment. If the segment cannot be matched against the active experiments, an empty response is returned. #### Randomization Unit This a required value for A/B experiments and optional for Switchback experiments (may only be applicable to randomized switchbacks, depending on how the project is configured). The value of the randomization unit in the request has an impact on the treatment generated. For example, this could be the pricing request id, which is used to randomly select a treatment from an A/B experiment's weighted list of treatment choices, where the weights are the traffic percentages assigned to the respective treatments. ### Experiment An experiment is the set of configurations and filters that allow for systematically varying some independent variables to impact some other dependent variables. Experiment definitions comprise 3 types of information: * Metadata such as the name, description, etc. * Segment definition * Treatment configurations #### Experiment Orthogonality Every request to XP to fetch a treatment for a given project and request parameters should be able to deterministically select no more than one experiment active in the given time. This is enforced by a property of the experiments called Orthogonality - for each pairwise combination of the experiments, there should be at least one segmenter in the experiments that has no overlapping values at the same "match strength". For more information on this and illustrations, please refer to the Experiment Hierarchy section below. When active experiments are created and when inactive experiments are activated, XP runs these checks and their failure will result in the failure of the experiment creation/update. #### Experiment Types XP support various experiment types. * **A/B Experiments** - Treatment assignment is randomized on the unit supplied in the request and one of the treatments in the experiment will be chosen at random, accounting for the traffic allocation for each treatment. * **Switchback Experiments** - The main idea behind switchback experiments is that the experiment engine switches back and forth between the control and treatment configurations, per configured time interval. In XP, switchback experiments can have one or more treatments and the engine cycles through them, selecting one treatment for all requests in every time interval. * **Randomized Switchback Experiments** - This is a hybrid between the A/B experiments and Switchbacks. These experiments are Switchbacks by nature (they have a time interval). In addition, they can have a traffic allocation on each of the treatments. Thus, at every new interval, the selection of the treatment is not cyclical, but randomized. All requests in a given time interval will receive the same treatment. #### Experiment Hierarchy One of the greatest benefits of using XP to manage experiments is that, prior to generating the treatment from an experiment's configurations, the system handles the more complex task of 'selecting the right experiment' to run. Multiple simultaneous experiments can be scheduled on XP and the correct one is chosen at runtime, by matching the request parameters against the active experiments' configurations. Where the incoming request may match multiple active experiments, the most granular experiment is chosen. To understand the workings, let us consider an example project that uses the segmenters `country`, `geo_area` and `service` (in that order, as chosen in the Project Settings), and the following active experiments: | Experiment Name | country | geo\_area | service | | --------------- | ------- | ---------- | --------------------- | | exp\_1 | ID | 3 (Bali) | 1 (Ride), 2 (Package) | | exp\_2 | ID | 3 (Bali) | - | | exp\_3 | ID | - | 1 (Ride) | | exp\_4 | ID | 15 (Batam) | - | | exp\_5 | ID | - | - | Notes: * `exp_1` is specific to Bali and the Ride/Package service types * `exp_2` is the fallback experiment for Bali * `exp_3` is the fallback experiment for the Ride service type * `exp_4` is the fallback experiment for Batam * `exp_5` is the fallback experiment for all Indonesia based requests When the Fetch Treatment API is called, the user would have to supply the country, latitude & longitude (which will be used to match the geo\_area) and the service in the request parameters. The following experiments will be chosen based on the transformed parameters. | Transformed Parameters | All Matched Experiments | Chosen Experiment | | ---------------------- | ---------------------------------- | ----------------- | | (ID, Bali, Ride) | `exp_1`, `exp_2`, `exp_3`, `exp_5` | `exp_1` | | (ID, Bali, Food) | `exp_2`, `exp_5` | `exp_2` | | (ID, Bandung, Ride) | `exp_3`, `exp_5` | `exp_3` | | (ID, Batam, Food) | `exp_4`, `exp_5` | `exp_4` | | (ID, Bandung, Package) | `exp_5` | `exp_5` | | (SG, Singapore, Ride) | - | - | **Optional Segmenters** Segmenters registered in a project may be required or optional. Those that are optional can be supplied values in the experiment definition or can be left unset in the experiment, in which case, the experiment will apply to all values of that segmenter and we may also say that the segmenter is optional to the experiment. **Inter-Segmenter Hierarchy** In the first row, among `exp_2` and `exp_3`, there are 2 different optional segmenters. If `exp_1` did not exist, `exp_2` will be chosen because it has an exact match of the higher priority segmenter `service` (the **inter-segmenter hierarchy** is decided by the order in which the segmenters are chosen in the Project Settings). **Revisiting Experiment Orthogonality** The validation rules for configuring experiments is such that, no more than 1 experiment may be chosen at the time of treatment generation. This means that zero or more experiments may be matched by the transformed parameters but only (zero or) 1 of them can be ultimately filtered by the Fetch Treatment request. To achieve this, the system makes it impossible to schedule the following experiments, if the above experiments are also active for (parts of) the same duration: | Experiment Name | country | geo\_area | service | | --------------- | ------- | -------------------- | -------- | | exp\_6 | ID | 3 (Bali), 15 (Batam) | 1 (Ride) | | exp\_7 | ID | - | - | * `exp_6` cannot be created because there is already an exact experiment `exp_1` for Bali+Ride * `exp_7` conflicts with `exp_5` - both geo\_area and service are optional in both experiments and the other segmenters (ID) overlap. But we can create the experiment below, because there is no other experiment with an exact match for ID+Batam+Ride or ID+Batam+Package: | Experiment Name | country | geo\_area | service | | --------------- | ------- | ---------- | -------------------- | | exp\_8 | ID | 15 (Batam) | 1 (Ride), 2(Package) | **Experiment Tiers** We may often have a long-running experiment (say, for several weeks) for a certain segment and would like to run a short spike (say, for 1 day) to quickly test the impact of a different set of treatments for that segment. In such a scenario, we can make use of experiment tiers. XP only allow(s) one of 2 tiers: * The **Default** tier which is the default value for all experiments * The **Override** tier which will override the default experiment, if exists. The Override experiments are not global overrides. They simply override the default experiment of a similar granularity. To illustrate this, let's consider the previous examples: * If `exp_1` was in the default tier and `exp_2` in the override tier and a treatment request was made for (ID, Bali, Ride), the system would still select `exp_1` because it is more granular. * If `exp_1` was in the default tier, `exp_6` can be created in the override tier (or vice versa). Of the 2, the experiment in the override tier will be chosen over the one in the default tier. **A Note on S2IDs** S2ID levels have an implicit hierarchy. The system accepts S2ID values at levels 10-14 and the more granular levels (14 is the most granular) will supersede the lower levels, by the same matching rules as above. **Fetch Treatment API Hierarchy Resolution** The following logic summarizes the experiment filtering mechanism adopted by the Fetch Treatment API: 1. Match all experiments for the given request. If 0 or 1 experiment matched, return. 2. Based on the inter-segmenter hierarchy, in that order, filter out weak matches if one or more exact matches exist for the segmenter. 3. If S2ID is used, select the experiment(s) with the most granular level among the matches. 4. At this point, we will either have one experiment or two (one in each tier). If we have 2 experiments, we pick the one in the override tier. This way, the API will select exactly 1 experiment at the end of steps 2-4. # Projects *** Project represents a namespace for a collection of CaraML resources, that belong to a specific team such as service accounts, Models, Routers, Pipelines etc. Project is one of the main building blocks for access control in CaraML. {% content-ref url="/pages/qpjnxq7zTdOEvxvMDM85" %} [Create a project](/user-guides/projects/create-project) {% endcontent-ref %} # Create a project ### Choose “Create Project” in the projects dropdown. ![](/files/xhSfjnnRC7b7drp0bmYw) ### Name your project. ![](/files/RG1XqAHpQWLwbudivQEC) ### Select a team. ![](/files/qRKN1pwwiyeM9Cur7YDa) ### Select a stream. ![](/files/doZuZmkcgWUZRcQFQ8uq) ### Define a list of project members that can access your project. These members can either be individual users or service accounts, defined by their emails and can be granted Administrator or Reader privileges. There are 2 required inputs. ![](/files/fyZi7ij97RNS4Vvgy1RR) **Administrators**: These users will have full access to your project. **Readers**: These users will have read-only access to your project. You can optionally add labels, which are key-value pairs, to your project. These labels will be applied to the Kubernetes resources of your project. ![](/files/2TWTkFMCrHKqlwfsDJ9x) ### Submit your project. Click “Submit” to create your project. If your project is successfully created, you will be taken to the Project Settings page of your newly created project. # Managing secrets CaraML provides capability for storing user's secret in the platform. The secret can be used by the user within CaraML workload such as batch prediction, router deployment, and pipeline. The secret is stored within a secret storage which can be configured in CaraML deployment. ## Creating Secret To create a secret, user can visit the project settings page and click the "Secret" tab. ![](/files/Nk0o1pCAnp2eywnXj70G) The secret tab will show the list of secret that has been created for the project. User can click the "Create Secret" button to create a new secret. ![](/files/bNB3AYxOWENTVnyvcs4D) ## Configuring Default Secret Storage By default, a CaraML installation will use internal secret storage. The internal secret storage is a simple key-value store that is stored in the database. For a more secure secret storage, CaraML can be configured to use external secret storage such as Hashicorp Vault. To configure the default secret storage, user can specify `defaultSecretStorage` configuration when deploying the [MLP component](https://github.com/caraml-dev/helm-charts/tree/main/charts/mlp). Currently, the supported secret storage is `internal` and `vault`. For example following configuration will configure CaraML to use Hashicorp Vault as the default secret storage. ```yaml defaultSecretStorage: name: vault type: vault config: vaultConfig: url: https://localhost:8200 role: my-role mountPath: secret pathPrefix: caraml-secret/{{ .Project }}/ authMethod: gcp gcpAuthType: gce ``` Below table shows the list of supported configuration when using vault secret storage. ### Vault Configurations | name | description | required | example | | ------------------- | ---------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------- | ----------------------------------------------------------- | | url | vault URL | y | | | role | role name to be used when authenticating against Vault | y (not required when `authMethod` is set to `token`) | my-role | | mountPath | [vault mount path](https://developer.hashicorp.com/vault/tutorials/enterprise/namespace-structure) | y | secret | | pathPrefix | path prefix of the secret in Vault. Path prefix must contains placeholder for project {{ .Project }} | y | caraml-secret/{{ .Project }}/ | | authMethod | authentication method | y | `gcp`, `token` (Do not use `token` for production use-case) | | gcpAuthType | GCP authentication type | required when `authMethod` is set to `gcp` | `gce`, `iam` | | serviceAccountEmail | Service account email to be used when using `iam` gcp auth | required when `authMethod` is set to `gcp` and `gcpAuthType` is `iam` | | | token | Vault token to be used for authentication when `authMethod` is set to `token` | required when `authMethod` is set to `token` | test-token | # Feature Store CaraML Feature Store are built upon another open source project [Feast](https://feast.dev/). From CaraML Maintainer's perspective, we have customized Feast to be more suitable for our use case, and lean more towards a production ready system. Work is in progress to provide documentations for CaraML's take of Feast and how it integrates with other CaraML components, stay tuned! # Models ## Python SDK The Merlin SDK can be installed directly using pip: ```bash pip install merlin-sdk ``` Users should then be able to connect to a Merlin deployment as follows {% code title="getting\_started.py" overflow="wrap" lineNumbers="true" %} ```python import merlin from merlin.model import ModelType # Connect to an existing Merlin deployment merlin.set_url("merlin.example.com") # Set the active model to the name given by parameter, if the model with the given name is not found, a new model will # be created. merlin.set_model("example-model", ModelType.PYFUNC) # Ensure that you're connected by printing out some Model Endpoints merlin.list_model_endpoints() ``` {% endcode %} ## Client Libraries Merlin provides [Go client library](https://github.com/caraml-dev/merlin/blob/main/api/client/client.go) to deploy and serve ML models. To connect to the Merlin deployment, the client needs to be authenticated by Google OAuth2. You can use `google.DefaultClient()` to get the Application Default Credential. {% code title="getting\_started.go" overflow="wrap" lineNumbers="true" %} ```go googleClient, _ := google.DefaultClient(context.Background(), "https://www.googleapis.com/auth/userinfo.email") cfg := client.NewConfiguration() cfg.BasePath = "http://merlin.dev/api/merlin/v1" cfg.HTTPClient = googleClient apiClient := client.NewAPIClient(cfg) ``` {% endcode %} # Create a Model ## Creating a Model A Model represents a machine learning model. Each Model has a type. Currently Merlin supports both standard model types (PyTorch, SKLearn, Tensorflow, and XGBoost) and user-defined models (PyFunc model). Merlin also supports custom models. More info can be found here: {% content-ref url="/pages/6kwE95Lt6Vke46x0PTxu" %} [Custom Model](/user-guides/01_getting_started/02_creating_a_model/01_custom_model) {% endcontent-ref %} Conceptually, a Model in Merlin is similar to a class in programming languages. To instantiate a Model, you’ll have to create a [Model Version](#creating-a-model-version). `merlin.set_model(, )` will set the active model to the name given by parameter. If the Model with given name is not found, a new Model will be created. {% code title="model\_creation.py" overflow="wrap" lineNumbers="true" %} ```python import merlin from merlin.model import ModelType merlin.set_model("tensorflow-sample", ModelType.TENSORFLOW) ``` {% endcode %} ## Creating a Model Version A Model Version represents a snapshot of A particular Model iteration. A Model Version might contain artifacts which are deployable to Merlin. You'll also be able to attach information such as metrics and tags to a given Model Version. {% code title="model\_version\_creation.py" overflow="wrap" lineNumbers="true" %} ```python with merlin.new_model_version() as v: merlin.log_metric("metric", 0.1) merlin.log_param("param", "value") merlin.set_tag("tag", "value") merlin.log_model(model_dir='tensorflow-sample') ``` {% endcode %} # Custom Model Custom model enables users to deploy any docker image that satisfy merlin requirements. Users are responsible to develop their own web service, build and publish the docker image, which later on can be deployed through Merlin. Users should consider to use custom model, if they have one of the following conditions: * Model needs custom complex transformations (preprocess and postprocess) and want to use other languages than Python. * Using non standard model, e.g using heuristic or other ml framework model that have not been introduced in merlin. * Having dependencies with some os distribution packages. ## Comparison With PyFunc Model In high level PyFunc and custom model has similarity, they both enable users to specify custom logic and dependencies. The difference is mostly on the flexibility level and performance. | Factor | Custom Model | Pyfunc Model | | ----------- | ------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------- | | Web Service |
  • Users can use any tech stack for web service
  • Users need to implement whole web service
| Use python server, and users only need to modify core logic of prediction (infer function in this case) | | Dependency | Users can specify any dependencies that is required. It can be os distribution package or library from specific programming language | Users can only specify python package dependencies | | Performance | Users has more control on the performance of model. Since there is no limitation on tech stack that can be used | Users only has control on the infer function. Performance is rather slow because of the performance of python | ## Web Service Implementation Users need to implement their own web service using any tech stack that suitable for their use case. Currently users can deploy web service using `HTTP_JSON` or `UPI_V1` protocol, both have different requirements that must be satisfied by the web server. ### HTTP\_JSON Custom Model Users can add the artifact (model or anything else) in addition to the docker image when uploading the model. During the deployment, these artifacts will be made available in the directory specified by `CARAML_ARTIFACT_LOCATION` environment variable. Web service must open and listen to the port number given by `CARAML_HTTP_PORT` environment variable. Web service MUST implement the following endpoints: | Endpoint | HTTP Method | Description | | --------------------------------- | ----------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `/v1/models/{model_name}:predict` | POST | For every inference or prediction calls, it will call this endpoint. Merlin will give the `CARAML_MODEL_FULL_NAME` environment variable, this value can be used as {model\_name} for this endpoint. | | `/v1/models/{model_name}` | GET | This endpoint will be used to check model healthiness. Model can serve after this API return 200 status code. | | `/` | GET | This endpoint will be used as server liveness. Return 200 if the model is healthy. | | `/metrics` | GET | This endpoint is used by prometheus to pull the metrics produced by the predictor. The implementation of this endpoint is handled by prometheus library, for example [this](https://prometheus.io/docs/guides/go-application/) is how to implement the endpoint with golang. | ### UPI\_V1 Custom Model Similar with `HTTP_JSON` custom model, users can add the artifact during model upload, and the uploaded artifacts will be available in the directory specified by `CARAML_ARTIFACT_LOCATION` environment variable. The web server must implement service that defined in the [UPI interface](https://github.com/caraml-dev/universal-prediction-interface/blob/main/proto/caraml/upi/v1/upi.proto#L11), also the server must open and listen to the port number given by `CARAML_GRPC_PORT` environment variable. If users want to emit metrics from this web server, they need to create scrape metrics REST endpoint. The challenge here, the knative (the underlying k8s deployment tools that merlin use) doesn't open multiple ports, hence the REST endpoint must be running on the same port as gRPC server (using port number given by `CARAML_GRPC_PORT`). Not every programming language can support running multiple protocol (gRPC and HTTP in this case) on the same port, for Go language users can use [cmux](https://github.com/soheilhy/cmux) to solve this problem, otherwise users can use push metrics to [pushgateway](https://prometheus.io/docs/instrumenting/pushing/) ### Environment Variables As mentioned in the previous section, there are several environment variables that will be supplied by Merlin control plane to the custom model. Below are the list of the variables | Name | Description | | -------------------------- | ------------------------------------------------------------------------------------------------------ | | STORAGE\_URI | Contains the URI where the `model` artifacts is remotely stored | | CARAML\_HTTP\_PORT | Port that must be openend when the model is deployed with `HTTP_JSON` protocol | | CARAML\_GRPC\_PORT | Port that must be opened when the model is deployed with `UPI_V1` protocol | | CARAML\_MODEL\_NAME | Name of merlin model | | CARAML\_MODEL\_VERSION | Merlin model version | | CARAML\_MODEL\_FULL\_NAME | Full name merlin model, per current version it use `{CARAML_MODEL_NAME}-{CARAML_MODEL_VERSION}` format | | CARAML\_ARTIFACT\_LOCATION | Local path where the model artifacts will be stored | ## Docker Image Docker image must contains web service application and dependencies that must be installed in order to run the web service. Users are responsible for building the docker image as well as for publishing it. Please make sure the k8s cluster (where model will be deployed) have access to pull the docker image. ## Deployment Using Merlin SDK ``` resource_request = ResourceRequest(1, 1, "1", "1Gi") model_dir = "model_dir" with merlin.new_model_version() as v: v.log_custom_model(image="ghcr.io/yourcustommodelimage", model_dir=model_dir) endpoint = merlin.deploy(v, resource_request= resource_request, protocol = Protocol.HTTP_JSON) # endpoint = merlin.deploy(v, resource_request= resource_request, protocol = Protocol.UPI_V1) if using UPI ``` Most of the method that used in the above snipped is commonly used by all the model deployment, but `log_custom_model` method. `log_custom_model` is method exclusively used to upload custom model. Below are the method parameters that can be specified during the invocation | Parameter | Description | Required | | ----------- | -------------------------------------------------------- | -------- | | `image` | Docker image that will be used as predictor | Yes | | `model_dir` | Directory that will be uploaded to MLFlow | No | | `command` | Command to run docker image | No | | `args` | Arguments that needs to be specified when running docker | No | ### Deployment Flow * Create new model version * Log custom model, specify image and model directory that contains artifacts that need to be uploaded * Deploy. There is no difference with other model deployments # Deploy a Model To learn about deploying a model, please visit the following docs. {% content-ref url="/pages/ZqhpigQML7kiLyhgV9GA" %} [Deploying a Model Version](/user-guides/01_getting_started/03_deploying_a_model/01_deploying_a_model_version) {% endcontent-ref %} {% content-ref url="/pages/53py0NIxYyhezszziU8y" %} [Severing a Model Version](/user-guides/01_getting_started/03_deploying_a_model/02_serving_a_model_version) {% endcontent-ref %} {% content-ref url="/pages/Khrm00aS00WrG5jbiWdP" %} [Configuring Transformer](/user-guides/01_getting_started/03_deploying_a_model/03_configuring_transformers) {% endcontent-ref %} {% content-ref url="" %} {% endcontent-ref %} # Deploying a Model Version To start sending inference requests to a model version, it must first be deployed. During deployment, different configurations can be chosen such as the number of replicas, CPU/memory requests, autoscaling policy, environment variables, etc. The set of these configurations that are used to deploy a model version is called a *deployment*. A model may have any number of versions. But, at any given time, only a maximum of **2** model versions can be deployed. When a model version is deployed, a Model Version Endpoint is created. The URL is of the following format: ``` http://-.. ``` For example a Model named `my-model` within Project named `my-project` with the base domain `models.id.merlin.dev` will have a Model Version Endpoint for version `1` as follows: ``` http://my-model-1.my-project.models.id.merlin.dev ``` A Model Version Endpoint has several states: * **pending**: The initial state of a Model Version Endpoint. * **running**: Once deployed, a Model Version Endpoint is in running state and is accessible. * **serving**: A Model Version Endpoint is in serving state if a Model Endpoint is created from it. * **terminated**: Once undeployed, a Model Version Endpoint is in terminated state. * **failed**: If an error occurred during deployment. ## Image Building Depending on the type of the model being deployed, there may be an intermediate step to build the Docker image (using Kaniko). This is applicable to PyFunc models. {% hint style="info" %} You can separately start the image building process without actually deploying your model by using `build_image()` function. For more details, you can check [Build Image](https://github.com/caraml-dev/docs/blob/main/module/model/user/generated/12_build_image.md). {% endhint %} ## Deploying a Model Version A model version can be deployed via the SDK or the UI. ### Deploying a Model Version via SDK Here's the example to deploy a Model Version Endpoint using Merlin Python SDK: {% code title="model\_version\_deployment.py" overflow="wrap" lineNumbers="true" %} ```python with merlin.new_model_version() as v: merlin.log_metric("metric", 0.1) merlin.log_param("param", "value") merlin.set_tag("tag", "value") merlin.log_model(model_dir='tensorflow-sample') merlin.deploy(v, environment_name="staging") ``` {% endcode %} ### Deploying a Model Version via UI The Deploy option can be selected from the model versions view. ![Deploy a Model Version](/files/ZouG76qxWMGjm89IuNjL) ## Deployment Modes Merlin supports 2 types of deployment mode: `SERVERLESS` and `RAW_DEPLOYMENT`. Under the hood, `SERVERLESS` deployment uses KNative as the serving stack. On the other hand `RAW_DEPLOYMENT` uses native [Kubernetes deployment resources](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/). The deployment modes supported by Merlin have their own advantages and disadvantages, listed below. * **Serverless Deployment:** * **Pros:** Supports more advanced autoscaling policy (RPS, Concurrency); supports scale down to zero. * **Cons:** Slower compared to `RAW_DEPLOYMENT` due to infrastructure overhead * **Raw Deployment:** * **Pros:** Relatively faster compared to `SERVERLESS` deployments; less infrastructure overhead and more cost efficient. * **Cons:** Supports only autoscaling based on CPU usage. ### Configuring Deployment Modes Users are able to configure the deployment mode of their model via the SDK or the UI. #### Configuring Deployment Mode via SDK Example below will configure the deployment mode to use `RAW_DEPLOYMENT` {% code title="deployment\_configuration.py" overflow="wrap" lineNumbers="true" %} ```python import merlin from merlin import DeploymentMode from merlin.model import ModelType # Deploy using raw_deployment merlin.set_url("merlin.example.com") merlin.set_project("my-project") merlin.set_model("my-model", ModelType.TENSORFLOW) model_dir = "test/tensorflow-sample" with merlin.new_model_version() as v: merlin.log_model(model_dir=model_dir) # Deploy using raw_deployment new_endpoint = merlin.deploy(v, deployment_mode=DeploymentMode.RAW_DEPLOYMENT) ``` {% endcode %} #### Configuring Deployment Mode via UI ![Deployment Mode](/files/3ssP5YEyqmvW3Ku0kvSk) ## Autoscaling Policy Merlin supports configurable autoscaling policy to ensure that users have complete control over the autoscaling behavior of their models. There are 4 types of autoscaling metrics in Merlin: * **CPU Utilization:** The autoscaling is based on the ration of model service's CPU usage and its CPU request. This autoscaling policy is available on all deployment mode. * **Memory Utilization:** The autoscaling is based on the ration of model service's Memory usage and its Memory request. This autoscaling policy is available only on `SERVERLESS` deployment mode. * **Model Throughput (RPS):** The autoscaling is based on RPS per replica of the model service. This autoscaling policy is available only on `SERVERLESS` deployment mode. * **Concurrency:** The autoscaling is based on number of concurrent request served by a replica of the model service. This autoscaling policy is available only on `SERVERLESS` deployment mode. ### Configuring Autoscaling Policy Users can update the autoscaling policy via the SDK or the UI. #### Configuring Autoscaling Policy via SDK Below is the example of configuring autoscaling policy of a `SERVERLESS` deployment to use `RPS` metrics. {% code title="autoscaling\_policy.py" overflow="wrap" lineNumbers="true" %} ```python import merlin from merlin import DeploymentMode from merlin.model import ModelType # Deploy using raw_deployment merlin.set_url("merlin.example.com") merlin.set_project("my-project") merlin.set_model("my-model", ModelType.TENSORFLOW) model_dir = "test/tensorflow-sample" with merlin.new_model_version() as v: merlin.log_model(model_dir=model_dir) # Deploy using raw_deployment endpoint = merlin.deploy(v1, deployment_mode=DeploymentMode.SERVERLESS, autoscaling_policy=merlin.AutoscalingPolicy( metrics_type=merlin.MetricsType.RPS, target_value=20)) ``` {% endcode %} #### Configuring Autoscaling Policy via UI ![Autoscaling Policy](/files/M03bR38ti5JxAH6y6O2o) ## CPU Limits By default, Merlin determines the CPU limits of all model deployments using platform-level configured values. These CPU limits can either be calculated as a factor of the user-defined CPU request value for each deployment (e.g. 2x of the CPU request value) or as a constant value across all deployments. However, users can override this platform-level configured value by setting this value explicitly on the UI or on the SDK. On the UI: ![CPU Limits](/files/u0wYxHK00WYXJVMSGCRL) On the SDK: ```python merlin.deploy( v, environment_name=some_name, resource_request=ResourceRequest(cpu_limit="2"), deployment_mode=some_deployment_mode, ) ``` ## Liveness Probe When deploying a model version, the model container will be built with a livenes probe by default. The liveness probe will periodically check that your model is still alive, and restart the pod automatically if it is deemed to be dead. However, should you wish to disable this probe, you may do so by providing an environment variable to the model service with the following value: ``` MERLIN_DISABLE_LIVENESS_PROBE="true" ``` This can be supplied via the deploy function. i.e. {% code title="liveness\_probe.py" overflow="wrap" lineNumbers="true" %} ```python merlin.deploy(v, env_vars={"MERLIN_DISABLE_LIVENESS_PROBE"="true"}) ``` {% endcode %} The liveness probe is also available for the transformer. More details can be found at: {% content-ref url="/pages/U6jfr8ecZ1dsgG0tjs7T" %} [Standard Transformer Expressions](/user-guides/01_getting_started/03_deploying_a_model/03_configuring_transformers/01_standard_transformer/01_standard_transformer_expressions) {% endcontent-ref %} # Severing a Model Version Model serving is the next step of model deployment. After deploying a model version, we can optionally start serving it. This creates a Model Endpoint which is a stable URL associated with a model, of the following format: ``` http://.. ``` For example a Model named `my-model` within Project named `my-project` with the base domain `models.id.merlin.dev` will have a Model Endpoint which look as follows: ``` http://my-model.my-project.models.id.merlin.dev ``` Having a Model Endpoint makes it easy to keep updating the model (creating a new model version, running it and then serving it) without having to modify the model URL used by the called system. ## Serving a Model Version A model version can be served via the SDK or the UI. ### Serving a Model Version via SDK To serve a model version, you can call `serve_traffic()` function from Merlin Python SDK. {% code title="model\_version\_serving.py" overflow="wrap" lineNumbers="true" %} ```python with merlin.new_model_version() as v: merlin.log_metric("metric", 0.1) merlin.log_param("param", "value") merlin.set_tag("tag", "value") merlin.log_model(model_dir='tensorflow-sample') version_endpoint = merlin.deploy(v, environment_name="staging") # serve 100% traffic at endpoint model_endpoint = merlin.serve_traffic({version_endpoint: 100}) ``` {% endcode %} ### Serving a Model Version via UI Once a model version is deployed (i.e., it is in the Running state), the Serve option can be selected from the model versions view. ![Serve Model Version](/files/vKmfoivreFvLNciqFzWq) # Configuring Transformer In the Merlin ecosystem, a Transformer is a service deployed in front of the model service which users can use to perform pre-processing / post-processing steps to the incoming request / outgoing response, to / from the model service. A Transformer allows the user to abstract the transformation logic outside of their model and even write it in a language more performant than python. Currently, Merlin supports two types of Transformer: Standard and Custom: {% content-ref url="/pages/rqUwryMOAPo1EMvaqBxk" %} [Standard Transformer](/user-guides/01_getting_started/03_deploying_a_model/03_configuring_transformers/01_standard_transformer) {% endcontent-ref %} {% content-ref url="/pages/uPspdAlZPnX6BpGG3S3T" %} [Custom Transformer](/user-guides/01_getting_started/03_deploying_a_model/03_configuring_transformers/02_custom_transformer) {% endcontent-ref %} # Standard Transformer Standard Transformer is a built-in pre and post-processing steps supported by Merlin. With standard transformer, it’s possible to enrich the model’s incoming request with features from feast and transform the payload so that it’s compatible with API interface provided by the model. Same transformation can also be applied against the model’s response payload in the post-processing step, which allow users to adapt the response payload to make it suitable for consumption. Standard transformer supports **http\_json** and **upi\_v1** protocol. For **http\_json** protocol the standard transformer server runs rest server on top of **http 1.1**, **upi\_v1 protocol** the server run grpc server. ## Concept Within standard transformer there are 2 process that user can specify: preprocess and postprocess. Preprocess is useful to perform transformation against model’s incoming request such as enriching the request with features from Feast and transforming the client’s request to a format accepted by model service. Post Processing is useful for performing transformation against model response so that it is more suitable for client consumption. Within both preprocess and postprocess, there are 3 stages that users can specify: * Input stage In the input stage, users specify all the data dependencies that are going to be used in subsequent stages. There are 2 operations available in these stages: variable declaration and table creation. * Transformation stage. In this stage, the standard transformers perform transformation to the tables created in the input stage so that its structure is suitable for the output. In the transformation stage, users operate mainly on tables and are provided with 2 transformation types: single table transformation and table join. * Output stage At this stage, both the preprocessing and postprocessing pipeline should create output payload which later on can be used as request payload for model predictor or the final response to be returned to downstream service/client. There are 3 types of output operation: * JSON Output. JSON output operation will return JSON output, this operation only applicable for **http\_json** protocol * UPIPreprocessOutput. UPIPreprocessOutput will return UPI Request interface payload in a protobuf.Message type * UPIPostprocessOutput. UPIPostprocessOutput will return UPI Response interface payload in a protobuf.Message type ![Standard Transformer](/files/i4MnxMJsQCVIRPz0aXfn) ## Jsonpath Jsonpath is a way to find value from JSON payload. Standard transformer using jsonpath to find values either from request or model response payload. Standard transformer using Jsonpath in several operations: * Variable declaration * Feast entity value * Base table * Column value in table * Json Output Most of the jsonpath configuration is like this ``` fromJson: jsonPath: # Json path in the incoming request / model response payload defaultValue: # (Optional) Default value if value for the jsonPath is nil or empty valueType: # Type of default value, mandatory to specify if default value is exist ``` but in some part of operation like variable operation and feast entity extraction, jsonPath configuration is like below ``` jsonPathConfig: jsonPath: # Json path in the incoming request / model response payload defaultValue: # (Optional) Default value if value for the jsonPath is nil or empty valueType: # Type of default value, mandatory to specify if default value is exist ``` ### Default Value In standard transformer, user can specify jsonpath with default value if the result of jsonpath is empty or nil. Cases when default value is used: * Result of jsonpath is nil * Result of jsonpath is empty array * Result of jsonpath is array, where some of its value is null Value Type | Value Type | Syntax | | ---------- | ------ | | Integer | INT | | Float | FLOAT | | Boolean | BOOL | | String | STRING | For example, if we have incoming request ``` { "signature_name" : "predict", "instances": [ {"sepal_length":2.8, "sepal_width":1.0, "petal_length":6.8, "petal_width":0.4}, {"sepal_length":0.1, "sepal_width":0.5, "petal_length":1.8, "petal_width":2.4} ], "instances_with_null": [ {"sepal_length":2.8, "sepal_width":1.0, "petal_length":6.8, "petal_width":0.4}, {"sepal_length":0.1, "sepal_width":0.5, "petal_length":1.8, "petal_width":null}, {"sepal_length":0.1, "sepal_width":0.5, "petal_length":1.8, "petal_width":0.5} ], "empty_array": [], "null_key": null, "array_object": [ {"exist_key":1}, {"exist_key":2} ] } ``` * Result of jsonpath is nil There are cases when `jsonpath` value is nil: * Value in JSON is nil * There is no such key in JSON Example: ``` fromJson: jsonPath: $.null_key defaultValue: -1 valueType: INT ``` the result of above Jsonpath is `-1` because `$.null_key` returning nil * Result of jsonpath is empty array ``` fromJson: jsonPath: $.empty_array defaultValue: 0.0 valueType: FLOAT ``` the result of above Jsonpath is `[0.0]` because `$.empty_array` returning empty so it will use default value * Result of jsonpath is array, where some of its value is null ``` fromJson: jsonPath: $.instances_with_null[*].petal_width defaultValue: -1 valueType: INT ``` the result of above Jsonpath is `[0.4,-1,0.5]`, because the original jsonpath result `[0.4,null,0.5]` containing null value, the default value is used to replace `null` value ## Expression An expression is a single line of code which should return a value. Standard transformer uses expression as a flexible way of calculating values to be used in variable initialization or any other operations. For example: Expression can be used for initialising variable value ``` variables: - currentTime: expression: now() - currentHour: expression: currentTime.Hour() ``` Expression can be used for updating column value ``` updateColumns: column: "s2id" expression: getS2ID(df.Col('lat'), df.Col('lon')) ``` For full list of standard transformer built-in functions, please check: {% content-ref url="/pages/U6jfr8ecZ1dsgG0tjs7T" %} [Standard Transformer Expressions](/user-guides/01_getting_started/03_deploying_a_model/03_configuring_transformers/01_standard_transformer/01_standard_transformer_expressions) {% endcontent-ref %} ## Input Stage At the input stage, users specify all the data dependencies that are going to be used in subsequent stages. There are 4 operations available in these stages: 1. Table creation * Table Creation from Feast Features * Table Creation from Input Request * Table Creation from File 2. Variable declaration 3. Encoder declaration 4. Autoload ### Table Creation Table is the main data structure within the standard transformer. There are 3 ways of creating table in standard transformer: #### Table Creation from Feast Features This operation creates one or more tables containing features from Feast. This operation is already supported in Merlin 0.10. The key change to be made is to adapt the result of operation. Previously, the features retrieved from feast is directly enriched to the original request body to be sent to the model. Now, the operation only outputs as internal table representation which can be accessible by subsequent transformation steps in the pipeline. Additionally, it should be possible for users to give the features table a name to ease referencing the table from subsequent steps. Following is the syntax: ``` feast: - tableName: # Specify the output table name project: # Name of project in feast where the features located source: # Source for feast (REDIS or BIGTABLE) entities: # List of entities - name: # Entity Id valueType: # Entity Value Type # The entity value will be retrieved either using jsonPath or expression configuration below: jsonPathConfig: jsonPath: # Json path in the incoming request container the entity value defaultValue: # (Optional) Default value if value for the jsonPath is nil or empty valueType: # Type of default value, mandatory to specify if default value is exist jsonPath: # Json path in the incoming request containing the entity value (Deprecated) expression: # Expression provided by user which return entity values features: # List of features to be retrieved - name: # feature name defaultValue: # default value if the feature is not available ``` below is the sample of feast input: ``` feast: - tableName: table_1 project: sample source: BIGTABLE entities: - name: merchant_uuid valueType: STRING jsonPathConfig: jsonPath: $.merchant_uuid defaultValue: -1 valueType: INT - name: customer_id valueType: STRING expression: customer_id features: - name: sample_driver:order_count valueType: DOUBLE defaultValue: '90909' ``` There are two ways to get/retrieve features from feast in merlin standard transformer: \* Getting the features values from feast GRPC URL \* By direcly querying from feast storage (Bigtable or Redis). For this, you need to add extra environment variables in standard transformer \* REDIS. Set `FEAST_REDIS_DIRECT_STORAGE_ENABLED` value to true \* BIGTABLE. Set `FEAST_BIGTABLE_DIRECT_STORAGE_ENABLED` value to true \ For detail explanation of environment variables in standard transformer, you can look [this section](#standard-transformer-environment-variables) #### Table Creation from Input Request This step is generic table creation that allows users to define one or more tables based on value from either JSON payload, result of built-in expressions, or an existing table. Following is the syntax for table input: ``` tables: - name: # Table name baseTable: # create a base table either from a JSON array of object or from existing table fromJson: # create a table based on an array of objects within JSON payload, the object key will be column name. jsonPath: # JSONPath to array of object in the JSON payload defaultValue: # Fallback value if value for jsonpath is nil or empty addRowNumber: # True/false, add column called "row_number" which contains row number fromTable: # Create base table from an existing table tableName: # Source table name columns: # List of columns to be added in the table, it's possible to have 0 columns. The columns will override existing column defined in the baseTable # The number of row in the first column determines the table size so put the longest column first - name: # Column name fromJson: # Get column values from json path jsonPath: # JSONPath to array of object in the JSON payload defaultValue: # Fallback value if value for jsonpath is nil or empty expression: # Assign result from expression to the column value ``` sample: ``` - tables: - name: table_2 baseTable: fromTable: tableName: table_1 columns: - name: col_1 fromJson: jsonPath: $.drivers[*].id - name: col_2 expression: table.Col('rating') - name: table_3 baseTable: fromJson: jsonPath: $.drivers[*] columns: - name: col_1 fromJson: jsonPath: $.drivers[*].id - name: col_2 expression: table.Col('rating') ``` #### Table Creation from File This operation allows user to create a static table from a file. For example, user might choose to load a table with a list of public holidays for the year. As the data will be loaded into memory, it is strongly advised to keep the total size of all files within 50mb. Also, each file shall only contain information for 1 table. **Supported File Format** There are 2 types of files are currently supported: * csv: For this file type, only comma (,) may be used as delimiter. The first line shall also contain a header, which gives each column a unique name. * parquet **Supported File Storage Location** Currently, files must first be uploaded to a preferred GCS bucket in gods-\* project. The file will be read once during deployment. **Supported Column Types** Only basic types for the columns are supported, namely: String, Integer, Float and Boolean The types of each column are auto-detected, but may be manually set by the user (please ensure type compatibility). **How to use** In order to use this feature, firstly, these files will have to be loaded into GCS buckets in gods-\* projects in order to be linked. Then, use the syntax below to define the specifications: ```` ``` tables: - name: # Table name baseTable: fromFile: format: CSV # others: PARQUET uri: # GCS uri to the location of the file in gods-* project schema: # this part is used to manually set column type - name: col_1 # name of column type: STRING #others: INT, FLOAT, BOOL … - name: col_2 type: INT ``` ```` ### Variable Variable declaration is used for assigning literal value or result of a function into a variable. The variable declaration will be executed from top to bottom and it’s possible to refer to the declared variable in subsequent variable declarations. Following are ways to set value to variable. * Literal Specifiying literal value to variable. By specifying literal values user needs to specify what is the type for that variable. Types that supported for this: * String * Int * Float * Bool for example: ``` - variables: - name: var_1 literal: intValue: 3 - name: var_2 literal: floatValue: 2.2 - name: var_3 literal: boolValue: true - name: var_4 literal: stringValue: stringVal ``` * Jsonpath Value of variable is obtained from request/model response payload by specifying jsonpath value, e.g ``` - variables: - name: var_5 jsonPathConfig: jsonPath: $.rating defaultValue: -1 valueType: INT - name: var_6 jsonPath: $rating # deprecated ``` * Expression Value of variable is obtained from expression, e.g ``` - variables: - name: var_7 jsonPathConfig: jsonPath: $.customer_id - name: var_8 expression: var_7 ``` ### Encoders In order to encode data in the transformation stage, we need to first define an encoder by giving it a name, and defining the associated configurations. The syntax of encoder declaration is as follows: ``` - encoders: - name: #name of encoder 1 - name: #name of encoder 2 ``` There are 2 types of encoder currently available: Ordinal encoder: For mapping column values from one type to another Cyclical encoder: For mapping column values that have a cyclical significance. For example, Wind directions, time of day, days of week #### Ordinal Encoder Specification The syntax to define an ordinal encoder is as follows: ``` ordinalEncoderConfig: defaultValue: #default value targetValueType: #target value type. i.e. INT, FLOAT, BOOL or STRING mapping: … ``` There are currently 4 types of target value supported. The following table shows the syntax to use for each type: | Value Type | Syntax | | ---------- | ------ | | Integer | INT | | Float | FLOAT | | Boolean | BOOL | | String | STRING | See below for a complete example on how to declare an ordinal encoder ``` - encoders: - name: vehicle_mapping ordinalEncoderConfig: defaultValue: '0' targetValueType: INT mapping: suv: '1' sedan: '2' ``` #### Cyclical Encoder Specification Cyclical encoder are useful for encoding columns that has cyclical significance. By encoding such columns cyclically, you can ensure that the values representing the end of a cycle and the start of the next cycle does not jump abruptly. Some examples of such data are: * Hours of the day * Days of the week * Months in a year * Wind direction * Seasons * Navigation Directions The syntax to define an cyclical encoder is as follows: ``` cyclicalEncoderConfig: ``` There are 2 ways to encode the column: 1. By epoch time: Unix Epoch time is the number of seconds that have elapsed since January 1, 1970 (midnight UTC/GMT). By using this option, we assume that the time zone to encode in will be UTC. In order to use this option you only need to define the period of your cycle to encode. 2. By range: This defines the base range of floating point values representing a cycle. For example, one might define wind directions to be in the range of 0 to 360 degrees, although the actual value may be >360 or <0. To encode by **epoch time**, use the following syntax: ``` cyclicalEncoderConfig: byEpochTime: periodType: HOUR #HOUR, DAY, WEEK, MONTH, QUARTER, HALF, YEAR ``` Period type defines the time period of a cycle. For example, HOUR means that a new cycle begins every hour and DAY means that a new cycle begins every day. ***NOTE: If you choose to encode by epoch time, the granularity is per seconds. If you need different granularity, you can modify the values in the epoch time column accordingly or choose to encode by range.*** To encode by **range**, use the following syntax: ``` cyclicalEncoderConfig: byRange: min: FLOAT max: FLOAT ``` Do note that the min and max values are Float. The range is inclusive for the min and exclusive for the max, since in a cycle min and max will represent the same phase. For example, you can encode the days of a week in the range of \[1, 8), where 8 and 1 both represents the starting point of a cycle. You can then represent Monday 12am as 1 and Sunday 12pm as 7.5 and so on. See below for complete examples on how to declare a cyclical encoder: *By epoch time:* ``` -encoders: -name: payday_trend cyclicalEncoderConfig: byEpochTime: periodType: MONTH ``` *By range:* ``` -encoders: -name: wind_dir cyclicalEncoderConfig: byRange: min: 0 max: 360 ``` **Input/Output Examples**\ By epoch time: Period of a day | col | col\_x | col\_y | remarks | | ---------- | ------ | ------ | ------------------------ | | 1644278400 | 1 | 0 | 8 Feb 2022 00:00:00 UTC | | 1644300000 | 0 | 1 | 8 Feb 2022 06:00:00 UTC | | 1644451200 | -1 | 0 | 8 Feb 2022 12:00:00 UTC | | 1644343200 | 0 | -1 | 8 Feb 2022 18:00:00 UTC | | 1644364800 | 1 | 0 | 9 Feb 2022 00:00:00 UTC | | 1644451200 | 1 | 0 | 10 Feb 2022 00:00:00 UTC | By range: 0 to 360 (For example wind directions) | col | col\_x | col\_y | | --- | ------ | ------ | | 0 | 1 | 0 | | 90 | 0 | 1 | | 180 | -1 | 0 | | 270 | 0 | -1 | | 360 | 1 | 0 | | 420 | 0 | 1 | | -90 | 0 | -1 | To learn more about cyclical encoding, you may find this page useful: [Cyclical Encoding](https://towardsdatascience.com/cyclical-features-encoding-its-about-time-ce23581845ca) ### Autoload Autoload declares tables and variables that need to be loaded to standard transformer runtime from incoming request/response. This operation is only applicable for **upi\_v1** protocol. Below is specification of autoload ```yaml autoload: tableNames: - table_name_1 - table_name_2 variableNames: - var_name_1 - var_name_2 ``` `tableNames` and `variableNames` are fields that list table name and variables declaration. If `autoload` is part of `preprocess` pipeline, it will try to load those declared table and variables from request payload, otherwise it will load from model response payload. ## Transformation Stage In this stage, the standard transformers perform transformation to the tables created in the input stage so that its structure is suitable for the output. In the transformation stage, users operate mainly on tables and are provided with 2 transformation types: single table transformation and table join. Each transformation declared in this stage will be executed sequentially and all output/side effects from each transformation can be used in subsequent transformations. There are two types of transformations in standard transformer: \* Table Transformation \* Table Join ### Table Transformation Table transformation performs transformation to a single input table and creates a new table. The transformation performed to the table is defined within the “steps” field and executed sequentially. ``` tableTransformation: inputTable: # name of the input table outputTable: # name of the output table steps: # list of transformation steps, it will be executed sequentially - - ``` Following are the operation available for table transformation: #### Drop Column This operation will drop one or more column ``` tableTransformation: inputTable: myTable outputTable: myTransformedTable steps: - dropColumns: ["id"] ``` #### Select Column This operation will reorder and optionally drop non-selected column ``` tableTransformation: inputTable: myTable outputTable: myTransformedTable steps: - selectColumns: ["lat", "lon", "total_trip"] ``` #### Sort Operation This operation will sort the table using the defined column and ordering ``` tableTransformation: inputTable: myTable outputTable: myTransformedTable steps: - sort: - column: id order: ASC - column: total_trip order: DESC ``` #### Rename Columns This operation will rename one column into another ``` tableTransformation: inputTable: myTable outputTable: myTransformedTable steps: - renameColumns: "total_trip": "totalTrip" ``` #### Update Columns Adding column or modifying column in-place using expressions ``` tableTransformation: inputTable: myTable outputTable: myTransformedTable steps: - updateColumns: - column: "s2id" expression: S2ID(myTable.Col('lat'), myTable.Col('lon'), 12) - column: "col2" conditions: - rowSelector: myTable.Col('col1') * 2 > 10 expression: myTable.Col('col1') - default: expression: -1 ``` There are two ways to update columns: * Update all rows in the column. You need to specify `column`and `expression`. `column` determines which column to be updated and `expression` determines the value that will be used to update the column. Value produced by the `expression` must be a scalar or a series that has the same length as the other columns. Following the example:: ``` - updateColumns: - column: "customer_id" expression: "cust_1" # the value is scalar and will be broadcasted to all the row - column: "s2id" expression: S2ID(myTable.Col('lat'), myTable.Col('lon'), 12) # the value is array or series that the length should be the same with the rest of the columns in a table ``` * Update subset of rows in the columns given some row selector condition. For this users can set multiple `rowSelector` with `expression` and also default value if none of conditions are match. For example users have following table | customer\_id | customer\_age | total\_booking\_1w | | ------------ | ------------- | ------------------ | | 1234 | 60 | 8 | | 4321 | 23 | 4 | | 1235 | 17 | 4 | Users want to create new column `customer_segment` with certain rules: 1. Customer that older than 55, the `customer_segment` will be `retired` 2. Customer that has age between 30 - 55, the `customer_segment` will be `matured` 3. Customer that has age between 22 - 30, the `customer_segment` will be `productive` 4. Customer that has age < 22, the `customer_segment` will be `non-productive` Based on those rules we can translate this to standard transformer config: ``` tableTransformation: inputTable: myTable outputTable: myTransformedTable steps: - updateColumns: - column: "customer_segment" conditions: - rowSelector: myTable.Col('customer_age') > 55 expression: "retired" - rowSelector: myTable.Col('customer_age') >= 30 expression: "matured" - rowSelector: myTable.Col('customer_age') >= 22 expression: "productive" - default: expression: "non-productive" ``` All `rowSelector` conditions are working like `if else` statement. `rowSelector` condition must be returning boolean or series of boolean, `default` will be executed if none of the `rowSelector` conditions are matched. #### Filter Row Filter row is an operation that will filter rows in a table based on given condition. Suppose users have this following table | customer\_id | customer\_age | total\_booking\_1w | | ------------ | ------------- | ------------------ | | 1234 | 60 | 8 | | 4321 | 23 | 4 | | 1235 | 17 | 4 | and users want to show only records that have `total_booking_1w` less than 5. To achieve that users need to use `filterRow` operation like below configuration: ``` tableTransformation: inputTable: myTable outputTable: myTransformedTable steps: - filterRow: condition: myTable.Col('total_booking_1w') < 5 ``` #### Slice Row Slice row is an operation to slice a table based on start(lower bound) and end index(upper bound) that given by the user. The result includes starting index but excluding end index. Below is the example of this operation ``` tableTransformation: inputTable: myTable outputTable: myTransformedTable steps: - sliceRow: start: 0 end: 4 ``` Value of `start` end `end` can be null or negative. Following are the behaviour: * Null value of `start` means that `start` value is 0 * Null value of `end` means that `end` value is number of rows in a table * Negative value of `start` or `end` means that the value will be (`number of row` + `start`) or (`number of row` + `end`). Suppose you set `start` -5 and `end` -1 and number of row is 10, so `start` value will be 5 and `end` will be 9 #### Encode Column This operation will encode the specified columns with the specified encoder defined in the input step. ``` tableTransformation: inputTable: myTable outputTable: myTransformedTable steps: - encodeColumns: - columns: - vehicle - previous_vehicle encoder: vehicle_mapping ``` #### Scale Column This operation will scale a specified column using scalers. At the moment 2 types of scalers are available: * Standard Scaler * Min-max Scaler Standard Scaler In order to use a standard scaler, the mean and standard deviation (std) of the respective column to be scaled should be computed beforehand and provided in the specification. The syntax for scaling a column with a standard scaler is as follows: ``` tableTransformation: inputTable: myTable outputTable: myTransformedTable steps: - scaleColumns: - column: rank standardScalerConfig: mean: 0.5 std: 0.2 ``` Min-Max Scaler In order to use a min-max scaler, the minimum and maximum value for the column to scale to must be defined in the specification. The syntax for scaling a column with a min-max scaler is as follows: ``` tableTransformation: inputTable: myTable outputTable: myTransformedTable steps: - scaleColumns: - column: rating minMaxScalerConfig: min: 1 max: 5 ``` ### Join Operation This operation joins 2 tables, as defined by “leftTable” and “rightTable” parameters, into 1 output table given a join column and method of join. The join column must exist in both the input tables. The available method of join are: \* Left join \* Concat Column \* Cross join \* Inner Join \* Outer join \* Right join ``` tableJoin: leftTable: merchant_table rightTable: customer_table outputTable: merchant_customer_table how: LEFT # LEFT, INNER , RIGHT, CROSS, OUTER, CONCAT_COLUMN onColumn: merchant_id ``` ## Output Stage At this stage, both the preprocessing and postprocessing pipeline should create an output. The output of preprocessing pipeline will be used as the request payload to be sent as model request, whereas output of the postprocessing pipeline will be used as response payload to be returned to downstream service / client. There are 3 types of output specifications: * JSON Output. Applicable for **http\_json** protocol and both preprocess and postprocess output * UPIPreprocessOutput. Applicable only for **upi\_v1** protocol and preprocess output * UPIPostprocessOutput. Applicable only for **upi\_v1** protocol and postprocess output ### JSON Output - User-defined JSON template Users are given freedom to specify the transformer’s JSON output structure. The syntax is as follows: ``` output: - jsonOutput: jsonTemplate: baseJson: # Base JSON Template, the value can be "fromJson" or "fromTable" fromJson: # Copy JSON object pointed by the source and jsonPath jsonPath: # Path to JSON field to be copied from in the source JSON fromTable: # Create json payload from a table tableName: # Source table name format: # json output format, possible format are (based on https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.to_json.html): RECORD, VALUES, SPLIT fields: # list of JSON field to be included, fields defined here will override JSON field in the base JSON - fieldName : # field name fromJson: # Copy json field from RAW_REQUEST / MODEL_RESPONSE jsonPath: # json path of the field to be copied from source JSON payload fromTable: # Create json from a table tableName: # Source table name format: # json output format, possible format are (based on https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.to_json.html): RECORD, VALUES, SPLIT expression: # populate the field with value from result of an expression - fieldName: fields: # it's also possible to have a nested JSON field - ``` Similar to the table creation specification, users can specify the “baseJson” as the base json structure and override it using “fields” configuration. The field\_value above can be configured to retrieve from 3 sources: * From JSON * From Table * From Expression #### From JSON In the example below, “output” field will be set to the “predictions” field from the model response. ``` jsonOutput: jsonTemplate: fields: - fieldName: output fromJson: jsonPath: $.model_response.predictions ``` #### From Table Users can populate JSON fields using values from a table. The table can be rendered into 3 JSON formats: RECORD, VALUES, and SPLIT. Note that if “fromTable” is used as “baseJson” it will use the table name as the json field. For example, given following customerTable: | customer\_id | customer\_age | total\_booking\_1w | | ------------ | ------------- | ------------------ | | 1234 | 34 | 8 | | 4321 | 23 | 4 | | 1235 | 17 | 4 | Depending on the json format, it will render different result JSON * RECORD Format ``` outputStage: jsonOutput: jsonTemplate: fields: - fieldName: instances fromTable: tableName: customerTable format: RECORD ``` JSON Result: ``` { "instances" : [ [ { "customer_id" : 1234, "customer_age" : 34, "total_booking_1w": 8 }, { "customer_id" : 4321, "customer_age" : 23, "total_booking_1w": 4 }, { "customer_id" : 1235, "customer_age" : 17, "total_booking_1w": 4 } ] ] } ``` * VALUES Format ``` outputStage: jsonOutput: jsonTemplate: fields: - fieldName: instances fromTable: tableName: customerTable format: VALUES ``` JSON Result: ``` { "instances":[ [ [1234, 34, 8], [4321, 23, 4], [1235, 17, 4] ] ] } ``` * SPLIT Format ``` outputStage: jsonOutput: jsonTemplate: fields: - fieldName: instances fromTable: tableName: customerTable format: SPLIT ``` JSON Result: ``` { "instances" : { "data": [ [1234, 34, 8], [4321, 23, 4], [1235, 17, 4] ], "columns" : ["customer_id", "customer_age", "total_booking_1w"] } } ``` ### UPIPreprocessOutput UPIPreprocessOutput is output specification only for **upi\_v1** protocol and preprocess step. This output specification will create operation that convert defined tables to UPI request interface. Below is the specification ```yaml upiPreprocessOutput: predictionTableName: table_1 transformerInputTableNames: - input_table_1 - input_table_2 ``` This specification will convert content of `predictionTableName` into UPI table ``` message Table { string name = 1; repeated Column columns = 2; repeated Row rows = 3; } message Column { string name = 1; Type type = 2; } message Row { string row_id = 1; repeated Value values = 2; } message Value { double double_value = 1; int64 integer_value = 2; string string_value = 3; bool is_null = 10; } ``` and then set field `prediction_table` from this UPI Request interface ``` message PredictValuesRequest { Table prediction_table = 1; TransformerInput transformer_input = 4; string target_name = 2; repeated Variable prediction_context = 3; RequestMetadata metadata = 10; } ``` `transformerInputTableNames` are list of table names that will be converted into UPI Table. These values will be assigned into field `transformer_input`.`tables` field. The rest of the fields will be carried on from the incoming reques payload. ### UPIPostprocessOutput UPIPostprocessOutput is output specification only for **upi\_v1** protocol and postprocess step. This output specification will create operation that convert defined tables to UPI response interface. Below is the specification ```yaml upiPostprocessOutput: predictionResultTableName: table_name_1 ``` This specification will convert content of `predictionResultTableName` into UPI table and assigned it to field `prediction_result_table` in this UPI Response interface like below ``` message PredictValuesResponse { Table prediction_result_table = 1; string target_name = 2; repeated Variable prediction_context = 3; ResponseMetadata metadata = 10; } ``` The rest of the fields will be carried on from model predictor response ### Deploy Standard Transformer using Merlin UI Once you logged your model and it’s ready to be deployed, you can go to the model deployment page. Here’s the short video demonstrating how to configure the Standard Transformer: ![Configure Standard Transformer](/files/n11TPLBc7R4znfXq2hVt) 1. As the name suggests, you must choose **Standard Transformer** as Transformer Type. 2. The **Retrieval Table** panel will be displayed. This panel is where you configure the Feast Project, Entities, and Features to be retrieved. 1. The list of Feast Entity depends on the selected Feast Project 2. Similarly, the list of Feast Feature also depends on the configured entities 3. You can have multiple Retrieval Table that can retrieve a different kind of entities and features and enrich the request to your model at once. To add it, simply click `Add Retrieval Table`, and new Retrieval Table panel will be displayed and ready to be configured. 4. You can check the Transformer Configuration YAML specification by clicking `See YAML configuration`. You can copy and paste this YAML and use it for deployment using Merlin SDK. 1. To read more about Transformer Configuration specification, please continue reading. 5. You can also specify the advanced configuration. These configurations are separated from your model. 1. Request and response payload logging 2. Resource request (Replicas, CPU, and memory) 3. Environment variables (See supported environment variables below) ### Deploy Standard Transformer using Merlin SDK Make sure you are using the supported version of Merlin SDK. ```bash > pip install merlin-sdk -U > pip show merlin-sdk Name: merlin-sdk Version: 0.10.0 ... ``` You need to pass `transformer` argument to the `merlin.deploy()` function to enable and deploy your standard transformer. {% code title="standard\_transformer\_deployment.py" overflow="wrap" lineNumbers="true" %} ```python from merlin.resource_request import ResourceRequest from merlin.transformer import StandardTransformer # Specify the path to transformer config YAML file transformer_config_path = "transformer_config.yaml" # Create the transformer resources requests config resource_request = ResourceRequest(min_replica=0, max_replica=1, cpu_request="100m", memory_request="200Mi") # Create the transformer object transformer = StandardTransformer(config_file=transformer_config_path, enabled=True, resource_request=resource_request) # Deploy the model alongside the transformer endpoint = merlin.deploy(v, transformer=transformer) ``` {% endcode %} ### Standard Transformer Environment Variables Below are supported environment variables to configure your Transformer. | Name | Description | Default Value | | ------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------- | | `LOG_LEVEL` | Set the logging level for internal system. It doesn’t effect the request-response logging. Supported value: DEBUG, INFO, WARNING, ERROR. | INFO | | `FEAST_FEATURE_STATUS_MONITORING_ENABLED` | Enable metrics for the status of each retrieved feature. | false | | `FEAST_FEATURE_VALUE_MONITORING_ENABLED` | Enable metrics for the summary value of each retrieved feature. | false | | `FEAST_BATCH_SIZE` | Maximum number of entities values that will be passed as a payload to feast. For example if you want to get features from 75 entities values and FEAST\_BATCH\_SIZE is set to 50, then there will be 2 calls to feast, first call request features from 50 entities values and next call will request features from 25 entities values. | 50 | | `FEAST_CACHE_ENABLED` | Enable cache response of feast request | true | | `FEAST_CACHE_TTL` | Time to live cached features, if TTL is reached the cached will be expired. The value has format like this \[$number]\[$unit] e.g 60s, 10s, 1m, 1h | 60s | | `CACHE_SIZE_IN_MB` | Maximum capacity of cache from allocated memory. Size is in MB | 100 | | `FEAST_REDIS_DIRECT_STORAGE_ENABLED` | Enable features retrieval by querying direcly from redis | false | | `FEAST_REDIS_POOL_SIZE` | Number of redis connection established in one replica of standard transformer | 10 | | `FEAST_REDIS_READ_TIMEOUT` | Timeout for read commands from redis. If reached commands will fails | 3s | | `FEAST_REDIS_WRITE_TIMEOUT` | Timeout for write commands to redis. If reached commands will fails | 3s | | `FEAST_BIGTABLE_DIRECT_STORAGE_ENABLED` | Enable features retrieval by querying direcly from bigtable | false | | `FEAST_BIGTABLE_POOL_SIZE` | Number of bigtable grpc connections established in one replica of standard transformer | | | `FEAST_TIMEOUT` | Timeout of feast request | 1s | | `FEAST_HYSTRIX_MAX_CONCURRENT_REQUESTS` | Maximum concurrent requests when calling feast | 100 | | `FEAST_HYSTRIX_REQUEST_VOLUME_THRESHOLD` | Threshold of error percentage, once breached circuit will be open | 100 | | `FEAST_HYSTRIX_SLEEP_WINDOW` | Sleep window is duration of rejecting calling feast once the circuit is open | 1s | | `FEAST_HYSTRIX_ERROR_PERCENT_THRESHOLD` | Threshold of number of request to model predictor | 25 | | `FEAST_SERVING_KEEP_ALIVE_ENABLED` | Flag to enable feast keep alive | true | | `FEAST_SERVING_KEEP_ALIVE_TIME` | Duration of interval between keep alive PING | 60s | | `FEAST_SERVING_KEEP_ALIVE_TIMEOUT` | Duration of PING that considered as TIMEOUT | 5s | | `MERLIN_DISABLE_LIVENESS_PROBE` | Disable liveness probe of transformer if set to true | | | `MODEL_TIMEOUT` | Timeout duration of model prediction | 1s | | `MODEL_HYSTRIX_MAX_CONCURRENT_REQUESTS` | Maximum concurrent requests when calling model predictor | 100 | | `MODEL_HYSTRIX_ERROR_PERCENTAGE_THRESHOLD` | Threshold of error percentage, once breached circuit will be open | 25 | | `MODEL_HYSTRIX_REQUEST_VOLUME_THRESHOLD` | Threshold of number of request to model predictor | 100 | | `MODEL_HYSTRIX_SLEEP_WINDOW_MS` | Sleep window is duration of rejecting calling model predictor once the circuit is open | 10 | | `MODEL_GRPC_KEEP_ALIVE_ENABLED` | Flag to enable UPI\_V1 model predictor keep alive | false | | `MODEL_GRPC_KEEP_ALIVE_TIME` | Duration of interval between keep alive PING | 60s | | `MODEL_GRPC_KEEP_ALIVE_TIMEOUT` | Duration of PING that considered as TIMEOUT | 5s | # Standard Transformer Expressions Standard Transformer provides several built-in functions that are useful for common ML use-cases. These built-in functions are accessible from within expression context. | Categories | Functions | | ---------- | ----------------------------------------------------------- | | Geospatial | [Geohash](#geohash) | | Geospatial | [S2ID](#s2id) | | Geospatial | [HaversineDistance](#haversinedistance) | | Geospatial | [HaversineDistanceWithUnit](#haversinedistancewithunit) | | Geospatial | [PolarAngle](#polarangle) | | Geospatial | [GeohashDistance](#geohashdistance) | | Geospatial | [GeohashAllNeighbors](#geohashallneighbors) | | Geospatial | [GeohashNeighborForDirection](#geohashneighborfordirection) | | JSON | [JsonExtract](#jsonextract) | | Statistics | [CumulativeValue](#cumulativevalue) | | Time | [Now](#now) | | Time | [DayOfWeek](#dayofweek) | | Time | [IsWeekend](#isweekend) | | Time | [FormatTimestamp](#formattimestamp) | | Time | [ParseTimestamp](#parsetimestamp) | | Time | [ParseDateTime](#parsedatetime) | | Series | [Get](#get) | | Series | [IsIn](#isin) | | Series | [StdDev](#stddev) | | Series | [Mean](#mean) | | Series | [Median](#median) | | Series | [Max](#max) | | Series | [MaxStr](#maxstr) | | Series | [Min](#min) | | Series | [MinStr](#minstr) | | Series | [Quantile](#quantile) | | Series | [Sum](#sum) | | Series | [Flatten](#flatten) | | Series | [Unique](#unique) | ## Geospatial ### Geohash Geohash calculates geohash of `latitude` and `longitude` with the given `precision`. #### Input | Name | Description | | --------- | ----------------------------------------------------------------- | | Latitude | Latitude of the object, in form of JSONPath, array, or variable. | | Longitude | Longitude of the object, in form of JSONPath, array, or variable. | | Precision | Character precision in integer. | #### Output `Geohash of location with the given precision.` #### Example ``` Input: { "latitude": 1.0, "longitude": 2.0 } Standard Transformer Config: variables: - name: geohash expression: Geohash("$.latitude", "$.longitude", 12) Output: `"s01mtw037ms0"` ``` ### S2ID S2ID calculates S2ID cell of `latitude` and `longitude` with the given `level`. #### Input | Name | Description | | --------- | ----------------------------------------------------------------- | | Latitude | Latitude of the object, in form of JSONPath, array, or variable. | | Longitude | Longitude of the object, in form of JSONPath, array, or variable. | | Level | S2ID level in integer. | #### Output `S2ID cell of the location in certain level.` #### Example ``` Input: { "latitude": 1.0, "longitude": 2.0 } Standard Transformer Config: variables: - name: s2id expression: S2ID("$.latitude", "$.longitude", 12) Output: `"1154732743855177728"` ``` ### HaversineDistance HaversineDistance calculates Haversine distance of two points (given by their latitude and longitude). #### Input | Name | Description | | ----------- | ----------------------------------------------------------------------- | | Latitude 1 | Latitude of the first point, in form of JSONPath, array, or variable. | | Longitude 1 | Longitude of the first point, in form of JSONPath, array, or variable. | | Latitude 2 | Latitude of the second point, in form of JSONPath, array, or variable. | | Longitude 2 | Longitude of the second point, in form of JSONPath, array, or variable. | #### Output `The haversine distance between 2 points in kilometer.` #### Example ``` Input: { "pickup": { "latitude": 1.0, "longitude": 2.0 }, "dropoff": { "latitude": 1.2, "longitude": 2.2 } } Standard Transformer Config: variables: - name: haversine_distance expression: HaversineDistance("$.pickup.latitude", "$.pickup.longitude", "$.dropoff.latitude", "$.dropoff.longitude") ``` ### HaversineDistanceWithUnit HaversineDistanceWithUnit calculates Haversine distance of two points (given by their latitude and longitude) and given the distance unit #### Input | Name | Description | | ------------- | ----------------------------------------------------------------------- | | Latitude 1 | Latitude of the first point, in form of JSONPath, array, or variable. | | Longitude 1 | Longitude of the first point, in form of JSONPath, array, or variable. | | Latitude 2 | Latitude of the second point, in form of JSONPath, array, or variable. | | Longitude 2 | Longitude of the second point, in form of JSONPath, array, or variable. | | Distance Unit | Unit of distance measurement, supported unit `km` and `m` | #### Output `The haversine distance between 2 points.` #### Example ``` Input: { "pickup": { "latitude": 1.0, "longitude": 2.0 }, "dropoff": { "latitude": 1.2, "longitude": 2.2 } } Standard Transformer Config: variables: - name: haversine_distance expression: HaversineDistanceWithUnit("$.pickup.latitude", "$.pickup.longitude", "$.dropoff.latitude", "$.dropoff.longitude", "m") ``` ### PolarAngle PolarAngle calculates polar angles between two points (given by their latitude and longitude) in radian. #### Input | Name | Description | | ----------- | ----------------------------------------------------------------------- | | Latitude 1 | Latitude of the first point, in form of JSONPath, array, or variable. | | Longitude 1 | Longitude of the first point, in form of JSONPath, array, or variable. | | Latitude 2 | Latitude of the second point, in form of JSONPath, array, or variable. | | Longitude 2 | Longitude of the second point, in form of JSONPath, array, or variable. | #### Output `The polar angles between 2 points in radian.` #### Example ``` Input: { "pickup": { "latitude": 1.0, "longitude": 2.0 }, "dropoff": { "latitude": 1.2, "longitude": 2.2 } } Standard Transformer Config: variables: - name: polar_angle expression: PolarAngle("$.pickup.latitude", "$.pickup.longitude", "$.dropoff.latitude", "$.dropoff.longitude") ``` ### GeohashDistance GeohashDistance will calculate haversine distance between two geohash. It will convert a geohash into the center point (latitude, longitude) of that geohash and calculate haversine distance based on that point. #### Input | Name | Description | | ------------- | --------------------------------------------------------- | | Geohash 1 | First geohash, in form of JSONPath, array | | Geohash 2 | Second geohash, in form of JSONPath, array | | Distance Unit | Unit measurement of distance, supported unit `km` and `m` | #### Output `Haversine Distance between two geohash calculated from the center point of that geohash` #### Example ``` Input: { "pickup_geohash": "qqgggnwxx", "dropoff_geohash": "qqgggnweb" } Standard Transformer Config: variables: - name: geohash_distance expression: GeohashDistance("$.pickup_geohash", "$.dropoff_geohash", "m") ``` ### GeohashAllNeighbors GeohashAllNeighbors will find all neighbors of geohash from all directions #### Input | Name | Description | | --------- | ------------------------------------ | | Geohash 1 | Geohash , in form of JSONPath, array | #### Output `List of neighbors of given geohash` #### Example ``` Input: { "pickup_geohash": "qqgggnwxx", "dropoff_geohash": "qqgggnweb" } Standard Transformer Config: variables: - name: geohash_distance expression: GeohashAllNeighbors("$.pickup_geohash") ``` ### GeohashNeighborForDirection GeohashNeighborForDirection will find a neighbor of geohash given the direction #### Input | Name | Description | | --------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Geohash 1 | Geohash , in form of JSONPath, array | | Direction | Direction of that neighbor relatively from geohash. List of accepted direction `north`, `northeast`, `northwest`, `south`, `southeast`, `southwest`, `west`, `east` | #### Output `Neighbor of given geohash` #### Example ``` Input: { "pickup_geohash": "qqgggnwxx", "dropoff_geohash": "qqgggnweb" } Standard Transformer Config: variables: - name: geohash_distance expression: GeohashNeighborForDirection("$.pickup_geohash", "north") ``` ## JSON ### JsonExtract Given a JSON string as value, you can use JsonExtract to extract JSON value from that JSON string. #### Input | Name | Description | | ----------------- | ------------------------------------------------------------------------------------ | | Parent's JSONPath | Path to JSON key that its value is a JSON string to be extracted. | | Nested's JSONPath | Path to JSON key inside of JSON string which extracted from Parent's JSONPath above. | #### Output `JSON value within a JSON string pointed by the first JSONPath argument.` #### Example ``` Input: { "details": "{\"merchant_id\": 9001}" } Standard Transformer Config: variables: - name: merchant_id valueType: STRING expression: JsonExtract("$.details", "$.merchant_id") Output: `"9001"` ``` ## Statistics ### CumulativeValue CumulativeValue is a function that accumulates values based on the index and its predecessors. E.g., `[1, 2, 3] => [1, 1+2, 1+2+3] => [1, 3, 6]`. #### Input | Name | Description | | ------ | ----------------- | | Values | Array of numbers. | #### Output `Array of cumulative values.` #### Example ``` Input: { "fares": [10000, 20000, 50000] } Standard Transformer Config: variables: - name: cumulative_fares expression: CumulativeValue($.fares) Output: `[10000, 30000, 80000]` ``` ## Time ### Now Return current local timestamp. #### Input `None` #### Output `Current local timestamp.` #### Example ``` Standard Transformer Config: variables: - name: currentTime expression: Now() ``` ### DayOfWeek Return number representations of the day in a week, given the timestamp and timezone. SUNDAY(0), MONDAY(1), TUESDAY(2), WEDNESDAY(3), THURSDAY(4), FRIDAY(5), SATURDAY(6). #### Input | Name | Description | | --------- | ------------------------------------------------------------------------------------------------ | | Timestamp | Unix timestamp value in integer or string format. It accepts JSONPath, arrays, or variable. | | Timezone | Timezone value in string. For example, `Asia/Jakarta`. It accepts JSONPath, arrays, or variable. | #### Output `Day number.` #### Example ``` Input: { "timestamp": "1637605459" } Standard Transformer Config: variables: - name: day_of_week expression: DayOfWeek("$.timestamp", "Asia/Jakarta") Output: `2` ``` ### IsWeekend Return 1 if given timestamp is weekend (Saturday or Sunday), otherwise 0. #### Input | Name | Description | | --------- | ------------------------------------------------------------------------------------------------ | | Timestamp | Unix timestamp value in integer or string format. It accepts JSONPath, arrays, or variable. | | Timezone | Timezone value in string. For example, `Asia/Jakarta`. It accepts JSONPath, arrays, or variable. | #### Output `1 if weekend, 0 if not` #### Example ``` Input: { "timestamp": "1637445044", "timezone": "Asia/Jakarta" } Standard Transformer Config: variables: - name: is_weekend expression: IsWeekend("$.timestamp", "$.timezone") Output: `1` ``` ### FormatTimestamp FormatTimestamp converts timestamp in given location into formatted date time string. #### Input | Name | Description | | --------- | --------------------------------------------------------------------------------------------------------- | | Timestamp | Unix timestamp value in integer or string format. It accepts JSONPath, arrays, or variable. | | Timezone | Timezone value in string. For example, `Asia/Jakarta`. It accepts JSONPath, arrays, or variable. | | Format | Targetted date time format. It follows Golang date time format (). | #### Output `Date time.` #### Examples ``` Input: { "timestamp": "1637691859" } Standard Transformer Config: variables: - name: datetime expression: FormatTimestamp("$.timestamp", "Asia/Jakarta", "2006-01-02") Output: `"2021-11-24"` ``` ### ParseTimestamp ParseTimestamp converts timestamp in integer or string format to time. #### Input | Name | Description | | --------- | ------------------------------------------------------------------------------------------- | | Timestamp | Unix timestamp value in integer or string format. It accepts JSONPath, arrays, or variable. | #### Output `Parsed timestamp.` #### Examples ``` Input: { "timestamp": "1619541221" } Standard Transformer Config: variables: - name: parsed_timestamp expression: ParseTimestamp("$.timestamp") Output: `"2021-04-27 16:33:41 +0000 UTC"` ``` ### ParseDateTime ParseDateTime converts datetime given with specified format layout (e.g. RFC3339) into time. #### Input | Name | Description | | --------- | ----------------------------------------------------------------------------------------------------- | | Date time | Date time value in string format. It accepts JSONPath, arrays, or variable. | | Timezone | Timezone value in string. For example, `Asia/Jakarta`. It accepts JSONPath, arrays, or variable. | | Format | Date time input format. It follows Golang date time format (). | #### Output `Parsed date time.` #### Examples ``` Input: { "datetime": "2021-11-30 15:00:00", "location": "Asia/Jayapura" } Standard Transformer Config: variables: - name: parsed_datetime expression: ParseDateTime("$.datetime", "$.location", "2006-01-02 15:04:05") Output: `"2021-11-30 15:00:00 +0900 WIT"` ``` ## Series Expression Series expression is function that can be invoked by series (column) values in a table ### Get `Get` will retrieve a row in series based on the given index #### Input | Name | Description | | ----- | ------------------------------ | | Index | Position of rows starts with 0 | #### Output Single series row #### Examples Suppose users have table `yourTableName` | restaurant\_id | avg\_order\_1\_day | avg\_cancellation\_rate\_30\_day | | -------------- | ------------------ | -------------------------------- | | 1 | 2000 | 0.02 | | 2 | 3000 | 0.005 | | 3 | 4000 | 0.006 | Users try to retrieve index 2 for series `avg_order_1_day` Standard Transformer Config: ``` variables: - name: total_order_1_day expression: yourTableName.Col("avg_order_1_day").Get(2) ``` Output: 4000 ### IsIn `IsIn` checks whether value in a row is part of the given array, the result will be a new series that has boolean type #### Input | Name | Description | | ---------- | -------------- | | Comparator | Array of value | #### Output New Series that has boolean type and same dimension with original series #### Examples Suppose users have table `yourTableName` | restaurant\_id | avg\_order\_1\_day | avg\_cancellation\_rate\_30\_day | | -------------- | ------------------ | -------------------------------- | | 1 | 2000 | 0.02 | | 2 | 3000 | 0.005 | | 3 | 4000 | 0.006 | Standard Transformer Config: ``` variables: - name: bool_series expression: yourTableName.Col("avg_order_1_day").IsIn([2000, 3000]) ``` Output: | bool\_series | | ------------ | | true | | true | | false | ### StdDev `StdDev` is a function to calculate standard deviation from series values. The output will be single value #### Input No Input #### Output Single value with float type #### Examples Suppose users have table `yourTableName` | restaurant\_id | avg\_order\_1\_day | avg\_cancellation\_rate\_30\_day | | -------------- | ------------------ | -------------------------------- | | 1 | 2000 | 0.02 | | 2 | 3000 | 0.005 | | 3 | 4000 | 0.006 | Standard Transformer Config: ``` variables: - name: std_dev expression: yourTableName.Col("avg_cancellation_rate_30_day").StdDev() ``` Output: 0.0068475461947247 ### Mean `Mean` is a function to calculate mean value from series values. The output will be single value #### Input No Input #### Output Single value with float type #### Examples Suppose users have table `yourTableName` | restaurant\_id | avg\_order\_1\_day | avg\_cancellation\_rate\_30\_day | | -------------- | ------------------ | -------------------------------- | | 1 | 2000 | 0.02 | | 2 | 3000 | 0.005 | | 3 | 4000 | 0.006 | Standard Transformer Config: ``` variables: - name: mean expression: yourTableName.Col("avg_order_1_day").Mean() ``` Output: 3000 ### Median `Median` is a function to calculate median value from series values. The output will be single value #### Input No Input #### Output Single value with float type #### Examples Suppose users have table `yourTableName` | restaurant\_id | avg\_order\_1\_day | avg\_cancellation\_rate\_30\_day | | -------------- | ------------------ | -------------------------------- | | 1 | 2000 | 0.02 | | 2 | 3000 | 0.005 | | 3 | 4000 | 0.006 | Standard Transformer Config: ``` variables: - name: median expression: yourTableName.Col("avg_order_1_day").Median() ``` Output: 3000 ### Max `Max` is a function to find max value from series values. The output will be single value #### Input No Input #### Output Single value with float type #### Examples Suppose users have table `yourTableName` | restaurant\_id | avg\_order\_1\_day | avg\_cancellation\_rate\_30\_day | | -------------- | ------------------ | -------------------------------- | | 1 | 2000 | 0.02 | | 2 | 3000 | 0.005 | | 3 | 4000 | 0.006 | Standard Transformer Config: ``` variables: - name: max expression: yourTableName.Col("avg_order_1_day").Max() ``` Output: 4000 ### MaxStr `MaxStr` is a function to find max value from series values. The output will be single value in string type #### Input No Input #### Output Single value with string type #### Examples Suppose users have table `yourTableName` | restaurant\_id | avg\_order\_1\_day | avg\_cancellation\_rate\_30\_day | | -------------- | ------------------ | -------------------------------- | | 1 | 2000 | 0.02 | | 2 | 3000 | 0.005 | | 3 | 4000 | 0.006 | Standard Transformer Config: ``` variables: - name: max_str expression: yourTableName.Col("avg_order_1_day").MaxStr() ``` Output: "4000" ### Min `Min` is a function to find minimum value from series values. The output will be single value in float type #### Input No Input #### Output Single value with float type #### Examples Suppose users have table `yourTableName` | restaurant\_id | avg\_order\_1\_day | avg\_cancellation\_rate\_30\_day | | -------------- | ------------------ | -------------------------------- | | 1 | 2000 | 0.02 | | 2 | 3000 | 0.005 | | 3 | 4000 | 0.006 | Standard Transformer Config: ``` variables: - name: min expression: yourTableName.Col("avg_order_1_day").Min() ``` Output: 2000 ### MinStr `MinStr` is a function to find minimum value from series values. The output will be single value in string type #### Input No Input #### Output Single value with string type #### Examples Suppose users have table `yourTableName` | restaurant\_id | avg\_order\_1\_day | avg\_cancellation\_rate\_30\_day | | -------------- | ------------------ | -------------------------------- | | 1 | 2000 | 0.02 | | 2 | 3000 | 0.005 | | 3 | 4000 | 0.006 | Standard Transformer Config: ``` variables: - name: min_str expression: yourTableName.Col("avg_order_1_day").MinStr() ``` Output: "2000" ### Quantile `Quantile` is a function to returns the sample of x such that x is greater than or equal to the fraction p of samples #### Input Fraction in float type #### Output Single value with float type #### Examples Suppose users have table `yourTableName` | rank | | ---- | | 1 | | 2 | | 3 | | 4 | | 5 | | 6 | | 7 | | 8 | | 9 | | 10 | Standard Transformer Config: ``` variables: - name: quantile_0.9 expression: yourTableName.Col("rank").Quantile(0.9) ``` Output: 9 ### Sum `Sum` is a function to sum all the values in the seriess. The output will be single value in float type #### Input No Input #### Output Single value with float type #### Examples Suppose users have table `yourTableName` | restaurant\_id | avg\_order\_1\_day | avg\_cancellation\_rate\_30\_day | | -------------- | ------------------ | -------------------------------- | | 1 | 2000 | 0.02 | | 2 | 3000 | 0.005 | | 3 | 4000 | 0.006 | Standard Transformer Config: ``` variables: - name: sum expression: yourTableName.Col("avg_order_1_day").Sum() ``` Output: 9000 ### Flatten `Flatten` is a function to flatten all values in a series, this is suitable for series that has list type, for non list the result will be the same with the original seriess #### Input No Input #### Output New Series that the value already flatten #### Examples Suppose users have table `yourTableName` | restaurant\_id | nearby\_restaurant\_ids | | -------------- | ----------------------- | | 1 | \[2, 3, 4] | | 2 | \[4, 5, 6] | | 3 | \[7, 8, 9] | Standard Transformer Config: ``` variables: - name: restaurant_ids expression: yourTableName.Col("nearby_restaurant_ids").Flatten() ``` Output: | restaurant\_ids | | --------------- | | 2 | | 3 | | 4 | | 4 | | 5 | | 6 | | 7 | | 8 | | 9 | ### Unique `Unique` is a function to return all values without duplication. #### Input No Input #### Output New Series that has unique value for each row #### Examples Suppose users have table `yourTableName` | restaurant\_id | rating | | -------------- | ---------- | | 1 | \[2, 2, 4] | | 2 | \[4, 5, 4] | | 1 | \[2, 2, 4] | Standard Transformer Config: ``` variables: - name: unique_restaurant_id expression: yourTableName.Col("restaurant_id").Unique() ``` Output: | unique\_restaurant\_id | | ---------------------- | | 1 | | 2 | ``` variables: - name: rating expression: yourTableName.Col("rating").Unique() ``` Output: | rating | | ---------- | | \[2, 2, 4] | | \[4, 5, 4] | # Standard Transformer UPI {% hint style="info" %} This guide assumes you have experience using standard transformer and are familiar with UPI contract. You can refer to to get details on the contract. {% endhint %} There are 2 key differences in Standard Transformer when it’s deployed using UPI protocol: 1. Autoload Feature 2. Separate Output Operation for Pre-process and Post-process ## Autoload Feature Autoload feature is the primary mechanism for importing values in the request payload as variables or tables into standard transformer. Previously, it is done by using JSONPath query in the HTTP mode. For example in HTTP model, if you want to declare a rating variable that should use value from `user_rating` field of the below incoming request ```json { "user_id": 12345, "user_rating": 4.9, "user_name": "jon_doe" } ``` then you have to declare following configuration in the input configuration of the standard transformer which will extract the data from incoming request. The drawback of this approach is that it could be complicated for a more complex request payload and for large number of variable/table to be imported. ```yaml - variables: - name: rating jsonPathConfig: jsonPath: $.user_rating defaultValue: -1 valueType: FLOAT ``` You can avoid it altogether by using autoload feature in UPI. To do so: #### Store the variable/table in `prediction_table` or `transformer_input` field of the `PredictValuesRequest` For example, when using Python SDK, you can do so by following code. In below example, we are storing `user_rating` as variable and `customer_df` as `customer_table` in `transformer_input`, as well as sending the `prediction_df` as `prediction_table`. {% code title="upi\_standard\_transformer\_deployment.py" overflow="wrap" lineNumbers="true" %} ```python from caraml.upi.v1 import type_pb2, upi_pb2_grpc, upi_pb2, variable_pb2 request = upi_pb2.PredictValuesRequest( # ... prediction_table=df_to_table(predict_df, "prediction_table"), transformer_input=upi_pb2.TransformerInput( variables=[ variable_pb2.Variable(name="user_rating", type=type_pb2.TYPE_DOUBLE, double_value=5.0), ], tables=[df_to_table(customer_df, "customer_table")] ), # ... ) ``` {% endcode %} #### Add autoload feature in the standard transformer config. Add all variables and tables that are going to be imported in the standard transformer. In below example we are importing `prediction_table`, `customer_table` , and `user_rating` that was sent by the client. ![UPI Autoloading](/files/ScHFcqji3G5663YrgqQ0) Which will add following config ```yaml transformerConfig: preprocess: inputs: - autoload: tableNames: - customer_table variableNames: - user_rating postprocess: {} ``` Note that, the table created using UPI autoload will have additional column `row_id` which will store the `row_ids` value of the associated table. The imported tables and variables from UPI Autoload can then be used for downstream transformation in the standard transformer’s preprocess and post-process. ## Preprocess & Postprocess Output Standard transformer’s preprocessing output in UPI mode must satisfy `PredictValuesRequest` structure. You can populate `prediction_table` field and tables in transformer\_inputs fields of the `PredictValuesRequest` that will be sent to model by defining its source tables. The source tables must be tables that have been declared in preprocessing pipeline. Example below shows a preprocessing pipeline which join prediction\_table and sample\_table to produce preprocessed\_table , and then use the preprocessed\_table as the prediction\_table of the `PredictValuesRequest` that is sent to model. ![UPI Standard Transformer Preprocessing Output](/files/D9yQT2OVi5DN6l5cbNEG) ```yaml transformerConfig: preprocess: inputs: - autoload: tableNames: - prediction_table variableNames: - sample_table transformations: - tableJoin: leftTable: prediction_table rightTable: sample_table outputTable: preprocessed_table how: LEFT onColumns: - row_id outputs: - upiPreprocessOutput: predictionTableName: preprocessed_table transformerInputTableNames: [] postprocess: {} ``` Similarly, post-processing output in UPI mode must satisfy `PredictValuesResponse`. You can populate prediction\_result\_table of the `PredictValuesResponse` that will be sent back to client by defining its source table. The source table can be a table declared both in preprocessing and post-processing pipeline. {% hint style="info" %} When pre-processing or post-processing pipeline is not defined, standard transformer will simply forward the request/response to its receiver. {% endhint %} # Custom Transformer In 0.8 release, Merlin adds support to the Custom Transformer deployment. This transformer type enables the users to deploy their own pre-built Transformer service. The user should develop, build, and publish their own Transformer Docker image. Similar to Standard Transformer, users can configure Custom Transformer from UI and SDK. The difference is instead of specifying the standard transformer configuration, users configure the Docker image and the command and arguments to run it. ### Deploy Custom Transformer using Merlin UI 1. As the name suggests, you must choose Custom Transformer as Transformer Type. 2. Specify the Docker image registry and name. 1. You need to push your Docker image into supported registries: public DockerHub repository and private GCR repository. 3. If your Docker image needs command or arguments to start, you can specify them on related input form. 4. You can also specify the advanced configuration. These configurations are separated from your model. 1. Request and response payload logging 2. Resource request (Replicas, CPU, and memory) 3. Environment variables ### Deploy Custom Transformer using Merlin SDK {% code title="custom\_transformer\_deployment.py" overflow="wrap" lineNumbers="true" %} ```python from merlin.resource_request import ResourceRequest from merlin.transformer import Transformer # Create the transformer resources requests config resource_request = ResourceRequest(min_replica=0, max_replica=1, cpu_request="100m", memory_request="200Mi") # Create the transformer object transformer = Transformer("gcr.io//", resource_request=resource_request) # Deploy the model alongside the transformer endpoint = merlin.deploy(v, transformer=transformer) ``` {% endcode %} # Deleting a Model ## Model Version Deletion A Merlin model version can be deleted only if it is not serving any endpoints and does not have any deployed endpoints or, if the base model is of the `pyfunc_v2` type, the model version must not have any active prediction jobs. Deleting a model version will result in the purging of the model version and its related entities, such as endpoints or prediction jobs, from the Merlin database. This action is **irreversible**. Model versions with related active prediction jobs or endpoints can not be deleted. ### Model Version Deletion via the SDK To delete a Model Version, you can call the `delete_model_version()` function from Merlin Python SDK. ```python merlin.set_project("test-project") merlin.set_model('test-model') version = merlin.active_model().get_version(id_version) version.delete_model_version() ``` ### Model Version Deletion via the UI To delete a model version from the UI, you can access the delete button directly on the model version list page. The dialog will provide information about entities that are blocking the deletion process or will be deleted along with the model version. * If the model version does not have any associated entities, a dialog like the one below will be displayed: ![Delete Model Version without linked entites](/files/jIGAY5xPnrWFvHvev620) * If the model version has any associated active entities, a dialog like the one below (showing the entities blocking the deletion process) will be displayed: ![Delete Model Version with linked active entites](/files/RP7H6Uh3EtVzWQq7tsYk) * If the model version has any associated inactive entities, a dialog like the one below (showing which entities will get deleted along with the deletion process) will be displayed: ![Delete Model Version with linked inactive entites](/files/qRDBPfO4kvR2uJHYfdon) ## Model Deletion {% hint style="info" %} This feature is currently behind a toggle and may or may not be enabled on the Merlin controller, by the maintainers. {% endhint %} A Merlin model can be deleted only if it is not serving any endpoints and does not have any deployed model versions or, if the model is of the `pyfunc_v2` type, none of its model versions must not have any active prediction jobs. Deleting a model will result in the purging of all the model versions associated with it, as well as related entities such as endpoints or prediction jobs (applicable for models of the `pyfunc_v2` type) from the Merlin database. This action is **irreversible**. A model with model versions that have any active prediction jobs or endpoints cannot be deleted. ### Model Deletion Via the SDK To delete a Model, you can call the `delete_model()` function from the Merlin Python SDK. ```python merlin.set_project("test-project") merlin.set_model('test-model') model = merlin.active_model() model.delete_model() ``` ### Model Deletion via the UI To delete a model from the UI, you can access the delete button directly on the model list page. The dialog will provide information about any entities that are blocking the deletion process. * If the model does not have any associated entities, a dialog like the one below will be displayed: ![Delete Model without linked entites](/files/rI58jKxqw6rJuQp8mBLB) * If the model has any associated active entities, a dialog like the one below will be displayed: ![Delete Model with linked active entites](/files/B9hOvuEzoPWanYSr2S2i) # Configuring Alerts {% hint style="info" %} This feature is currently behind a toggle and may or may not be enabled on the Merlin controller, by the maintainers. {% endhint %} Merlin uses a GitOps based alerting mechanism. Alerts can be configured for a model, on the Model Endpoint (i.e., for the model version that is in the 'Serving' state), from the models list UI. ![Configure Alerts on Model Endpoint](/files/Etac64FY0AZOtF92RgZA) ## Metrics Alerting based on the following metrics are supported. For all metrics below, the transformer metrics, if exists, will also be taken into account. * **Throughput:** This alert is triggered when the number of requests per second received by the model is lower than the threshold. * **Latency:** This alert is triggered when the latency of model response time is higher than the threshold. * **Error Rate:** This alert is triggerred when the percentage of erroneous responses from the model is more than the threshold. * **CPU:** This alert is triggered when the percentage of CPU utilization is more than the threshold. * **Memory:** This alert is triggered when the percentage of memory utilization is more than the threshold. ![Alert Configuration](/files/eGwtrXuNITixKfTgpf1N) # Batch Prediction The batch prediction job will be executed as a Spark Application running in a Spark cluster on top of Kubernetes. ## Prediction Job Prediction Job is the resource introduced in Merlin for executing batch prediction. A Prediction Job is owned by the corresponding Model Version. One Model Version can have several Prediction Jobs and it maintains the history of all jobs ever created. Prediction Job has several important properties: 1. **Id**: Unique ID of the prediction job 2. **Model / Model version**: Reference to the model version from which the prediction job is created 3. **Config**: config will contain the source, sink, secret configuration of the prediction job. It could also contain additional config for resource requests or spark-specific configuration 4. **State**: Current state of the prediction job (see lifecycle section) 5. **Error**: Detailed error message if the prediction job is unsuccessful 6. **Logs**: Link to the log location 7. **Monitoring URL**: Link to the monitoring dashboard ## Lifecycle Prediction Job has several state during its lifetime: 1. **Pending**: Prediction job is in this state once it is created / submitted. It will enter the running state if the spark application is started successfully, otherwise it will enter a failed state. 2. **Running**: Prediction jobs will move to the running state once the underlying spark application for executing the prediction job is created. The prediction job will be in this state until the spark application is completed (in which case it moves to completed state) or failed (in which case the prediction job entered failed state). Users can manually stop the prediction job and it will enter the terminating state. 3. **Completed**: Prediction job enter the completed state if it’s completed successfully 4. **Failed**: Any kind of failure preventing the prediction job not being able to complete will make it enters the failed state. 5. **Terminating**: Prediction jobs enter a terminating state if a user manually cancels a pending/running prediction job. 6. **Terminated**: Once the termination process is completed the prediction job will enter the terminated state. ![Prediction Job Lifecycle](/files/5BtfTRX7mN1dLGKwckdl) ## Creating Secret/Service Account To be able to run a Prediction Job you’ll need a service account and store the key inside the MLP Project using secret management API. The service account must have following authorization: 1. BigQuery Job User (`roles/bigquery.jobUser`) in the project where service account is created 2. BigQuery Read Session User (`roles/bigquery.readSessionUser`) in the project where service account is created 3. BigQuery Data Viewer (`roles/bigquery.dataViewer`) in the source dataset 4. BigQuery Data Editor (`roles/bigquery.dataEditor`) in the destination dataset 5. Storage Writer (`roles/storage.legacyBucketWriter`) 6. Storage Object Admin (`roles/storage.objectAdmin`) ## Configuring Source You can specify the source configuration of your prediction job by creating an instance of `BigQuerySource`. This class’s constructor accept following parameters: 1. `table`: source table ID with format gcp\_project.dataset\_name.table\_name 2. `features`: list of features to be used for prediction, it has to match the column name in the source table. 3. `options`: is dictionary containing additional options that could be used to customize the source. Following are option that can be used. | Property | Description | | ---------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `parentProject` | The Google Cloud Project ID of the table to bill for the export.(Optional. Defaults to the project of the Service Account being used) | | `maxParallelism` | The maximal number of partitions to split the data into. Actual number may be less if BigQuery deems the data small enough. If there are not enough executors to schedule a reader per partition, some partitions may be empty. **Important**: The old parameter (`parallelism`) is still supported but in deprecated mode. It will ve removed in version 1.0 of the connector. (Optional. Defaults to one partition per 400MB. See [Configuring Partitioning](https://github.com/GoogleCloudDataproc/spark-bigquery-connector#configuring-partitioning).) | | `viewsEnabled` | Enables the connector to read from views and not only tables. Please read the [relevant section](https://github.com/GoogleCloudDataproc/spark-bigquery-connector#reading-from-views) before activating this option.(Optional. Defaults to `false`) | | `viewMaterializationProject` | The project id where the materialized view is going to be created(Optional. Defaults to view's project id) | | `viewMaterializationDataset` | The dataset where the materialized view is going to be created(Optional. Defaults to view's dataset) | | `readDataFormat` | Data Format for reading from BigQuery. Options: `ARROW`, `AVRO`. Unsupported Arrow filters are not pushed down and results are filtered later by Spark. (Currently Arrow does not suport disjunction across columns).(Optional. Defaults to `AVRO`) | | `optimizedEmptyProjection` | The connector uses an optimized empty projection (select without any columns) logic, used for count() execution. This logic takes the data directly from the table metadata or performs a much efficient `SELECT COUNT(*) WHERE...` in case there is a filter. You can cancel the use of this logic by setting this option to `false`. (Optional, defaults to `true`) | Source: ### Reading from View To use view as data source instead of table you’ll have to set viewsEnabled to true and specify `viewMaterializationProject` and `viewMaterializationDataset`. Since the materialization of view will create a table, the service account should also have `roles/bigquery.dataEditor` in the pointed dataset. Below is an example: {% code title="bq\_source.py" overflow="wrap" lineNumbers="true" %} ```python bq_source = BigQuerySource("project.dataset.table_iris", features=["sepal_length", "sepal_width", "petal_length", "petal_width"], options={ "viewsEnabled" : "true", "viewMaterializationProject" : "project", "viewMaterializationDataset" : "dsp" }) ``` {% endcode %} ## Configuring Sink To configure the destination of prediction job you can create an instance of `BigQuerySink`. The class accepts following parameters: 1. `table`: destination table ID with format gcp\_project.dataset\_name.table\_name 2. `staging_bucket`: GCS staging bucket that will be used as temporary storage for storing prediction result before loading it to the destination table. 3. `result_column`: Column name in the destination table that will be used to store the prediction result. Note that it has to be a string and not list of string even though if you specify ARRAY as the result. 4. `save_mode`: SaveMode is used to specify the expected behavior of saving the prediction result into destination table. Following are the possible values: * `ERRORIFEXISTS`: it will throw error if the destination table already exists (default). * `OVERWRITE`: it will overwrite the destination table if it exists. * `APPEND`: it will append the new result into destination table if it exists. * `IGNORE`: it will not write the prediction result if the destination table exists. 5. `options`: Dictionary of strings that can be used to specify additional configuration. Following are the available parameters. | Property | Description | | ----------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `createDisposition` |

Specifies whether the job is allowed to create new tables. The permitted values are:

  1. CREATE\_IF\_NEEDED - Configures the job to create the table if it does not exist.
  2. CREATE\_NEVER - Configures the job to fail if the table does not exist.

This option takes place only in case Spark has decided to write data to the table based on the SaveMode. (Optional. Default to CREATE\_IF\_NEEDED).

| | `intermediateFormat` | The format of the data before it is loaded to BigQuery, values can be either "parquet" or "orc". (Optional. Defaults to `parquet`). On write only. | | `partitionField` | If not set, the table is partitioned by pseudo column, referenced via either `'_PARTITIONTIME' as TIMESTAMP` type, or `'_PARTITIONDATE' as DATE` type. If field is specified, the table is instead partitioned by this field. The field must be a top-level TIMESTAMP or DATE field. Its mode must be **NULLABLE** or **REQUIRED**. (Optional). | | `partitionExpirationMs` | Number of milliseconds for which to keep the storage for partitions in the table. The storage in a partition will have an expiration time of its partition time plus this value. (Optional). | | `partitionType` | The only type supported is `DAY`, which will generate one partition per day. (Optional. Default to `DAY`). | | `clusteredFields` | Comma separated list of non-repeated, top level columns. Clustering is only supported for partitioned tables (Optional). | | `allowFieldAddition` | Adds the [ALLOW\_FIELD\_ADDITION](https://googleapis.dev/java/google-cloud-clients/latest/com/google/cloud/bigquery/JobInfo.SchemaUpdateOption.html#ALLOW_FIELD_ADDITION) SchemaUpdateOption to the BigQuery LoadJob. Allowed vales are `true` and `false`. (Optional. Default to `false`). | | `allowFieldRelaxation` | Adds the ALLOW\_FIELD\_RELAXATION SchemaUpdateOption to the BigQuery LoadJob. Allowed vales are `true` and `false`. (Optional. Default to `false`). | Source: ## Configuring Resource Request Class `PredictionJobResourceRequest` is useful to configure the resource request for running prediction job. It contains several configurable parameters of the underlying Spark application. Broadly, configurations for the following Spark components are exposed: * Driver - The driver is responsible for orchestration of the computation. It is the central coordinator that manages the execution of the Spark application. * Executor - The executors execute the tasks assigned to them by the driver. These are the worker nodes responsible for performing the actual computations. The executor resources play a crucial role in the performance of the prediction jobs. The driver resources, on the other hand, are more relevant to driver-intensive tasks (when a lot of aggregations are involved) and are thus less important for the prediction jobs. For both the driver and the executor, increasing the CPU allocation can result in better parallelism and faster computations. Similarly, increasing the memory allocation allows for larger amounts of data to be retained in memory (as opposed to saving chunks onto the disk) which in turn improves the processing time. The below configurations of the driver and executor may be modified. 1. `driver_cpu_request` : Driver CPU request. e.g: 1, 1500m , 500m. 2. `driver_memory_request`: Driver memory request. e.g. 1Gi, 512Mi 3. `executor_cpu_request`: executor CPU request. e.g: 1, 1500m , 500m 4. `executor_memory_request`: executor memory request. e.g. 1Gi, 512Mi 5. `executor_replica`: number of executor replica. e.g. 1, 2 The same configurations may also be applied from the UI when submitting a batch job: ![Batch resource configuration](/files/XLHPeNseeWjuv22LSTBU) Without specifying any resources, the prediction job will run with the system default as follows: ``` executor_replica: 3 driver_cpu_request: "2" driver_memory_request: "2Gi" executor_cpu_request: "2" executor_memory_request: "2Gi" ``` This default configuration is good enough for most cases. However, it might not be sufficient for cases where you have large model size, the dataset has a wide table (a lot of column), or the processing requires a lot of memory. In such cases, you might want to increase the `executor_memory_request` to a larger value. You might also want to make the prediction job to complete faster by increasing the `executor_cpu_request` and `executor_replica`. However, **it will increase the cost significantly**. **Note:** When optimizing the resource configurations, the best values can be determined by observing the resource usage of the components in the monitoring dashboard. However, we must also account for spikes instead of simply relying on the average value over a time window. ![Monitoring Dashboard](/files/NfTBMl1XEvedwY3mxdIp) ## Known Issues ### Type Conversion Error When BQ Source Has Date Column #### Symptom Following error is thrown during batch prediction execution ``` > raise AttributeError("Can only use .dt accessor with datetimelike " "values") E AttributeError: Can only use .dt accessor with datetimelike values ../../../.local/share/virtualenvs/merlin-pyspark-n4ybPFnE/lib/python3.8/site-packages/pandas/core/indexes/accessors.py:324: AttributeError Assertion failed ``` Check whether your BQ source table has DATE type column. If so the workaround might help. #### Root Cause #### Work Around Add `pyarrow==0.11.1` and `pandas==0.24.1` to conda `environment.yaml` of your model. # Model Schema Model schema is a specification of input and output of a model, such as what are the features columns, prediction columns and also ground truth columns. Following are the fields in model schema: | Field | Type | Description | Mandatory | | ---------- | --------------- | --------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------- | | `id` | int | Unique identifier for each model schema | Not mandatory, if ID is not specified it will create new model schema otherwise it will update the model schema with corresponding ID | | `model_id` | int | Model ID that correlate with the schema | Not mandatory, if not specified the SDK will assign it with the model that user set | | `spec` | InferenceSchema | Detail specification for model schema | True | Detail specification is defined by using `InferenceSchema` class, following are the fields: | Field | Type | Description | Mandatory | | ------------------------- | --------------------- | ------------------------------------------------------------------------------------------------------------------------- | --------- | | `feature_types` | Dict\[str, ValueType] | Mapping between feature name with the type of the feature | True | | `model_prediction_output` | PredictionOutput | Prediction specification that differ between model types, e.g BinaryClassificationOutput, RegressionOutput, RankingOutput | True | | `session_id_column` | str | The column name that is unique identifier for a request | True | | `row_id_column` | str | The column name that is unique identifier for a row in a request | True | | `tag_columns` | Optional\[List\[str]] | List of column names that contains additional information about prediction, you can treat it as metadata | False | From above we can see `model_prediction_output` field that has type `PredictionOutput`, this field is a specification of prediction that is generated by the model depending on it's model type. Currently we support 3 model types in the schema: * Binary Classification * Regression * Ranking Each model type has it's own model prediction output specification. ## Binary Classification Model prediction output specification for Binary Classification type is `BinaryClassificationOutput` that has following fields: | Field | Type | Description | Mandatory | | ------------------------- | ----- | ---------------------------------------------------------------------------------------------------- | -------------------------------------------------- | | `prediction_score_column` | str | Column that contains prediction score value of a model. Prediction score must be between 0.0 and 1.0 | True | | `actual_label_column` | str | Name of the column containing the actual class | False, because not all model has the ground truth | | `positive_class_label` | str | Label for positive class | True | | `negative_class_label` | str | Label for negative class | True | | `score_threshold` | float | Score threshold for prediction to be considered as positive class | False, if not specified it will use 0.5 as default | ## Regression Model prediction output specification for Regression type is `RegressionOutput` that has following fields: | Field | Type | Description | Mandatory | | ------------------------- | ---- | ------------------------------------------------------ | ------------------------------------------------- | | `prediction_score_column` | str | Column that contains prediction score value of a model | True | | `actual_score_column` | str | Name of the column containing the actual score | False, because not all model has the ground truth | ## Ranking Model prediction output specification for Ranking type is `RankingOutput` that has following fields: | Field | Type | Description | Mandatory | | ---------------------------- | ---- | ------------------------------------------------------------------- | --------- | | `rank_score_column` | str | Name of the column containing the ranking score of the prediction | True | | `prediction_group_id_column` | str | Name of the column containing the prediction group id | True | | `relevance_score_column` | str | Name of the column containing the relevance score of the prediction | True | ## Define model schema From the specification above, users can create the schema for their model. Suppose that users have binary classification model, that has 4 features * featureA that has float type * featureB that has int type * featureC that has string type * featureD that has float type With positive class `complete` and negative class `non_complete` and the threshold for positive class is 0.75. Actual label is stored under column `target`, `prediction_score` under column `score` `prediction_id` under column `prediction_id`. From that specification, users can define the model schema and put it alongside version creation. Below is the example snipped code ```python from merlin.model_schema import ModelSchema from merlin.observability.inference import InferenceSchema, ValueType, BinaryClassificationOutput model_schema = ModelSchema(spec=InferenceSchema( feature_types={ "featureA": ValueType.FLOAT64, "featureB": ValueType.INT64, "featureC": ValueType.STRING, "featureD": ValueType.BOOLEAN }, session_id_column="session_id", row_id_column="row_id", model_prediction_output=BinaryClassificationOutput( prediction_score_column="score", actual_label_column="target", positive_class_label="complete", negative_class_label="non_complete", score_threshold=0.75 ) )) with merlin.new_model_version(model_schema=model_schema) as v: .... ``` The above snipped code will define model schema and attach it to certain model version, the reason is the schema for each version is possible to differ. # Model Observability Model observability enable model's owners to observe and analyze their model in production by looking at the performance and drift metrics. Performance indicate how well your model to do prediction compare to the actual output, and drift indicate the difference of distribution between two datasets. To calculate those metrics the model observability system needs the following data: * Features data. The features data that is supplied to the model to do prediction * Prediction data. The prediction as output of your model * Ground truth / Actual data. The actual value of thing that your model try to predict Those data can be collected from training phase and serving phase (production). Data that is collected on the training phase is used as the baseline dataset, we can refer it as training dataset. For data during serving phase we can refer it as production dataset, this data must be emitted by the model. By default the merlin model is not emitting any of those data, hence model observability is not enabled by default. However, merlin provides a way so model can emit such data but currently it is limited only for PyFunc model. The way is to turn on the flag of `ENABLE_MODEL_OBSERVABILITY` and modify the PyFunc model to returning model input (features) and model output (prediction output), more detail will be explained in the `Onboarding` section. ## Architecture ![architecture](/files/pBQ4BFHxYkKozxJ84PDa) From above architecture diagram, we can see that there are three places where the data is published to model observability system * Model deployment workflow. Especially after model training step is completed. This step is publishing training dataset as baseline dataset * Model serving. PyFunc model will emit features and predictions data to a topic in a kafka cluster, and separate kafka consumer consume corresponding topic and publish the data to model observability system. Kafka consumer also store the data into separate BQ table that later will be used to be joined with user ground truth BQ table. * Ground truth collector workflow. This workflow primary objective is to publish ground truth or actual for a prediction ## Onboarding As the architecture diagram illustrate, the end to end model onboarding to model observability needs to involving several components. The scope of this section is limited to merlin model modification. ### PyFunc modification PyFunc model should implements class `PyFuncV3Model` instead of `PyFuncModel`. This `PyFuncV3Model` has difference method signature that must be implemented. Following are the new methods: | Method Name | Description | | ---------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------- | | `preprocess(self, request: dict, **kwargs) -> ModelInput` | Doing preprocessing that returning all the required features for prediction. Must be implemented if using `HTTP_JSON` protocol | | `postprocess(self, model_output: ModelOutput, request: dict) -> dict` | Postprocessing basically do additional processing to construct end result of the overall model. Must be implemented if using `HTTP_JSON` protocol | | `upiv1_preprocess(self, request: upi_pb2.PredictValuesRequest, context: grpc.ServicerContext) -> ModelInput` | Preprocess method signature that only called when using `UPI_V1` protocol. Must be implemented if using `UPI_V1` protocol | | `upiv1_postprocess(self, model_output: ModelOutput, request: upi_pb2.PredictValuesRequest) -> upi_pb2.PredictValuesResponse` | Postprocess method signature that only callend when using `UPI_V1` protocol. Must be implemented if using `UPI_V1` protocol | Beside changes in signature, you can see some of those methods returning new type, `ModelInput` and `ModelOutput`. `ModelInput` is a class that represents input information of the models, this class contains following fields: | Field | Type | Description | | ------------ | ------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `row_ids` | List\[str] | Unique identifier for each row in prediction | | `features` | Union\[Values, pandas.DataFrame] | Features value that is used by the model to generate prediction. Length of features should be the same with `row_ids` | | `entities` | Optional\[Union\[Values, pandas.DataFrame]] | Additional data that are not used for prediction, but this data is used to retrieved another features, e.g `driver_id`, we can retrieve features associated with certain `driver_id` | | `session_id` | str | Identifier for the request. This value will be used together with `row_ids` as prediction identifier in model observability system | `ModelInput` data is essential for model observability since it contains features values and identifier of prediction. Features values are used to calculate feature drift, and identifier is used as join key between features, prediction data with ground truth data. On the other hand, `ModelOutput` is the class that represent raw model prediction output, not the final output of PyFunc model. `ModelOutput` class contains following fields: | Field | Type | Description | | ------------ | ---------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `prediction` | Values | `predictions` contains prediction output from ml\_predict, it may contains multiple columns e.g for multiclass classification or for binary classification that contains prediction score and label | | `row_ids` | List\[str] | Unique identifier for each row in prediction output | Same like `ModelInput`, `ModelOutput` is also essential for model observability, it can be used to calculate prediction drift but more importantly it can calculate performance metrics. ### Standard Model Only xgboost model is supported at the moment. To enable model observability, user will need to enable request response logging for the model. 1% of the predictions will be sampled, and this is not configurable at the moment. ### Configure Model Schema Model schema is essential for model observability because it is used by the kafka consumer to choose which columns that is relevant to model observability and do necessary preprocessing before publishing the data to model observability system. Users can see more detail of configuring model schema [here](https://github.com/caraml-dev/docs/blob/main/module/model/user/templates/09_model_schema.md) ### Deployment There is not much change on the deployment part, users just needs to set `enable_model_observability` parameter to `True` during model deploy. For clarity, we take one use case for model observability example, suppose a model has 4 features: * featureA that has float type * featureB that has int type * featureC that has string type * featureD that has float type The model type is ranking with prediction group id information is located in `session_id` column, row id in `row_id` column, rank score in `score` column and `relevance_score_column` in `relevance_score`. Below is the snipped of the python code ```python class ModelObservabilityModel(PyFuncV3Model): def preprocess(self, request: dict, **kwargs) -> ModelInput: return ModelInput( session_id="session_id", row_ids=["prediction_1", "prediction_2"], features=pd.DataFrame([[0.7, 200, "ID", True], [0.99, 250, "SG", False]], columns=["featureA", "featureB", "featureC", "featureD"]), ) def infer(self, model_input: ModelInput) -> ModelOutput: return ModelOutput( row_ids=model_input.row_ids, predictions=Values(columns=["score"], data=[[0.5], [0.9]]), ) def postprocess(self, model_output: ModelOutput, request: dict) -> dict: return {"predictions": model_output.predictions.data} model_schema = ModelSchema(spec=InferenceSchema( feature_types={ "featureA": ValueType.FLOAT64, "featureB": ValueType.INT64, "featureC": ValueType.STRING, "featureD": ValueType.BOOLEAN }, session_id_column="session_id", row_id_column="row_id", model_prediction_output=RankingOutput( rank_score_column="score", prediction_group_id_column="session_id", relevance_score_column="relevance_score" ) )) with merlin.new_model_version(model_schema=model_schema) as v: v.log_pyfunc_model(model_instance=ModelObservabilityModel(), conda_env="env.yaml", code_dir=["src"], artifacts={"model": ARTIFACT_PATH}) endpoint = merlin.deploy(v, enable_model_observability=True) ``` For standard model, we will also need to add an additional field, `feature_orders` to the inference schema: ```python InferenceSchema( ... feature_orders=["featureA", "featureB", "featureC", "featureD"], ) ``` The features in the list should follow the same order as the expected input to the standard model. The json request body to the standard model is also expected to have these additional fields: * `session_id (String)` - Identifier for the request. * `row_ids (Array)` - Unique identifier for each row in prediction. Optional if you only have one row per prediction. # Routers ## What is Turing? Turing is a fast, scalable and extensible system that can be used to design, deploy and evaluate ML experiments in production. It takes care of the core Engineering aspects of experimentation such as traffic routing, outcome logging, system monitoring, etc. and is designed to work with pluggable Experiment Engines, pre and post processors. It is backed by existing systems like [Merlin](https://github.com/caraml-dev/merlin) for model endpoints. ## Features * Low-latency, high-throughput traffic routing to an unlimited number of ML models. * Experimentation rules based on incoming requests to determine the treatment to be applied. The experiment engines currently supported are closed source for now (we are working on this!). * Feature enrichment of incoming requests through [Feast](https://github.com/feast-dev/feast) (planned) and arbitrary pre-processors. * Dynamic ensembling of models for each treatment. This could be selecting one of the models' response, custom ensembling of responses from two or more models or any other arbitrary post-processing. * Reliable and safe fallbacks in case of timeouts. * Simple response and outcome tracking. ## How It Works 1. The Turing router receives incoming requests from the client. 2. Enrichment of request with features from external sources can be done if required by the Enricher. 3. The unit ID is extracted and passed to an Experiment Engine to determine the treatment. Simultaneously, the request is forwarded to all model endpoints via the configured routes. 4. The Ensembler is called with the original Turing request, the implementation of exploration policies from the experiment engine response and the model responses. 5. A tracking ID is appended to the ensembled response and it is logged (together with the individual model responses and the original request) and it is returned to the client. 6. The client will then be able to log the outcome with the tracking ID. ## When To Use Turing * You need to send traffic to multiple model endpoints * You need to ensemble the resulting response based on the experiment configuration. * You want request-response pairs to be logged. # Creating a Router A Router in the Turing system represents an ML experiment and holds the configuration for the traffic routing, pre/post-processors, incorporating the response from the Experiment engine and logging. It is built on our Fiber traffic routing library written in Golang, and sources its configuration when it starts up. ### Setting Up An Experiment In Turing, your will need to create a router with an optional Experiment Engine, Pre-processor (Enricher) and Post-processor (Ensembler) configured based on your requirements. After configuring and deploying the router, you will receive a URL to which you are able to send your experimentation requests. ### Navigate to the Create Router UI * Open the Turing homepage. * Choose a project in which you want to create your router. If such a project does not exist, you can [create a project](https://docs.caraml.dev/user-guides/projects/create-project) ![](/files/4S9kU9p5oE1hcrNzK9rn) * Choose “Create Router”. ![](/files/Hie6EZb9e92sfkeX1O7r) ### Configure router Now that we have navigated to the create a router page, we can continue to configure the router. {% content-ref url="/pages/3QAaFmUnKpzWQTreSzPb" %} [Configure general settings](/user-guides/router/create-a-router/configure-general-settings) {% endcontent-ref %} {% content-ref url="/pages/tQMF20UP3KSoatUI8i6z" %} [Configure routes](/user-guides/router/create-a-router/configure-routes) {% endcontent-ref %} {% content-ref url="/pages/MH8vP72a7GHWdaasNZpk" %} [Configure traffic rules](/user-guides/router/create-a-router/configure-traffic-rules) {% endcontent-ref %} {% content-ref url="/pages/lADlysQlUlb0w4A5oqfc" %} [Configure autoscaling](/user-guides/router/create-a-router/configure-autoscaling) {% endcontent-ref %} {% content-ref url="/pages/8XCMQXxVTxbkGkZBp2SI" %} [Configure experiment engine](/user-guides/router/create-a-router/configure-experiment-engine) {% endcontent-ref %} {% content-ref url="/pages/HIZsncyfu7gTvSSwdaLQ" %} [Configure enricher](/user-guides/router/create-a-router/configure-enricher) {% endcontent-ref %} {% content-ref url="/pages/Ij8w5mRVEyY7XjcW7Ayv" %} [Configure ensembler](/user-guides/router/create-a-router/configure-ensembler) {% endcontent-ref %} {% content-ref url="" %} {% endcontent-ref %} # Configure general settings There are 4 required inputs: **Environment**: A drop down menu of the target environment your router will be deployed to. This is set [here](https://github.com/caraml-dev/merlin/blob/main/charts/merlin/values.yaml#L102-L130). An example used in Turing is [here](https://github.com/caraml-dev/turing/blob/main/infra/docker-compose/dev/merlin/deployment-config.yaml). As Turing manages multiple deployment environments, you are free to choose which environment your router will be deployed in. **Protocol**: Protocol of router, currently HTTP and UPI (gRPC) are supported. **Name**: Name of your router deployment. **Timeout**: Overall timeout, which when exceeded, the request execution by your Turing router will be terminated. ![](/files/lJ9SGAhOwoLCNU0GfDrH) # Configure routes Routes are an essential part of your Turing Router setup. Each route is defined by its ID and the endpoint. A route can be your deployed ML model, exposed via HTTP interface, or an arbitrary non-ML web-service. Each router should be configured with at least one route and each route's ID should be unique among other routes of this router. The request sent to the each route is constructed from the request headers to the Router (including an identifier `Turing-Req-Id` that is uniquely assigned to each request received by the Router), and either the request body to the Router (if there is no Enricher) or the response body from the Enricher (if enabled). For UPI Routers, the endpoint must support [UPI interface](https://github.com/caraml-dev/universal-prediction-interface). The default service method is `/caraml.upi.v1.UniversalPredictionService/PredictValues` and is not configurable. ![](/files/eejWOgMuEX3gmNF56dVh) {% hint style="info" %} You should also configure timeouts for each of the routes. The request execution will be terminated, when this timeout is exceeded during a call from Turing to the route's endpoint. {% endhint %} # Configure traffic rules {% hint style="info" %} This step is **optional** and the default behaviour will not discriminate between requests. {% endhint %} ![](/files/B2Dyt5tw37LuLjyBDTSU) It is also possible to configure your router such that each request is only dispatched to a subset of the configured routes, based on some request specific conditions. For example, you might have some models, trained on geography-specific data. So, in this scenario, you want to call `mock` if the request contains header `"X-Region: region-A"` or `model-b` if `X-Region` header equals `"region-B"`. It's possible to achieve this by configuring traffic rules on your router. **Note**: Nested paths in the payload can be specified using `.` as the separator. This also works for array indices. Eg: The first session id in the below structure can be specified as `session.[0].id`. ```json { "session": [ {"id": 123, "data": {}}, {"id": 456, "data": {}} ] } ``` Traffic rules define which routes should be "activated" for a particular request to your router. Each rule is defined by one or more request conditions and one or more routes that would be activated if the request satisfies the conditions of the rule. When traffic rules are configured, the following behaviour is expected: * A Default traffic rule is required, which has no request conditions and acts as a fallback rule for requests that don't match any of the configured traffic rule conditions. At least 1 route should be configured for the Default traffic rule. * All routes should be used by at least 1 traffic rule, i.e no dangling routes. * When a Nop or Standard Ensembler is configured (See: [Configure Ensembler](/user-guides/router/create-a-router/configure-ensembler)), the final/fallback route must be present in every traffic rule, including the default rule. Rules are matched against the incoming request in the order in which they are defined. The UI supports the drag-and-drop functionality to reorder priority of the rules. This property can be used to create one or more specific rules over a general rule. Consider the following example, where a router has rules defined such that requests are routed by the country of origin (ID, SG, etc.). In addition, if the routing logic must be altered for a certain service type in a country, this rule (the 'specific' rule) can be defined before the general rule. ![](/files/InezgMSKfUK5uUyzZPTI) In addition, orthogonality checks have been put in-place to disallow overlapping rules. Some examples are as described below: | Rule | Conditions | | ---- | ------------------------------------------ | | 1 | country\_code=\[ID], geo\_area=\[1] | | 2 | country\_code=\[ID,SG], geo\_area=\[1,2,3] | | 3 | country\_code=\[SG], geo\_area=\[1] | | 4 | country\_code=\[ID], geo\_area=\[2] | From the above rules, rule 2 is overlapping with all other rules since there are no unique combinations that would let rules 3 and 4 to be called, and rule 2 will not be called for the same conditions met by rule 1. | Rule | Conditions | | ---- | ----------------------------------- | | 1 | country\_code=\[ID], geo\_area=\[1] | | 2 | country\_code=\[SG], geo\_area=\[1] | From the above rules, both rule 1 and 2 are not overlapping because there's at least 1 condition parameter i.e country\_code that is exclusive. ### Conditions Each rule should have at least one condition configured on it. If there are multiple conditions configured on the same rule, then this rule will be triggered only if each and every condition is satisfied. Rule condition can be defined on either request header or request payload (assuming payload is a valid JSON object). For UPI routers, it would be header or [prediction context](https://github.com/caraml-dev/universal-prediction-interface/blob/main/proto/caraml/upi/v1/upi.proto) (name of the variable and value). For each condition, you should specify: * **Condition source**: either `Header` or `Payload` for HTTP router. `Header` or `Prediction Context` for UPI router. * **Condition key**: if condition's source is `Header` – then the name of a request Header (example: `X-Session-ID`), or else, if condition's source is `Payload` – a valid JSON path of the property from the request's JSON payload (example: `service_type.id` or `users.0.name`). If condition source is `Prediction Context` - the name of the variable within the Prediction Context proto. * **Condition values**: one or more values that the extracted condition key is expected to match. Condition will be satisfied if key matches at least one of the configured values.
{% hint style="warning" %} Provided values are case-sensitive. {% endhint %} ### Routes You should also select one or more routes from the drop-down list that would be activated if the rule is triggered. A route can be attached to zero or more traffic rules. * If a route is attached to some traffic rule, then Turing will only send request to this route if the request meets this rule's conditions. * If a route is attached to multiple rules and the request satisfies more than one rule, then Turing will decide what group of routes should receive this request based on the order in which the traffic rules are defined. # Configure autoscaling You can also configure the amount of resources (CPU/Memory) allocated for each replica of your router as well as lower and upper limits for the autoscaling. ![](/files/DPPOEfu6Nuq6i7gcoD7m) {% hint style="warning" %} In most situations, 500m CPU and 0.5Gi memory should suffice, however, you can fine tune this configuration based on the data from Turing monitoring dashboard. {% endhint %} ## Autoscaling Policy By default, Turing uses request concurrency as a measure to monitor performance. For situations that require more fine-grained control, you can also choose a different autoscaling metric and adjust the target value according to prior load tests. Currently, 4 metrics are supported - Concurrency, RPS, CPU and Memory. ![](/files/oaEbXzOSvoA4ZvL1EDL9) # Configure experiment engine {% hint style="info" %} This step is **optional** and the default behaviour will be to send the request to all of the configured routes, but the response from Turing will depend on the Ensembler configuration. {% endhint %} The request sent to the experiment engine is constructed from the request headers to the Router (including an identifier `Turing-Req-Id` that is uniquely assigned to each request received by the Router), and either the request body to the Router (if there is no Enricher) or the response body from the Enricher (if enabled). If you intend to determine how to select/combine responses from the individual routes (typically in the Ensembler) to send back as the Turing response, configure the Experiment Engine as shown in Configure Experiment Engine. ![](/files/ed7vI1Y1SfxXaKiNmXQt) # Configure enricher {% hint style="info" %} This step is **optional** and the default behaviour will send the original request to the configured routes. {% endhint %} The Turing enricher has the ability to perform arbitrary transformations on the incoming request for feature enrichment. It can deploy any enricher image (implemented by a user) and integrate it with the rest of the system. Currently, there are 3 options available - no enricher, Docker and an external enricher that is coming soon. ### No Enricher Original request will be sent to configured routes without enrichment. ### Docker Turing will deploy specified Docker image as a pre-processor and will send original request to it for enrichment. The request to the enricher is constructed from the request headers to the Router, and an identifier `Turing-Req-Id` that is uniquely assigned to each request received by the Router. To configure a Docker enricher, there are 3 sections to fill. Configure the Docker Container. There are 4 required inputs. ![](/files/IFEDbXYiUzTJiy7a64SE) **Docker Image**: The image is formed of 2 parts. Select the registry to where your image is stored. Following that, enter the name of the image. **Endpoint**: Relative URL endpoint of the enricher. **Port**: Port number exposed by your container. **Timeout**: Request timeout, which when exceeded, the request to enricher will be terminated **Service Account**: You can **optionally** mount a service account for your Docker deployment. ![](/files/d9kapJWVwFHnSJI5nHRL) Configure any environment variables required by the Docker container. You need to input the name and corresponding value of each input. ![](/files/6hsdc42bnZ8fKzP29Vfe) Configure the resources required for the enricher. There are 3 required inputs, with default values provided for each. ![](/files/Hkt3HuTH8upNovrim8ER) **CPU**: Total amount of CPU available for your enricher. **Memory**: Total amount of RAM available for your enricher. **Min/Max Replicas**: Min/max number of replicas for your enricher. Scaling of the enricher based on traffic volume will be automatically done for you. **CPU Limit**: By default, Turing determines the CPU limits of all deployed components using platform-level configured values. These CPU limits is calculated as a factor of the user-defined CPU request value for each component (e.g. 2x of the CPU request value). However, you can override this platform-level configured value by setting this value explicitly on the UI (as seen above) or via the SDK. Optionally, modify the autoscaling policy on the enricher. ![](/files/oaEbXzOSvoA4ZvL1EDL9) **Metric**: The autoscaling metric to monitor. Currently, 4 metrics are supported - Concurrency, RPS, CPU and Memory. **Target**: The target value of the chosen metric for each replica, after which autoscaling should be triggered. ### External Enricher Coming soon. # Configure ensembler Turing currently supports ensemblers in the same fashion as the enrichers. The ensembling is controlled by the policy from the rule engine. Currently, there are 4 options available - no ensembler, a standard ensembler, Docker and Pyfunc ensembler. ## No Ensembler The router will return a response from the route configured to act as the final response. This option is available only when **no experiment engine** is configured in Configure Experiment Engine. ![](/files/7HJ5Opv6YlhgnQOar2cX) {% hint style="info" %} It is not possible to select as the final response a route that has traffic rules associated to it. {% endhint %} ## Standard Ensembler There are two types of Standard Ensemblers available, one that is available with Standard Experiment Engines that have experiment selection enabled, and the other with Custom Experiment Engines. Both types of standard ensemblers support two modes of routing - 'Selective' (where the experiment engine is called upfront, and only the route that will be chosen as the final response will be invoked), and 'Exhaustive' (where all applicable routes and the experiment engine will be called in parallel) - the former is more cost efficient while the latter is more performant. ### Standard Experiment Engines For routers configured with Standard Experiment Engines that have experiment selection enabled, the router will return a response from one of the routes based on the configured mapping between routes and experiment treatments. At run time, the treatment returned by the engine will be used to select the corresponding route’s response. ![](/files/Z6JMHKZXFgakYnkCIjmi) In addition, a fallback route may be configured whose results will be used at runtime when the call to the experiment engine fails or if a route mapping for the treatment generated by the experiment engine does not exist. ![](/files/zANeOBsYznhFhHKEwd9y) {% hint style="info" %} It is not possible to select as the fallback response a route that has traffic rules associated to it. {% endhint %} ### Custom Experiment Engines For routers configured with Custom Experiment Engines, the router will return a response from one of the routes corresponding to the route name that is found within the treatment configuration returned by the experiment engine. At run time, the router will attempt to access the route name within the treatment configuration received via a user-configured path. A fallback route also has to be configured in order to capture cases whereby the route name found in the treatment configuration does not correspond to any of the routes configured, or when the user-configured path is invalid with respect to the treatment configuration received. ## Docker Turing will deploy the specified image as a post-processor and will send in the request payload the following, for ensembling - the original request, responses from all routes, and the treatment configuration (if an Experiment Engine is selected, in the Configure Experiment Engine step). The ensembler's request headers will contain the original request headers sent to Turing, merged with the enricher's response headers (if there are duplicates, the value in the enricher's response headers will take precedence), and an identifier `Turing-Req-Id` that is uniquely assigned to each request received by the Router. To configure a Docker ensembler, there are 3 sections to be filled. Configure the Docker Container. There are 4 required inputs. ![](/files/IFEDbXYiUzTJiy7a64SE) **Docker Image**: The image is formed of 2 parts. Select the registry to where your image is stored. Following that, enter the name of the image. **Endpoint**: Relative URL endpoint of the ensembler. **Port**: Port number exposed by your container. **Timeout**: Request timeout, which when exceeded, the request to the ensembler will be terminated **Service Account**: You can **optionally** mount a service account for your Docker deployment. Configure any environment variables required by the Docker container. You need to fill in the name and corresponding value of each input. ![](/files/6hsdc42bnZ8fKzP29Vfe) Configure the resources required for the ensembler. There are 3 required inputs, with default values provided for each.
![](/files/Hkt3HuTH8upNovrim8ER) **CPU**: Total amount of CPU available for your ensembler. **Memory**: Total amount of RAM available for your ensembler. **Min/Max Replicas**: Min/max number of replicas for your ensembler. Scaling of the ensembler based on traffic volume will be automatically done for you. **CPU Limit**: By default, Turing determines the CPU limits of all deployed components using platform-level configured values. These CPU limits is calculated as a factor of the user-defined CPU request value for each component (e.g. 2x of the CPU request value). However, you can override this platform-level configured value by setting this value explicitly on the UI (as seen above) or via the SDK. Optionally, modify the autoscaling policy on the ensembler. ![](/files/oaEbXzOSvoA4ZvL1EDL9) **Metric**: The autoscaling metric to monitor. Currently, 4 metrics are supported - Concurrency, RPS, CPU and Memory. **Target**: The target value of the chosen metric for each replica, after which autoscaling should be triggered. ## Pyfunc Ensembler Turing will deploy a previously registered Pyfunc ensembler (refer to [the samples](https://github.com/caraml-dev/turing/tree/main/sdk/samples) in the SDK section for more information on how to deploy one) as a containerised web service. This allows you to simply define the logic required for the ensembling step by implementing a Python `mlflow`-based interface, and rely on Turing API to containerise and package your implementation as an entire web service automatically. Similar to requests sent to a Docker Ensembler, the request payload sent to a Pyfunc ensembler will contain the original request, responses from all routes, and the treatment configuration (if an Experiment Engine is selected, in the Configure Experiment Engine step). The ensembler's request headers will contain the original request headers sent to Turing, merged with the enricher's response headers (if there are duplicates, the value in the enricher's response headers will take precedence), and an identifier `Turing-Req-Id` that is uniquely assigned to each request received by the Router. **Note on compatibility**: The Pyfunc servers are compatible with `protobuf>=3.12.0,<5.0.0`. Users whose ensemblers have a strong dependency on Protobuf `3.x.x` are advised to pin the library version in their conda environment, when submitting the ensembler. If using Protobuf `3.x.x` with the UPI protocol, users can do one of the following: * Use `protobuf>=3.20.0` - these versions support simplified class definitions and this is the recommended approach. * If you must use `protobuf>=3.12.0,<3.20.0`, please pin `caraml-upi-protos<=0.3.6` in your ensembler’s conda environment. To configure your router with a Pyfunc ensembler, simply select from the drop down list your desired ensembler, registered in your current project. You'll also need to indicate your desired timeout value and resource request values: ![](/files/8lv0Iwpzg8NZOK0wc1Dp) **Pyfunc Ensembler**: The name of the Pyfunc ensembler that has been deployed in your *current* project **Timeout**: Request timeout, which when exceeded, the request to the ensembler will be terminated **CPU**: Total amount of CPU available for your ensembler. **Memory**: Total amount of RAM available for your ensembler. **Min/Max Replicas**: Min/max number of replicas for your ensembler. Scaling of the ensembler based on traffic volume will be automatically done for you. **CPU Limit**: By default, Turing determines the CPU limits of all deployed components using platform-level configured values. These CPU limits is calculated as a factor of the user-defined CPU request value for each component (e.g. 2x of the CPU request value). However, you can override this platform-level configured value by setting this value explicitly on the UI (as seen above) or via the SDK. Optionally, modify the autoscaling policy on the ensembler. ![](/files/oaEbXzOSvoA4ZvL1EDL9) **Metric**: The autoscaling metric to monitor. Currently, 4 metrics are supported - Concurrency, RPS, CPU and Memory. **Target**: The target value of the chosen metric for each replica, after which autoscaling should be triggered. ## External Ensembler Coming Soon. The router will send responses from all routes, together with treatment configuration to the external URL for ensembling. ## Ensembler Request Payload Format When the ensembler type is Docker/Pyfunc/External, the ensembler will receive the following information in the request payload and the behaviour of the ensembler is up to the implementer. ```json { // original request payload unmodified "request":{}, "response": { "route_responses": [ { "route": "control", "data": { //... } }, { "route": "xgboost-ordinal", "data": { //... } }, ], "experiment": { // response from Experiment Engine unmodified "configuration": { //... }, // populated if error occurs "error": "", } } } ``` # Configure logging {% hint style="info" %} This step is **optional** and the default behaviour is not to log any request-response pair. {% endhint %} Turing currently supports logging request-treatment-response data to BigQuery and Kafka. ## BigQuery Configure the BigQuery destination. There are 2 required inputs. ![](/files/8YsoLtP4XAL4EImRdnpA) **BigQuery Table**: Specify the name of the BigQuery Table in the format of `project_name.dataset.table`. If the table does not exist, it will be created automatically at the deployment. **Service Account**: Choose a service account from the ones provided that has both JobUser and DataEditor privileges and write access to the configured BigQuery dataset. ## Kafka Select Kafka as the Results Logging Destination and configure the required values. ![](/files/5JjRnMCHh6kDjJxRZBKK) **Brokers**: A comma-separated list of one or more Kafka brokers **Topic**: A valid Kafka topic name on the server. The data will be written to this topic. **Serialization Format**: The message serialization format to be used. This can be JSON or Protobuf. When Protobuf serialization is used, the message published to the topic is of type `TuringResultLogMessage` and the message key is of type `TuringResultLogKey`. When JSON serialization is used, the `TuringResultLogMessage`'s JSON representation is published to the topic. The protocol buffers can be found [here](https://github.com/caraml-dev/turing/blob/main/engines/router/missionctl/log/resultlog/proto/turing/TuringResultLog.proto). # Viewing Routers Once the router has been created, you will be able to view the configuration of the deployed router version and other versions in the router. ### Navigate to Router Details Open the Turing homepage. Select the project in which the router was created. ![](/files/4S9kU9p5oE1hcrNzK9rn) Click on the row that contains the router. ![](/files/E2LDH8Gi3Vpv2xO7ZGhz) You will now be able to see the Router Details View with details about the router. ![](/files/7tYSHuuh4yX8NUjkyJso) **Name and Status Badge**: You will see the name of the router alongside a badge that indicates the status of the router deployment. **Endpoint**: Dynamically generated URL to which users can send their experimentation requests. **Environment**: The environment in which the router has been deployed. **Created At**: This is when the first version of the router was created. **Updated At**: This marks the latest occurrence of a different version being deployed to the router (either via Editing Router or Redeploying Router). | Status | Description | Badge | | ------------ | ------------------------------------------------------------------------------------------------- | -------------------------------- | | Deployed | The configuration provided for the router is valid and has been successfully deployed. | ![](/files/CD4rbnk7WxS4VItTn0zM) | | Not Deployed | There are no resources deployed in the cluster for the router. | ![](/files/JQOQEx9gtJfMiybKnfCl) | | Updating | The newly provided configuration for the router is being validated and will be deployed if valid. | ![](/files/qHgCMHWPHMw4WXQKxrAF) | | Failed | All configurations provided for the router are invalid and deployment has been unsuccessful. | ![](/files/GxzYWpo7eO4Rbh70ON4W) | There are 5 tabs that can be viewed in the homepage a more-actions button that will be explained more below. However we have omitted alerts and moniroting as there is no easy way to set those up. This feature will be added in a future release. {% content-ref url="/pages/4UJ6Y1l8GILvyz0K8uaq" %} [Configuration](/user-guides/router/viewing-routers/configuration) {% endcontent-ref %} {% content-ref url="/pages/ByEJSsRXl4WxivHYibEZ" %} [History](/user-guides/router/viewing-routers/history) {% endcontent-ref %} {% content-ref url="/pages/vkHhDblgKfpF2Ijg2ygz" %} [Logs](/user-guides/router/viewing-routers/logs) {% endcontent-ref %} {% content-ref url="/pages/eTrWq5PUyClHXoiS6DJs" %} [More actions](/user-guides/router/viewing-routers/more-actions) {% endcontent-ref %} # Configuration The configuration tab displays the configuration of the deployed version of the router. These values are either configured when creating a router or editing a router. ![](/files/jB0LswNTD5k8ZCCbEyPa) If there are traffic rules (See: [Configure Traffic Rules](/user-guides/router/create-a-router/configure-traffic-rules)) configured on the router, then you will see a ![](/files/OxEB2iB7vwiW1xVag50r) button next to the routes that are part of a traffic-split configuration. Click on the button to expand the row and see traffic rules and their conditions: ![](/files/EZr69ogk7vnJziEpwZyA) Routes details panel also supports grouping by traffic rules, where you can easily see what traffic rule conditions have been configured for the Router. ![](/files/0EX6nwVH9YCoG1rKJxBs) # History The history tab displays all versions and activity log associated with the router. ### Versions ![](/files/cVuVPdV367lLxRaLkl9j) You will see a summary of each version being displayed. **Version**: You will see the version number alongside applicable badges. There are 2 possible badges: ``` * **Current**: Indicates the deployed version if the router is deployed or the latest deployed version if the router is undeployed. * **New**: Indicates if the version was created in the past hour ``` **Created**: This is when the router version was created. **Updated**: This marks the latest occurrence of the router version being deployed (either when the version was created or when the version was redeployed). **Actions**: Represents the actions that can be done to the router version: ``` * **Compare**: It's possible to compare any two router versions to see what has been changed. See [Comparing Versions](./history.md#comparing-versions) * **Deploy**: A router version which is not currently deployed can be deployed to the router. * **Delete**: A router version which is not currently deployed can be deleted. ``` **Status**: Similar to the router status, there are 4 possible status values. | Status | Description | Badge | | ------------ | --------------------------------------------------------------------------------------------------------------------- | -------------------------------- | | Deployed | The configuration provided in the router version is valid and currently deployed. | ![](/files/IkUKNKQjQw3h9uBTlCuf) | | Not Deployed | The configuration provided in the router version is valid. | ![](/files/UXaOdwTSzaY2pfm8EEs3) | | Updating | The configuration provided in the router version is being validated and will be deployed to the router if successful. | ![](/files/4v8Gam4q1itoKMe5sw4z) | | Failed | The configuration provided in the router version is invalid. | ![](/files/wmE4FZRvYWWSgUpfJLLB) | To view the configuration of the version in detail, proceed to you can click on the version to find out the details of that particular version. ### Comparing Versions You can compare any two versions of router and check which part of the configuration has been changed. To do so, select two versions from the versions list and click on `Compare` button: ![](/files/NGJFJi17IfX1rhG8KWYy) ### Activity Log The activity log displays information regarding the progress of the deployment and potential reasons of failed deployments. ![](/files/HEgFOAbg9F9IPQqosuq3) # Logs The logs tab displays the container logs of the router, enricher or ensembler (if an enricher or ensembler) is configured. Click on the `Router`, `Enricher` or `Ensembler` to toggle between the containers. ![](/files/BcLBwmFtKzzfkp4AnpQM) # More actions Opening the More Actions panel displays the actions that can be done to the specified router. These actions are: * **Edit Router**: An existing router configuration can be edited and deployed as a new version within the router. * **Undeploy Router**: A deployed router can be undeployed. * **Redeploy Router**: An undeployed router can be redeployed. * **Delete Router**: An undeployed router can be deleted. # Edit Routers Heading to the `Edit Router` page allows you to make changes to the configuration of the selected router. You will be able to view panels corresponding to various components of a Turing Router. ![](/files/smPr2ZyLqlLiRikg3RF0) Once you are done editing the configuration of your router, click on `Next` to proceed to the following step: ![](/files/aXu5n5kpkIhcrhJDWarp) ## Version Comparison Clicking on next brings you to a version comparison page which shows the differences in the current router configuration as well as your proposed changes: ![](/files/Ycz9ueN4xivp4YcuyHLl) If you are content with the proposed changes and wish to proceed, there are 2 options available: * `Save` your proposed changes * `Deploy` your proposed changes Saving your router simply creates a new router version without deploying the new configuration. Your existing router version remains active. You can visit the `History` tab to find the new router version that you have saved. Note that this new router version will be in a `Not Deployed` state. Deploying a router will create a new router version **and** deploy the new version immediately. If the deployment is successful, instances of the router with the current version will be taken down and be replaced by the new version. If you wish to make any additional changes instead, click on the `Previous` button. You will be returned to the previous page where you can continue making changes. # Monitoring router Turing provides a convenient user interface for monitoring the router deployments and getting the standard metrics out-of-the-box, and configuring alerts based on the performance of the routers. To take advantage of these features, an appropriate monitoring / alerting backend should have been configured. {% content-ref url="/pages/WCp8B9O44h6TYUZUuUWs" %} [Monitor Router Performance](/user-guides/router/monitor-a-router/monitor-router-performance) {% endcontent-ref %} {% content-ref url="/pages/Nan1RNovCavo9MvGTrXh" %} [Configure Alerts](/user-guides/router/monitor-a-router/configure-alerts) {% endcontent-ref %} # Monitor Router Performance {% hint style="warning" %} This document assumes that an appropriate monitoring solution has been integrated to the deployment of Turing. The actual functionality may be different, according to the deployment configurations. {% endhint %} 1. Navigate to the Router Details View of your router. 2. Click on the monitoring link of your router under Router Actions. ![monitoring\_tab](/files/86nqPJmprA2iDgGsrX34) 3. This link will bring you to the dashboard with standard router metrics such as throughput, latency and error rate of your Router. ![monitoring\_dashboard](/files/HPtlvBXi5IcZNFT8DLuI) # Configure Alerts {% hint style="warning" %} This document assumes that an appropriate alerting solution has been integrated to the deployment of Turing. The actual functionality may be different, according to the specific deployment configurations. {% endhint %} You can configure alerts for the critical metrics of your deployed router. An alert will be fired if a metric is out of range, to the destination set up in the alerting backend. 1. Navigate to the Router Details View of your router. 2. Click on the Alerts Tab and then, the Configure Alerts button. ![configure\_alerts](/files/kHdNfZsrDjPZdNtQkEaU) 3. Select a team that must be informed of any alerts associated with your router. You can choose a provided team from the dropdown or enter a custom team. You will only be able to configure a **single team** for all your alerts. ![alerts\_team](/files/3VhMfVTkzKllXpjEIB28) 4. Enable the alert by toggling the switch of the metric you want to be alerted for. ![toggle\_alert](/files/25TtGdKqhMfXiPMBMkgY) 5. Configure the details of the alert. ![alert\_details](/files/kbQ9KQwZP1XEfTKkFAWM) 6. Click on Update Alerts. ![update\_alerts](/files/2wd5ULIhgtyH1P6PoBd3) 1. If there is an error in validation, you will be taken back to the first section with an error and will be able to see all the errors in the form. You will need to rectify all errors before clicking on the Update button again. ![alert\_validation](/files/d2po1TWBjMrGPmORIEK0) 7. Once the alerts have been successfully validated and configured, you will be taken to the Alerts Details page and will see the alerts you have configured. ![alert\_config](/files/gPasnzbwbYByc1sArnbV) # Undeploying Router Undeploying a router stops a deployed router and removes any resources deployed in the cluster for that router. This action is reversible and the router can be redeployed. Navigate to the Router Details View of your router. Click on the More Actions button and select Undeploy router. Only a **deployed** router, with the status badge `Deployed` can be undeployed. ![](/files/xusqM2swiw3lntkV0jQ6) Confirm if you want to undeploy your router. ![](/files/qYyXXg9e8MqCFnE6QWZe) Once the router has been successfully undeployed, you will see the status of your router has changed to `Not Deployed` and will no longer be monitored. # Redeploying Router You can redeploy an undeployed router or redeploy a version of your router which is not currently deployed. Resources in the cluster that were removed from the router or router version will be reallocated to the router or router version. There are three different ways to deploy a router. {% content-ref url="/pages/BE6j1MC95BkkMoiY0ndV" %} [Redeploy undeployed router](/user-guides/router/redeploy-a-router/redeploy-undeployed-router) {% endcontent-ref %} {% content-ref url="/pages/uZMEJFZFDxhJlHlgVTku" %} [Redeploy version from history](/user-guides/router/redeploy-a-router/redeploy-version-from-history) {% endcontent-ref %} {% content-ref url="/pages/WpivbjCWyHdsDsaPmW2s" %} [Redeploy version from version details page](/user-guides/router/redeploy-a-router/redeploy-from-version-detail) {% endcontent-ref %} # Redeploy undeployed router Navigate to the Router Details View of your router. Click on the More Actions button and select Redeploy router. Only an **undeployed** router (indicated by the router status `Not deployed`) can be redeployed and Turing will automatically redeploy the most recently deployed version for you. ![](/files/CcjpOe9sQ0djHVwv31bP) Confirm if you want to redeploy your router. ![](/files/VgOCWRCkCf79eEuc7Fo6) Once the router is in the process of redeployment, you will see that the status of your router has changed from `Not deployed` to `Updating`. ![](/files/6WpeGrJjPJxHPG6jv0Up) # Redeploy version from history Navigate to the Router Details View of your router and click the `history` tab. Only a version that is not currently deployed (indicated by the `Not Deployed` version status) can be deployed. If you would like to see the details of your router version before deploying, proceed to deploy a router version from version details page. If not, click on the deploy button. ![](/files/CadskxChDygDmvtEbQXQ) Confirm if you want to deploy the specified version of your router. ![](/files/HiBqCsHZwTd7sQklWGI7) Once the specified version is in the process of the redeployment, you will be taken to the Version Details View of the deploying version and the status of the version will be changed to `Updating`. ![](/files/6hBCK0nHZ1lprwoEqZ5G) # Redeploy version from version details page Clicking on the row of your version in the history tab will bring you to the version details page. You should be able to see the existing version configuration here. Within the More Actions panel, click Deploy This Version. ![](/files/QSYHEYtMjTVpFOyj2Gbq) Confirm if you want to deploy the specified version of your router. ![](/files/HiBqCsHZwTd7sQklWGI7) Once the specified version is in the process of redeployment, the status of the version will be changed to Updating and it will be in the process of deploying your router. ![](/files/ig5XW15L3Q9UMtPLoHzq) # Deleting Router A router version that is not deployed or in the process of deploying can be deleted. Deleting a router or deleting a router version will result in the purging of the router or router version metadata from the Turing database. This action is **irreversible**. {% content-ref url="" %} {% endcontent-ref %} {% content-ref url="" %} {% endcontent-ref %} {% content-ref url="" %} {% endcontent-ref %} # Deleting router versions Any version that is not currently deployed or in the process of deployment can be deleted. This action will remove the version from the router’s history and can no longer be accessed. Navigate to the Router Details View of your router. Click on the History tab. Only a version that is **not currently deployed** (indicated by the absence of the `Deployed` status) can be deleted. If you would like to see the details of your router version before deleting, proceed to Deleting a Router Version from Version Details View. If not, click on the Delete button of the version you would like to delete in the Actions column. ![](/files/WPZbqmaK0HmHBDF7XiwL) Confirm if you want to delete the specified version of your router. ![](/files/jDLWKYZUJb5eLk2FJyq0) Once the specified version has been successfully deleted, you will no longer be able to see the deleted version in the History tab. ![](/files/MgUkhMWE4QHS7ZN7RLF2) # Deleting router versions from details page Click on the row of your version. You should be able to see the existing version configuration. Within the More Actions panel, click on Delete This Version. ![](/files/yGy7RwBPAQLMXtdVx8dJ) Confirm if you want to delete the specified version of your router. ![](/files/jDLWKYZUJb5eLk2FJyq0) Once the specified version has been successfully deleted, you will be taken to the History Tab of the Router Details page and will no longer be able to see the deleted version. ![](/files/MgUkhMWE4QHS7ZN7RLF2) # Deleting routers A router that is not currently deployed or in the process of deployment can be deleted. This action will remove the router from the Turing database and the router and all associated versions can no longer be accessed. Navigate to the Router Details View of your router. Click on the More Actions button and select Delete Router. Only a router that is **not currently deployed or in the process of deployment** (indicated by the absence of `deployed` and `updating` status badges) can be deleted. If you want to delete a router that is currently deployed, undeploy your router first. ![](/files/IUQC8hqD5Fcwgm9f1dqR) Confirm if you want to delete the router. ![](/files/qPDFvugesDqZFeaAhEau) Once the router has been successfully deleted, you will be taken to Routers List View and will no longer be able to see the router in the list. ![](/files/tt0UiJmBbwa9XkhRAZH5) # Deleting Emsemblers An ensembler can only be deleted if it is not in active use by any routers or ensembling jobs. Deleting such an ensembler will result in the purging of all related inactive router versions and ensembling jobs, as well as the ensembler metadata from the Turing database. This action is **irreversible**. {% content-ref url="/pages/RpYXrVyA9nDNOAH71T2P" %} [Delete an Ensembler without related entity](/user-guides/router/delete-an-ensembler/delete-an-ensembler) {% endcontent-ref %} {% content-ref url="/pages/pKwddPtnix0jCG2jKjxL" %} [Delete an Ensembler with active entities](/user-guides/router/delete-an-ensembler/delete-an-ensembler-active) {% endcontent-ref %} {% content-ref url="/pages/3k4r8SPsoeyYqqKIXtNP" %} [Delete an Ensembler with inactive entities](/user-guides/router/delete-an-ensembler/delete-an-ensembler-inactive) {% endcontent-ref %} # Delete an Ensembler without related entity This page describes the process of deleting an ensembler **without any related entities** (ensembling jobs or router versions). Navigate to the Ensemblers page. Click on the 'Delete' button: ![](/files/aLfQpICUJK3iPE0qAqPt) Type the ensembler's name in the text bar to confirm your decision: ![](/files/7nHjdW5Fx9LCtQrpcOv0) ![](/files/FvysTXPnOZhQs03cAkGi) Once the specified ensembler has been successfully deleted, you will no longer be able to see the ensembler on the Ensemblers page: ![](/files/Xzsg8uiZKx2ItamXWMpf) # Delete an Ensembler with active entities This page describes the process of deleting ensemblers **with related active entities** (ensembling jobs and/or router versions). Ensemblers with related active router versions or ensembling jobs cannot be deleted. Ensemblers that are currently used by any routers also cannot be deleted. Navigate to the Ensemblers page. Click on the delete button: ![](/files/aLfQpICUJK3iPE0qAqPt) The ensembler cannot be deleted since there are active router versions or ensembling jobs using the ensembler. Hence, the dialog will show the related entity that blocks the deletion process: ![](/files/PxCN6fVwxINSiZF8geDJ) If you still wish to delete the ensembler, please follow the instructions shown in the dialog. # Delete an Ensembler with inactive entities This page describes the process of deleting of ensemblers **with related inactive entities** (ensembling jobs and/or router versions). Deleting the ensembler will also delete any inactive entities related to the ensembler. Navigate to the Ensemblers page. Click on the delete button: ![](/files/aLfQpICUJK3iPE0qAqPt) The related router versions and the ensembling jobs shown in the dialog will get deleted together with the ensembler since they are inactive: ![](/files/kVnqMofwgo4wf1UgNWgj) Type the ensembler's name in the text bar to confirm your decision: ![](/files/X7WzuJDnBhGBTMmKcnVm) Once the specified ensembler has been successfully deleted, you will no longer be able to see the ensembler on the Ensemblers page: ![](/files/Xzsg8uiZKx2ItamXWMpf) # Experiments ## Onboarding Process For access to XP, users can self-onboard via the MLP UI. 1. Visit MLP Landing Page, open the sidebar and click `Experiments`. ![MLP Landing Turing Experiments](/files/MPLT834H4oFMg9so7Lwr) 2. You should see the following Landing page, if you have yet to setup the project for Experiments. ![Experiments Landing](/files/sXz19AcO3Rbeu5xcUwqQ) 3. Upon clicking 'here' in the previous page, you should see a form to input the necessary settings. ![Experiments Settings Create Form](/files/7saQ871NciOda2X2d3TV) 4. Enter a name for the Randomization Key and select the Segmenters. * The order of the segmenters determines the priority of the segmenters when optional segmenters are used. For example, if the chosen segmenters are `s2_ids`, and `days_of_week` (in that order) and a given request matches 2 experiments - one where the `s2_ids` is optional and another one where the `days_of_week` is optional, the s2\_ids experiment (where there is an exact match of the s2\_ids) will be chosen. For more information and examples, please refer to the [Experiment Hierarchy](/introduction/core-concepts/concepts-1#Experiment-Hierarchy) section in the Introduction page. * Where the segmenter may be computed from several different (groups of) variables at runtime, also select the desired variable mapping. For example, `s2_ids` may be supplied as `s2_id` or computed from `latitude,longitude`. This must be specified in the settings. 5. Click on Save. And voila! The onboarding is complete and you should see the configured settings. The project credentials (in particular, the `passkey`) would be required for running experiments ([Turing](https://github.com/caraml-dev/turing/tree/main/docs) takes care of this if you are running the experiments through its routers). ![Experiments Settings Details](/files/XAr4tPSdYkzEFobLHPou) # View Experiment Settings From the Experiments List page, clicking the 'arrow' and 'Settings' icon would bring you to the Settings Details page. ![Settings Details Button](/files/mGmYAQ57xX2cwBvA2Q1x) ## General Configuration ![Settings General View](/files/CNC05PbTN9Vtvrn3l6Ta) 1. General Info: General settings of the project. 2. Experimentation: Segmenters and experiment variables of the project. 3. Randomization: Randomization key of the project. ## Validation Configuration ![Settings Validation View](/files/hKVrhwAMPOlkMibGMoUC) 1. External Validation: URL of external validation to validate experiments. 2. Treatment Validation Rules: Rules used to validate treatment configuration of experiments. ### Treatment Validation Rules Treatment Validation Rules allows users to specify rules that the treatment configuration of experiments must comply with. XP uses [Go Template](https://pkg.go.dev/text/template) and the [Sprig library](http://masterminds.github.io/sprig/) as the template function. The treatment configuration will be templated with the predicate provided and the final output must return `true` for the operation to be permitted. This is useful to prevent careless mistakes while creating the treatment configuration, such as creating rules to ensure the weight of the models sums to 1 or to ensure that a control is set for every treatment. #### Sample rules * Checking if a field matches some string and float type provided ``` {{- or (and (eq .field1 "abc") (eq .field2 "def") (contains "float" (typeOf .field3.field4))) (and (eq .field1 "xyz") (eq .field2 "def") (contains "int" (typeOf .field3.field4))) -}} ``` * Checking sum of weights of models is 1 ``` {{- $sum := 0 -}} {{- range $index, $item := .config_details.models -}} {{- $sum = (addf $sum $item.weight) -}} {{- end -}} {{- eq $sum 1.0 -}} ``` # Modify Experiment Settings In the Settings Details Page, click 'More Actions' -> 'Edit Settings' ![Modify Settings Edit Button](/files/cVs5x8NDCHswNZJzCBdh) In the Edit Experiment Page, you can edit the `randomization_key`, the list of `segmenters` and their variable mappings. ![Modify Settings Edit Page](/files/KQ0DarHGM9l1D8PuMEak) When modifying settings, please take note of the following pointers: 1. Segmenters with asterisk(\*) are required segmenters that cannot be removed from the selected list. 2. Removing Segmenters will not be allowed when there are 1 or more 'active' and 'yet-to-end' experiments that are not defined granularly enough. The following scenario is an example: * Currently selected segmenters: `country`, `service` * Currently created experiments: ``` // Experiment 1 { "name": "test-exp-1", "created_at": "2022-06-27T10:00:00.0000Z", "segment": { "country": ["SG"], "service": [1] }, ... } // Experiment 2 { "name": "test-exp-2", "created_at": "2022-06-27T10:00:00.0000Z", "segment": { "country": ["SG"], "service": [2] }, ... } ``` Suppose we want to remove `service` segmenter from the project, this would make it impossible to distinguish between both experiments when XP tries to fetch a treatment because the only segmenter is `country` and both experiments have the same value, i.e, `SG` in this case. ## Edit Validation Validation configuration can be edited and configuration can be tested in the playground provided in the Edit Validation View. Both External Validation URL and Treatment Validation Rules can be left blank. From the Validation Detail page, click 'More Actions' -> 'Configure Validation'. ![Edit Validation Button](/files/Ne1qxy6YzERxsiOwFKQv) ### Edit Validation Page 1. In the Edit Validation Page, you can edit the Validation's configuration. ![Edit Validation](/files/DwgssFNDtIf9GZOK3e5W) 2. External Validation: URL of external validation to validate experiments. * Note that the external validation endpoint will be called whenever experiments or treatments are created or edited. XP will send a POST request to the following HTTP endpoint with the following payload that has to be handled by the custom web service: ``` POST /validate-experiment { "entity_type": // oneOf {experiment, treatment} "operation": // oneOf {create, update} "data": { } // The json object representing the new experiment or treatment being created or saved "context": { "current_data": {} // Current entity, in update operations } // Optional context information } ``` * A response with status code of 200 will be deemed as successful. A timeout of 5 seconds is also configured to treat any hanging call to the validation URL as a failure. 3. Treatment Validation Rules: Rules, specified in the form of [Go Templates](https://pkg.go.dev/text/template), used to validate treatment configuration of experiments. In addition to the default Go Template operations, those from the [Sprig library](http://masterminds.github.io/sprig/) are also supported. All rules will be evaluated on the treatment configuration and must return `true` for the operation to be permitted. Some examples below. To test that the data satisfies one of the following conditions: * `.field1`=`"abc"` and `.field2`=`"def"` and `.field3.field4` is of type `float` * (OR) `.field1`=`"xyz"` and `.field2`=`"def"` and `.field3.field4` is of type `int` ``` {{- or (and (eq .field1 "abc") (eq .field2 "def") (contains "float" (typeOf .field3.field4))) (and (eq .field1 "xyz") (eq .field2 "def") (contains "int" (typeOf .field3.field4))) -}} ``` To test that a certain property `weight` in an array field `.config_details.models` sums to 1 across all elements: ``` {{- $sum := 0 -}} {{- range $index, $item := .config_details.models -}} {{- $sum = (addf $sum $item.weight) -}} {{- end -}} {{- eq $sum 1.0 -}} ``` ### Creating or Updating an Experiment Sample experiment schema: ```json { "id":0, "project_id":1, "name":"test-experiment-create", "description":"Test description", "type":"Switchback", "interval":60, "tier":"default", "treatments":[ { "configuration":{ "meta":{ "created-by":"test" }, "weight":0.2 }, "name":"treatment", "traffic":100 } ], "segment":{ "country":[ "ID", "SG" ] } } ``` * The entire experiment will be sent to the validation url for validation * *Each* treatment configuration within the experiment will be validated with *all* of the treatment validation rules Both of these conditions must be fulfilled before the experiment can be created or updated successfully. ### Creating or Updating a Treatment Sample treatment schema: ```json { "field1":"abc", "field2":"def", "field3":{ "field4":0.1 } } ``` * The treatment configuration will be sent to the validation url for validation * The treatment configuration will be validated with *all* of the treatment validation rules Both of these conditions must be fulfilled before the treatment can be created or updated successfully. ## Validation Playground To test entities such as treatment configuration, before saving them, you can access the validation playground page by clicking the blue button `Playground` at the top right hand corner of the `Edit Validation Page`. ![Validation Playground Button](/files/N9uPVqrwsXfXcT5ztRDM) 1. External Validation: Select the `External Validation` radio button under `Validation Type` and fill in the `Sample Data` text field with a sample **payload** that you would like to send your validation url. 2. Treatment Validation Rules: Select the `Treatment Validation Rules` radio button under `Validation Type` and fill in the `Sample Data` text field with a **treatment configuration** that you would like to validate against the treatment validation rules. Click on validate once you are done to perform the validation. ![Validation Playground Confirmation](/files/Z4IudZwT3c9gtFx5I1EF) # Creating Experiments ## Type of experiments ### A/B Experiments A/B experiments require the traffic allocation to be specified for each treatment, and the sum of the traffic for all treatments should be 100. ### Switchback Experiments Switchback experiments, in the simplest form, are cyclical and require that the treatments do not carry any traffic specification. At every new time interval, the treatment that is chosen is the next in the list of treatments and the same treatment will be applied to all incoming requests. ## Experiment Creation Experiments can be created from the experiments landing page. ## 0. Create Experiment Click on the "Create Experiment" button on the landing page. ![Create Experiment Landing](/files/2YxObky0zkLuytC7cbUl) ## 1. Configure Experiment's General Settings a. In the Create Experiment's general settings page, fill up the form. The Switchback Configuration section will be shown for Switchback experiments. ![Create Experiment General](/files/NijXNW3QD8Q4MIHend9r) 1. **Name**: Name of experiment. 2. **Experiment** Type: A/B or Switchback. 3. **Description**: Description of experiment. 4. **Status**: Active or inactive experiment. Experiment status can be toggled later. 5. **Tier**: Default or override experiment. The tier makes it possible to schedule 2 experiments on the same segment (one in each tier) where the value of the tier serves as the tie-breaker (the override experiment is given preference). This is useful to schedule short spikes to temporarily override a long-running experiment. 6. **Duration**: Start and end time of experiment. For all experiments, start time must be in the future. 7. **Switchback Interval**: Duration for which each treatment is alternately applied in successive time intervals. b. Click the "Next" button. ## 2. Configure Experiment's Segmenters a. In the Create Experiment's segmenter page, fill up the segmenters configuration. The segmenters shown will be based on the project settings. Segmenters marked with an asterisk(\*) are required and cannot be left unset. All other segmenters are optional and where a value is not supplied, it results in a "weak" match and where it is supplied, there may be an "exact" match or a no match. For more information on optional segmenters and the matching behavior, please refer to the [Experiment Hierarchy](/introduction/core-concepts/concepts-1#Experiment-Hierarchy) section in the Introduction page. You may choose to select a Pre-configured Segment from the drop down as highlighted in red below and edit them in-place for use. ![Create Experiment Segment](/files/pRJe2YXvdI6vfbd7ZrTJ) Upon selection, the chosen Segment template values will be loaded into the respective segmenter fields configured for the project. ![Create Experiment Segment](/files/URb22PLVy7eISd89K686) 1. **s2\_ids**: S2 ids of experiment, delimited by newline. The values can be set at levels 10-14. 2. **days\_of\_the\_week**: Days of the week to run the experiment. 3. **hours\_of\_the\_week**: Hours of the week to run the experiment. b. Click the "Next" button. ## 3. Configure Experiment's Treatments a. Fill in the treatment(s) configurations. While selecting treatment(s) for the Experiment, you may create custom configuration or select a template. If a template is selected, the treatment fields will be auto-populated, further edit is possible (See `Creating Treatments` section to understand more about Treatments template). Likewise to an Experiment's Segment, when configuring treatments for the Experiment, you may choose to select a Pre-configured Treatment from the dropdown as highlighted in red below and edit them in-place for use. ![Create Experiment Treatments](/files/8v6TUNdiHNKb4uU6nKtL) Upon selection, the chosen Treatment template values will be loaded into the respective treatment-related fields. ![Create Experiment Treatments Fields](/files/S1Ap2IossMZ5Fz3K5WuW) 1. **Treatment Name**: Name of treatment (Input). 2. **Traffic Percentage**: Traffic allocation for treatment. Sum of traffic for all treatments should be 100. Traffic configuration is optional for switchback experiments. 3. **Configuration**: Treatment configuration JSON. b. Click "Save" to create the experiment. # Viewing Experiments Once the experiment has been created, you will be able to view the experiment's configuration on the landing page. ![View Experiment](/files/BDwQAw8M0XwWLy6TIKW9) ## Navigate to Experiment Details 1. Click on the row that contains the experiment. 2. You will now be able to see the Experiment Details View. ![View Experiment Details](/files/wGgPuFvH9LOdnBUNEznO) At the top row, you will be able to see your experiment name and a badge that indicates the status of experiment | Status | Description | Badge | | ----------- | ---------------------------------------------------- | ---------------------------------------------------------------- | | Running | Experiment is active and currently running | ![View Experiment Status Running](/files/kb24ZG8IBYctGYetsGqc) | | Scheduled | Experiment is active and start time is in the future | ![View Experiment Status Scheduled](/files/AMLVNpXyVux9qDX6vMcL) | | Completed | Experiment is active and end time is the past | ![View Experiment Status Completed](/files/MJGsb6r1PJ26H447qP6N) | | Deactivated | Experiment is inactive | ![View Experiment Status Inactive](/files/ARkF3deMJOjT9E0uaMpc) | ### Configuration The Configuration tab displays the selected experiment's details. These values are configured from creating or editing an experiment. 1. General Info: General settings of the experiment. 2. Activity: Activity details of experiment. 3. Segment: Segmenters of experiment. 4. Treatments: Treatments Configurations for 1 or more registered Treatment(s). ### Searching The UI supports two types of search - Basic and Advanced. Advanced search options enable filtering the experiments by different attributes. ## Basic Search 1. In the search panel, enter the experiment name or description to filter by. ![View Experiment Search Simple](/files/ePKC4l3aKOX5VNJHcqvl) ## Advanced Search 1. Click "Search Options", this will open up the Filters Panel. ![View Experiment Search Filter](/files/Egjz5a27X975zn03Eols) 2. In the Filters Panel, select the respective filters to apply. A "Filtered" badge will be shown beside experiment name to indicate that the experiments are filtered. ![View Experiment Search Filter](/files/Ppui2GdvSHVwrLh72ZNZ) ### History When an experiment is modified (edited / activated / deactivated) its existing configurations are saved as a historical version. All versions can be viewed from the **History** tab of the Experiment Detail view. ![View Experiment History](/files/97906ODgpe9pT1PeG0xI) The versions are ordered in the descending order of creation (the most recent version appearing on top). The Created and Updated dates of the version symbolize the duration that the configuration was applied in the experiment. Selecting a row opens the details of the version. ![View Experiment History](/files/zkeo4FuzHD2YRAeTWprW) # Modifying Experiments In the Experiment Details Page, click 'Edit Experiment'. ![Modify Experiment Edit Button](/files/LiYM3bqr0N6FAE0vJ5jR) In the Edit Experiment Page, you can edit the experiment's configuration. All fields except **Name** and **Experiment Type** are allowed to be modified. ![Modify Experiment Edit Page](/files/icR7aCxeMEreGx3BrL8F) ## Enabling / Disabling Experiments This impacts the `status` field in the experiment. Experiments are only considered to be running when the following conditions are met: * Current time falls within the experiment start and end times * Experiment status is 'active' Experiments can be created in the 'active' state and disabled later on, or vice versa, which can help instantly modify its behavior. 1. In the Experiment Details Page, you can click 'Activate Experiment' or 'Deactivate Experiment' button beside the 'Edit Experiment' button to change the status. The corresponding button will be shown based on the status. ![Modify Experiment Deactivate Button](/files/saaOAp3pkculzeHsqx8L) 2. Upon clicking the button, a confirmation window will be shown. ![Modify Experiment Activate Modal](/files/fXtVxpgrAhKyWDMw9sQ1) ![Modify Experiment Deactivate Modal](/files/dEHIeFPUR8upWzOX5hoM) ## Experiment History When an experiment is edited, the existing details in the experiment prior to the edit would be saved as a historical version and can be viewed from the **History** tab in the Experiment Details view. Note that status changes via the Activate / Deactivate action would still create a historical version but would not increment the version number. ## Deleting Experiments Experiments in XP cannot be deleted - users can simply disable experiments to prevent them from being run. # Running Experiments Experiments can be run using [routers](https://github.com/caraml-dev/turing/blob/main/docs/concepts.md) or independently. Based on the segmenters enabled for the project and the experiment variables mapped to them (in the project's settings), all the variable(s) must be provided in the fetch treatment request body. For each segmenter below, one of the specified (group of) variables can be configured in the project settings and subsequently, included in the fetch treatment call. | Segmenter | Fetch Treatment Input (One of) | | ---------------- | ------------------------------- | | S2ID | `s2id`, `(latitude, longitude)` | | Days of the Week | `tz` (timezone), `day_of_week` | | Hours of the Day | `tz` (timezone), `hour_of_day` | ## Running Experiments with Turing Routers When deploying a Turing router, the experiment engine can be configured to 'Turing Experiments'. For more information, check [Turing - Creating a Router](https://github.com/caraml-dev/turing/tree/main/docs/how-to/create-a-router). ## Running Experiments with API Experiments can be run independently using the POST endpoint (See details on the Treatment Swagger, in [Getting Started](/user-guides/01_getting_started-1)), with the required segmenter values and randomization unit (which may be optional for some Switchback experiments) in the request body. # Monitoring Experiments The Experiments may be monitored on Prometheus. This includes the following metrics. * Performance of the app (latency, throughput, error rate, etc.) and various components (such as calls to DB) * Resource utilization * Experiment Lookup * Treatment assignment (Matching, non-matching experiments) ## Treatment logs Treatment request and response log are available to be written to BigQuery or Kafka. # Creating Treatments From the Experiments List page, click on 'Treatments'. ![Create Treatment Navigation](/files/HvwXiDvKh0022prYAXGS) Treatments can be created from the treatments landing page. Created treatments can be used as a template, to initialise one or more treatments in an experiment (See `Creating Experiments - Configure Experiment's Treatments` for more information). ## 0. Create Treatment a. Click on the "Create Treatment" button on the landing page. ![Create Treatment Landing](/files/62XAhTzfuqqExwoHElCU) ## 1. Configure Treatment's General Settings a. In the Create Treatment's general settings page, you may either input the fields from scratch or choose to select a pre-configured Treatment template where upon selection, would fill up the respective fields of the form. Subsequently, you may continue editing the fields if necessary. ![Create Treatment General](/files/1vceYjdDxKUN0A562Y5X) 1. **Name**: Name of treatment. 2. **Status**: Active or inactive treatment. Treatment status can be toggled later. (You may refer to #Enabling/Disabling section in [Viewing Treatments](/user-guides/01_getting_started-1/10_viewing_treatments) document for more information.) 3. **Configuration**: Configuration of treatment. This needs to be a valid JSON string. b. Click "Save" to create the treatment. # Viewing Treatments Once the treatment has been created, you will be able to view the treatment's configuration on the landing page. ![View Treatment](/files/ofSYGjsz2t6TUjDj4DFh) ## Navigate to Treatment Details 1. Click on the row that contains the treatment. 2. You will now be able to see the Treatment Details View. ![View Treatment Details](/files/3oWWzS5UKlX0jRCTBPy2) At the top row, you will be able to see your treatment name. ### Configuration The Configuration tab displays the selected treatment's details. These values are configured from creating or editing a treatment. The treatment can be deleted from this page as well, which will also remove all of its historical versions. 1. Configuration: General settings of the treatment. 2. Activity: Activity details of treatment. ### Searching The UI supports searching by treatment name. In the search panel, enter the treatment name to filter by. ![View Treatment Search](/files/TEjTSwp02AqJh9KMVtNe) ### History When a treatment is modified (edited / activated / deactivated) its existing configurations are saved as a historical version. All versions can be viewed from the **History** tab of the Treatment Details view. ![View Treatment History](/files/tzETMxCmvebJII4619Vr) The versions are ordered in the descending order of creation (the most recent version appearing on top). The Created and Updated dates of the version symbolize the duration that the configuration was applied in the treatment. Selecting a row opens the details of the version. ![View Treatment History](/files/QU74FlUPWlaqQLiDAg7y) # Modifying Treatments 1. In the Treatment Details Page, click 'Edit Treatment' ![Modify Treatment Edit Button](/files/YuGIp8Z0hEM1O2QEl7cx) 2. In the Edit Treatment Page, you can edit the treatment's configuration. All fields except **Name** are allowed to be modified. Similar to creating a new Treatment template, you may choose to select an existing one and pre-fill some fields first rather than starting from scratch. ![Modify Treatment Edit Page](/files/6O9FLU2AV378YCqzud2P) ## Treatment History When a treatment is edited, the existing details in the treatment prior to the edit would be saved as a historical version and can be viewed from the History tab in the Treatment Details view. ## Deleting Treatments Treatments in XP are templates that can be deleted, deleting them prevents usage in new Experiments, but do not affect existing Experiments which were created with it. # Creating Segments From the Experiments List page, click on 'Segments'. ![Create Segment Navigation](/files/Bkdup9ptk7J2Y0CHJToA) Segments can be created from the segments landing page. Created segments can be used as a template, to initialise one or more segments in an experiment (See [`Creating Experiments - Configure Experiment's Segmenters`](https://github.com/caraml-dev/docs/blob/main/module/experiment/how-to/04_creating_experiments/README.md#2-Configure-Experiments-Segmenters) for more information). ## 0. Create Segment a. Click on the "Create Segment" button on the landing page. ![Create Segment Landing](/files/F528X8tWP8bO5HY4iwPp) ## 1. Configure Segment's General Settings 1. In the Create Segment's general settings page, you may either input the fields from scratch or choose to select a pre-configured Segment template where upon selection, would fill up the respective fields of the form. Subsequently, you may continue editing the fields if necessary. ![Create Segment General](/files/uk5GEOeEpnHu0UKbIYMB) 2. **Name**: Name of Segment. 3. **Configuration**: Configuration of Segment. The dropdowns that are shown are dependent on the project's segmenters. b. Click "Save" to create the Segment. # Viewing Segments Once the segment has been created, you will be able to view the segment's configuration on the landing page. ![View Segment](/files/mxnYI8rFLCWoQkHaf0X1) ## Navigate to Segment Details 1. Click on the row that contains the segment. 2. You will now be able to see the Segment Details View. If one or more of the project's segmenters have been removed since an experiment has been created, the segmenter highlighted in red is no longer in use for the experiment. ![View Segment Details](/files/8c858yr9fIXa2cgcI1j4) At the top row, you will be able to see your segment name. ### Configuration The Configuration tab displays the selected segment's details. These values are configured from creating or editing a segment. The segment can be deleted from this page as well, which will also remove all of its historical versions. 1. Activity: Activity details of segment. 2. Segment: Different segmenter settings of the Segment. ### Searching The UI supports searching by segment name. In the search panel, enter the segment name to filter by. ![View Segment Search](/files/ZDh3OD19LvdwZmStP4N6) ### History When a segment is modified (edited / activated / deactivated) its existing configurations are saved as a historical version. All versions can be viewed from the **History** tab of the Segment Detail view. ![View Segment History](/files/s2irYFbAtdOJQ7dzGjXZ) The versions are ordered in the descending order of creation (the most recent version appearing on top). The Created and Updated dates of the version symbolize the duration that the configuration was applied in the segment. Selecting a row opens the details of the version. ![View Segment History](/files/IxyjIWSvWoj4ieXVhTXz) # Modifying Segments 1. In the Segment Details Page, click 'Edit Segment'. ![Modify Segment Edit Button](/files/UFeXUDfF0fy1IMtdDTGJ) 2. In the Edit Segment Page, you can edit the segment's configuration. All fields except **Name** are allowed to be modified. Similar to creating a new Segment template, you may choose to select an existing one and pre-fill some fields first rather than starting from scratch. ![Modify Segment Edit Page](/files/jo9rHj85J0EYTFhTi6Sq) ## Segment History When a segment is edited, the existing details in the segment prior to the edit would be saved as a historical version and can be viewed from the History tab in the Segment Details view. ## Deleting Segments Segments in XP are templates that can be deleted, deleting them prevents usage in new Experiments, but do not affect existing Experiments which were created with it. # Creating Custom Segmenters From the Settings page, click on the 'Segmenters' tab. ![Create Custom Segmenter Navigation](/files/ZhckOP1iimEu6jRTmXqY) Created segmenters can be used just as global segmenters, but only in the project in which they are created in. ## 0. Create Segmenter 1. Click on 'More Actions', followed by the 'Create Segmenter' button on the landing page. ![Create Custom Segmenter Landing](/files/DPyg2WOj4Na3IUqdtOo1) ## 1. Configure Segmenter's General Settings 1. In the Create Segmenter's general settings page, fill up the given form ![Create\_Custom\_Segmenter General](/files/rTP5a8WUiBIrrSryBydz) 1. **Name**: Name of segmenter. 2. **Type**: Type of the segmenter (string, bool, integer or real). 3. **Description**: Description of segmenter. 4. **Required**: Indicates whether the segmenter must be selected in experiments. 5. **Multi-Valued**: Indicates whether the segmenter has multiple values. 6. **Options**: Name-value pairs for the segmenter values. This field needs to be a valid JSON object. 7. **Constraints**: Constraints for the segmenter when prerequisite conditions are met. 1. **Pre-Requisite Segmenter Values**: An array of objects with the fields `segmenter_name` and `segmenter_values`. This field needs to be a valid JSON array. 1. **Segmenter\_Name**: A string indicating the name of the segmenter. Note that **none** of the objects specified should have a `segmenter_name` with the same name as the segmenter that you are creating. 2. **Segmenter Values**: An array of segmenter values corresponding to the segmenter. 2. **Allowed Values**: An array of allowed segmenter values. Note that the values specified here should form a subset of the values specified in 'Options' and cannot be empty. This field needs to be a valid JSON array. 3. **Values Override**: New name-value mappings to overwrite the names for the values under 'Options'. Note that the values specified here should form a subset of the values specified in 'Options'. This field should specify the name-value mappings for each and every value in the allowed values field, and it needs to be a valid JSON object. 2. Click "Save" to create the Segmenter. # Viewing Custom Segmenters Once the segmenter has been created, you will be able to view the segmenter's configuration on the list page. ![View Segmenter](/files/w5wVKzluA0YRS0nG7B1P) ## Navigate to Segmenter Details 1. Click on the row that contains the segmenter. 2. You will now be able to see the Segmenter Details View. ![View Segmenter Details](/files/mwCu8Atg8rx2Ci4WexHn) At the top row, you will be able to see your segmenter name. ### Configuration The Configuration tab displays the selected segmenter's details, configured when creating or editing a segmenter. 1. General Info: General settings of the segmenter. 2. Activity: Activity details of segmenter. 3. Options: Options of segmenter. 4. Treatment Request Fields: Experiment variables to be supplied in the `FetchTreatment` request. 5. Constraints: Constraints of segmenter. # Modifying Custom Segmenters ## Modifying Segmenters 1. In the Segmenter Details Page, click 'More Actions' followed by 'Edit Segmenter'. ![Modify Custom Segmenter Edit Button](/files/dNViZkEn3N9xnpNPpz29) 2. In the Edit Segmenter Page, you can edit the segmenter's configuration. All fields except **Name** and **Type** are allowed to be modified. ![Modify Custom Segmenter Edit Page](/files/hHmbAQMCvORse1MHahWR) ## Deleting Segmenters Custom segmenters (as opposed to global segmenters) can be deleted by clicking on the 'Delete Segmenter' button in the 'More Actions' dropdown list. Note that custom segmenters can only be deleted if they have been deactivated, i.e. removed from the project settings. # Pipelines CaraML Pipelines are built upon another open source project [Flyte](https://flyte.org/). Work is in progress to provide documentations for CaraML's take of Flyte and how it integrates with other CaraML components, stay tuned! # Model Sample Notebooks Examples of using Merlin for different purposes are available to be tried out as Jupyter notebooks in the links below. You may want to clone the examples to your local directory and run them using Jupyter notebook. {% content-ref url="/pages/twls97uJkZuoajQyPjTG" %} [Deploy Standard Models](/tutorial-and-examples/07_examples/01_standard_model) {% endcontent-ref %} {% content-ref url="/pages/avpSoRZ45Tpt7RL1Z8uY" %} [Deploy PyFunc Model](/tutorial-and-examples/07_examples/02_pyfunc_model) {% endcontent-ref %} {% content-ref url="/pages/msFdiaLC999yDV8ppYhb" %} [Using Transformers](/tutorial-and-examples/07_examples/03_transformer) {% endcontent-ref %} {% content-ref url="/pages/iRwPmF8pDCJCi4YVqGHr" %} [Run Batch Prediction Job](/tutorial-and-examples/07_examples/04_batch_prediction) {% endcontent-ref %} {% content-ref url="/pages/h0CRJvbifboW9irxZijr" %} [Others examples on Models](/tutorial-and-examples/07_examples/05_others) {% endcontent-ref %} # Deploy Standard Models Try out the notebooks below to learn how to deploy different types of Standard Models to Merlin. ## Deploy SKLearn Model {% embed url="" %} ## Deploy XGBoost Model {% embed url="" %} ## Deploy Tensorflow Model {% embed url="" %} ## Deploy Pytorch Model {% embed url="" %} # Deploy PyFunc Model Try out the notebooks below to learn how to deploy PyFunc Models to Merlin. **Note on compatibility**: The Pyfunc servers are compatible with `protobuf>=3.12.0,<5.0.0`. Users whose models have a strong dependency on Protobuf `3.x.x` are advised to pin the library version in their conda environment, when submitting the model version. If using Protobuf `3.x.x`, users can do one of the following: * Use `protobuf>=3.20.0` - these versions support simplified class definitions and this is the recommended approach. * If you must use `protobuf>=3.12.0,<3.20.0`, other packages used in the Pyfunc server need to be downgraded as well. Please pin the following in your model’s conda environment: ```yaml dependencies: - pip: - protobuf==3.15.6 # Example older protobuf version - caraml-upi-protos<=0.3.6 - grpcio<1.49.0 - grpcio-reflection<1.49.0 - grpcio-health-checking<1.49.0 ``` ## Deploy PyFunc Model {% embed url="" %} ## Deploy PyFunc Model with Custom Prometheus Metrics {% embed url="" %} # Using Transformers Try out the notebooks below to learn how to deploy models with each type of transformers in Merlin. ## Deploy PyFunc Model with Standard Transformer {% embed url="" %} ## Deploy PyFunc Model with Custom Transformer {% embed url="" %} ## Deploy PyTorch Model with Custom Transformer {% embed url="" %} ## Deploy PyFunc Model with Feast Enricher Transformer {% embed url="" %} # Run Batch Prediction Job Try out the notebooks below to learn how to run batch prediction jobs using PyFunc V2 in Merlin. ## Run Iris Classifier Batch Prediction Job {% embed url="" %} ## Run New York Taxi Fare Batch Prediction Job {% embed url="" %} # Others examples on Models Try out the notebooks below to learn about other features of Merlin. ## Requesting CPU and Memory Resources {% embed url="" %} ## Working with Model Endpoint {% embed url="" %} # Router Examples More examples on how to use CaraML routers are on the way, stay tuned! # Feature Store Examples More examples on how to use CaraML feature store are on the way, stay tuned! # Pipeline Examples More examples on how to use CaraML pipelines are on the way, stay tuned! # Performing load test in CaraML Stay tuned for our guideline on how to conduct load test in CaraML # Best practice for CaraML Stay tuned for our guideline on CaraML best practices! # Feature Store SDK Feature Store SDK documents are on their way, stay tuned! # Models SDK Models SDK documents are on their way, stay tuned! # Routers SDK Routers SDK documents are on their way, stay tuned! # Pipeline SDK Pipeline SDK documents are on their way, stay tuned! # CaraML System FAQ In this section you will find some frequently asked questions regarding CaraML and its components. # Models FAQ You may refer to the below FAQs about CaraML Models component. {% content-ref url="/pages/h7lZ5yBknhzxizqSTIS9" %} [System Limitations](/troubleshooting-and-faqs/models-faq/08_limitations) {% endcontent-ref %} {% content-ref url="/pages/ubniuiaNoRwFZPUJNqav" %} [E2E Test](/troubleshooting-and-faqs/models-faq/e2e-test) {% endcontent-ref %} # System Limitations This article is an aggregation of the limits imposed on various components of the Merlin platform. ## Project ### Project Name A project name can only contain letters `a-z` (lowercase), numbers `0-9` and the dash `-` symbol. The maximum length of a project name is `50` characters. An example of a valid project name would be `gojek-project-01`. ## Model ### Model Name A model name can only contain letters `a-z` (lowercase), numbers `0-9` and the dash `-` symbol. The maximum length of a model name is `25` characters. An example of a valid model name would be `gojek-model-01`. ### Model Deployment The maximum number of model versions that can be deployed in an environment is `2` per model. ### Resources The maximum amount of CPU cores that can be allocated to a model is `4`. The maximum amount of memory that can be allocated to a model is `8GB`. ## Autoscaling Policy ### Autoscaling Autoscaling is enabled for both staging and production environment. User can set minimum and maximum number of replica during deployment. ### Scale Down to Zero "Scaling down to zero" is a feature in Merlin, which automatically reduces the number of model deployments to zero when they haven't received any traffic for 10 minutes. To make the model available again, it must receive HTTP traffic, which triggers a scale-up. This feature is only applicable when your autoscaling policy is set to either `RPS` or `Concurrency`." Note that, to utilise this feature, the minimum replicas for the deployment should be set to `0`. ### Logs ### Log History Users can only view the logs that are still in the model’s container. Link to the associated Stackdriver dashboard is provided in the log page to access past log. # Troubleshooting Deployment Errors This page discusses scenarios you may encounter during model deployment that will require troubleshooting, including: * Image building errors * Deployment errors ## Troubleshooting views Merlin provides the following views on the UI to troubleshoot a deployment: * **Logs** - the live console output when the image is building or the deployment is running * **History** - the list of deployment history status and message You can navigate to these views from the Model Version page by clicking on the Logs tab or History tab. ![Model Version's Logs & History tabs](/files/zUOH5gVHoEoD7xFqDfeo) ## Known Errors ### OOMKilled The "OOMKilled" error occurs when a container is terminated due to out-of-memory conditions. This typically happens when a container exceeds its allocated memory limit and the system is unable to provide additional memory. When this occurs, the container will be killed with exit code 137 to free up resources. This error can affect both image building and deployment steps. To resolve the OOMKilled error, follow these steps: 1. Check which components that got OOMKilled 2. Check affected component memory limits 3. Monitor memory usage 4. Optimize model memory usage 5. Adjust memory limits ![Failed image building due to OOMKilled](/files/FiifP9nkqtjC9My39sNy) ### Liveness or readiness probe failures Liveness and readiness probes are essential for ensuring the health and availability of Model services. The liveness probe is used to determine if a model is initialized and running properly, while the readiness probe indicates if a model is ready to serve traffic. When these probes fail, it can lead to service disruptions and impact the overall stability of the application. Troubleshooting steps: 1. For standard model type, check pre-trained model size 2. For pyfunc model type, check how model was initialized 3. Inspect model logs 4. Monitor resource utilization ![Model service in crash loop backoff state](/files/JZblf6NVAtRLeAWsTKuH) ### Image not found The "Image Not Found" error occurs when Merlin is unable to locate the specified container image. This can happen for various reasons, such as the image not being available in the specified registry, incorrect image name, or network issues preventing the image pull. To troubleshoot and resolve the "Image Not Found" error, follow these steps: 1. Verify image name and tag 2. Check image registry 3. Test image pull manually ![Image not found](/files/D8E0FBhBYwnQuVYYZ9EW) --- [Next Page](/llms-full.txt/1)