Custom Model

Custom model enables users to deploy any docker image that satisfy merlin requirements. Users are responsible to develop their own web service, build and publish the docker image, which later on can be deployed through Merlin.

Users should consider to use custom model, if they have one of the following conditions:

  • Model needs custom complex transformations (preprocess and postprocess) and want to use other languages than Python.

  • Using non standard model, e.g using heuristic or other ml framework model that have not been introduced in merlin.

  • Having dependencies with some os distribution packages.

Comparison With PyFunc Model

In high level PyFunc and custom model has similarity, they both enable users to specify custom logic and dependencies. The difference is mostly on the flexibility level and performance.

Factor
Custom Model
Pyfunc Model

Web Service

  • Users can use any tech stack for web service

  • Users need to implement whole web service

Use python server, and users only need to modify core logic of prediction (infer function in this case)

Dependency

Users can specify any dependencies that is required. It can be os distribution package or library from specific programming language

Users can only specify python package dependencies

Performance

Users has more control on the performance of model. Since there is no limitation on tech stack that can be used

Users only has control on the infer function. Performance is rather slow because of the performance of python

Web Service Implementation

Users need to implement their own web service using any tech stack that suitable for their use case. Currently users can deploy web service using HTTP_JSON or UPI_V1 protocol, both have different requirements that must be satisfied by the web server.

HTTP_JSON Custom Model

Users can add the artifact (model or anything else) in addition to the docker image when uploading the model. During the deployment, these artifacts will be made available in the directory specified by CARAML_ARTIFACT_LOCATION environment variable.

Web service must open and listen to the port number given by CARAML_HTTP_PORT environment variable.

Web service MUST implement the following endpoints:

Endpoint
HTTP Method
Description

/v1/models/{model_name}:predict

POST

For every inference or prediction calls, it will call this endpoint. Merlin will give the CARAML_MODEL_FULL_NAME environment variable, this value can be used as {model_name} for this endpoint.

/v1/models/{model_name}

GET

This endpoint will be used to check model healthiness. Model can serve after this API return 200 status code.

/

GET

This endpoint will be used as server liveness. Return 200 if the model is healthy.

/metrics

GET

This endpoint is used by prometheus to pull the metrics produced by the predictor. The implementation of this endpoint is handled by prometheus library, for example this is how to implement the endpoint with golang.

UPI_V1 Custom Model

Similar with HTTP_JSON custom model, users can add the artifact during model upload, and the uploaded artifacts will be available in the directory specified by CARAML_ARTIFACT_LOCATION environment variable. The web server must implement service that defined in the UPI interface, also the server must open and listen to the port number given by CARAML_GRPC_PORT environment variable.

If users want to emit metrics from this web server, they need to create scrape metrics REST endpoint. The challenge here, the knative (the underlying k8s deployment tools that merlin use) doesn't open multiple ports, hence the REST endpoint must be running on the same port as gRPC server (using port number given by CARAML_GRPC_PORT). Not every programming language can support running multiple protocol (gRPC and HTTP in this case) on the same port, for Go language users can use cmux to solve this problem, otherwise users can use push metrics to pushgateway

Environment Variables

As mentioned in the previous section, there are several environment variables that will be supplied by Merlin control plane to the custom model. Below are the list of the variables

Name
Description

STORAGE_URI

Contains the URI where the model artifacts is remotely stored

CARAML_HTTP_PORT

Port that must be openend when the model is deployed with HTTP_JSON protocol

CARAML_GRPC_PORT

Port that must be opened when the model is deployed with UPI_V1 protocol

CARAML_MODEL_NAME

Name of merlin model

CARAML_MODEL_VERSION

Merlin model version

CARAML_MODEL_FULL_NAME

Full name merlin model, per current version it use {CARAML_MODEL_NAME}-{CARAML_MODEL_VERSION} format

CARAML_ARTIFACT_LOCATION

Local path where the model artifacts will be stored

Docker Image

Docker image must contains web service application and dependencies that must be installed in order to run the web service. Users are responsible for building the docker image as well as for publishing it. Please make sure the k8s cluster (where model will be deployed) have access to pull the docker image.

Deployment

Using Merlin SDK

resource_request = ResourceRequest(1, 1, "1", "1Gi")
model_dir = "model_dir"
with merlin.new_model_version() as v:
    v.log_custom_model(image="ghcr.io/yourcustommodelimage", model_dir=model_dir)

endpoint = merlin.deploy(v, resource_request= resource_request, protocol = Protocol.HTTP_JSON)
# endpoint = merlin.deploy(v, resource_request= resource_request, protocol = Protocol.UPI_V1) if using UPI

Most of the method that used in the above snipped is commonly used by all the model deployment, but log_custom_model method. log_custom_model is method exclusively used to upload custom model. Below are the method parameters that can be specified during the invocation

Parameter
Description
Required

image

Docker image that will be used as predictor

Yes

model_dir

Directory that will be uploaded to MLFlow

No

command

Command to run docker image

No

args

Arguments that needs to be specified when running docker

No

Deployment Flow

  • Create new model version

  • Log custom model, specify image and model directory that contains artifacts that need to be uploaded

  • Deploy. There is no difference with other model deployments

Last updated