Custom Model
Custom model enables users to deploy any docker image that satisfy merlin requirements. Users are responsible to develop their own web service, build and publish the docker image, which later on can be deployed through Merlin.
Users should consider to use custom model, if they have one of the following conditions:
Model needs custom complex transformations (preprocess and postprocess) and want to use other languages than Python.
Using non standard model, e.g using heuristic or other ml framework model that have not been introduced in merlin.
Having dependencies with some os distribution packages.
Comparison With PyFunc Model
In high level PyFunc and custom model has similarity, they both enable users to specify custom logic and dependencies. The difference is mostly on the flexibility level and performance.
Web Service
Users can use any tech stack for web service
Users need to implement whole web service
Use python server, and users only need to modify core logic of prediction (infer function in this case)
Dependency
Users can specify any dependencies that is required. It can be os distribution package or library from specific programming language
Users can only specify python package dependencies
Performance
Users has more control on the performance of model. Since there is no limitation on tech stack that can be used
Users only has control on the infer function. Performance is rather slow because of the performance of python
Web Service Implementation
Users need to implement their own web service using any tech stack that suitable for their use case. Currently users can deploy web service using HTTP_JSON
or UPI_V1
protocol, both have different requirements that must be satisfied by the web server.
HTTP_JSON Custom Model
Users can add the artifact (model or anything else) in addition to the docker image when uploading the model. During the deployment, these artifacts will be made available in the directory specified by CARAML_ARTIFACT_LOCATION
environment variable.
Web service must open and listen to the port number given by CARAML_HTTP_PORT
environment variable.
Web service MUST implement the following endpoints:
/v1/models/{model_name}:predict
POST
For every inference or prediction calls, it will call this endpoint. Merlin will give the CARAML_MODEL_FULL_NAME
environment variable, this value can be used as {model_name} for this endpoint.
/v1/models/{model_name}
GET
This endpoint will be used to check model healthiness. Model can serve after this API return 200 status code.
/
GET
This endpoint will be used as server liveness. Return 200 if the model is healthy.
/metrics
GET
This endpoint is used by prometheus to pull the metrics produced by the predictor. The implementation of this endpoint is handled by prometheus library, for example this is how to implement the endpoint with golang.
UPI_V1 Custom Model
Similar with HTTP_JSON
custom model, users can add the artifact during model upload, and the uploaded artifacts will be available in the directory specified by CARAML_ARTIFACT_LOCATION
environment variable. The web server must implement service that defined in the UPI interface, also the server must open and listen to the port number given by CARAML_GRPC_PORT
environment variable.
If users want to emit metrics from this web server, they need to create scrape metrics REST endpoint. The challenge here, the knative (the underlying k8s deployment tools that merlin use) doesn't open multiple ports, hence the REST endpoint must be running on the same port as gRPC server (using port number given by CARAML_GRPC_PORT
). Not every programming language can support running multiple protocol (gRPC and HTTP in this case) on the same port, for Go language users can use cmux to solve this problem, otherwise users can use push metrics to pushgateway
Environment Variables
As mentioned in the previous section, there are several environment variables that will be supplied by Merlin control plane to the custom model. Below are the list of the variables
STORAGE_URI
Contains the URI where the model
artifacts is remotely stored
CARAML_HTTP_PORT
Port that must be openend when the model is deployed with HTTP_JSON
protocol
CARAML_GRPC_PORT
Port that must be opened when the model is deployed with UPI_V1
protocol
CARAML_MODEL_NAME
Name of merlin model
CARAML_MODEL_VERSION
Merlin model version
CARAML_MODEL_FULL_NAME
Full name merlin model, per current version it use {CARAML_MODEL_NAME}-{CARAML_MODEL_VERSION}
format
CARAML_ARTIFACT_LOCATION
Local path where the model artifacts will be stored
Docker Image
Docker image must contains web service application and dependencies that must be installed in order to run the web service. Users are responsible for building the docker image as well as for publishing it. Please make sure the k8s cluster (where model will be deployed) have access to pull the docker image.
Deployment
Using Merlin SDK
Most of the method that used in the above snipped is commonly used by all the model deployment, but log_custom_model
method. log_custom_model
is method exclusively used to upload custom model. Below are the method parameters that can be specified during the invocation
image
Docker image that will be used as predictor
Yes
model_dir
Directory that will be uploaded to MLFlow
No
command
Command to run docker image
No
args
Arguments that needs to be specified when running docker
No
Deployment Flow
Create new model version
Log custom model, specify image and model directory that contains artifacts that need to be uploaded
Deploy. There is no difference with other model deployments
Last updated