Custom Model
Custom model enables users to deploy any docker image that satisfy merlin requirements. Users are responsible to develop their own web service, build and publish the docker image, which later on can be deployed through Merlin.
Users should consider to use custom model, if they have one of the following conditions:
Model needs custom complex transformations (preprocess and postprocess) and want to use other languages than Python.
Using non standard model, e.g using heuristic or other ml framework model that have not been introduced in merlin.
Having dependencies with some os distribution packages.
Comparison With PyFunc Model
In high level PyFunc and custom model has similarity, they both enable users to specify custom logic and dependencies. The difference is mostly on the flexibility level and performance.
Web Service Implementation
Users need to implement their own web service using any tech stack that suitable for their use case. Currently users can deploy web service using HTTP_JSON
or UPI_V1
protocol, both have different requirements that must be satisfied by the web server.
HTTP_JSON Custom Model
Users can add the artifact (model or anything else) in addition to the docker image when uploading the model. During the deployment, these artifacts will be made available in the directory specified by CARAML_ARTIFACT_LOCATION
environment variable.
Web service must open and listen to the port number given by CARAML_HTTP_PORT
environment variable.
Web service MUST implement the following endpoints:
UPI_V1 Custom Model
Similar with HTTP_JSON
custom model, users can add the artifact during model upload, and the uploaded artifacts will be available in the directory specified by CARAML_ARTIFACT_LOCATION
environment variable. The web server must implement service that defined in the UPI interface, also the server must open and listen to the port number given by CARAML_GRPC_PORT
environment variable.
If users want to emit metrics from this web server, they need to create scrape metrics REST endpoint. The challenge here, the knative (the underlying k8s deployment tools that merlin use) doesn't open multiple ports, hence the REST endpoint must be running on the same port as gRPC server (using port number given by CARAML_GRPC_PORT
). Not every programming language can support running multiple protocol (gRPC and HTTP in this case) on the same port, for Go language users can use cmux to solve this problem, otherwise users can use push metrics to pushgateway
Environment Variables
As mentioned in the previous section, there are several environment variables that will be supplied by Merlin control plane to the custom model. Below are the list of the variables
Docker Image
Docker image must contains web service application and dependencies that must be installed in order to run the web service. Users are responsible for building the docker image as well as for publishing it. Please make sure the k8s cluster (where model will be deployed) have access to pull the docker image.
Deployment
Using Merlin SDK
Most of the method that used in the above snipped is commonly used by all the model deployment, but log_custom_model
method. log_custom_model
is method exclusively used to upload custom model. Below are the method parameters that can be specified during the invocation
Deployment Flow
Create new model version
Log custom model, specify image and model directory that contains artifacts that need to be uploaded
Deploy. There is no difference with other model deployments
Last updated