CaraML Docs
CaraML Homepage
  • Introduction
    • What is CaraML?
    • Architecture
      • Feature Store Architecture
      • Models Architecture
      • Routers Architecture
      • Experiments Architecture
      • Pipelines Architecture
    • Core Concepts
      • Models Concepts
      • Router Concepts
      • Experiment Concepts
  • User guides
    • Projects
      • Create a project
      • Managing secrets
    • Feature Store
    • Models
      • Create a Model
        • Custom Model
      • Deploy a Model
        • Deploying a Model Version
        • Severing a Model Version
        • Configuring Transformer
          • Standard Transformer
            • Standard Transformer Expressions
            • Standard Transformer UPI
          • Custom Transformer
        • Redeploying a Model Version
      • Deleting a Model
      • Configuring Alerts
      • Batch Prediction
      • Model Schema
      • Model Observability
    • Routers
      • Creating a Router
        • Configure general settings
        • Configure routes
        • Configure traffic rules
        • Configure autoscaling
        • Configure experiment engine
        • Configure enricher
        • Configure ensembler
        • Configure logging
      • Viewing Routers
        • Configuration
        • History
        • Logs
        • More actions
      • Edit Routers
      • Monitoring router
        • Monitor Router Performance
        • Configure Alerts
      • Undeploying Router
      • Redeploying Router
        • Redeploy undeployed router
        • Redeploy version from history
        • Redeploy version from version details page
      • Deleting Router
        • Deleting router versions
        • Deleting router versions from details page
        • Deleting routers
      • Deleting Emsemblers
        • Delete an Ensembler without related entity
        • Delete an Ensembler with active entities
        • Delete an Ensembler with inactive entities
    • Experiments
      • View Experiment Settings
      • Modify Experiment Settings
      • Creating Experiments
      • Viewing Experiments
      • Modifying Experiments
      • Running Experiments
      • Monitoring Experiments
      • Creating Treatments
      • Viewing Treatments
      • Modifying Treatments
      • Creating Segments
      • Viewing Segments
      • Modifying Segments
      • Creating Custom Segmenters
      • Viewing Custom Segmenters
      • Modifying Custom Segmenters
    • Pipelines
  • Tutorial and Examples
    • Model Sample Notebooks
      • Deploy Standard Models
      • Deploy PyFunc Model
      • Using Transformers
      • Run Batch Prediction Job
      • Others examples on Models
    • Router Examples
    • Feature Store Examples
    • Pipeline Examples
    • Performing load test in CaraML
    • Best practice for CaraML
  • CaraML SDK
    • Feature Store SDK
    • Models SDK
    • Routers SDK
    • Pipeline SDK
  • Troubleshooting and FAQs
    • CaraML System FAQ
    • Models FAQ
      • System Limitations
      • Troubleshooting Deployment Errors
      • E2E Test
    • Routers FAQ
    • Experiments FAQ
    • Feature Store FAQ
    • Pipelines FAQ
    • CaraML Error Messages
  • Deployment Guide
    • Deploying CaraML
      • Local Development
    • Monitoring and alerting
      • Configure a monitoring backend
      • Configure an alerting backend
    • Prerequisites and Dependencies
    • System Benchmark results
    • Experiment Treatment Service
  • Release Notes
    • CaraML Release Notes
Powered by GitBook
On this page
  • Type of experiments
  • A/B Experiments
  • Switchback Experiments
  • Experiment Creation
  • 0. Create Experiment
  • 1. Configure Experiment's General Settings
  • 2. Configure Experiment's Segmenters
  • 3. Configure Experiment's Treatments
  1. User guides
  2. Experiments

Creating Experiments

PreviousModify Experiment SettingsNextViewing Experiments

Last updated 2 years ago

Type of experiments

A/B Experiments

A/B experiments require the traffic allocation to be specified for each treatment, and the sum of the traffic for all treatments should be 100.

Switchback Experiments

Switchback experiments, in the simplest form, are cyclical and require that the treatments do not carry any traffic specification. At every new time interval, the treatment that is chosen is the next in the list of treatments and the same treatment will be applied to all incoming requests.

Experiment Creation

Experiments can be created from the experiments landing page.

0. Create Experiment

Click on the "Create Experiment" button on the landing page.

1. Configure Experiment's General Settings

a. In the Create Experiment's general settings page, fill up the form. The Switchback Configuration section will be shown for Switchback experiments.

  1. Name: Name of experiment.

  2. Experiment Type: A/B or Switchback.

  3. Description: Description of experiment.

  4. Status: Active or inactive experiment. Experiment status can be toggled later.

  5. Tier: Default or override experiment. The tier makes it possible to schedule 2 experiments on the same segment (one in each tier) where the value of the tier serves as the tie-breaker (the override experiment is given preference). This is useful to schedule short spikes to temporarily override a long-running experiment.

  6. Duration: Start and end time of experiment. For all experiments, start time must be in the future.

  7. Switchback Interval: Duration for which each treatment is alternately applied in successive time intervals.

b. Click the "Next" button.

2. Configure Experiment's Segmenters

You may choose to select a Pre-configured Segment from the drop down as highlighted in red below and edit them in-place for use.

  1. s2_ids: S2 ids of experiment, delimited by newline. The values can be set at levels 10-14.

  2. days_of_the_week: Days of the week to run the experiment.

  3. hours_of_the_week: Hours of the week to run the experiment.

b. Click the "Next" button.

3. Configure Experiment's Treatments

a. Fill in the treatment(s) configurations. While selecting treatment(s) for the Experiment, you may create custom configuration or select a template. If a template is selected, the treatment fields will be auto-populated, further edit is possible (See Creating Treatments section to understand more about Treatments template).

Likewise to an Experiment's Segment, when configuring treatments for the Experiment, you may choose to select a Pre-configured Treatment from the dropdown as highlighted in red below and edit them in-place for use.

  1. Treatment Name: Name of treatment (Input).

  2. Traffic Percentage: Traffic allocation for treatment. Sum of traffic for all treatments should be 100. Traffic configuration is optional for switchback experiments.

  3. Configuration: Treatment configuration JSON.

b. Click "Save" to create the experiment.

a. In the Create Experiment's segmenter page, fill up the segmenters configuration. The segmenters shown will be based on the project settings. Segmenters marked with an asterisk(*) are required and cannot be left unset. All other segmenters are optional and where a value is not supplied, it results in a "weak" match and where it is supplied, there may be an "exact" match or a no match. For more information on optional segmenters and the matching behavior, please refer to the section in the Introduction page.

Upon selection, the chosen Segment template values will be loaded into the respective segmenter fields configured for the project.

Upon selection, the chosen Treatment template values will be loaded into the respective treatment-related fields.

Create Experiment General
Create Experiment Segment
Create Experiment Treatments Fields
Create Experiment Segment
Create Experiment Treatments
Create Experiment Landing
Experiment Hierarchy