Standard Transformer Expressions

Standard Transformer provides several built-in functions that are useful for common ML use-cases. These built-in functions are accessible from within expression context.

Geospatial

Geohash

Geohash calculates geohash of latitude and longitude with the given precision.

Input

Output

Geohash of location with the given precision.

Example

Input:
{
  "latitude": 1.0,
  "longitude": 2.0
}

Standard Transformer Config:
variables:
- name: geohash
  expression: Geohash("$.latitude", "$.longitude", 12)

Output: `"s01mtw037ms0"`

S2ID

S2ID calculates S2ID cell of latitude and longitude with the given level.

Input

Output

S2ID cell of the location in certain level.

Example

Input:
{
  "latitude": 1.0,
  "longitude": 2.0
}

Standard Transformer Config:
variables:
- name: s2id
  expression: S2ID("$.latitude", "$.longitude", 12)

Output: `"1154732743855177728"`

HaversineDistance

HaversineDistance calculates Haversine distance of two points (given by their latitude and longitude).

Input

Output

The haversine distance between 2 points in kilometer.

Example

Input:
{
  "pickup": {
    "latitude": 1.0,
    "longitude": 2.0
  },
  "dropoff": {
    "latitude": 1.2,
    "longitude": 2.2
  }
}

Standard Transformer Config:
variables:
- name: haversine_distance
  expression: HaversineDistance("$.pickup.latitude", "$.pickup.longitude", "$.dropoff.latitude", "$.dropoff.longitude")

HaversineDistanceWithUnit

HaversineDistanceWithUnit calculates Haversine distance of two points (given by their latitude and longitude) and given the distance unit

Input

Output

The haversine distance between 2 points.

Example

Input:
{
  "pickup": {
    "latitude": 1.0,
    "longitude": 2.0
  },
  "dropoff": {
    "latitude": 1.2,
    "longitude": 2.2
  }
}

Standard Transformer Config:
variables:
- name: haversine_distance
  expression: HaversineDistanceWithUnit("$.pickup.latitude", "$.pickup.longitude", "$.dropoff.latitude", "$.dropoff.longitude", "m")

PolarAngle

PolarAngle calculates polar angles between two points (given by their latitude and longitude) in radian.

Input

Output

The polar angles between 2 points in radian.

Example

Input:
{
  "pickup": {
    "latitude": 1.0,
    "longitude": 2.0
  },
  "dropoff": {
    "latitude": 1.2,
    "longitude": 2.2
  }
}

Standard Transformer Config:
variables:
- name: polar_angle
  expression: PolarAngle("$.pickup.latitude", "$.pickup.longitude", "$.dropoff.latitude", "$.dropoff.longitude")

GeohashDistance

GeohashDistance will calculate haversine distance between two geohash. It will convert a geohash into the center point (latitude, longitude) of that geohash and calculate haversine distance based on that point.

Input

Output

Haversine Distance between two geohash calculated from the center point of that geohash

Example

Input:
{
  "pickup_geohash": "qqgggnwxx",
  "dropoff_geohash": "qqgggnweb"
}

Standard Transformer Config:
variables:
- name: geohash_distance
  expression: GeohashDistance("$.pickup_geohash", "$.dropoff_geohash", "m")

GeohashAllNeighbors

GeohashAllNeighbors will find all neighbors of geohash from all directions

Input

Output

List of neighbors of given geohash

Example

Input:
{
  "pickup_geohash": "qqgggnwxx",
  "dropoff_geohash": "qqgggnweb"
}

Standard Transformer Config:
variables:
- name: geohash_distance
  expression: GeohashAllNeighbors("$.pickup_geohash")

GeohashNeighborForDirection

GeohashNeighborForDirection will find a neighbor of geohash given the direction

Input

Output

Neighbor of given geohash

Example

Input:
{
  "pickup_geohash": "qqgggnwxx",
  "dropoff_geohash": "qqgggnweb"
}

Standard Transformer Config:
variables:
- name: geohash_distance
  expression: GeohashNeighborForDirection("$.pickup_geohash", "north")

JSON

JsonExtract

Given a JSON string as value, you can use JsonExtract to extract JSON value from that JSON string.

Input

Output

JSON value within a JSON string pointed by the first JSONPath argument.

Example

Input:
{
  "details": "{\"merchant_id\": 9001}"
}

Standard Transformer Config:
variables:
- name: merchant_id
  valueType: STRING
  expression: JsonExtract("$.details", "$.merchant_id")

Output: `"9001"`

Statistics

CumulativeValue

CumulativeValue is a function that accumulates values based on the index and its predecessors. E.g., [1, 2, 3] => [1, 1+2, 1+2+3] => [1, 3, 6].

Input

Output

Array of cumulative values.

Example

Input:
{
  "fares": [10000, 20000, 50000]
}

Standard Transformer Config:
variables:
- name: cumulative_fares
  expression: CumulativeValue($.fares)

Output: `[10000, 30000, 80000]`

Time

Now

Return current local timestamp.

Input

None

Output

Current local timestamp.

Example

Standard Transformer Config:
variables:
- name: currentTime
  expression: Now()

DayOfWeek

Return number representations of the day in a week, given the timestamp and timezone.

SUNDAY(0), MONDAY(1), TUESDAY(2), WEDNESDAY(3), THURSDAY(4), FRIDAY(5), SATURDAY(6).

Input

Output

Day number.

Example

Input:
{
  "timestamp": "1637605459"
}

Standard Transformer Config:
variables:
- name: day_of_week
  expression: DayOfWeek("$.timestamp", "Asia/Jakarta")

Output: `2`

IsWeekend

Return 1 if given timestamp is weekend (Saturday or Sunday), otherwise 0.

Input

Output

1 if weekend, 0 if not

Example

Input:
{
  "timestamp": "1637445044",
  "timezone": "Asia/Jakarta"
}

Standard Transformer Config:
variables:
- name: is_weekend
  expression: IsWeekend("$.timestamp", "$.timezone")

Output: `1`

FormatTimestamp

FormatTimestamp converts timestamp in given location into formatted date time string.

Input

Output

Date time.

Examples

Input:
{
  "timestamp": "1637691859"
}

Standard Transformer Config:
variables:
- name: datetime
  expression: FormatTimestamp("$.timestamp", "Asia/Jakarta", "2006-01-02")

Output: `"2021-11-24"`

ParseTimestamp

ParseTimestamp converts timestamp in integer or string format to time.

Input

Output

Parsed timestamp.

Examples

Input:
{
  "timestamp": "1619541221"
}

Standard Transformer Config:
variables:
- name: parsed_timestamp
  expression: ParseTimestamp("$.timestamp")

Output: `"2021-04-27 16:33:41 +0000 UTC"`

ParseDateTime

ParseDateTime converts datetime given with specified format layout (e.g. RFC3339) into time.

Input

Output

Parsed date time.

Examples

Input:
{
  "datetime": "2021-11-30 15:00:00",
  "location": "Asia/Jayapura"
}

Standard Transformer Config:
variables:
- name: parsed_datetime
  expression: ParseDateTime("$.datetime", "$.location", "2006-01-02 15:04:05")

Output: `"2021-11-30 15:00:00 +0900 WIT"`

Series Expression

Series expression is function that can be invoked by series (column) values in a table

Get

Get will retrieve a row in series based on the given index

Input

Output

Single series row

Examples

Suppose users have table yourTableName

Users try to retrieve index 2 for series avg_order_1_day

Standard Transformer Config:

variables:
- name: total_order_1_day
  expression: yourTableName.Col("avg_order_1_day").Get(2)

Output: 4000

IsIn

IsIn checks whether value in a row is part of the given array, the result will be a new series that has boolean type

Input

Output

New Series that has boolean type and same dimension with original series

Examples

Suppose users have table yourTableName

Standard Transformer Config:

variables:
- name: bool_series
  expression: yourTableName.Col("avg_order_1_day").IsIn([2000, 3000])

Output:

StdDev

StdDev is a function to calculate standard deviation from series values. The output will be single value

Input

No Input

Output

Single value with float type

Examples

Suppose users have table yourTableName

Standard Transformer Config:

variables:
- name: std_dev
  expression: yourTableName.Col("avg_cancellation_rate_30_day").StdDev()

Output: 0.0068475461947247

Mean

Mean is a function to calculate mean value from series values. The output will be single value

Input

No Input

Output

Single value with float type

Examples

Suppose users have table yourTableName

Standard Transformer Config:

variables:
- name: mean
  expression: yourTableName.Col("avg_order_1_day").Mean()

Output: 3000

Median

Median is a function to calculate median value from series values. The output will be single value

Input

No Input

Output

Single value with float type

Examples

Suppose users have table yourTableName

Standard Transformer Config:

variables:
- name: median
  expression: yourTableName.Col("avg_order_1_day").Median()

Output: 3000

Max

Max is a function to find max value from series values. The output will be single value

Input

No Input

Output

Single value with float type

Examples

Suppose users have table yourTableName

Standard Transformer Config:

variables:
- name: max
  expression: yourTableName.Col("avg_order_1_day").Max()

Output: 4000

MaxStr

MaxStr is a function to find max value from series values. The output will be single value in string type

Input

No Input

Output

Single value with string type

Examples

Suppose users have table yourTableName

Standard Transformer Config:

variables:
- name: max_str
  expression: yourTableName.Col("avg_order_1_day").MaxStr()

Output: "4000"

Min

Min is a function to find minimum value from series values. The output will be single value in float type

Input

No Input

Output

Single value with float type

Examples

Suppose users have table yourTableName

Standard Transformer Config:

variables:
- name: min
  expression: yourTableName.Col("avg_order_1_day").Min()

Output: 2000

MinStr

MinStr is a function to find minimum value from series values. The output will be single value in string type

Input

No Input

Output

Single value with string type

Examples

Suppose users have table yourTableName

Standard Transformer Config:

variables:
- name: min_str
  expression: yourTableName.Col("avg_order_1_day").MinStr()

Output: "2000"

Quantile

Quantile is a function to returns the sample of x such that x is greater than or equal to the fraction p of samples

Input

Fraction in float type

Output

Single value with float type

Examples

Suppose users have table yourTableName

Standard Transformer Config:

variables:
- name: quantile_0.9
  expression: yourTableName.Col("rank").Quantile(0.9)

Output: 9

Sum

Sum is a function to sum all the values in the seriess. The output will be single value in float type

Input

No Input

Output

Single value with float type

Examples

Suppose users have table yourTableName

Standard Transformer Config:

variables:
- name: sum
  expression: yourTableName.Col("avg_order_1_day").Sum()

Output: 9000

Flatten

Flatten is a function to flatten all values in a series, this is suitable for series that has list type, for non list the result will be the same with the original seriess

Input

No Input

Output

New Series that the value already flatten

Examples

Suppose users have table yourTableName

Standard Transformer Config:

variables:
- name: restaurant_ids
  expression: yourTableName.Col("nearby_restaurant_ids").Flatten()

Output:

Unique

Unique is a function to return all values without duplication.

Input

No Input

Output

New Series that has unique value for each row

Examples

Suppose users have table yourTableName

Standard Transformer Config:

variables:
- name: unique_restaurant_id
  expression: yourTableName.Col("restaurant_id").Unique()

Output:

variables:
- name: rating
  expression: yourTableName.Col("rating").Unique()

Output: