Schema and Mappings

EventNative is a "schemaless" by design. Any JSON structure can be sent to our backend. If the destination is a relational database, EventNative will automatically adjust the schema, apply typecast, and add the necessary columns.

Workflow

Workflow Schema

Step1: Collection

The event comes from our Web or App collector. It has following JSON structure:

Structure
Example
Structure
{
"api_key": "63c13a7c-7d61-4a25-b9de-ab14341be4e5", //API key
"event_type": "3rdparty",
//If event was intercepted from GA or Segment, name of source.
//'eventn' otherwise
"src": "ga|eventn|ajs",
//If event was intercepted a copy of intercepted data
"src_payload": {...},
//Data of event (url, id and so on)
"eventn_ctx": {...},
"eventn_data": {...}
}
Example
{
"api_key": "63c13a7c-7d61-4a25-b9de-ab14341be4e5", //API key
"event_type": "3rdparty",
"src": "ga|eventn|ajs",
"src_payload": {
"gjid": "",
"ga_protocol_version": "1",
"ga_property": "UA-139969969-2",
"client_id": "1484854471.1594305592",
"ga_user_id": "GA1.1.524175970.1596394943",
"screen_size": "1792x1120",
"viewport_size": "1792x986",
"document_encoding": "UTF-8",
"screen_color": "30-bit",
"user_language": "en-us",
"event_type": "pageview",
"url": "http://localhost/tracker-demo/demo.html?_ijt=u5jsi606103iugi7mjphngrc7i",
"document_title": "kSenseTrackingDemo",
"java_installed": "0"
},
"eventn_ctx": {
"event_id": "dq4s1q6rrg",
"user": {
"anonymous_id": "dorhlsckco"
},
"user_agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.89 Safari/537.36",
"utc_time": "2020-08-03T14:35:58.638Z",
"local_tz_offset": 240,
"referer": "",
"url": "http://localhost:63342/tracker-demo/demo.html?_ijt=u5jsi606103iugi7mjphngrc7i",
"page_title": "kSenseTrackingDemo"
},
"eventn_data": {
"customer_group": "test_group"
}
}

Step 2: Enrichment

The event is enriched with additional data. The following fields are added inside eventn_ctx node:

"eventn_ctx": {
"location": {
"ip": "127.0.0.1",
"country": "US",
//other location nodes: city, region, state etc will be addede as well
},
"parsed_ua": {
"ua_family": "Chrome",
"ua_version": "83.0.4103",
"os_family": "Mac OS X"
//other user agent nodes: os version, device family, brand, model
}
}

Please note, that geo data fields will be resolved only if a MaxMind file is provided (see Geo Resolution page)

Step 3: Mapping

A special section in the destination configuration is designed to define how JSON is transformed before it's sent to the target. Currently, two operations are supported:

  • Rename: /src/path/to/node -> /dst/path will rename JSON node

  • Delete node: /src/path/to/node -> will remove node

  • Typecast: /src/path/to/node -> (integer)/dst/pathwill change the type of node to integer. Following typecasts are supported: integer, double, string, timestamp

destinations:
destination_name:
data_layout:
mapping:
- "/src/path1 -> /dst/path2" #rename node
- "/src/path1 -> " #delete node
- "/src/path1 -> (integer) /src/path1" #cast to int
- "/src/path1 -> (double) /dst/path2" #rename node an cast to double
- "/src/path1 -> (string) /dst/path2" #rename node an cast to string
- "/src/path1 -> (timestamp) /dst/path2" #rename node an cast to timestamp

Please note, that typecast from string to timestamp is supported only for string layout:

"2006-01-02T15:04:05.000000Z"

We also have a strict mapping that allows defining only those fields that will be stored in the database. To enable this mode, use mapping_type configuration field of data_layout.

destinations:
destination_name:
data_layout:
mapping_type: strict
mapping:
- "/src/path1 -> /dst/path1"
- "/src/path2 -> /dst/path3"

With such config, only /src/path1 and /src/path2 values will be saved.

Step 3.5 (optional): Data Flattening, Schema

Most data warehouses work as traditional relational databases, meaning a data schema is required. However, EventNative doesn't require the schema to be defined.

First, it applies JSON flattening, converting JSON into a table like structure. Node /a/b/c turns into field a_b_c. For example:

JSON
Table
JSON
{
"api_key": "63c13a7c7",
"event_type": "3rdparty",
"src": "ga",
"src_payload": {
"client_id": "1484854471.1594305592",
"screen_size": "1792x1120",
"viewport_size": "1792x986"
}
}
Table

api_key

event_type

src

src_payload_client_id

src_payload_screen_size

src_payload_viewport_size

63c13a7c7

3rdparty

ga

1484854471.1594305592

1792x1120

1792x986

After the JSON object is flattened, EventNative makes sure that all columns are present in destination table. Otherwise, missing columns will be created.

For configuring Segment like schema please see our wiki page.

Step 4: Upload

Transformed data is uploaded to destination table

Please, note that data will not upload in the destination table immediately. Due to the nature of data warehouses / data lakes, we send data in batches. Batch size / frequency is defined by log section in config (5 mins by default)