Configuration Guide

EventNative is configured through a single YAML file. We follow convention over configuration so the majority of parameters are optional. To get a base file, clone it from GitHub.

Our config file consists of following four sections:

  • server — General configuration parameters such as address, tokens, etc.

  • logEventNative logs all events locally and sends them to their destination in batches. This is where you configure your local temporary path and push frequency.

  • geo — Geo resolution data (extracting city/state information from the IP address). We currently only support MaxMind as a data provider.

  • destinations — A set of targets where the final version of events will be stored.

Example:

server:
port: 8081
auth: '193b6281-f211-47a9-b384-102cf4cd2d55'
public_url:
log:
path: /home/eventnative/logs/
metrics.prometheus.enabled: true
geo.maxmind_path: /home/eventnative/app/res/
log:
path: /home/eventnative/logs/events
rotation_min: 5
destinations:
redshift:
bigquery:

Server

EventNative supports collecting Prometheus metrics. If enabled - metrics will be available by /prometheus endpoint.

EventNative supports reloadable authorization from http[s]://, file:// sources and static authorization from YAML configuration. JSON payload must have the following structure:

{
"tokens": [ #array of json objects
{
"id": "uniq_id_1", #Unique identifier of token instance
"client_secret": "123dasbc", #Optional. Token for js integration
"server_secret": "abcc22", #Optional. Token for api integration
"origins": ["abc.com", "*.yourdomain.com"] #Optional. Allowed origins
}
]
}

Examples:

Plain String
Yaml objects
HTTP
File
Plain String
server:
auth: '193b6281-f211-47a9-b384-102cf4cd2d55'
#this token will be used as client_secret
Yaml objects
server:
auth:
-
id: 'unique_tokenId'
client_secret: 'bd33c5fa-d69f-11ea-87d0-0242ac130003'
server_secret: '5f15eba2-db58-11ea-87d0-0242ac130003'
origins:
- '*abc.com'
- 'efg.com'
-
id: unique_tokenId2
client_secret: 123jsy213c5fa-c20765a0-d69f003
-
id: unique_tokenId3
server_secret: 231dasds-3211kb3rdf-412dkjnabf
HTTP
server:
#will be reloaded every server.auth_reload_sec (30 secons by default)
auth: 'https://token-source.ru/path'
File
server:
auth_reload_sec: 10 #will be reloaded every 10 seconds
auth: 'file:///home/eventnative/app/res/tokens.json'

Destinations

Each config can contain multiple destinations for one instance of EventNative. "Destination" are targets where all events are stored (currently we support RedShift, BiqQuery, Postgres, ClickHouse, Snowflake and S3).

EventNative supports reloadable destinations from http[s]://, file:// sources and static destinations from YAML configuration. JSON payload must have the following structure:

{
"destinations": { #json object where inner keys - destinations unique names
"redshift_dab213ibda": { #destination config object
"type": "redshift",
....
},
"clickhouse_in31o31": {
"type": "clickhouse",
...
}
}
}

Destination config fields are the same in JSON payload and in YAML configuration and it has the following parameters:

destinations:
{destination unique name}:
type: #if not set, type will be taken from destination name
only_tokens: #if not set, data from all tokens will be stored
mode: stream #Optional. Available mode: [batch, stream]. Default value: batch
data_layout:
mapping:
table_template_name: #events by default

Each destination should have a name. If you set your destination name different than the provider name, you need to set up a type (example: destination redshift will have redshift type by default. But if name=redshift_1 type should be explicitly defined as redshift).

Examples:

RedShift
BigQuery
Postgres
ClickHouse
S3
Snowflake+s3
Snowflake+gcp
reloadable
RedShift
destinations:
my_redshift:
type: redshift
only_tokens: ['api_key1', 'api_key2']
datasource:
host: redshift.amazonaws.com
db: my-db
schema: myschema #public by default
username: user
password: pass
s3:
access_key_id: abc123
secret_access_key: secretabc123
bucket: my-bucket
region: us-west-1
folder: my_redshift_destination #optional
data_layout:
mapping:
table_template_name: #events by default
BigQuery
destinations:
bigquery:
only_tokens: ['api_key1', 'api_key3']
google:
gcs_bucket: google_cloud_storage_bucket
bq_project: big_query_project
bq_dataset: big_query_dataset # 'default' by default
key_file: path_to_bqkey.json # or json string of key e.g. "{"service_account":...}"
data_layout:
table_name_template: #events by default
Postgres
destinations:
my_postgres:
type: postgres
only_tokens: ['api_key1', 'api_key2']
mode: stream
datasource:
host: my_postgres_host
db: my-db
schema: myschema #public by default
port: 5432 #5432 by default
username: user
password: pass
parameters: #optional postgres connect db parameters (see https://www.postgresql.org/docs/9.1/libpq-connect.html)
sslmode: disable
connect_timeout: 300
data_layout:
mapping:
table_template_name: #events by default
ClickHouse
destinations:
clickhouse_ksense:
type: clickhouse
only_tokens: ['api_key1', 'api_key2']
clickhouse:
dsns:
- "https://username:[email protected]:8443/mydb?read_timeout=5m&timeout=5m&enable_http_compression=1&tls_config=maincert"
- "https://username:[email protected]:8443/mydb?read_timeout=5m&timeout=5m&enable_http_compression=1&tls_config=maincert"
- "https://username:[email protected]:8443/mydb?read_timeout=5m&timeout=5m&enable_http_compression=1&tls_config=maincert"
db: mydb
cluster: mycluster #required if dsns > 1
engine: #optional
raw_statement: 'ENGINE = ReplacingMergeTree(_timestamp) ORDER BY (eventn_ctx_event_id)' #optional
non_null_fields: #required if raw_statement is provided
- _timestamp
- eventn_ctx_event_id
#if raw_statement is provided - below parameters from 'engine' section will be skipped
partition_fields: #optional. Override PARTITION BY in CREATE TABLE statement
- function: toYYYYMMDD #optional. It is used in 'PARTITION BY (toYYYYMMDD(_timestamp), event_type)'
field: _timestamp
- field: event_type
order_fields: #optional. Override ORDER BY in CREATE TABLE statement
- function: intHash32 #optional. It is used in 'ORDER BY intHash32(id)'
field: id
primary_keys: #optional. Override PRIMARY KEY in CREATE TABLE statement
- eventn_ctx_event_id
tls: #optional
maincert: path_to_crt_file
S3
destinations:
my_s3:
type: s3
only_tokens: ['api_key1', 'api_key2']
s3:
access_key_id: abc123
secret_access_key: secretabc123
bucket: my-bucket
region: us-west-1
folder: my_s3_events #optional
data_layout:
mapping:
table_template_name: #events by default. Will be used for s3 file naming
Snowflake+s3
destinations:
my_snowflake_via_aws_s3:
type: snowflake
only_tokens: ['api_key1', 'api_key2']
s3:
access_key_id: abc123
secret_access_key: secretabc123
bucket: my-snowflake-bucket
region: us-west-1
folder: my_snowflake_via_aws_s3_destination #optional
snowflake:
account: hha56552.us-east-1
schema: MYSCHEMA #PUBLIC by default
warehouse: my_warehouse
db: mydb
username: user
password: pass
parameters: #optional
name: value
data_layout:
mapping:
table_template_name: #events by default
Snowflake+gcp
destinations:
my_snowflake_via_gcp:
type: snowflake
only_tokens: ['api_key1', 'api_key2']
google:
gcs_bucket: en-snowflake-test
key_file: /home/eventnative/app/res/bqsnow.json
snowflake:
account: hha56552.us-east-1
schema: MYSCHEMA #PUBLIC by default
warehouse: my_warehouse
db: mydb
username: user
password: pass
stage: mystage #required when gcp integration
parameters: #optional
name: value
data_layout:
mapping:
table_template_name: #events by default
reloadable
destinations: 'https://mydestinations/'
#will be reloaded every server.destinations_reload_sec (40 secons by default)

mode sets the communication mode of destination. EventNative supports stream and batch value (batch is a default value). In batch mode, data will be collected in a log file and then stored in the destination in one transaction. In stream mode, data will be stored immediately: one object in one transaction.

table_template_name sets the name of the destination table. The name can be either static, or partitioned (based on events properties). For example: 'data_{{.event_type}}_{{.timestamp.Format "YYYY_"}}' will name tables as "data_pageview_2020_10" (see text/template syntax for more detail). Any variable from event JSON (please read Schema and Mapping section) can be referenced. Example:

{
...,
"field": {
"field2": 10
}
...
}

10 can be referenced as {{.field.field2}} in the name template.

mapping sets the rules on how original events should be transformed before sending them to the Data Warehouse. Please, refer to the specific section of Schema and Mapping manual.