ClickHouse destination

EventNative supports ClickHouse as a destination. For more information about ClickHouse see docs.

Configuration

ClickHouse destination config consists of following schema:

destinations:
clickhouse_ksense:
type: clickhouse
only_tokens: ['api_key1', 'api_key2']
clickhouse:
dsns:
- "https://username:[email protected]:8443/mydb?read_timeout=5m&timeout=5m&enable_http_compression=1&tls_config=maincert"
- "https://username:[email protected]:8443/mydb?read_timeout=5m&timeout=5m&enable_http_compression=1&tls_config=maincert"
- "https://username:[email protected]:8443/mydb?read_timeout=5m&timeout=5m&enable_http_compression=1&tls_config=maincert"
db: mydb
cluster: mycluster
engine:
raw_statement: 'ENGINE = ReplacingMergeTree(_timestamp) ORDER BY (eventn_ctx_event_id)' #optional
non_null_fields:
- _timestamp
- eventn_ctx_event_id
partition_fields:
- function: toYYYYMMDD
field: _timestamp
- field: event_type
order_fields:
- function: intHash32
field: id
primary_keys:
- eventn_ctx_event_id
tls:
maincert: path_to_crt_file

clickhouse

Field (*required)

dsns*

Array of connection strings. Must contain at least one connection string.

db*

Database name.

cluster**

Cluster name. Required if dsns count > 1.

engine

Tables engine configuration.

tls

TLS configuration. Map of cert names and paths to cert files. Cert names will be used in tls_config query parameter in dsns.

If engine wasn't provided default one (depends on cluster configuration) will be used:

default values
#if ClickHouse single server (dsns = 1)
ReplacingMergeTree(_timestamp) PARTITION BY toYYYYMM(_timestamp) ORDER BY (eventn_ctx_event_id)
#if ClickHouse cluster (dsns > 1)
ReplicatedReplacingMergeTree('/clickhouse/tables/{shard}/$db/$tablename', '{replica}', _timestamp) PARTITION BY toYYYYMM(_timestamp) ORDER BY (eventn_ctx_event_id)

engine

Field (*required)

raw_statement

Engine raw statement which will be used in

CREATE TABLE db.name (a type,b type) $raw_statement.

Must begin with 'ENGINE='. Could include ORDER BY, PARTITION BY, etc.

non_null_fields**

Array of non-null fields. Required if raw_statement is provided. All fields from ENGINE, PARTITION BY, ORDER BY and other clauses must be non-null.

partition_fields

Array of field objects will be used in PARTITION BY ($partition_fields) statement

order_fields

Array of field objects will be used in ORDER BY ($order_fields) statement

primary_keys

Array of field names will be used in PRIMARY KEY ($primary_keys) statement

Please note, if raw_statement exists than config keys partition_fields, order_fields, primary_keys will be skipped.

field object

Field (*required)

function

ClickHouse function name which will be applied to the field (e.g. toYYYYMMDD, intHash32)

field*

Field name