This is the template YAML connector file. To make it useful, it needs to be populated. In the next section we will go over the different sections of the connector configuration file.
# connector.yaml
# Version of the schema of the connector config
# Valid value is `0.1.0`
apiVersion: 0.1.0
meta:
name:
version:
type:
topic:
# optional
producer:
# optional
linger:
# optional
batch-size:
# optional
compression:
# optional
consumer:
# optional
partition:
# optional
max_bytes:
# optional
secrets:
- name: secret_1
# optional
transforms:
- uses: smartmodule_name
with:
param_name: param_value
# Type specific configs
# key depends on connector
# <custom_key>:
# foo: bar
#
# eg.
# http:
# endpoint: https://example.com
The apiVersion
is the version of the connector API that the connector uses to parse the configuration file. The current only accepted version is 0.1.0
.
The meta
section contains the metadata for the connector:
name
is the name of the connector. e.g. my-connector
.type
is the type of the connector. e.g. http-source
, http-sink
, mqtt-source
.
version
is the version of the connector. e.g. 0.2.0
.topic
is the topic that the connector will connect to. e.g. my-topic
. The topic will be created automatically if it does not exist.secrets
(optional) is a list of secrets that the connector will use. This accepts a list of objects with the key name
.
producer
(optional) is the producer configuration for the connector. Currently, this is only used for source
/inbound
connectors. The current supported configurations are linger
, compression
and batch_size
. All configurations are optional. See examples to a list of valid values for each configuration.consumer
(optional) is the consumer configuration for the connector. Currently, this is only used for sink
/outbound
connectors. The current supported configurations are partition
and max_bytes
. Both configurations are optional. See examples to a list of valid values for each configuration.An example with all the keys filled for a http-source
connector:
apiVersion: 0.1.0
meta:
name: my-http-source-connector
type: http-source
version: 0.2.0
topic: my-topic
producer:
linger: 1ms
batch_size: "44.0 MB"
# possible values: `none`, `gzip`, `snappy` and `lz4`
compression: gzip
secrets:
- name: MY_SECRET
An example with all the keys filled for a http-sink
connector:
apiVersion: 0.1.0
meta:
name: my-http-sink-connector
type: http-sink
version: 0.1.0
topic: my-topic
consumer:
max_bytes: 3 MB
partition: 0
secrets:
- name: MY_SECRET
Connectors support transforms
. Records can be modified before they are sent to the topic. The transforms
section is a list of transform
objects. Each transform
object has an uses
and a with
section.
uses
is the reference to the SmartModule used in the transform.with
is the configuration for the transform
See the Transformations section for more information.
See Tutorials for examples of using transforms.