Connector Configuration File
This is a template YAML connector file that needs to be populated to be functional. In the next section, we will explore the different sections of the connector configuration file and how to properly configure each one.
apiVersion: 0.1.0
meta:
name:
version:
type:
topic:
# optional (inbound connectors)
producer:
# optional
linger:
# optional
batch-size:
# optional
compression:
# optional (outbound connectors)
consumer:
# optional as default, but required if offset strategy is provided
id:
# optional
offset:
strategy:
start:
flush:
# optional
partition:
# optional
max_bytes:
# optional
secrets:
- name: secret_1
# optional
transforms:
- uses: smartmodule_name
with:
param_name: param_value
Connector apiVersion
configuration
The apiVersion
is the version of the connector API that the connector uses to parse the configuration file. The current accepted version is 0.1.0
.
Connector meta
configuration
The meta
section contains the metadata for the connector:
- The
name
is the name of the connector. e.g.my-connector
. - The
type
is the type of the connector. e.g.http-source
,http-sink
,mqtt-source
.- See the connectors section for the full list of connectors supported.
- The
version
is the version of the connector. e.g.0.1.0
. - The
topic
is the topic that the connector will connect to. e.g.my-topic
. The topic will be created automatically if it does not exist. - The
secrets
(optional) is a list of secrets that the connector will use. This accepts a list of objects with the keyname
.- See the [secrets] section for more information.
- The
producer
(optional) is the producer configuration for the connector. Currently, this is only used forsource
/inbound
connectors. The current supported configurations arelinger
,compression
andbatch_size
. All configurations are optional. See examples to a list of valid values for each configuration. - The
consumer
(optional) is the consumer configuration for the connector. Currently, this is only used forsink
/outbound
connectors. The current supported configurations areid
,partition
,max_bytes
andoffset
. All configurations are optional. See examples to a list of valid values for each configuration.
At minimum connector configuration would look like:
apiVersion: 0.1.0
meta:
name: my-connector
type: my-connector-type
version: x.y.z
topic: my-topic
All other fields are optional.
The x.y.z
version should be updated with the version of the latest connector in the Hub.
Connector transforms
configuration
Connectors support transforms
. Records can be modified before they are sent to the topic. The transforms
section is a list of transform
objects. Each transform
object has an uses
and a with
section.
uses
is the reference to the SmartModule used in the transform.with
is the configuration for the transform
Consumer with Offset
Consumers can be configured with different start
offsets to determine where to begin consuming records from a topic.
The following options are available for the start
offset:
absolute: x
: Starts consuming from the absolute indexx
of the topic.beginning
: Starts consuming from the very beginning of the topic.from-beginning: x
: Starts consuming from indexx
from the beginning of the topic.end
: Starts consuming from the end of the topic.from-end: x
: Starts consuming from indexx
from the end of the topic.
Offset Management
Offset management is a Fluvio feature that stores the offset of the last consumed record. This is especially useful when a connector is stopped and restarted, as it allows the connector to resume consumption from the last saved offset rather than starting over.
The behavior of offset management depends on the strategy
field. When strategy
is set to auto
or manual
, offset management is enabled, and the start
rule is based on the saved offset rather than the entire topic.
The strategy
field supports the following values:
auto
: Automatically commits the offset after each record is consumed.manual
: Requires the user to commit the offset manually.none
: Disables offset management entirely.
Example Configuration
A consumer connector with offset management might look like this:
apiVersion: 0.1.0
meta:
name: my-connector
type: my-connector-type
version: x.y.z
topic: my-topic
consumer:
id: my-consumer
offset:
strategy: auto
start: beginning
flush: 10s
References
The following references will provide additional information for:
- SmartModules for additional information on transformations
- Tutorials for end-to-end examples
- Offsets for more information on offsets