Generate

Generate a Connector

CDK generate helps developers build a sample Connector project by answering a few simple questions.

Use cdk generate to create a new connector project:

$ cdk generate
🤷   Project Name: my-connector
🤷   Please set a group name: acme
🤷   Which type of Connector would you like [source/sink]? · source
🤷   Will your Connector be public? · false
[1/8]   Done: .gitignore
[2/8]   Done: Cargo.toml
[3/8]   Done: Connector.toml
[4/8]   Done: README.md
[5/8]   Done: sample-config.yaml
[6/8]   Done: src/config.rs
[7/8]   Done: src/main.rs
[8/8]   Done: src

The generator created Rust project ready to compile:

$ tree
.
├── Cargo.toml
├── Connector.toml
├── README.md
├── sample-config.yaml
└── src
    ├── config.rs
    └── main.rs

This a simple connector with the code in src/main.rs:

mod config;
use config::CustomConfig;

use fluvio::{RecordKey, TopicProducerPool};
use fluvio_connector_common::{
    connector,
    Result
};

#[connector(source)]
async fn start(config: CustomConfig, producer: TopicProducerPool) -> Result<()> {
    println!("Starting my-connector source connector with {config:?}");
    for i in 1..1000 {
        let value = format!("Hello, Fluvio - {i}");
        producer.send(RecordKey::NULL, value).await?;
        producer.flush().await?;
        std::thread::sleep(std::time::Duration::from_millis(1000));
    }
    Ok(())
}

Connectors may also have define configuration parameters as shown in src/config.rs:

use fluvio_connector_common::connector;

#[connector(config)]
#[derive(Debug)]
pub(crate) struct CustomConfig {
    #[allow(dead_code)]
    foo: String,
}

The Connector.toml file contains the definition of the Connector parameters required to load the file in the Cluster and publish it to Connector Hub.

[package]
name = "my-connector"
group = "acme"
version = "0.1.0"
apiVersion = "0.1.0"
fluvio = "0.10.0"
description = ""
license = "Apache-2.0"
visibility = "private"

[direction]
source = true

[deployment]
binary = "my-connector"

[custom.properties.foo]
title = "Foo"
description = "Foo"
type = "string"

Sections

• package is used to build the connector FQDN acme/my-connector@0.1.0, and the description to publish to Hub. The group name is equivalent to the package owner in the Hub.
• direction is used to declare the direction data flows through the connector, with respect to the Fluvio cluster. An inbound connector uses source = true, and and outbound connector uses sink = true
• custom.properties.foo defines a user configuration key foo that can be used in the logic of the connector.

The project is ready to build and test.