Skip to main content
Version: latest

SmartModules Overview

SmartModules are powerful components that enhance the functionality and flexibility of the Fluvio platform. They are programmable data streaming functions that serve as intelligent building blocks which enable users to apply transformations. By leveraging smartmodules, users can automate repetitive tasks, orchestrate data flows, and enable advanced functionalities without requiring deep technical knowledge or extensive coding.

By utilizing WebAssembly, SmartModules provide a secure and isolated environment for executing custom code. This ensures that the modules run safely within the Fluvio ecosystem, preventing potential security risks or conflicts with the underlying system.

The beauty of SmartModules lies in their flexibility and adaptability. Users can configure them to suit their specific needs by adjusting parameters and settings. Producers and consumers can use SmartModules in order to shape data on both the client-side or server-side. This empowers individuals to tailor the behavior of the modules to match their unique requirements, all without writing complex code or dealing with intricate technical details.

You may use SmartModules in consumer, producers, as well as inbound and outbound connectors.

You may use pre-build SmartModules or build your own:

Using a SmartModules​

Smartmodules in are define in transformation files in yaml format. For example, the following yaml file performs a jolt transformation on a json object to rename animal.fact to fact and pass along length.

transforms:
 - uses: infinyon/jolt@x.y.z
   with:
     spec:
       - operation: shift
         spec:
           fact: "animal.fact"
           length: "length"

Chaining SmartModules​

You can define a SmartModule chain to use with fluvio or smdk with a yaml file.

transforms:
 - uses: infinyon/jolt@x.y.z
   with:
     spec:
       - operation: shift
         spec:
           fact: "animal.fact"
           length: "length"
 - uses: infinyon/regex-filter@x.y.z
   with:
     regex: "[Cc]at"

You can pass this file to the CLI with the --transforms-file option.

Example Pipelines​

Examples on how to use SmartModules in a pipeline:

  • [Build HTTP to SQL Pipeline]
  • [Build MQTT to SQL Pipeline]

References​

SmartModules features:

Smartmodule sample repositories in github: