Filter Operator

The Filter Operator is a powerful tool that allows you to filter messages based on a condition. It is a simple operator that takes a single input and returns a boolean value. If the input meets the condition, the operator will return true. Otherwise, it will return false. The filter operator will be used to detect whether or not a sentence is a question.

Prerequisites

This guide uses local Fluvio cluster. If you need to install it, please follow the instructions at here.

Transformation

Here is a simple filter operator. It takes a string as input and returns a boolean value. If the input string contains a question mark, the operator will return true. Otherwise, it will return false.

    transforms:
      - operator: filter
        run: |
          fn filter_questions(input: String) -> Result<bool> {
            Ok(input.contains("?"))
          }

Running the Example

Copy and paste following config and save it as dataflow.yaml.

# dataflow.yaml
apiVersion: 0.5.0
meta:
  name: filter-example
  version: 0.1.0
  namespace: examples

config:
  converter: raw

topics:
  sentences:
    schema:
      value:
        type: string

  questions:
    schema:
      value:
        type: string

services:
  filter-service:
    sources:
      - type: topic
        id: sentences

    transforms:
      - operator: filter
        run: |
          fn filter_questions(input: String) -> Result<bool> {
            Ok(input.contains("?"))
          }

    sinks:
      - type: topic
        id: questions

To run example:

$ sdf run

Produce sentences to in sentence topic:

$ echo "Hello world" | fluvio produce sentences
$ echo "Are you there?" | fluvio produce sentences

Consume topic questions to retrieve the result in another terminal:

$ fluvio consume questions -Bd
Are you there?

Only questions are returned.

Cleanup

Exit sdf terminal and clean-up. The --force flag removes the topics:

$ sdf clean --force

Conclusion

We just covered one of the most basic operators in SDF, the Filter Operator. Combining with other basic operators, you can build more complex dataflows to process your data.