Skip to main content
Version: latest

Create an Data Pipeline from Webhook to Slack

This guide shows an end-to-end event pipeline that reads an event from a webhook, generates a formatted string, and publishes the result to Slack. While this is a simple example, it has many event notification use cases, such as:

  • submission from website forms (via Cloudflare workers or your custom backend).
  • activity from e-commerce platforms on purchases and shopping carts.
  • notifications from github on your projects' activities.
  • alerts from financial products on your transactions.
  • notifications from any product that can invoke a webhook.

This pipeline uses the following features:

  • webhook: that creates a public API to receive external events, transform them, and publish them to a topic.
  • http-sink: that listens to the same topic and publishes them on Slack.

Objective

Show an example of how to build an event streaming pipeline that receives webhook events, transforms the input into a readable form, and generates an alert. We assume the events are generated by a user submitting a form, and we'll format it accordingly.

Prerequisites

Step-by-Step

  1. Create webhook configuration file
  2. Create http-sink configuration file
  3. Download SmartModules
  4. Start Webhook and Connector
  5. Test Data Pipeline

Create webhook configuration file

Create a webhook configuration file called form-webhook.yaml:

info

All versions are marked with x.y.z. To find the latest version, run:

  • fluvio hub connector list
  • fluvio hub smartmodule list
meta:
name: form-webhook
topic: form-events
webhook:
outputParts: body
outputType: json
transforms:
- uses: infinyon-labs/json-formatter@x.y.z
with:
spec:
match:
- key: "/type"
value: "subscribe"
format:
with: "📢 {} ({}) subscribed on {}"
using:
- "/name"
- "/email"
- "/source"
output: "/formatted"
- key: "/type"
value: "use-case"
format:
with: ":confetti_ball: {} ({}) wants to solve the following '{}' use-case:\n>{}"
using:
- "/name"
- "/email"
- "/source"
- "/description"
output: "/formatted"
default:
format:
with: "{} ({}) submitted a request"
using:
- "/name"
- "/email"
output: "/formatted"

The webhook reads the JSON body, applies the json-formatter smartmodule to generate readable text, and writes the new record to a topic called form-events. Checkout labs-json-formatter-sm in github for additional information.

Create http-sink configuration file

Create an HTTP source connector configuration file called slack-form-alerts.yaml :

apiVersion: 0.1.0
meta:
version: x.y.z
name: slack-form-alerts
type: http-sink
topic: form-events
secrets:
- name: SLACK_USER_ALERTS
http:
endpoint: "https://hooks.slack.com/services/${{ secrets.SLACK_USER_ALERTS }}"
headers:
- "Content-Type: application/json"
transforms:
- uses: infinyon/jolt@x.y.z
with:
spec:
- operation: shift
spec:
"formatted": "text"

The sink connector reads from the form-events topic and uses the jolt smartmodule to shift the formatted string into a field called text per the Slack instructions. Checkout fluvio-jolt in github for additional information.

Add Slack webhook token to InfinyOn Secrets

The Slack webhook link is sensitive information, let's add the Access Token part to secret in InfinyOn Cloud :

$ fluvio cloud secret set SLACK_USER_ALERTS <webhook-token>

Check out Slack Webhooks on how to create the webhook token.

Download SmartModules

Download the smartmodules used by the webhook ad the connector:

$ fluvio hub download infinyon/jolt@x.y.z
$ fluvio hub download infinyon-labs/json-formatter@x.y.z

Check fluvio smartmodule list to ensure they've been downloaded.

Start Webhook and Connector

Start webhook listener:

$ fluvio cloud webhook create --config form-webhook.yaml

Check fluvio cloud webhook list to ensure it has been successfully provisioned. In checkout the webhook link that we'll use to test the pipeline: https://infinyon.cloud/webhooks/v1/[token]

Start sink connector:

$ fluvio cloud connector create -c slack-form-alerts.yaml

Check fluvio cloud connector list to ensure it has been successfully provisioned.

Test Data Pipeline

Use curl to send a POST request with a fictitious user request to our webhook link. In production environments, this iw what a website would send:

$ curl -X POST https://infinyon.cloud/webhooks/v1/<token> \
-H "Content-Type: application/json" \
-d '{ "email": "alice@acme.com", "name": "Alice Liddell", "type": "subscribe", "source": "front-page" }'

The following alert is displayed in Slack:

`📢 Alice Liddell ("alice@acme.com) subscribed on front-page` will show-up in your slack channel.

That's all folks!

Reference