Skip to main content
Version: latest

Create a Data Pipeline from GitHub to Slack

Create an end-to-end event pipeline that detects changes in github stars & forks and publishes the result to Slack. This guide uses two connectors:

  • http-source: to read periodically from a github, parse the fields from the json output, and publishes the result to a topic.
  • http-sink: to listen to the same topic, detect changes, and publish the result to Slack.

Let's get started.

Prerequisites​

Step-by-Step​

  1. Create http-source configuration file
  2. Create http-sink configuration file
  3. Download smartmodules
  4. Start Connectors
  5. Test Data Pipeline

Create http-source configuration file​

Create an HTTP source connector configuration file called github.yaml :

info

All versions are marked with x.y.z. To find the latest version, run:

  • fluvio hub connector list
  • fluvio hub smartmodule list
apiVersion: 0.1.0
meta:
  version: x.y.z
  name: github-stars-in
  type: http-source
  topic: stars-forks
  secrets:
    - name: GITHUB_TOKEN
http:
  endpoint: 'https://api.github.com/repos/infinyon/fluvio'
  method: GET
  headers:
    - 'Authorization: token ${{ secrets.GITHUB_TOKEN }}'
  interval: 30s
transforms:
  - uses: infinyon/jolt@x.y.z
    with:
      spec:
        - operation: shift
          spec:
            "stargazers_count": "stars"
            "forks_count": "forks"

Github rate-limits API requests to 60 per hour, which you an extend to 5000 by creating an application token. Check out github documentation on how to create an Access Tokens.

Add the access token secret in InfinyOn Cloud :

$ fluvio cloud secret set GITHUB_TOKEN <access-token>

Create http-sink configuration file​

Create an HTTP source connector configuration file called slack.yaml :

apiVersion: 0.1.0
meta:
  version: x.y.z
  name: slack-stars-out
  type: http-sink
  topic: stars-forks
  secrets:
    - name: SLACK_TOKEN
http:
  endpoint: "https://hooks.slack.com/services/${{ secrets.SLACK_TOKEN }}"
  headers:
    - "Content-Type: application/json"
transforms:
  - uses: infinyon-labs/stars-forks-changes@x.y.z
    lookback:
      last: 1
  - uses: infinyon/jolt@x.y.z
    with:
      spec:
        - operation: shift
          spec:
            "result": "text"

Check out Slack Webhooks on how to create a channel webhook token.

Add the access token secret in InfinyOn Cloud :

$ fluvio cloud secret set SLACK_TOKEN <webhook-token>

Download Smartmodules​

Download the smartmodules used by the connectors to your cluster:

$ fluvio hub sm download infinyon/jolt@x.y.z
$ fluvio hub sm download infinyon-labs/stars-forks-changes@x.y.z

Start Connectors​

Start source & sink connectors:

$ fluvio cloud connector create -c github.yaml
$ fluvio cloud connector create -c slack.yaml

Check fluvio cloud connector log to ensure they have been successfully provisioned.

Test Data Pipeline​

Check the last values generated by the github connector:

$ fluvio consume -dT 1 stars-forks
{"stars":1770,"forks":138}

Produce a new value

$ fluvio produce stars-forks
> {"stars":1769,"forks":138}
OK

An alert with :star2: 1769 will show-up in your slack channel.

Reference​