Skip to main content
Version: latest

Mirroring IoT Data from Edge to Cloud

InfinyOn IoT edge is a ~14 Mb binary that runs on ARMv7 chips on less than 256 MB memory. We are working with teams building the future of monitoring dynamic assets to push the boundaries of edge data stream processing.

  • If connected, InfinyOn IoT edge sends telemetry and events to the InfinyOn Cloud in real-time using mirroring.

  • If disconnected, the InfinyOn IoT edge stream processor caches events locally. When the connection resumes, the InfinyOn IoT edge stream processor brings InfinyOn Cloud up to date and continues mirroring until the subsequent connection loss.

Benefits​

The benefits of the InfinyOn solution are as follows:

  • Reliable edge-to-cloud synchronization:
    • Real-time publishing when connected.
    • Automatic synchronization after reconnect.
      • Edge devices can be offline for extended periods (days).
  • Edge collection without downtime when disconnected.
    • Reliable local caching for gigabytes of data.
    • Simplified logic for edge clients.
      • Edge cluster provides a reliable connection to the local clients.
  • Intelligent processing at the edge with InfinyOn Smartmodules
    • filter
    • transform
    • enrich
  • Hierarchical processing, where you decide where to apply the transformations.
  • Built-in cloud connectors to push events to databases and other core products.

Installation​

In this tutorial we'll use VM emulator to create and edge endpoint and mirror traffic to InfinyOn Cloud.

Let's get started.

Setup InfinyOn Cloud​

Mirroring is an experimental feature using a development cluster. Please get in touch with us on Discord to request access for your organization. Upon approval, please continue as follows:

Create a Cloud account​

Using your web browser, navigate to https://infinyon.cloud/ui/signup, where this experimental feature is available.

After the account is created, you will be placed in the Dashboard. You may choose to create a cluster in the GUI. In this tutorial, we'll create a cluster using the CLI later.

Download fluvio binary​

Use curl to download and install:

curl -fsS https://hub.infinyon.cloud/install/install.sh | bash

Make sure to add .fluvio/bin to the $PATHas specified in the installation script.

Login to InfinyOn Cloud​

Login to InfinyOn Cloud:

fluvio cloud login --use-oauth2

Leave out --use-oauth2 if you prefer username/password method.

Provision a new Cluster​

Let's provision a new cloud cluster:

fluvio cloud cluster create

Check the result with:

fluvio cluster status

Next, we'll configure the cluster to receive traffic from the edge clusters.

Register Edge cluster​

Let's register the edge cluster edge1 to inform our home cluster from the cloud to accept connection requests from the remote device:

fluvio remote register edge1

Create the mirror topic​

Each edge cluster mirror connects a partition of a topic, where each partition has a 1-to-1 relationship with the edge cluster.

Create a partition assignment file with an array of edge mirrors we expect to connect this cluster:

echo '[
    "edge1"
]' > assignment_file.json

Apply the configuration file to create the topic:

fluvio topic create edge-topic --mirror-apply assignment_file.json

List partitions to check the assignment:

fluvio partition list

It should display all partitions:

  TOPIC       PARTITION  LEADER  MIRROR        REPLICAS  RESOLUTION  SIZE  HW  LEO  LRS  FOLLOWER OFFSETS
  edge-topic  0          5001    edge1         []        Online      0 B   0   0    0    0                 []

List remote clusters to check their status:

fluvio remote list

It should show the following:

  REMOTE   SC STATUS  SPU STATUS  LAST SEEN  ERRORS
  edge1    Waiting    Waiting     -          -

Create a new directory​

In the next step, we'll create a configuration file that we'll need to pass along to the edge device. It's easier if we make a clean directory and pass it along to the VM emulator:

mkdir -p ~/local/projects/mirror; cd ~/local/projects/mirror

Generate metadata for Edge Cluster​

Each edge cluster requires a unique metadata file that informs the edge cluster how to connect with the home cluster. Create the config file by running the following command:

fluvio cloud remote export edge1 --file edge1.json

The Cloud cluster configuration is now complete. Next, we'll create an edge cluster that synchronizes data to the Cloud.

Install Edge Cluster on Local VM​

We'll start an edge cluster on our local computer in a VM using OrbStack.

Instal OrbStack​

We'll use OrbStack for the VM management:

  1. Install OrbStack

  2. Start Ubuntu VM machine.

  3. Click the VM to open a terminal.

  4. Using the terminal, navigate to your data directory:

cd local/projects/mirror

All files we've generated on the local machines should be visible here.

Download fluvio binaries​

Download binaries:

curl -fsS https://hub.infinyon.cloud/install/install.sh | bash

Add to path:

echo 'export PATH="${HOME}/.fluvio/bin:${PATH}"' >> ~/.bashrc
source ~/.bashrc

Run the following command to double check:

fluvio version

Start Edge Cluster​

First we will start the cluster:

fluvio cluster start

Then, we'll connect to the Cloud with the metadata edge1:

fluvio home connect --file edge1.json

Let's check the partitions:

fluvio partition list

The edge device should show the following partition:

  TOPIC       PARTITION  LEADER  MIRROR                REPLICAS  RESOLUTION  SIZE  HW  LEO  LRS  FOLLOWER OFFSETS
  edge-topic  0          5001    0:public_endpoint     []        Online      0 B   0   0    0    0                 []

Test 1: Mirroring from VM Edge to Cloud​

Let's produce on the Edge VM cluster and consume from the Cloud cluster.

Produce on Edge​

Produce on the edge terminal:

fluvio produce edge-topic
> 1
Ok!
> 2
Ok!

Consume from Cloud​

Consume on the cloud terminal:

fluvio consume edge-topic --mirror edge1 -B
1
2

Mirror test is successful.

Test 2: Cloud Cluster Offline​

To simulate a disconnect, we'll perform the following steps:

  1. Turn off the network connection to the internet.

  2. Produce records on the edge terminal.

fluvio produce edge-topic
> 3
Ok!
> 4
Ok!
> 5
Ok!
  1. Turn on the network connection and check that the data is synced.

The topic on the Cloud cluster should automatically synchronize with the edge cluster.

  1. Consume from the cloud terminal:

Wait for the connection retry interval to trigger for the new records to arrive, then consume:

fluvio consume edge-topic --mirror edge1 -B
1
2
3
4
5

The disconnect test was successful.

Test 3: Edge Cluster Offline​

This test ensures that the edge cluster will preserve all cached data following a power loss.

Restart Edge Cluster​

On the edge terminal, shutdown the cluster:

fluvio cluster shutdown --local

Restart the cluster:

fluvio cluster upgrade

Consume from edge cluster​

First, on the edge terminal, check the status of the home cluster from the Cloud:

fluvio cluster home
HOME  ROUTE            SC STATUS  SPU STATUS  LAST SEEN  ERRORS
home  localhost:30003  Connected  Connected   1s         -

Then, consume from the edge cluster:

fluvio consume edge-topic -B
1
2
3
4
5

Produce records and observe that the mirror will resume the synchronization.

🎉 Congratulations! These tests confirm that the synchronization from Edge to Cloud works as expected. It is now time to roll it out in your environment.

Join us on Discord if you have questions, or would like to suggest new improvements.