Mirroring IoT Data from Raspberry Pi
This advanced tutorial requires a Raspberry Pi and a local installation of your collector cluster running on Local.
Raspberry Pi to Local Cluster
This section will use a Raspberry Pi v3 running Ubuntu as the remote device and our local machine for the home cluster.
Let's start with installing and configuring the home cluster.
Download fluvio binary
Use curl to download and install:
curl -fsS https://hub.infinyon.cloud/install/install.sh | bash
Make sure to add .fluvio/bin to the $PATHas specified in the installation script.
Start home cluster
Use the fluvio binary to start the cluster:
fluvio cluster start
Check the result with:
fluvio cluster status
Remote Raspberry Cluster:
fluvio remote register remote-raspberry
Create the mirror topic
Mirror topics on the home cluster have multiple partitions, where each partition has a 1-to-1 relationship with the remote cluster.
Create the mirror topic, then add the remote to it:
fluvio topic create mirror-topic --mirror
fluvio topic add-mirror mirror-topic remote-raspberry
or apply a JSON file with an array of remotes that you want to assign when creating a topic:
echo '["remote-raspberry"]' > assignment_file.json
fluvio topic create mirror-topic --mirror-apply assignment_file.json
List partitions to check the assignment:
fluvio partition list
It should display all partitions:
TOPIC PARTITION LEADER MIRROR REPLICAS RESOLUTION SIZE HW LEO LRS FOLLOWER OFFSETS
mirror-topic 0 5001 remote-raspberry [] Online 0 B 0 0 0 0 []
List remote clusters to check their status:
fluvio remote list
It should show the following:
REMOTE SC STATUS SPU STATUS LAST SEEN ERRORS
remote-raspberry Waiting Waiting - -
Generate Metadata for a Remote Cluster
Each remote cluster requires a unique metadata file that gives the remote cluster the information to connect to the home cluster and the topic/mirror where the data is synchronized.
The home remote device is a Raspberry Pi device. You may skip this if you don't have such a device.
The IP address of our machine where the home server is running is 192.168.1.42. Please identify your own IP address and replace it with the command below.
fluvio remote export remote-raspberry --public-endpoint 192.168.1.42 --file remote2.json
We'll transfer these files to remote devices in the following sections.
Install Remote Cluster on Raspberry Pi
We'll use the same procedure as before to mirror from the Raspberry Pi to the same home cluster. The test below was performed on a Raspberry Pi v3 running an Ubuntu image.
Download the metadata file
We'll use the metadata file remote-raspberry.json that we've exported above to provision this device.
Identify the IP address of your Raspberry Pi device and it replaces below
Using the home terminal, let's use the scp command to send the remote-raspberry.json file to the remote device:
scp remote-raspberry.json fluvio@192.168.1.50:~
Login to the remote device
Spawn a new terminal and login into the Raspberry Pi:
ssh fluvio@192.168.1.50
Download fluvio binaries
On the Raspberry Pi, run the following command:
curl -fsS https://hub.infinyon.cloud/install/install.sh | bash
Run fluvio version to double-check.
Start cluster
First we will start the cluster:
fluvio cluster start
Then, we'll use the metadata file on the Raspberry Pi to connect:
fluvio home connect --file remote-raspberry.json
Let's check the partitions:
fluvio partition list
The remote device should show the following partition::
TOPIC PARTITION LEADER MIRROR REPLICAS RESOLUTION SIZE HW LEO LRS FOLLOWER OFFSETS
mirror-topic 0 5001 home_name:0:public_endpoint [] Online 0 B 11 11 11 0 []
Test 1: Mirror from Raspberry Pi Remote to Home
Let's produce on the Raspberry Pi and consume from the home cluster.
Produce for a remote cluster
Produce on the pi terminal:
fluvio produce mirror-topic
> A
Ok!
> B
Ok!
Consume from home
Consume on the home terminal:
fluvio consume mirror-topic --mirror remote-raspberry -B
A
B
Mirror test is successful.
Test 2: Home Cluster Offline
Shutdown the home cluster and check that the remote cluster can continue receiving records. Then, resume the home cluster and ensure the data is synchronized and can be consumed on both sides.
Shutdown the home cluster
On the home terminal, shutdown the cluster:
fluvio cluster shutdown --local
Ensure the cluster is off with:
fluvio cluster status
Produce on the remote cluster
Produce a few more records on the pi terminal:
fluvio produce mirror-topic
C
D
E
Reconnect home cluster & consume from topic
On the home terminal, restart the cluster:
fluvio cluster upgrade --local
The topic on the home cluster should automatically synchronize with the remote cluster.
Wait for the connection retry interval to trigger for the records to arrive.
Let's consume:
fluvio consume mirror-topic --mirror remote-raspberry -B
A
B
C
D
E
The disconnect test was successful.
Test 3: Remote Cluster Offline
This test ensures that the remote cluster will not lose data following a power loss.
Restart the remote cluster
On the remote terminal, shutdown the cluster:
fluvio cluster shutdown --local
Restart the cluster:
fluvio cluster upgrade
Consume from a remote cluster
First, on the pi terminal, check the status of the target cluster:
fluvio cluster home
HOME ROUTE STATUS SC STATUS SPU STATUS LAST SEEN ERRORS
home localhost:30003 Online Connected Connected 1s -
Then, consume from the remote cluster:
fluvio consume mirror-topic -B
A
B
C
D
E
Produce records and observe that the mirror will resume the synchronization.
🎉 Congratulations! You have successfully tested remote mirroring using the Raspberry Pi. It is now time to roll it out in your environment.