1

I'm running a k3s cluster on RPi4, with heterogenous config (a node has a high capacity but slow hdd, another has a ssd drive, a third only has a sd card).

I have persistent volumes & claims of kind "local-path", attached to nodes & pods depending on my needs.

I'm facing a situation where I need to schedule a pod on the node with no disk to process data stored in the node with the ssd disk (re-encode some video files to mp4 using ffmpeg, and as this is an expensive process I'd like to do this on an idle node and not slow the node running the ssd).

Is it possible to transparently mount a PV from a different node ? Do I need to use some nfs ? Is there a more evoluted type of volume that can be used in bare-metal RPi4 to do what I want ?

Looking at the docs didn't help much (there is tons of different persistent volume type, with not many use-case described).

Thanks

spi
  • 113
  • 4

1 Answers1

3

I think you may be interested in Project Longshorn.
It is a persistent storage implementation for any Kubernetes cluster and it uses the existing disks of your Kubernetes nodes to provide replicated and stable storage for your Kubernetes pods.

As described on Longhorn github:

Longhorn is lightweight, reliable, and powerful. You can install Longhorn on an existing Kubernetes cluster with one kubectl apply command or using Helm charts. Once Longhorn is installed, it adds persistent volume support to the Kubernetes cluster.

And best of all, you can use intuitive GUI dashboard to customize Longhorn to your needs.

Note: At this time Longhorn only supports amd64.

Installation

In the below example, I will show you how to install Longshorn.

First of all you must have open-iscsi installed on each cluster nodes. You need to install it using for example:

apt-get install open-iscsi

Next apply the longhorn.yaml to install Longhorn:

kubectl apply -f https://raw.githubusercontent.com/longhorn/longhorn/master/deploy/longhorn.yaml

Then create a longhorn StorageClass (you can customize it later using ConfigMap)

kubectl apply -f https://raw.githubusercontent.com/rancher/longhorn/master/deploy/longhorn.yaml

That's all we need to start using Longhorn :)

Longhorn UI

As I described before, we can use helpful GUI dashboard, here you can find how to access it.
In addition, you may need to expose the dashboard to the outside world using Ingress as described here

Using Longhorn UI dashboard is pretty intuitive, e.g. in "Nodes" tab , you can easily specify Nodes dedicated to provide storage.

longhorn StorageClass

You can customize longhorn StorageClass by editing longhorn-storageclass ConfigMap in longhorn-system namespace For example I edited numberOfReplicas parameter from 3 (default) to 1:

root@kmaster:~# kubectl describe sc longhorn
Name:            longhorn
IsDefaultClass:  No
Annotations:     longhorn.io/last-applied-configmap=kind: StorageClass
apiVersion: storage.k8s.io/v1
...
parameters:
  numberOfReplicas: "1"
  staleReplicaTimeout: "2880"
  fromBackup: ""
...
matt_j
  • 350
  • 2
  • 6
  • this is compelling, and I think that is exactly what I need, Too bad the arm64 package is still experimental. But you deserve a vote :) – spi Jan 15 '21 at 13:51
  • after playing a little with that tool, even though arm64 is experimental, this works very well with little knowledge and so far, I didn't face any difficulty in making it work with Rpi4. Very very nice tool! – spi Jan 15 '21 at 19:38