I am exploring the Kubernetes toolset and have now my dedicated 3 nodes cluster up and running.
Next steps are migrating our company projects to the cluster. As we use MongoDB and ElasticSearch and want to host multiple projects within the cluster, I am not sure how to handle the persistance correctly.
I will use glusterfs with replication running directly on the nodes (not in kubernetes) and mount volumes to the pods using
https://github.com/kubernetes/examples/tree/master/staging/volumes/glusterfs
Should I install the MongoDB as ReplicaSet directly on the nodes as well or run one MongoDB instance for each project inside a corresponding pod using glusterfs for replicated storage and backup?
As a requirement the lost of one (worker-) node should allow running the lost pods on other nodes without the loss of data.
How should I preserve the database data among multiple nodes? Even replication (MongoDB ReplicaSet e.q.) needs a persistant storage for each pod.