1

I deployed an Elasticsearch cluster on GKE, using this project as a starting point: https://github.com/pires/kubernetes-elasticsearch-cluster

I would like to configure two Curator jobs by using Kubernetes CronJob objects, to snapshot the indices, and then to delete/prune old indices.

I would like to store the snapshots on a GCS bucket. I created a new service account and downloaded the JSON credentials key, to be used with the elasticsearch keystore. See plugin docs: https://www.elastic.co/guide/en/elasticsearch/plugins/current/repository-gcs-usage.html

I am not sure how/where to add this key, to be used by the Curator CronJob performing the backup. The Elasticsearch documentation mentions running the elasticsearch-keystore binary on the credentials key file.

curator.yaml:

apiVersion: batch/v1beta1
    kind: CronJob
    metadata:
      name: curator
    spec:
      schedule: 0 11 * * *
      jobTemplate:
        spec:
          template:
            spec:
              containers:
              - name: curator
                image: quay.io/pires/docker-elasticsearch-curator:5.4.1
                args:
                - --config
                - /etc/config/config.yml
                - /etc/config/action_file.yml
                env:
                  - name:
                volumeMounts:
                  - name: config-volume
                    mountPath: /etc/config
              volumes:
                - name: config-volume
                  configMap:
                    name: curator-config
              restartPolicy: OnFailure

curator-config.yaml:

apiVersion: v1
kind: ConfigMap
metadata:
  name: curator-config
data:
  action_file.yml:
    # Remember, leave a key empty if there is no value.  None will be a string,
    # not a Python "NoneType"
    #
    # Also remember that all examples have 'disable_action' set to True.  If you
    # want to use this action as a template, be sure to set this to False after
    # copying it.
    actions:
  1:
    action: snapshot
    options:
      repository: gcs_repository
      name: ${SNAPSHOT_NAME:snapshot-%Y-%m-%d}
      continue_if_exception: false
    filters:
      - filtertype: age
        source: name
        direction: older
        timestring: '%Y-%m-%d'
        unit: days
        unit_count: ${DAYS}
  2:
    action: delete_indices
    options:
      continue_if_exception: false
    filters:
      - filtertype: age
        source: name
        direction: older
        timestring: '%Y-%m-%d'
        unit: days
        unit_count: ${DAYS}

$kubectl get pods

NAME                                             READY     STATUS    RESTARTS   AGE
cerebro-59648dc47c-vr964                         1/1       Running   0          25d
es-client-7bff44b8f5-2wqcs                       1/1       Running   0          12d
es-client-7bff44b8f5-vnrhg                       1/1       Running   0          12d
es-data-0                                        1/1       Running   0          52d
es-data-1                                        1/1       Running   0          52d
es-data-2                                        1/1       Running   0          52d
es-master-6bf767f949-8fpjl                       1/1       Running   0          52d
es-master-6bf767f949-brjpq                       1/1       Running   0          52d
es-master-6bf767f949-gx2jp                       1/1       Running   0          52d
fluentd-gcp-v2.0-7mncl                           1/1       Running   0          43m
fluentd-gcp-v2.0-rsfmc                           1/1       Running   0          43m
fluentd-gcp-v2.0-tbh9t                           1/1       Running   0          43m
kibana-595858b4b7-5npcr                          1/1       Running   0          52d
nginx-ingress-controller-86c8447687-z4rjq        1/1       Running   2          52d
nginx-ingress-default-backend-6664bc64c9-q2hnm   1/1       Running   338        52d 
VAS
  • 370
  • 1
  • 9
syst0m
  • 131
  • 1
  • 4

0 Answers0