0

ElastiCache Cluster was in red state. Below is the out of cluster health Status

{
"cluster_name" : "graylog",
"status" : "red",
"timed_out" : false,
"number_of_nodes" : 1,
"number_of_data_nodes" : 1,
"active_primary_shards" : 0,
"active_shards" : 0,
"relocating_shards" : 0,
"initializing_shards" : 4,
"unassigned_shards" : 24,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 2,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 0.0

}

This is output of 

red    open   graylog_6   4   0
red    open   graylog_1   4   0
red    open   graylog_0   4   0
red    open   graylog_5   4   0
red    open   graylog_4   4   0
red    open   graylog_3   4   0
red    open   graylog_2   4   0

I have only diskspace of 75 GB

40M  ./graylog_0
6.1G  ./graylog_1
40G ./graylog_2
21G ./graylog_3
220K  ./graylog_4
2.3M  ./graylog_5
3.3G  ./graylog_6
70G total

From which I concluded that I am running out of space.Is it safe to indexes. What is the solution if it is risky to delete the indexes

tessie
  • 101
  • 5

2 Answers2

1

you can freely delete your indexes (from the indexes panel) in either graylog or elasticsearch, just recalculate your indexes after.

you may want to update your retention policy to fit your storage availability.

I would rotate them based on space, and then only keep the number of indexes you have room for.

example, rotate every 10G, keep last 7, assuming you have about 85G of free space. (not exactly numbers but a good baseline)

Select System, Indexes

system

Click UPdate Configuration

settings

Set rotation type to Size

storage

Jacob Evans
  • 7,636
  • 3
  • 25
  • 55
  • Thank you.I am actually getting 502 on the web interface which I assume is because of the cluster status red. How could we enable ' rotate them based on space, and then only keep the number of indexes you have room ' this – tessie Mar 08 '17 at 18:03
  • what version do you have – Jacob Evans Mar 08 '17 at 18:08
  • graylog 2.2.2-1 – tessie Mar 08 '17 at 18:13
  • @tes added, the elasticsearch cluster can be red and graylog would still operate, you might have a storage issue on graylog as those logs will queue in a journal until processed. – Jacob Evans Mar 08 '17 at 18:19
  • Thank you. What could be the reason I am getting a 502 . I am using nginx as a reverse proxy. http://serverfault.com/questions/836949/502-bad-gatway-graylog – tessie Mar 08 '17 at 18:22
  • Thank you. It was because mongodb stopped working due to memmory issue.Restarting it fixed the issue – tessie Mar 09 '17 at 05:53
0

The issue got fixed. Check all the services including mongodb. Mongodb has stopped working due to memory issue. After deleting some of the older index restarting mongodb it start working

tessie
  • 101
  • 5