I am training faster_rcnn_inception module for object detection on custom dataset. In training directory,we found folder called eval_0 and tensorflow generated events.out.tfevents.xxxxxx files.
Training Directory structure as follows
+training_dir
+eval_0
-events.out.tfevents.1542309785.instance-1 1.2GB
-events.out.tfevents.1542367255.instance-1 5.3GB
-events.out.tfevents.1542369886.instance-1 3.6GB
-events.out.tfevents.1542624154.instance-1 31MB
-events.out.tfevents.1543060258.instance-1 19MB
-events.out.tfevents.1543066775.instance-2 1.6GB
-events.out.tfevents.1542308099.instance-1 17MB
-events.out.tfevents.1542308928.instance-1 17MB
-events.out.tfevents.1542366369.instance-1 17MB
-events.out.tfevents.1542369000.instance-1 17MB
-events.out.tfevents.1542623262.instance-1 17MB
-events.out.tfevents.1543064936.instance-2 17MB
-events.out.tfevents.1543065796.instance-2 17MB
-events.out.tfevents.1543065880.instance-2 17MB
-model.ckpt-96004.data-00000-of-00001
-model.ckpt-96004.data-00000-of-00001
-model.ckpt-96004.index
-model.ckpt-96004.meta
-model.ckpt-96108.data-00000-of-00001
-model.ckpt-96108.index
-model.ckpt-96108.meta
As per my understanding, tfevents files in eval_0 folder are summery files of evaluation and tfevents files in training_dir are summery files of training.
I have interrupted training process several times and continued from recent checkpoint. I also understand restarting training process generates new tfevents files.
My Questions as follows:
Why training tfevents_files have same size, but in case if eval_0/tfevents_files size varies ?
Why interrupting training generates new tfevents_file in training folder, but same not observed in case of eval_0?
Can I delete all tfevents files in eval_0 except latest one? Does it affect on training or evolution history?
