I wrote a simple tool to upload logs to HDFS. And I found some curious phenomenon.
If I run the tool in foreground and close it with "Ctrl - C", there will be some data in HDFS.
If I run the tool in background and kill the process with "kill -KILL pid
", the data has been processed is lost and leaves an empty file in HDFS.
My tool has tried to do sync (by invoking SequenceFile.Writer.syncFs()
) frequently (every 1000 lines).
And I just couldn't figure out why the data was lost. If my tool has run all day but the machine crashed suddenly, will all the data be lost?
My tool is used to collect logs from different servers and then upload to HDFS (aggregating all log to a single file every day).