How to merge multiple files based on modified date and ignoring the first line in every file

Question

I have the following files:

-rw-r--r--  1 xyz xyz    122 May 01 01:00  Test1
-rw-r--r--  1 xyz xyz    122 May 01 01:00  Test2
-rw-r--r--  1 xyz xyz    122 May 05 05:00  Test3
-rw-r--r--  1 xyz xyz    122 May 05 05:00  Test4

The contents of the files contains a description row, followed by data, separated by commas. E.g.:

Test1:

ID, Name, Job, Address
1111, John, Janitor, 1234 Corson Ave

Test2:

ID, Name, Job, Address
2222, Tim, Cashier, 3245 Elliot St

Can someone please suggest a command that would concatenate the files above based on modified date while ignoring the first line in the file, which is the description (ID, Name, Job, Address).

E.g. This is what I would like after the script:

May_1_file_after_concatenate:

1111, John, Janitor, 1234 Corson Ave
2222, Tim, Cashier, 3245 Elliot St

Same thing for May5_file_after_concatenate. The directory I have will contain many millions of files. I will need for it to automatically sort out base on modified date and concatenate them accordingly.

I am on Centos 7.

Answer 1

In bash, you can use the following script:

#! /bin/bash
for file in "$@" ; do
    date=$(stat -c%y "$file" | cut -f1 -d' ')
    tail -n+2 "$file" >> "$date"
done

It iterates over the given files, retrieves the last modification date from it, and then adds everything starting from line 2 to a file named by the date.

If there are two many files to fit on a command line or into parameters, call the script on smaller groups of files sequentially (with xargs or manually).

Converting the date format should be easy :-)