lnav not recognizing our logs. Custom format not seeming to be applied.

I am working on adding a lnav format to parse the logs of my companies java application. I am running into difficulty getting lnav to recognize them. Here is what my json format file looks like so far:

{
    "company_serverlog" : {
        "title" : "Company Server Log",
        "description" : "Log format used by the application.",
        "url" : "http://www.google.com",
        "regex" : {
            "std" : {
                "pattern" : "^(?<timestamp>\\d\\d\\d\\d-\\d\\d-\\d\\d \\d\\d:\\d\\d:\\d\\d,\\d\\d\\d)\\|(?<level>DEBUG|ERROR|WARN|HEADER)\\|(?<orig>.+:)\\|(?<method>.+\\|)(?<body>.+)$"
            }
        },
        "level-field" : "level",
        "level" : {
            "error" : "ERROR",
            "warning" : "WARN"
        },
        "value" : {
            "orig" : {
                "kind" : "string",
                "identifier" : true
            },
            "method" : {
                "kind" : "string",
                "identifier" : true
            },
            "body" : {
                "kind" : "string",
                "identifier" : true
            }
        },
        "sample" : [
            {
                "line" : "2016-05-25 16:07:12,367|DEBUG|source of log msg:|com.company.package.file| Log message body here"
            }
        ]
    }
}

lnav is still picking up our log files as generic log files. Any assistance and/or recommendations would be greatly appreciated.

Andrew

Posted 2016-06-07T20:35:19.983

Reputation: 213

Answers

The issue was with my regex. I needed to address the case in our logs where the origin capturing group did not have a : character for ERROR or WARN log levels. After adding the quantifier ? for the : character, I was able to get lnav to pick up our companies log file and apply this custom format.

This oversight highlights the need to be very careful when crafting regex.

Andrew

Posted 2016-06-07T20:35:19.983

Reputation: 213

It might also help others in future to go into what/where the correct regex bits were – Journeyman Geek – 2016-06-14T23:59:58.137

Thanks for the critique. I edited the 'answer' to reflect your suggestions – Andrew – 2016-06-15T17:09:17.123

It sounds like lnav is not finding the config file. Are you placing the format file in a sub-directory of the ~/.lnav/formats directory? For example:

~/.lnav/formats/company_serverlog/format.json

You can enable debugging in lnav with the '-d <file>' option. That should give you an idea of what config files are being found and loaded.

Timothy Stack

Posted 2016-06-07T20:35:19.983

Reputation: 56

Hmm... I double checked the things you mentioned. I checked the output of the lnav -d output and it looks like my format file is getting loaded. It turned out there was an issue with my regex – Andrew – 2016-06-14T21:12:05.537

You should add a representative set of lines in the "sample" list of the format configuration to catch any issues with the regexes. On startup, lnav will check the sample lines against the regexes to verify that they are correct and, if they are not correct, it will print out some helpful messages showing where there was a mismatch. – Timothy Stack – 2016-06-15T21:44:04.230