As pointed by @kiyoto-tamura, fluentd can partition output files by day and this is the default behaviour.
And, as pointed by @vaab, fluentd cannot delete old files. Hence, the obvious solution would be to disable the partitioning of fluentd and let logrotate to handle partitioning along with watching for the number of files.
However, this might introduce an unnecessary complexity, especially for simple cases: one will have to install, configure, and monitor an additional service, which is logrotate.
Moreover, it can be problematic to get analogous of logrotate on Windows (unless one is comfortable with installing cygwin or using 0.0.0.x versions in production).
So, another feasible solution would be let fluentd to partition output files by day as usual, but delete old files periodically.
In UNIX-like systems this is a no-brainer and can be achieved by writing a one-liner shell script invoking find and scheduling it via cron.
The same logic applies to Windows environment. For example, one can write a PowerShell script and schedule it via the system task scheduler (see the Gist for readability).
The following listing defines such a script as an example:
# delete-old-service-logs.ps1
[CmdletBinding()]
param(
[Parameter(Position=0, Mandatory=$true)]
[ValidateScript({
if( -Not ($_ | Test-Path) ){
throw "Specified logs dir path does not exist (path='$_')"
}
if(-Not ($_ | Test-Path -PathType Container) ){
throw "Specified logs dir path does not point to a directory (path='$_')"
}
return $true
})]
[string]$LogsDirPath,
[Parameter(Position=1)]
[int]$LogsFileMaxN = 31,
[Parameter(Position=2)]
[ValidateNotNullOrEmpty()]
[string]$LogsFileNamePattern = "service.????-??-??.log"
)
[string[]]$FileNamesToRemove = Get-ChildItem -Path $LogsDirPath -Filter $LogsFileNamePattern |
Sort-Object -Property CreationTime -Descending |
Select-Object -Skip $LogsFileMaxN |
Select -ExpandProperty "Name"
$Shell = new-object -comobject "Shell.Application"
$LogsDir = $Shell.Namespace($LogsDirPath)
Foreach ($FileName in $FileNamesToRemove)
{
$Item = $LogsDir.ParseName($FileName)
$Item.InvokeVerb("delete")
}
The logic is pretty straightforward:
- Take the path to the dir with logs.
- Take the max number of files to keep.
- Take the file pattern to search logs by.
- Find all files in the logs dir, reverse-sort by creation date and take names of old files.
- Delete old files if any.
Invocation example:
./delete-old-service-logs.ps1 "path/to/logs/dir"
or:
./delete-old-service-logs.ps1 -LogsDirPath "path/to/logs/dir"
As creating a scheduled task in Windows via GUI can be a pain, one can have a script, which automates this:
# create-task_delete-old-service-logs.ps1
[CmdletBinding()]
param(
[Parameter(Position=0, Mandatory=$true)]
[ValidateNotNullOrEmpty()]
[string]$LogsDirPath,
[Parameter(Position=1)]
[int]$LogsFileMaxN = 31,
[Parameter(Position=2)]
[ValidateNotNullOrEmpty()]
[string]$LogsFileNamePattern = "service.????-??-??.log",
[Parameter(Position=3)]
[ValidateNotNullOrEmpty()]
[string]$TaskScriptFilePath = "delete-old-service-logs.ps1",
[Parameter(Position=4)]
[ValidateNotNullOrEmpty()]
[string]$TaskName = "SERVICE NAME - Delete Old Logs",
[Parameter(Position=5)]
[ValidateNotNullOrEmpty()]
[string]$TaskDescription = "Delete old logs of SERVICE NAME aggregated via Fluentd",
[Parameter(Position=6)]
[ValidateNotNullOrEmpty()]
[string]$TaskTriggerTime = "5:00:00 AM"
)
try {
$LogsDirPath = Resolve-Path -Path $LogsDirPath
}
catch {
throw "Specified logs dir path does not exist (path='$LogsDirPath')"
}
if(-Not ($LogsDirPath | Test-Path -PathType Container) ){
throw "Specified logs dir path does not point to a directory (path='$LogsDirPath')"
}
try {
$TaskScriptFilePath = Resolve-Path -Path $TaskScriptFilePath
}
catch {
throw "Specified task script file path does not exist (path='$TaskScriptFilePath')"
}
if( -Not ($TaskScriptFilePath | Test-Path -PathType Leaf) ){
throw "Specified task script file path is not a file (path='$TaskScriptFilePath')"
}
$TaskAction = New-ScheduledTaskAction -Execute "C:\Windows\system32\WindowsPowerShell\v1.0\powershell.exe" `
-Argument "-NoProfile -WindowStyle Hidden -command ""$TaskScriptFilePath -LogsDirPath ""$LogsDirPath"" -LogsFileMaxN $LogsFileMaxN -LogsFileNamePattern ""$LogsFileNamePattern"""""
$TaskTrigger = New-ScheduledTaskTrigger -Daily -At $TaskTriggerTime
Register-ScheduledTask -TaskName $TaskName -Description $TaskDescription -Action $TaskAction -Trigger $TaskTrigger
The actual logic happens at the last 3 lines, where an action, a trigger and a task are created and registered. In overall, the workflow is as follows:
- Take params for
delete-old-service-logs.ps1.
- Take path to
delete-old-service-logs.ps1 script.
- Take task name, description and time to trigger the script.
- Try to resolve and validate paths.
- Create an action for invoking PowerShell with args to run
delete-old-service-logs.ps1.
- Create a daily trigger.
- Register the task.
Invocation example, assuming both scripts are in the same directory:
./create-task_delete-old-service-logs.ps1 "path/to/logs/dir"
Or:
./create-task_delete-old-service-logs.ps1 -LogsDirPath "path/to/logs/dir" -TaskScriptFilePath "path/to/delete-old-service-logs.ps1"
