It doesn't look like there's a good way filter this like you are thinking of, but you could add a couple actions to your task to disable logging first, enable it after.
There is a chance that there is another scheduled task trying to write to the log at that time, so you would need to be okay with some potential logging loss on that end.
See below script from here
$logName = 'Microsoft-Windows-TaskScheduler/Operational'
$log = New-Object System.Diagnostics.Eventing.Reader.EventLogConfiguration
$logName
$log.IsEnabled=$true
$log.SaveChanges()
The only other way I can think of doing this is disabling it across the board and then for the events you want logging you could log manually by encapsulating your actual commands in a powershell script or something of the like that writes all output to text or even to a custom event log.