I was not able to find the root cause yet but I figured out how to clean it up.
When I copied one of the files to analyse it I found out that the invalid handle was "reused" or "refreshed" and properly closed. It seems that operations on file like open, copy, delete fix the handle. So I created the powershell script that first get the list of handles using util Handle v3.51 and open affected files. After first run the number of handles descreased, physical memory usage started to decrease as well and after a few runs it looks ok. The clean up is scheduled nightly.
$handlesLog = .\handle.exe -p 4 # 4 is System process id
foreach ($line in $handlesLog)
{
if ($line -match "<here is the pattern of affected files>")
{
$fileToCopy = <full path to the file>
if ([System.IO.File]::Exists($fileToCopy))
{
try
{
$fileStr = New-Object System.IO.FileStream($fileToCopy,[System.IO.FileMode]::Open, [System.IO.FileAccess]::Read)
}
finally
{
$fileStr.Close()
$fileStr.Dispose()
}
}
}
}