6

I would like to setup monitoring for system resources on my Windows servers. I've noticed a common configuration in Linux is to use collectd daemon to get system metrics information. From collectd data can read by logstash and ultimately put into Elastic Search to be viewed with Kibana.

This is nice and works well in the Linux world. However I'm stuck with Windows server and I need some advice to the best tools to achieve a similar workflow. As a side node I'm already using Nxlog to send IIS logs to logstash.

angaran
  • 355
  • 2
  • 3
  • 11

4 Answers4

4

Elastic now offers a tool called topbeat that does what you're looking for. It sends cpu, memory and disk stats directly into Elasticsearch or into Logstash.

Example metrics are on github at elastic/topbeat.

tommy_o
  • 141
  • 4
  • 2
    Topbeat seems deprecated now and the linked repository does not exist anymore. It seems to be now superseded by Metricbeat https://www.elastic.co/guide/en/beats/metricbeat/current/index.html – angaran Nov 09 '16 at 12:02
2

The 'Tools that works with Graphite' page http://graphite.readthedocs.org/en/latest/tools.html list several. I have tried the PowerShell script 'Graphite PowerShell Functions' https://github.com/MattHodge/Graphite-PowerShell-Functions and it works well.

edit I mis-read your question, you were talking only about Logstash and Kibana but not about Graphite. I don't use Logstash+Kibana for system metrics, but I use Statsd+Graphite. So not sure if my answer is valid to you but if you use the Graphite Logstash input http://logstash.net/docs/1.4.2/ you could use these tools.

daks
  • 673
  • 6
  • 23
  • Thanks, seems like this could be a valid and convenient solution. – angaran Dec 02 '14 at 09:42
  • Ok, so specifically we configure the Logstash [wmi input](http://www.logstash.net/docs/1.4.2/inputs/wmi) after installing the [contrib-plugins](http://www.logstash.net/docs/1.4.2/contrib-plugins) – CrazyPyro Apr 02 '15 at 05:35
  • And for version 1.4.2, you may need to manually comment-out a line per [issue#123](https://github.com/elastic/logstash-contrib/issues/123) if you get the error `LoadError: no such file to load -- logstash/inputs/eventlog/racob_fix` – CrazyPyro Apr 02 '15 at 05:51
  • @CrazyPyro if you use the specified Powershell script you don't need the wmi input of logstash. But running logstash directly on your server is in fact another solution, but it implies you need to run Java too (even if you can just copy Java binaries without installing it). – daks Apr 28 '15 at 11:54
0

Im using Powershell 5 and Filebeat to resolve this issue. Note this has not been tested more then a few hours and its a proof of concept.

#Version 0.2

#Todo:
#* Notify on fail
#Function to get Sql Server Counters
function Get-SqlServerData()
{
    $Data = get-counter ($SqlServerCounterPrefix + ":Buffer Manager\Buffer cache hit ratio"), 
                        ($SqlServerCounterPrefix + ":Buffer Manager\Page life expectancy"), 
                        ($SqlServerCounterPrefix + ":Access Methods\Page splits/sec")
    #$Data
    $TransformedData = $Data.CounterSamples | Select-Object -Property Path, CookedValue

    $object = New-Object psobject

    $object | Add-Member -NotePropertyName 'Timestamp' -NotePropertyValue $Data.Timestamp
    foreach ($row in $TransformedData)
    {
        $path = $row.Path
        $name = $path.Substring($path.LastIndexOf("\") + 1)
        $object | Add-Member -NotePropertyName $name -NotePropertyValue $row.CookedValue
    }
    $object
}


#Parameters
$SqlServerCounterPrefix = '\MSSQL$MSSQL_2008'
$Type = "SQLServerStatistics"
$Data = Get-SqlServerData
$Path = "Z:\Temp\PowershellTest\"
$AddTimeStamp = $false
$NumberOfDaysToKeepFiles = 7
$FileExtension = "csv"

#Variables (do not change)
$Date = Get-Date -format yyyy-MM-dd
$Timestamp = Get-Date
$FilenameRegex = "^" + $Type + "_(?<Date>\d{4}-\d{2}-\d{2})(?:\(\d\))?\." + $FileExtension + "$"
$Suffix = ''
$Counter = 0
$Done = $false

if ($AddTimeStamp -eq $true)
{
    $Data | ForEach-Object {
        $_ | Add-Member -NotePropertyName 'Timestamp' -NotePropertyValue $Timestamp
    }
}

#Try to export file if it fails (the headers have changed) add a (number)
while($Done -eq $false -and $Counter -le 9)
{
    Try
    {
        $Filename = $Type + "_" + $Date + $Suffix + "." + $FileExtension
        Write-Host "Trying to write $Filename"
        $FilePath = $Path + $Filename
        $Data | Export-Csv -Path $FilePath -Delimiter ";" -NoTypeInformation -Append
        $Done = $true
    }
    Catch [Exception]
    {
        Write-Host "Failed to create file " + $_.Exception.GetType().FullName, $_.Exception.Message
        $Counter++
        $Suffix = '(' + $Counter + ')'
    }
}

#Notify if we failed
if ($Done -eq $false)
{
    #Todo: Notify that we failed
}

#Cleanup
$Files = Get-ChildItem $Path -Filter ("*." + $FileExtension)
$Files | Foreach-Object {
    $FilePath = $_.FullName
    $Filename = [System.IO.Path]::GetFileName($FilePath)

    $Match = [regex]::Match($Filename, $FilenameRegex)

    Write-Host $FilePath
    Write-Host $Filename

    if ($Match.Success -eq $true)
    {
        Write-Host $Match.Groups["Date"].Value
        $FileDate = [datetime]::ParseExact($Match.Groups["Date"].Value, "yyyy-MM-dd", $null)
        $TimeSince = New-TimeSpan -Start $FileDate -End (Get-Date).Date
        if ($TimeSince.TotalDays -ge $NumberOfDaysToKeepFiles)
        {
            Remove-Item $FilePath
        }
    }
}

This will create a csv file in the $Path directory, if you change the counters then it will create a new file with (1-9) in the name.

A updated version can be found at: https://gist.github.com/AnderssonPeter/8261e255630d6f672ba5bc80f51b017f

Peter
  • 353
  • 2
  • 3
  • 9
  • Did you consider Metricbeat? https://www.elastic.co/guide/en/beats/metricbeat/current/index.html – angaran Nov 09 '16 at 12:02
  • @angaran we are using `Metricbeat` but there is no module for `windows performance counters` https://github.com/elastic/beats/issues/2473 – Peter Nov 09 '16 at 13:28
-1

There is now a logstash input plugin for Windows Performance Monitor.

NickRamirez
  • 165
  • 8