4

I'm trying to add a query that will match a request that ends with a slash, like this one:

n.n.n.n - - [16/Oct/2013:16:40:41 +0100] "GET / HTTP/1.1" 200 25058 "-" "Mozilla/5.0 (iPad; CPU OS 7_0_2 like Mac OS X) AppleWebKit/537.51.1 (KHTML, like Gecko) Version/7.0 Mobile/11A501 Safari/9537.53"

I'm using the Lucene query type.

If my query is set to *, I see the event.

If I set it to request:"css", I see CSS requests, as expected.

However, all of the following yield no results:

  • request:"/"
  • request:"\/"
  • request:"\\/"

I tried a Lucene regular expression, with no luck:

  • request:/\//

I note that someone else is getting what appears to be a similar issue, although that's on Kibana 2: https://github.com/rashidkpc/Kibana/issues/401

How can I query for requests that end with a / character?

G Mawr
  • 173
  • 1
  • 3
  • 6
  • If you are using logstash 1.3.x, every field will come with a virtual '.raw' field, in your case probably 'request.raw'. This field is not analyzed and you should be able to search for request.raw:"/". See also: http://www.elasticsearch.org/blog/logstash-1-3-1-released/ – Stefan Förster Jan 18 '14 at 21:06
  • @stefan-forster: `request.raw:"/"` works. `request.raw:"/some/path/"` works. `request.raw:"*/"` does not work. – G Mawr Jan 29 '14 at 16:09
  • request.raw:/.*\// perhaps? – Stefan Förster Jan 29 '14 at 18:58
  • @StefanFörster No, that doesn't do it. I'm having difficulty making regular expression queries work at all, even those not restricted to particular fields. When using regular expression, these all return results: `.*`, `.`, `..`, `\s{2}`. These do **not** return results: `\/`, `\s{3}`, `GET`. I think the `_all` field is not being populated, but am unsure how to check or fix that. – G Mawr Jan 30 '14 at 10:14

3 Answers3

0

What mapping have you defined?

Depending on the mapping you have defined on the [request] field, it is possible that the slash '/' is not stored in the elasticsearch index.

If you add a term panel to kibana for the [request] field, do you see the full request values, or do you see those values being split into keywords/term?

yahiko
  • 1
  • I haven't defined any mapping. I just followed the instructions at http://logstash.net/docs/1.2.1/tutorials/getting-started-centralized. I have found the `elasticsearch.yml` file, but haven't made any changes to it. I did add a term panel, and it does indeed appear that the request values are being split into multiple terms, dropping the `/`. – G Mawr Oct 18 '13 at 14:42
0

I have managed to work around my problem by adding a field before records are output to elasticsearch.

In my indexer.conf file, I have added this code:

filter {
  if [request] =~ /\/$/ {
    mutate {
      add_field => {
        'file_type' => 'html'
      }
    }
  }
}

I can now pick out the records that I'm interested in with the query file_type:"html".

This may actually be a better way of doing it, since there is warning about using leading wildcards here:

Allowing a wildcard at the beginning of a word (eg "*ing") is particularly heavy, because all terms in the index need to be examined, just in case they match.

source:http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-query-string-query.html#_wildcards

So, I'm probably going to add tests for images, JavaScript, CSS, etc..

G Mawr
  • 173
  • 1
  • 3
  • 6
0

Using parenthesis around .* works fine for me.

request.raw:/(.*)\//

It returns me all the url ending with /.

Jean-Claude Bernard
  • 1