Kapaski's Blog

not that russian anti-virus software

Logstash to Parse Json With Json Arrays in Values

Logstash has a known issue that it doesn’t convert json array into hash but just return the array.

Example

{a:[11,22,33]} gives you a = [11,22,33] << this is correct {a:[{foo:11}, {foo:22}]} gives you a = [{foo:11}, {foo:22}] << this is not flat enough, especially some queries are requiring to use keys like a.foo = 11

By all means, there a couple of pull request to the Logstash github : https://github.com/elasticsearch/logstash/pull/1185 but neither of them has been merged. There are still below comments in source code today:

  # TODO(sissel): Note, this will not successfully handle json lists
  # like your text is '[ 1,2,3 ]' json parser gives you an array (correctly)
  # which won't merge into a hash. If someone needs this, we can fix it
  # later.

So standing in middle of nowhere, with what’s on hand, the Ruby filter seems the last resort, and below code works, so no need to use the out of box ‘json’ filter anymore

  input {
    stdin{}
  }

  filter {
    grok {
      match => ["message","(?<json_raw>.*)"]
    }

    ruby {
      init => "
        def parse_json obj, pname=nil, event

          obj = JSON.parse(obj) unless obj.is_a? Hash
          obj = obj.to_hash unless obj.is_a? Hash

          obj.each {|k,v|
            p = pname.nil?? k : pname
            if v.is_a? Array
              v.each_with_index {|oo,ii|

                parse_json_array(oo,ii,p,event)
              }
            elsif v.is_a? Hash
              parse_json(v,p,event)
            else
              p = pname.nil?? k : [pname,k].join('.')
              event[p] = v
            end
          }

        end

        def parse_json_array obj, i,pname, event

          obj = JSON.parse(obj) unless obj.is_a? Hash
          pname_ = pname
          if obj.is_a? Hash
            obj.each {|k,v|

              p=[pname_,i,k].join('.')
              if v.is_a? Array
                v.each_with_index {|oo,ii|
                  parse_json_array(oo,ii,p,event)
                }
              elsif v.is_a? Hash
                parse_json(v,p, event)
              else
                event[p] = v
              end
            }
          else
            n = [pname_, i].join('.')
            event[n] = obj
          end
        end
      "
      code => "parse_json(event['json_raw'].to_s,nil,event) if event['json_raw'].to_s.include? ':'"
    }


  }

  output {
    stdout{codec => rubydebug}
  }

Test json structure

{"id":123, "members":[{"i":1, "arr":[{"ii":11},{"ii":22}]},{"i":2}], "im_json":{"id":234, "members":[{"i":3},{"i":4}]}}

and this is whats output

      {
           "message" => "{\"id\":123, \"members\":[{\"i\":1, \"arr\":[{\"ii\":11},{\"ii\":22}]},{\"i\":2}], \"im_json\":{\"id\":234, \"members\":[{\"i\":3},{\"i\":4}]}}",
          "@version" => "1",
        "@timestamp" => "2014-07-25T00:06:00.814Z",
              "host" => "Leis-MacBook-Pro.local",
          "json_raw" => "{\"id\":123, \"members\":[{\"i\":1, \"arr\":[{\"ii\":11},{\"ii\":22}]},{\"i\":2}], \"im_json\":{\"id\":234, \"members\":[{\"i\":3},{\"i\":4}]}}",
                "id" => 123,
       "members.0.i" => 1,
"members.0.arr.0.ii" => 11,
"members.0.arr.1.ii" => 22,
       "members.1.i" => 2,
           "im_json" => 234,
       "im_json.0.i" => 3,
       "im_json.1.i" => 4
      }

Trick With Kibanan Query String

Since we already did this couple of times, maybe put it in a post for others convenience:

Github Issue Ref

This will allow the user can add a ?query=foo to search with keywords ‘foo’

img

You must put above string in one query field and save the dashboard to make it happen, however the defect will be next time when you try to save the dashboard and forget put the samething in the query field, it will be lost.

Quick walkaround will be — pin it then save, and make your panels listen to this pinned query dedicatedly, and then you will have a dynamic panel showing search result based on http query strings!

The Secret Life of Walter Mitty

The Secret Life of Walter Mitty. 豆瓣评分8.1的影片

check this image

爱做白日梦的主角,屌丝,女神。 专注的做一件事,逆袭什么的总会发生的。 总之是一种说不出的共鸣,很久没看到类似的电影了。

Chef and His Cookbooks

Chef is fun when it has all carefully written cookbooks and a functioning ‘knife’.

ps 发现用Atom来写blog真是方便,git无缝集成.

Octopress Customisation

Added a category list to this blog. Essentially, a sidebar, a plugin and a flag in site config.

Second Post

This is still a test post that showing the share and comment added lately

友言读取速度比较慢的说 如果回复的人不多以后不如改成disqus了

First Post!

First Post!

Still trying to figure out how to deploy the samething on to heroku


Will write some markdown things here:

this is a blog

a leet blog

1
2
#this is code
puts 'i am actually a code block'