Kibana would no longer load, the lower frame just went missing, or even the page would not load at all.

Reviewing the apache log exposed an error 500 that was not accompanied by an error in kibana.log

But I found that the elasticsearch storage logfile (i.e. logstash-01.log, not the elasticsearch.log) contained this message:

 

[2016-01-13 03:23:11,218][INFO ][rest.suppressed          ] /.kibana/index-pattern/_search Params: {index=.kibana, fields=, type=index-pattern}
Failed to execute phase [query_fetch], all shards failed; shardFailures {[xxx][.kibana][0]: RemoteTransportException[[Joseph]
[127.0.0.1:9300][indices:data/read/search[phase/query+fetch]]]; nested: QueryPhaseExecutionException[Result window is too large, from + size must be 
less than or equal to: [10000] but was [2147483647]. See the scroll api for a more efficient way to request large data sets. This limit can be set by 
changing the [index.max_result_window] index level parameter.]; }

As you can see this message saying "all shards failed", along with a large backtrace, is flagged as an "INFO" level, so while it fully breaks your application it's still assumed to be not a problem.

As such, you only the the 500 but nothing else :/

 

That mess is caused by a setting in 

/opt/kibana/src/ui/public/index_patterns/_get_ids.js

The related bug is:

https://github.com/elastic/kibana/issues/5287

I changed the value to 10000 but could not make it take effect since I don't actually know how to "redeploy" a node.js application.



Instead I had to upgrade to Kibana 4.3.1, which kind of equals a redeployment :-)