Once configuration has been ruled out as the likely cause of slowness and you have a repeatably slow query on your hands, a useful next step is to perform some light profiling to capture a recording of your Druid Broker or Historical (or any other java) processes for analysis.
There are three steps:
- Download Swiss Java Knife (SJK) onto the servers that host the Druid processes you'd like to analyze. This will generally be a Broker or Historical, or perhaps an indexing task.
- Use SJK to capture stack dumps.
- Use SJK to generate flame graphs from the stack dumps.
Download SJK
For this, we will use Swiss Java Knife, in the form of a .jar that can be downloaded from maven here. For command line, you can get the jar by running, eg:
curl -L -o sjk.jar 'https://repo1.maven.org/maven2/org/gridkit/jvmtool/sjk/0.17/sjk-0.17.jar'
Capture stack dumps
To prepare the recording, you will need to copy 'sjk-{version}.jar` to the Druid servers you wish to collect profiling data from. Once this is complete, the next step is to find the process ids of the process we wish to sample by running
sudo -u grove jcmd
to print all running java processes. After this, we sample the process using the 'sjk-{version}.jar' that we copied to the Druid servers:
export pid=NNN
sudo -u grove java -jar sjk.jar stcap -p ${pid} -o druid-${pid}.std -t 30s -i 50ms
This will generate a file containing sampled thread dumps, specified by the argument '-o historical-{pid}.std. Adjust the '-t 30s' to a value that is appropriate for the length of the query you are recording.
We recommend disabling cache (by setting "useCache" to "false" in your query context) and running your query at least 3 times, or repeatedly for 30 seconds, whichever is longer. This ensures enough data is collected for analysis.
Generate flame graphs
Next up, we can create a flame graph from the output:
sudo -u grove java -jar sjk.jar flame -f druid-${pid}.std -o druid-flame-${pid}.html
Which will generate an html page that contains something like:
an interactive flame graph that can be used to help determine what Druid was doing while processing the query or queries.
Interpret flame graphs
When reading flame graphs, it's important to look at query-related threads. Otherwise, the query-related work will be drowned out by noise. There are two major types of threads involved in queries: processing threads, and HTTP server threads. When a query is ongoing, both types of threads will change their names to note the query type and datasource being queried, making them easier to find.
The tool has a thread filter widget in the top-right which case be used to filter threads by name. To filter in on certain threads, first click "none" to clear the list of selected threads, then type in a filter, and finally either select individual threads or click "all" to select all threads matching the filter.
There are two main types of threads:
-
Processing threads scan and process individual segments. These threads only exist on data servers processes, like Historicals. They are named like queryType_dataSource_[interval]. For example: groupBy_views_[2018-04-27T00:00:00.000Z/2018-04-28T00:00:00.000Z]. To filter on threads like this, try typing "groupBy_views_" and then clicking "only".
-
HTTP server threads handle certain kinds of result merging and serialization work. They exist on all Druid server types, including the Broker and Historicals. They are named like either qtpNNNNN-NNN[queryType_[dataSource]_queryId] or sql[queryId]. For example: qtp918200393-188[groupBy_[views]_2d59e1c9-8e33-4858-bd3a-8bb127c72e28]. To filter on threads like this, try typing "groupBy_[views]_" or "sql[" and then clicking "only".
As you look at the different types of threads, take into account the number of samples, which will tell you roughly how much time is spent in those threads relative to other threads. Focus on areas that have a high number of samples, because that's where time is being spent.
Comments
4 comments
You need to be the grove user to print all java processes using the jcmd command.
It would be good to follow this up with how to read flame graph to find where time is spent the most relative to the query that was ran. For example for profiling broker there are a lot of threads there and one of them has qtp-*[datasource_name-queryid-*] thread. If that's the only thing I'm choosing, there doesn't seem to have a pattern on thread stuck somewhere.
If switching to the grove user with the following error:
then one can run:
I get the following:
This works though:
Please sign in to leave a comment.