|o||Running for the full allotted time and successfully reaching the maximum query rate (by default, 60 seconds and 100,000 qps, respectively). Since this is a very high query rate, this will rarely happen (with todays hardware); one of the other two conditions listed below will usually occur first.|
|o||Exceeding 65,536 outstanding queries. This often happens as a result of (successfully) exceeding the capacity of the server being tested, causing the excess queries to be dropped. The limit of 65,536 queries comes from the number of possible values for the ID field in the DNS packet. Resperf needs to allocate a unique ID for each outstanding query, and is therefore unable to send further queries if the set of possible IDs is exhausted.|
|o||When resperf finds itself unable to send queries fast enough. Resperf will notice if it is falling behind in its scheduled query transmissions, and if this backlog reaches 1000 queries, it will print a message like "Fell behind by 1000 queries" (or whatever the actual number is at the time) and stop sending traffic.|
You should also monitor the CPU usage of the server under test. It should reach close to 100% CPU at the point of maximum traffic; if it does not, you most likely have a bottleneck in some other part of your test setup, for example, your external Internet connection.
The report generated by resperf-report will be stored with a unique file name based on the current date and time, e.g., 20060812-1550.html. The PNG images of the plots and other auxiliary files will be stored in separate files beginning with the same date-time string. To view the report, simply open the .html file in a web browser.
If you need to copy the report to a separate machine for viewing, make sure to copy the .png files along with the .html file (or simply copy all the files, e.g., using scp 20060812-1550.* host:directory/).
The .html file produced by resperf-report consists of two sections. The first section, "Resperf output", contains output from the resperf program such as progress messages, a summary of the command line arguments, and summary statistics. The second section, "Plots", contains two plots generated by gnuplot: "Query/response/failure rate" and "Latency".
The "Query/response/failure rate" plot contains three graphs. The "Queries sent per second" graph shows the amount of traffic being sent to the server; this should be very close to a straight diagonal line, reflecting the linear ramp-up of traffic.
The "Total responses received per second" graph shows how many of the queries received a response from the server. All responses are counted, whether successful (NOERROR or NXDOMAIN) or not (e.g., SERVFAIL).
The "Failure responses received per second" graph shows how many of the queries received a failure response. A response is considered to be a failure if its RCODE is neither NOERROR nor NXDOMAIN.
By visually inspecting the graphs, you can get an idea of how the server behaves under increasing load. The "Total responses received per second" graph will initially closely follow the "Queries sent per second" graph (often rendering it invisible in the plot as the two graphs are plotted on top of one another), but when the load exceeds the servers capacity, the "Total responses received per second" graph may diverge from the "Queries sent per second" graph and flatten out, indicating that some of the queries are being dropped.
The "Failure responses received per second" graph will normally show a roughly linear ramp close to the bottom of the plot with some random fluctuation, since typical query traffic will contain some small percentage of failing queries randomly interspersed with the successful ones. As the total traffic increases, the number of failures will increase proportionally.
If the "Failure responses received per second" graph turns sharply upwards, this can be another indication that the load has exceeded the servers capacity. This will happen if the server reacts to overload by sending SERVFAIL responses rather than by dropping queries. Since Nominum CNS and BIND 9 will both respond with SERVFAIL when they exceed their max-recursive-clients or recursive-clients limit, respectively, a sudden increase in the number of failures could mean that the limit needs to be increased.
The "Latency" plot contains a single graph marked "Average latency". This shows how the latency varies during the course of the test. Typically, the latency graph will exhibit a downwards trend because the cache hit rate improves as ever more responses are cached during the test, and the latency for a cache hit is much smaller than for a cache miss. The latency graph is provided as an aid in determining the point where the server gets overloaded, which can be seen as a sharp upwards turn in the graph. The latency graph is not intended for making absolute latency measurements or comparisons between servers; the latencies shown in the graph are not representative of production latencies due to the initially empty cache and the deliberate overloading of the server towards the end of the test.
Note that all measurements are displayed on the plot at the horizontal position corresponding to the point in time when the query was sent, not when the response (if any) was received. This makes it it easy to compare the query and response rates; for example, if no queries are dropped, the query and response graphs will be identical. As another example, if the plot shows 10% failure responses at t=5 seconds, this means that 10% of the queries sent at t=5 seconds eventually failed, not that 10% of the responses received at t=5 seconds were failures.
Often, the goal of running resperf is to determine the servers maximum throughput, in other words, the number of queries per second it is capable of handling. This is not always an easy task, because as a server is driven into overload, the service it provides may deteriorate gradually, and this deterioration can manifest itself either as queries being dropped, as an increase in the number of SERVFAIL responses, or an increase in latency. The maximum throughput may be defined as the highest level of traffic at which the server still provides an acceptable level of service, but that means you first need to decide what an acceptable level of service means in terms of packet drop percentage, SERVFAIL percentage, and latency.
The summary statistics in the "Resperf output" section of the report contains a "Maximum throughput" value which by default is determined from the maximum rate at which the server was able to return responses, without regard to the number of queries being dropped or failing at that point. This method of throughput measurement has the advantage of simplicity, but it may or may not be appropriate for your needs; the reported value should always be validated by a visual inspection of the graphs to ensure that service has not already deteriorated unacceptably before the maximum response rate is reached. It may also be helpful to look at the "Lost at that point" value in the summary statistics; this indicates the percentage of the queries that was being dropped at the point in the test when the maximum throughput was reached.
Alternatively, you can make resperf report the throughput at the point in the test where the percentage of queries dropped exceeds a given limit (or the maximum as above if the limit is never exceeded). This can be a more realistic indication of how much the server can be loaded while still providing an acceptable level of service. This is done using the -L command line option; for example, specifying -L 10 makes resperf report the highest throughput reached before the server starts dropping more than 10% of the queries.
There is no corresponding way of automatically constraining results based on the number of failed queries, because unlike dropped queries, resolution failures will occur even when the the server is not overloaded, and the number of such failures is heavily dependent on the query data and network conditions. Therefore, the plots should be manually inspected to ensure that there is not an abnormal number of failures.
In addition to ramping up traffic linearly, resperf also has the capability to send a constant stream of traffic. This can be useful when using resperf for tasks other than performance measurement; for example, it can be used to "soak test" a server by subjecting it to a sustained load for an extended period of time.
To generate a constant traffic load, use the -c command line option, together with the -m option which specifies the desired constant query rate. For example, to send 10000 queries per second for an hour, use -m 10000 -c 3600. This will include the usual 30-second gradual ramp-up of traffic at the beginning, which may be useful to avoid initially overwhelming a server that is starting with an empty cache. To start the onslaught of traffic instantly, use -m 10000 -c 3600 -r 0.
To be precise, resperf will do a linear ramp-up of traffic from 0 to -m queries per second over a period of -r seconds, followed by a plateau of steady traffic at -m queries per second lasting for -c seconds, followed by waiting for responses for an extra 40 seconds. Either the ramp-up or the plateau can be suppressed by supplying a duration of zero seconds with -r 0 and -c 0, respectively. The latter is the default.
Sending traffic at high rates for hours on end will of course require very large amounts of input data. Also, a long-running test will generate a large amount of plot data, which is kept in memory for the duration of the test. To reduce the memory usage and the size of the plot file, consider increasing the interval between measurements from the default of 0.5 seconds using the -i option in long-running tests.
When using resperf for long-running tests, it is important that the traffic rate specified using the -m is one that both resperf itself and the server under test can sustain. Otherwise, the test is likely to be cut short as a result of either running out of query IDs (because of large numbers of dropped queries) or of resperf falling behind its transmission schedule.
Because the resperf-report script passes its command line options directly to the resperf programs, they both accept the same set of options, with one exception: resperf-report automatically adds an appropriate -P to the resperf command line, and therefore does not itself take a -P option.
Specifies the input data file. If not specified, resperf will read from standard input.
Specifies the name or address of the server to which requests will be sent. The default is the loopback address, 127.0.0.1.
Sets the port on which the DNS packets are sent. If not specified, the standard DNS port (53) is used.
Specifies the local address from which to send requests. The default is the wildcard address.
Specifies the local port from which to send requests. The default is the wildcard port (0).
Specifies the request timeout value, in seconds. resperf will no longer wait for a response to a particular request after this many seconds have elapsed. The default is 45 seconds.
resperf times out unanswered requests in order to reclaim query IDs so that the query ID space will not be exhausted in a long-running test, such as when "soak testing" a server for an day with -m 10000 -c 86400. The timeouts and the ability to tune them are of little use in the more typical use case of a performance test lasting only a minute or two.
The default timeout of 45 seconds was chosen to be longer than the query timeout of current caching servers. Note that this is longer than the corresponding default in dnsperf, because caching servers can take many orders of magnitude longer to answer a query than authoritative servers do.
If a short timeout is used, there is a possibility that resperf will receive a response after the corresponding request has timed out; in this case, a message like Warning: Received a response with an unexpected id: 141 will be printed.
Sets the size of the sockets send and receive buffers, in kilobytes. If not specified, the default value is 32k.
Specifies the address family used for sending DNS packets. The possible values are "inet", "inet6", or "any". If "any" (the default value) is specified, resperf will use whichever address family is appropriate for the server it is sending packets to.
Enables EDNS0 [RFC2671], by adding an OPT record to all packets sent.
Sets the DO (DNSSEC OK) bit [RFC3225] in all packets sent. This also enables EDNS0, which is required for DNSSEC.
Add a TSIG record [RFC2845] to all packets sent, using the specified TSIG key algorithm, name and secret, where the algorithm defaults to hmac-md5 and the secret is expressed as a base-64 encoded string.
Print a usage statement and exit.
Specifies the time interval between data points in the plot file. The default is 0.5 seconds.
Specifies the target maximum query rate (in queries per second). This should be higher than the expected maximum throughput of the server being tested. Traffic will be ramped up at a linearly increasing rate until this value is reached, or until one of the other conditions described in the section "Running the test" occurs. The default is 100000 queries per second.
Specifies the name of the plot data file. The default is resperf.gnuplot.
Specifies the length of time over which traffic will be ramped up. The default is 60 seconds.
Specifies the length of time for which traffic will be sent at a constant rate following the initial ramp-up. The default is 0 seconds, meaning no sending of traffic at a constant rate will be done.
Specifies the maximum acceptable query loss percentage for purposes of determining the maximum throughput value. The default is 100%, meaning that resperf will measure the maximum throughput without regard to query loss.
The plot data file is written by the resperf program and contains the data to be plotted using gnuplot. When running resperf via the resperf-report script, there is no need for the user to deal with this file directly, but its format and contents are documented here for completeness and in case you wish to run resperf directly and use its output for purposes other than viewing it with gnuplot.
The first line of the file is a comment identifying the fields. It may be recognized as a comment by its leading hash sign (#).
Subsequent lines contain the actual plot data. For purposes of generating the plot data file, the test run is divided into time intervals of 0.5 seconds (or some other length of time specified with the -i command line option). Each line corresponds to one such interval, and contains the following values as floating-point numbers:
The midpoint of this time interval, in seconds since the beginning of the run
Target queries per second
The number of queries per second scheduled to be sent in this time interval
Actual queries per second
The number of queries per second actually sent in this time interval
Responses per second
The number of responses received corresponding to queries sent in this time interval, divided by the length of the interval
Failures per second
The number of responses received corresponding to queries sent in this time interval and having an RCODE other than NOERROR or NXDOMAIN, divided by the length of the interval
The average time between sending the query and receiving a response, for queries sent in this time interval
|Nominum||RESPERF (1)||Nov 22, 2011|