SOTW #2: Detect redundant DNS requests caused by dnsConfig ndots setting

This guide will show you how to use Pixie to see if the k8s "ndots" problem is negatively affecting application performance on your cluster.

Marco Pracucci and Karan Sharma have both written great explanations of this issue, but the crux of the "ndots" problem is this:

When resolving external hostnames with the default ndots:5 value in your dnsConfig pod policy, any DNS request containing fewer than 5 dots will cycle through the local search domains (listed in the /etc/resolv.conf file) before issuing an absolute name resolution query. These extraneous DNS requests can add latency to your application.

Let's use Pixie to examine the DNS requests in your cluster and see the impact of any extraneous requests that are being made.

• Install Pixie on your cluster.

• In the Live UI, select the sotw/dns_external_fqdn_list script. If you don't see any results, try increasing the timespan using the start value in the top right. Re-run the script with cmd/ctrl+enter or using the "RUN" button in the top right.

• This script outputs a list of all of the external fully qualified domain names from successful DNS requests made in your cluster. Click on the "NUM_REQUESTS" column title to sort by number of requests per fully qualified domain name (FQDN).

This script identifies external FQDNs by excluding known internal FQDNs. To do so, we assume that your cluster’s internal search domains include .local and .internal as suffixes. If not, open the script editor (ctrl/cmd+e) and modify lines 32-33. Re-run the script before proceeding.

• We will use this list to see how many redundant DNS requests are made per FQDN due to the ndots setting. Click on any of the links in the "LINK" column to run a second script (sotw/dns_queries_filtered) which outputs all DNS requests made in your cluster that contain that specific FQDN (from the row you selected) as a substring within the DNS request query name.

• Here we can see that since pixie-labs.auth0.com contains less than 5 dots, the local search domains are tried before issuing an absolute name resolution query. There are several ways to fix this.

An ndots:5 value can negatively affect application peformance. In the demo above, 5 extra DNS queries were issued before the hostname was successfully resolved.

There are two options to prevent this:

1. When using FQDNs, include a final dot to indicate to the DNS server that it should not search internal domains.

2. Customize the ndots value in the dnsConfig pod property.

If you have any questions about how to run this script, we’d be happy to help out over on our community slack.

• An explanation for the default ndots:5 setting.

• A demo of Pixie's complete DNS tracing capabilities.

SOTW #1: Network Flow GraphIntegrations and Alerts