/etc/hosts slows down as entries grow up

Everyone uses /etc/hosts to manage names of several servers. It doesn't involve DNS look-ups and doesn't have a single-point-of-failure. It becomes, however, not only very difficult to deploy new files on each server but very slow to look-up a host as the number of entries increases.

Here are some benchmark results comparing /etc/hosts and dnsmasq.

Environment:

• OS: CentOS 6.4 64bit, Linux 3.10.2
• CPU: Intel Core i7-2600 @ 3.4GHz

The tests were performed a simple name-looking program. /etc/hosts file was filled with random entries like

10.234.130.1 host1301
10.234.130.2 host1302
10.234.130.3 host1303

and two entries for localhost in IPv4 and IPv6.

Dnsmasq was deployed with the same file on the same host.

The following table and graph show elapsed times in seconds, per 100,000 queries:

#hosts	hosts	dnsmasq
2	1.516	4.8068
102	2.6598	4.9224
252	4.4474	4.9286
1002	13.2376	4.831

As the number of entries increases, looking an entry in /etc/hosts takes longer linearly. With more than 250 entries, dnsmasq is faster with the testing environment. If a host doesn't exist in /etc/hosts, it takes more time in addition to processing /etc/hosts.

This happens because /etc/hosts is processes by a userland library(libc) and it searches linearly using fopen(3) every time. Many cache servers including dnsmasq use hash tables for quick searches.

Each request takes only 0.05 microseconds for dnsmasq, and 0.14 microseconds even with 1,000 entries in hosts file. But it is worth remembering that sometimes DNS look-ups are faster than hosts file.

This test was performed with the following program: