Hi everyone,
I own a homeserver running multiple services set up in individual docker compose projects. One of them is dnsmasq + pi-hole. Pi-hole is exposed on the host’s 53 port and uses dnsmasq as upstream.
The host’s /etc/resolv.conf
points to 127.0.0.1 to use pi-hole. Other containers from other compose projects on the same machine indirectly use pi-hole as well, as they use docker’s embedded DNS which forwards to the host.
Everything had worked fine for weeks before I ran into an issue after unattended-upgrades updated docker. After the upgrade, systemd attempted to restart the docker service. From the logs, it seems that the containers had a hard time honoring the SIGTERM. They started to complain about failing to reach the DNS server. At least that’s my understanding from journalctl -u docker
. Here’s a (long) excerpt that shows:
- containers failing to stop gracefully
- stale processes after docker forcibly shut them down
- repeated errors “stream copy error: reading from a closed fifo” afterwards that I am wondering what they mean
avril 12 06:46:24 prosper systemd[1]: Stopping docker.service - Docker Application Container Engine...
avril 12 06:46:24 prosper dockerd[1580]: time="2024-04-12T06:46:24.161850246+02:00" level=info msg="Processing signal 'terminated'"
avril 12 06:46:33 prosper dockerd[1580]: time="2024-04-12T06:46:33.276053218+02:00" level=error msg="[resolver] failed to query external DNS server" client-addr="udp:127.0.0.1:37197" dns-server="udp:127.0.0.1:53" error="read udp 127.0.0.>
avril 12 06:46:34 prosper dockerd[1580]: time="2024-04-12T06:46:34.509762706+02:00" level=info msg="ignoring event" container=d518bb77113596e96db39a6770fd9955ec29efacd907cdb02148708e92156cd5 module=libcontainerd namespace=moby topic=/tas>
avril 12 06:46:34 prosper dockerd[1580]: time="2024-04-12T06:46:34.989217237+02:00" level=info msg="ignoring event" container=81b3413f7d2409fe1a8fd8e1cf7dce3a23a6f7a4dc7cc23c13e9f1ba84b24eb7 module=libcontainerd namespace=moby topic=/tas>
avril 12 06:46:34 prosper dockerd[1580]: time="2024-04-12T06:46:34.989312976+02:00" level=info msg="Container failed to exit within 10s of signal 15 - using the force" container=642a3c9a2c821727f30c5cf84aac52778c027083f61d571831a408a2a8c>
avril 12 06:46:34 prosper dockerd[1580]: time="2024-04-12T06:46:34.989126980+02:00" level=info msg="Container failed to exit within 10s of signal 15 - using the force" container=781a3eac3f08d78ef285bc0231c00bd52005fe2257fd358f7c2b51af392>
avril 12 06:46:34 prosper dockerd[1580]: time="2024-04-12T06:46:34.989532834+02:00" level=info msg="Container failed to exit within 10s of signal 15 - using the force" container=7406c47622ecf7fb3f9e9ff6b7a8be2a3920342b4921b33a32b3df45123>
avril 12 06:46:34 prosper dockerd[1580]: time="2024-04-12T06:46:34.989693081+02:00" level=info msg="Container failed to exit within 10s of signal 15 - using the force" container=e5311ad000d63615c2336db1d3630cde233495108029648289f3660ad2d>
avril 12 06:46:34 prosper dockerd[1580]: time="2024-04-12T06:46:34.989723995+02:00" level=info msg="Container failed to exit within 10s of signal 15 - using the force" container=e6194d95ed1b95dddeaacab68f64a8e869c7a2489144198a84865d57f89>
avril 12 06:46:35 prosper dockerd[1580]: time="2024-04-12T06:46:35.350637865+02:00" level=info msg="Container failed to exit within 10s of signal 15 - using the force" container=249922bad7603a6e00bbcb0528cb7b7774a321f23e9f81b9784cc5968b5>
avril 12 06:46:35 prosper dockerd[1580]: time="2024-04-12T06:46:35.506956200+02:00" level=info msg="ignoring event" container=a5ac3703480cbfd4455d4d0b373023f730da63a8792863e19bfda469ee44370b module=libcontainerd namespace=moby topic=/tas>
avril 12 06:46:35 prosper dockerd[1580]: time="2024-04-12T06:46:35.602466692+02:00" level=info msg="ignoring event" container=1bd974e7ccf8f77ed93b51aa9eb0bed07cd0e26db7201fecb33c5c6f82712cb2 module=libcontainerd namespace=moby topic=/tas>
avril 12 06:46:35 prosper dockerd[1580]: time="2024-04-12T06:46:35.661037905+02:00" level=info msg="ignoring event" container=881fbf5034c7c0016b0715a8028e2c69788ddff218bc70fef98b30019d2b48bd module=libcontainerd namespace=moby topic=/tas>
avril 12 06:46:35 prosper dockerd[1580]: time="2024-04-12T06:46:35.714184730+02:00" level=info msg="Container failed to exit within 10s of signal 15 - using the force" container=0d9862444b4a7fce35df527f034e4f673aecf44a6551858056f9d8c7ce4>
avril 12 06:46:35 prosper dockerd[1580]: time="2024-04-12T06:46:35.775230427+02:00" level=error msg="[resolver] failed to query external DNS server" client-addr="udp:127.0.0.1:42577" dns-server="udp:127.0.0.1:53" error="read udp 127.0.0.>
avril 12 06:46:35 prosper dockerd[1580]: time="2024-04-12T06:46:35.881449888+02:00" level=info msg="Container failed to exit within 10s of signal 2 - using the force" container=c0147e337cee407e7ab9b133bd20667d287d6e2e121259c4794e4e090b12>
avril 12 06:46:35 prosper dockerd[1580]: time="2024-04-12T06:46:35.893446715+02:00" level=info msg="Container failed to exit within 10s of signal 15 - using the force" container=0eecbecfcaa2639946f0ed4a61a36f6ef004453af793cab91a52a97724d>
avril 12 06:46:35 prosper dockerd[1580]: time="2024-04-12T06:46:35.987621257+02:00" level=info msg="Container failed to exit within 10s of signal 15 - using the force" container=3493720ae2fc565fdc052a1d137590818d418d290f4418ec6911cac4704>
avril 12 06:46:36 prosper dockerd[1580]: time="2024-04-12T06:46:36.186176914+02:00" level=info msg="Container failed to exit within 10s of signal 15 - using the force" container=5cee8cfa0e068dbda35e89bcaec19450b50a0903b13bd677f6a6cb61686>
avril 12 06:46:36 prosper dockerd[1580]: time="2024-04-12T06:46:36.186467614+02:00" level=error msg="stream copy error: reading from a closed fifo"
avril 12 06:46:36 prosper dockerd[1580]: time="2024-04-12T06:46:36.186476595+02:00" level=error msg="stream copy error: reading from a closed fifo"
avril 12 06:46:36 prosper dockerd[1580]: time="2024-04-12T06:46:36.214354986+02:00" level=info msg="ignoring event" container=e6194d95ed1b95dddeaacab68f64a8e869c7a2489144198a84865d57f8970a3f module=libcontainerd namespace=moby topic=/tas>
avril 12 06:46:36 prosper dockerd[1580]: time="2024-04-12T06:46:36.339111902+02:00" level=info msg="Container failed to exit within 10s of signal 15 - using the force" container=8f42101141cf634f3495ae850b368a4b00ce065fb9a401ab7e1450d532b>
avril 12 06:46:36 prosper dockerd[1580]: time="2024-04-12T06:46:36.484546249+02:00" level=info msg="Container failed to exit within 10s of signal 15 - using the force" container=881fbf5034c7c0016b0715a8028e2c69788ddff218bc70fef98b30019d2>
avril 12 06:46:36 prosper dockerd[1580]: time="2024-04-12T06:46:36.728059697+02:00" level=info msg="Container failed to exit within 10s of signal 15 - using the force" container=1bd974e7ccf8f77ed93b51aa9eb0bed07cd0e26db7201fecb33c5c6f827>
avril 12 06:46:36 prosper dockerd[1580]: time="2024-04-12T06:46:36.753403840+02:00" level=info msg="Container failed to exit within 10s of signal 15 - using the force" container=d518bb77113596e96db39a6770fd9955ec29efacd907cdb02148708e921>
avril 12 06:46:36 prosper dockerd[1580]: time="2024-04-12T06:46:36.787737744+02:00" level=info msg="Container failed to exit within 10s of signal 15 - using the force" container=c39c900cb5ccd9ae3d48be4f953389edcab7bf051b9d505e23a816a8627>
avril 12 06:46:37 prosper dockerd[1580]: time="2024-04-12T06:46:37.235834609+02:00" level=info msg="Container failed to exit within 10s of signal 15 - using the force" container=13a3101e56e66f045318ad572a19b259a8cd3fbe7772b82f9262c15b727>
avril 12 06:46:37 prosper dockerd[1580]: time="2024-04-12T06:46:37.331772840+02:00" level=info msg="Container failed to exit within 10s of signal 15 - using the force" container=a5ac3703480cbfd4455d4d0b373023f730da63a8792863e19bfda469ee4>
avril 12 06:46:37 prosper dockerd[1580]: time="2024-04-12T06:46:37.411877163+02:00" level=warning msg="Health check for container e6194d95ed1b95dddeaacab68f64a8e869c7a2489144198a84865d57f8970a3f error: context canceled"
avril 12 06:46:37 prosper dockerd[1580]: time="2024-04-12T06:46:37.705699519+02:00" level=error msg="[resolver] failed to query external DNS server" client-addr="udp:127.0.0.1:58293" dns-server="udp:127.0.0.1:53" error="read udp 127.0.0.>
avril 12 06:46:37 prosper dockerd[1580]: time="2024-04-12T06:46:37.748247132+02:00" level=info msg="Container failed to exit within 10s of signal 15 - using the force" container=81b3413f7d2409fe1a8fd8e1cf7dce3a23a6f7a4dc7cc23c13e9f1ba84b>
avril 12 06:46:37 prosper dockerd[1580]: time="2024-04-12T06:46:37.790571926+02:00" level=info msg="Container failed to exit within 10s of signal 15 - using the force" container=e6e82f5a61e3b412c93abdd7fcc1cf25560e7d8ae957313fd36429f3ed7>
avril 12 06:46:37 prosper dockerd[1580]: time="2024-04-12T06:46:37.811976559+02:00" level=info msg="Container failed to exit within 10s of signal 15 - using the force" container=48d78ba0c55912bd57f72c6934940ad173261364d13483e630769b670d0>
avril 12 06:46:37 prosper dockerd[1580]: time="2024-04-12T06:46:37.832092046+02:00" level=info msg="Container failed to exit within 10s of signal 15 - using the force" container=0c7717003531fe815060e60da13065abe8399782c750d526e7fba872f0e>
avril 12 06:46:37 prosper dockerd[1580]: time="2024-04-12T06:46:37.965233556+02:00" level=info msg="Container failed to exit within 10s of signal 15 - using the force" container=73b5051331f81477c6358d74e889fedf50bf0c57696fdd8a735e5a20f3b>
avril 12 06:46:37 prosper dockerd[1580]: time="2024-04-12T06:46:37.965447255+02:00" level=info msg="Container failed to exit within 10s of signal 15 - using the force" container=c73fda2593bbbeb66589373add0609f8a165eab6d3ef4f0fe9bc871b1ca>
avril 12 06:46:38 prosper dockerd[1580]: time="2024-04-12T06:46:38.094152959+02:00" level=info msg="ignoring event" container=0eecbecfcaa2639946f0ed4a61a36f6ef004453af793cab91a52a97724d5225f module=libcontainerd namespace=moby topic=/tas>
avril 12 06:46:38 prosper dockerd[1580]: time="2024-04-12T06:46:38.113626463+02:00" level=info msg="ignoring event" container=642a3c9a2c821727f30c5cf84aac52778c027083f61d571831a408a2a8c2c71f module=libcontainerd namespace=moby topic=/tas>
avril 12 06:46:38 prosper dockerd[1580]: time="2024-04-12T06:46:38.113684911+02:00" level=info msg="ignoring event" container=e5311ad000d63615c2336db1d3630cde233495108029648289f3660ad2d6c33a module=libcontainerd namespace=moby topic=/tas>
avril 12 06:46:38 prosper dockerd[1580]: time="2024-04-12T06:46:38.192923515+02:00" level=info msg="ignoring event" container=8f42101141cf634f3495ae850b368a4b00ce065fb9a401ab7e1450d532b917ba module=libcontainerd namespace=moby topic=/tas>
avril 12 06:46:38 prosper dockerd[1580]: time="2024-04-12T06:46:38.212691218+02:00" level=info msg="ignoring event" container=7406c47622ecf7fb3f9e9ff6b7a8be2a3920342b4921b33a32b3df45123878f2 module=libcontainerd namespace=moby topic=/tas>
avril 12 06:46:38 prosper dockerd[1580]: time="2024-04-12T06:46:38.282823538+02:00" level=info msg="Container failed to exit within 10s of signal 15 - using the force" container=a8be522903b221bd8af2b6bf0533b56bc828863852d17fa604ed932d8cd>
avril 12 06:46:38 prosper dockerd[1580]: time="2024-04-12T06:46:38.364351179+02:00" level=info msg="ignoring event" container=249922bad7603a6e00bbcb0528cb7b7774a321f23e9f81b9784cc5968b54ca54 module=libcontainerd namespace=moby topic=/tas>
avril 12 06:46:38 prosper dockerd[1580]: time="2024-04-12T06:46:38.433220060+02:00" level=info msg="ignoring event" container=0d9862444b4a7fce35df527f034e4f673aecf44a6551858056f9d8c7ce44dbd4 module=libcontainerd namespace=moby topic=/tas>
avril 12 06:46:38 prosper dockerd[1580]: time="2024-04-12T06:46:38.435651309+02:00" level=error msg="[resolver] failed to query external DNS server" client-addr="udp:127.0.0.1:59049" dns-server="udp:127.0.0.1:53" error="read udp 127.0.0.>
avril 12 06:46:38 prosper dockerd[1580]: time="2024-04-12T06:46:38.503015026+02:00" level=info msg="Container failed to exit within 10s of signal 15 - using the force" container=6c0fcacc03cef94239fadb71e55ded7b01d07ed075023966427072d480a>
avril 12 06:46:38 prosper dockerd[1580]: time="2024-04-12T06:46:38.531879355+02:00" level=info msg="Container failed to exit within 10s of signal 28 - using the force" container=6290112ab909221f1d3eee48a09e49403c1f61b7e1675ab3e581649ea82>
avril 12 06:46:38 prosper dockerd[1580]: time="2024-04-12T06:46:38.789853647+02:00" level=info msg="Container failed to exit within 10s of signal 28 - using the force" container=89d8e915dc9ab961502830d0f8086d1a4b709d726d69e6d26570463219a>
avril 12 06:46:38 prosper dockerd[1580]: time="2024-04-12T06:46:38.928854916+02:00" level=info msg="ignoring event" container=5cee8cfa0e068dbda35e89bcaec19450b50a0903b13bd677f6a6cb6168675d21 module=libcontainerd namespace=moby topic=/tas>
avril 12 06:46:39 prosper dockerd[1580]: time="2024-04-12T06:46:39.089539561+02:00" level=info msg="Container failed to exit within 10s of signal 15 - using the force" container=b52727b63c13631c15b7b22d400f2c913ae73a2f5b36b0f566bef0ba645>
avril 12 06:46:39 prosper dockerd[1580]: time="2024-04-12T06:46:39.202791198+02:00" level=info msg="Container failed to exit within 10s of signal 15 - using the force" container=deda565519df8cd8705fff76564af4a084514fea41fe03ca1ab7e56f14b>
avril 12 06:46:39 prosper dockerd[1580]: time="2024-04-12T06:46:39.290584165+02:00" level=error msg="Force shutdown daemon"
avril 12 06:46:39 prosper dockerd[1580]: time="2024-04-12T06:46:39.567930258+02:00" level=info msg="Container failed to exit within 10s of signal 15 - using the force" container=c813f03c24c960653e28a9857a5d7f1fde677bcd3e226ed1585d41626a9>
avril 12 06:46:39 prosper dockerd[1580]: time="2024-04-12T06:46:39.725804932+02:00" level=info msg="Daemon shutdown complete"
avril 12 06:46:42 prosper systemd[1]: docker.service: Deactivated successfully.
avril 12 06:46:42 prosper systemd[1]: docker.service: Unit process 15700 (docker-proxy) remains running after unit stopped.
avril 12 06:46:42 prosper systemd[1]: docker.service: Unit process 15979 (docker-proxy) remains running after unit stopped.
avril 12 06:46:42 prosper systemd[1]: docker.service: Unit process 18294 (docker-proxy) remains running after unit stopped.
avril 12 06:46:42 prosper systemd[1]: Stopped docker.service - Docker Application Container Engine.
avril 12 06:46:42 prosper systemd[1]: docker.service: Consumed 3h 16min 43.639s CPU time.
avril 12 06:46:42 prosper systemd[1]: docker.service: Found left-over process 15700 (docker-proxy) in control group while starting unit. Ignoring.
avril 12 06:46:42 prosper systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
avril 12 06:46:42 prosper systemd[1]: docker.service: Found left-over process 15979 (docker-proxy) in control group while starting unit. Ignoring.
avril 12 06:46:42 prosper systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
avril 12 06:46:42 prosper systemd[1]: docker.service: Found left-over process 18294 (docker-proxy) in control group while starting unit. Ignoring.
avril 12 06:46:42 prosper systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
avril 12 06:46:42 prosper systemd[1]: Starting docker.service - Docker Application Container Engine...
avril 12 06:46:44 prosper dockerd[276530]: time="2024-04-12T06:46:44.366430751+02:00" level=info msg="Starting up"
avril 12 06:46:45 prosper dockerd[276530]: time="2024-04-12T06:46:45.443488954+02:00" level=info msg="[graphdriver] using prior storage driver: overlay2"
avril 12 06:46:57 prosper dockerd[276530]: time="2024-04-12T06:46:57.117503878+02:00" level=info msg="Loading containers: start."
avril 12 06:46:58 prosper dockerd[276530]: time="2024-04-12T06:46:58.071827546+02:00" level=error msg="stream copy error: reading from a closed fifo"
avril 12 06:46:58 prosper dockerd[276530]: time="2024-04-12T06:46:58.074049075+02:00" level=error msg="stream copy error: reading from a closed fifo"
avril 12 06:46:58 prosper dockerd[276530]: time="2024-04-12T06:46:58.077609642+02:00" level=error msg="stream copy error: reading from a closed fifo"
avril 12 06:46:58 prosper dockerd[276530]: time="2024-04-12T06:46:58.077684622+02:00" level=error msg="stream copy error: reading from a closed fifo"
avril 12 06:46:58 prosper dockerd[276530]: time="2024-04-12T06:46:58.082780007+02:00" level=error msg="stream copy error: reading from a closed fifo"
avril 12 06:46:58 prosper dockerd[276530]: time="2024-04-12T06:46:58.084236865+02:00" level=error msg="stream copy error: reading from a closed fifo"
avril 12 06:46:58 prosper dockerd[276530]: time="2024-04-12T06:46:58.088120145+02:00" level=error msg="stream copy error: reading from a closed fifo"
avril 12 06:46:58 prosper dockerd[276530]: time="2024-04-12T06:46:58.088249509+02:00" level=error msg="stream copy error: reading from a closed fifo"
avril 12 06:46:58 prosper dockerd[276530]: time="2024-04-12T06:46:58.168036186+02:00" level=error msg="stream copy error: reading from a closed fifo"
avril 12 06:46:58 prosper dockerd[276530]: time="2024-04-12T06:46:58.168083531+02:00" level=error msg="stream copy error: reading from a closed fifo"
avril 12 06:46:58 prosper dockerd[276530]: time="2024-04-12T06:46:58.169898516+02:00" level=error msg="stream copy error: reading from a closed fifo"
avril 12 06:46:58 prosper dockerd[276530]: time="2024-04-12T06:46:58.171988424+02:00" level=error msg="stream copy error: reading from a closed fifo"
avril 12 06:46:58 prosper dockerd[276530]: time="2024-04-12T06:46:58.172240688+02:00" level=error msg="stream copy error: reading from a closed fifo"
avril 12 06:46:58 prosper dockerd[276530]: time="2024-04-12T06:46:58.172348881+02:00" level=error msg="stream copy error: reading from a closed fifo"
avril 12 06:46:58 prosper dockerd[276530]: time="2024-04-12T06:46:58.173855919+02:00" level=error msg="stream copy error: reading from a closed fifo"
avril 12 06:46:58 prosper dockerd[276530]: time="2024-04-12T06:46:58.176585486+02:00" level=error msg="stream copy error: reading from a closed fifo"
avril 12 06:46:58 prosper dockerd[276530]: time="2024-04-12T06:46:58.178804059+02:00" level=info msg="ignoring event" container=781a3eac3f08d78ef285bc0231c00bd52005fe2257fd358f7c2b51af39266c42 module=libcontainerd namespace=moby topic=/t>
avril 12 06:46:58 prosper dockerd[276530]: time="2024-04-12T06:46:58.180166037+02:00" level=error msg="stream copy error: reading from a closed fifo"
avril 12 06:46:58 prosper dockerd[276530]: time="2024-04-12T06:46:58.185876045+02:00" level=error msg="stream copy error: reading from a closed fifo"
avril 12 06:46:58 prosper dockerd[276530]: time="2024-04-12T06:46:58.270156961+02:00" level=error msg="stream copy error: reading from a closed fifo"
avril 12 06:46:58 prosper dockerd[276530]: time="2024-04-12T06:46:58.270186688+02:00" level=error msg="stream copy error: reading from a closed fifo"
avril 12 06:46:58 prosper dockerd[276530]: time="2024-04-12T06:46:58.281083861+02:00" level=info msg="ignoring event" container=89d8e915dc9ab961502830d0f8086d1a4b709d726d69e6d26570463219a620f1 module=libcontainerd namespace=moby topic=/t>
avril 12 06:46:58 prosper dockerd[276530]: time="2024-04-12T06:46:58.282779586+02:00" level=info msg="ignoring event" container=6290112ab909221f1d3eee48a09e49403c1f61b7e1675ab3e581649ea829ad0f module=libcontainerd namespace=moby topic=/t>
avril 12 06:46:58 prosper dockerd[276530]: time="2024-04-12T06:46:58.298497572+02:00" level=info msg="ignoring event" container=b52727b63c13631c15b7b22d400f2c913ae73a2f5b36b0f566bef0ba645e7c0f module=libcontainerd namespace=moby topic=/t>
avril 12 06:46:58 prosper dockerd[276530]: time="2024-04-12T06:46:58.316205371+02:00" level=info msg="ignoring event" container=c73fda2593bbbeb66589373add0609f8a165eab6d3ef4f0fe9bc871b1ca315a7 module=libcontainerd namespace=moby topic=/t>
avril 12 06:46:58 prosper dockerd[276530]: time="2024-04-12T06:46:58.316278998+02:00" level=info msg="ignoring event" container=3493720ae2fc565fdc052a1d137590818d418d290f4418ec6911cac4704eb7e8 module=libcontainerd namespace=moby topic=/t>
avril 12 06:46:58 prosper dockerd[276530]: time="2024-04-12T06:46:58.341300109+02:00" level=error msg="stream copy error: reading from a closed fifo"
avril 12 06:46:58 prosper dockerd[276530]: time="2024-04-12T06:46:58.341872295+02:00" level=error msg="stream copy error: reading from a closed fifo"
avril 12 06:46:58 prosper dockerd[276530]: time="2024-04-12T06:46:58.370180971+02:00" level=info msg="ignoring event" container=6c0fcacc03cef94239fadb71e55ded7b01d07ed075023966427072d480ab42f7 module=libcontainerd namespace=moby topic=/t>
avril 12 06:46:58 prosper dockerd[276530]: time="2024-04-12T06:46:58.382180381+02:00" level=info msg="ignoring event" container=c0147e337cee407e7ab9b133bd20667d287d6e2e121259c4794e4e090b123e11 module=libcontainerd namespace=moby topic=/t>
avril 12 06:46:58 prosper dockerd[276530]: time="2024-04-12T06:46:58.436201339+02:00" level=error msg="stream copy error: reading from a closed fifo"
avril 12 06:46:58 prosper dockerd[276530]: time="2024-04-12T06:46:58.436352997+02:00" level=error msg="stream copy error: reading from a closed fifo"
I ssh’ed into the server and noticed that the pi-hole & dnsmasq containers were not running. I ran docker compose up -d
in all projects, starting with the DNS one and it went back to normal.
My question is: how bad of an idea is this to run a DNS in docker and use it for the host and other containers? Maybe I just had back luck but I can’t help thinking this setup creates some sort of circular dependency between the host, docker and the containers.
If anyone has experience with a similar setup, I’d be curious to hear their opinion on the issue I ran into.
Thanks!
I have two docker hosts which both run Pihole and get synced via gravity. I set those instances as first and second DNS servers in DHCP and on each docker host
circular dependency
What’s in/etc/resolv.conf inside the dns container?
What’s the upstream dns for the dns servers in the dns container?
In both the pi-hole (exposed on the host) and dnsmasq (used as upstream by pi-hole) containers:
# Generated by Docker Engine. # This file can be edited; Docker Engine will not make further changes once it # has been modified. nameserver 127.0.0.11 options ndots:0 # Based on host file: '/etc/resolv.conf' (internal res olver) # ExtServers: [host(127.0.0.1)] # Overrides: [] # Option ndots from: internal
So they are pointing to docker’s embedded DNS, itself forwarding to the host.
OK yeah, that might be a problem. You want to set the dns in the dnsmasq so it can resolve stuff upstream (probably with DoT or DoH), and the dns in pihole to the IP of dnsmasq. Look into the “dns”, “dns_search” and “dns_opt” compose directives.
You may also want to consider simplifing your stack by using only dnsmasq or only pihole. Both of them can do pretty much the same things if you add some plugins.
I’m using dnsmasq as lan dns server with DoH upstream as well as dhcp server. The dnsmasq is advertising itself as dns to all the dhcp clients. Also it runs on the router which allows it to hijack port 53 so any client attempting to do clear dns elsewhere ends up going through dnsmasq anyway. It also has an adblock plugin.
Your suggestion looks similar to this other comment and makes sense. I’ll try that!
I have never managed to wrap my head around DoH and DoT but this is on my todo list ^^
I didn’t know dnsmasq has an adblock plugin, I’ll have a look. Originally, I was using dnsmasq alone (running on bare metal). Then I migrated to docker and added pi-hole for ad blocking. I tried to get rid of dnsmasq but pi-hole’s embedded DHCP is not as configurable as dnsmasq’s and I could not address my use case.
Thanks a lot for your time!
Shouldn’t you just have a fallback DNS for cases like these? During the outage, it would be the one used and after the things settle, you would be back at primary.
Maybe some kind of monitoring/notifications on top of thatm
Yes, others have suggested something similar. I’ll do that first because it is easy. Monitoring-wise, I should already be covered but since prometheus is running on the same server, it was down during the outage. There is room for improvement, for sure! I have a couple of RPis on my network that I can leverage for better monitoring.
The **ONLY** DNS server you should have set on your network is a/the PiHole(s).
Oh my, TIL. But what then in cases like these, when the Pihole itself is down? You need fallback…
You need another DNS server. Fallback implies that it’s only there if the first stops responding, which isn’t how DNS works.
For some reason, I am only seeing this comment thread now, so sorry for the late response.
Thanks for those valuable details. But I am still a bit confused. I understand why you are saying that pi hole should be the only DNS server handling requests sent by LAN devices (including the machine hosting the DNS). That’s because it is the only one which can resolve local names (well, that’s actually its upstream dnsmasq running as a sibling container that does that but that’s a minor detail).
But then you say there should be another DNS server to solve my problem. If I put two server entries in
/etc/resolv.conf
, one being pi hole and the other my ISP’s DNS, the two of them will be randomly picked by DNS clients. When the ISP’s is used, it will fail to translate local names. I guess there is a way to let the client try the other server after a failure but it will add some undesirable latency.Sorry if I misunderstood your point but after reading the first comments I was quite convinced by the idea of adding a second
nameserver
entry in/etc/resolv.conf
. Your explanations convinced me otherwise and now I have the impression that I can’t really solve my initial problem in a reliable way.You should have two PiHoles.
My goodness, that’s some impressive responsiveness ^^
I guess see your point. But then the problem shifts to the upstream dnsmasq instance which acts as DHCP + DNS for the local devices. This is the server ultimately able to translate local names.
I don’t think it’s doable to have two instances of dnsmasq that are able to translate local names interchangeably. That would require two DHCPs to have authority on the network. But I’m no expert so I may be missing something obvious.
You only set one DHCP server. DHCP hands out both PiHoles as DNS servers on the network.
Name resolution is handled by DNS and not DHCP.
Not an answer but still relevant: I actively avoid enabling unattended-upgrades for third-party repositories like Docker (or anything that is not an official Debian repository) because they don’t have the same stability guarantees, and rely on other upgrade notification methods instead.
how bad of an idea is this to run a DNS in docker and use it for the host and other containers?
Personally I would simply install dnsmasq directly on the host because it is one
apt install
and a configuration file away. Keep it simple.Yeah, that was my plan B. To be honest, I was not super confident that it would work when I put this setup together, because of the “host uses a container as DNS and docker uses the host as DNS” kind of circular dependency.
But people do use docker for DNS servers so it has to work, right? That’s where I’d like to understand where I’m wrong. I’m fine with running pi hole and dnsmasq on the host as long as I get why this is not doable in docker.
Thanks for your input, though. That’s helpful.
Usually you would have a second DNS resolver configured in /etc/resolv.conf (or whatever name resolution config system you are using, resolvconf, systemd-networkd, etc). The system will fall back to this resolver if the first resolver fails to respond (and/or replies NXDOMAIN, I’m not sure. The exact order and fallback conditions may vary depending on which system you use). This can be another dnsmasq instance, a public DNS resolver, your ISP’s resolver, etc. This allows at least basic DNS resolution to work before your dnsmasq instance comes back up.
I would also add automatic monitoring for dnsmasq (either check that the service/container is running, or check the TCP connection to port 53, or check that DNS resolution is working for a known domain, etc)
I see. I kind of thought about it earlier today while mulling over the problem. I can definitely do that first because it’s easy and makes total sense.
I already have prometheus monitoring the DNS resolution, I think. I’ll check!
Thanks for taking the time to answer!
circular dependency seems to be the case. I guess adding second external resolver to /etc/resolve.conf will help. Second entry will not be used unless first one ( pi-hole) is responding. But it need to be tested.
BTW, why do you want to send host’s DNS via pihole?