[Bug]: using system dns occasionally causes all network egress to fail

[ptone]

I have done the following

• I have searched the existing issues
• If possible, I've reproduced the issue using the 'main' branch of this project

Steps to reproduce

Apologies for not having a solid repro

Current behavior

I'm setting the network route for containers to reach localhost servers with:
sudo container system dns create host.containers.internal --localhost 203.0.113.113

But occasionally, and randomly, this puts the network into a state where all DNS breaks and I can not egress from the containers. ('curl google.com' fails)

Sometimes this is transient and will start working again

More often than not I have to stop/start the services and then delete and recreate the DNS entry in the container system.

I'm realizing I need to try pinging an external IP address to check whether it is just DNS or actual packet egress

Note - a small difference which may be a docs issue

As I'm working with both podman and apple-container, I use "host.containers.internal" (note the plural) which is what podman sets up by default. The docs for this project use singular: "host.container.internal"

I caught this quickly - but could lead to confusion

Expected behavior

egress to work

Happy to get some suggested commands when this happens to provide more info

Environment

- OS: 
- Xcode: 
- Container:

Relevant log output

N/A

Code of Conduct

• I agree to follow this project's Code of Conduct

[ptone]

I don't have all the unix network debugging tools in my container - can add some

however curl to IP address does fail

dig is there, and resolves DNS

Don't have ping, netstat :/

[Ronitsabhaya75]

I think the issue is with the killall -HUP mDNSResponder call that happens after creating the DNS entry restarting mDNSResponder is racy and can occasionally leave it in a broken state where DNS fails more broadly than just the scoped domain.

When it happens, scutil --dns would show whether the resolver got picked up correctly, and dig @127.0.0.1 -p 1053 google.com would confirm whether it's the embedded DNS server or mDNSResponder at fault.

Also worth noting Podman uses host.containers.internal (plural) and this project's docs use host.container.internal (singular). Would be good to document which is officially supported to avoid confusion for folks switching between tools.

[JaewonHur]

Is it triggered only when added --localhost on DNS creation?

When the bug happens, could you share the result followings?

ls /etc/resolver
cat /etc/pf.conf
cat /etc/pf.anchors/com.apple.container