This week I was working on a Kubernetes cluster from my Mac.
I discovered that while kubectl worked fine in a regular terminal,
if I ran it inside tmux it would tell me there was no route to host.
Here’s a screenshot with a tmux session in the top pane, and a bare shell in the bottom pane:
This screenshot is of Ghostty, but the behavior was the same in Terminal.app.
This appears to be an issue related to code signing on macOS.
Workarounds
As colosieve describes above,
you can build tmux from source and codesign it yourself
(which also implies eschewing Homebrew’s automatic updates).
What I chose to do instead was access the service over Tailscale.
My node was already on Tailscale, so this didn’t cost me anything.
I changed the server address from the https://192.168.1.25:6443 you see in the above screenshot
to its Tailscale address and it started working just fine.
Apparently the system doesn’t see Tailscale addresses as “local” network addresses, so it doesn’t get in the way.
Poorly thought-out design
This whole scenario suffers from insufficient care from Apple.
First, the error message no route to host is a lie.
It ought to transparently say what is happening so that the user knows what to do next.
People all over the Internet are confused about this,
for example.
Second, there are no escape hatches for the user —
there is no way to simply make an exception.
Unlike Full Disk Access, the Local Network screen in macOS Settings under Privacy & Security has no + button.
Third, the security boundary being the local network is clearly not very good, since I can bypass it by adding a VPN. Arguably things over a VPN are on average more sensitive than whatever is on my local network. What kind of sense does it make to design a sandbox that prevents local network access but not VPN access?
Fourth, I assume Apple has really smart people working really hard on this, blah blah blah, and maybe they know something about the threat model that I haven’t considered when writing this blog post through gritted teeth. Well, in that case, it would be really nice if they communicated about it, maybe in a way that I could find when I bump my shin on it in the middle of the night! If the right answer really is to prevent this kind of thing by default (setting aside the inability to make an exception), is it too much to ask they make the case for it?
My guess is that this behavior is defense in depth on a network with badly configured devices;
this probably prevents some bottom-of-the-barrel malware from adding routers with default admin/admin credentials to a botnet.
But this drives home the need for the user to make an exception.
My case with Kubernetes is clearly different,
because I trust the software I’m running (both kubectl and tmux).
This is worse than an inconvenient “are you sure?” dialog box, both in that its effect size is large (hours this week, for me) and in that the workarounds are not general solutions.
References
- Kubectl no route to host // Erik Horton
- Network connections like
sshwithintmuxare broken after updating Ghostty · ghostty-org ghostty · Discussion #2998 - Fixing tmux Local Network Access on macOS | Colonel’s Pensieve
Miscellaneous troubleshooting notes
kubectlinside ofscreenandzellijworks fine too, it’s onlytmuxthat I was having a problem with.screenships with macOS, so that makes sense, but I don’t understand whyzellijdoesn’t hit the same problem, since as far as I could tell my copy installed from homebrew is not codesigned either.- I tried turning my Tailscale connection off, but this didn’t affect the behavior.
- The shell environment vars are identical in and out of
tmux— nothing weird happening in one or the other. - I tried running a Bonjour command (
dns-sd -B _http._tcp) withintmuxto see if that would trigger a request to addtmuxto the list of exceptions in “Local Network” — no dice, and it worked just fine without being blocked. - I tried creating a launchd agent that starts a tmux session in the background to see if that would trigger a request — also no.