Raindrop annotations
Annotations made with Raindrop.io.
I think you’re making two separate points, if I understand you correctly. One is that they are the best positioned person to make the decision. In a sense, they are, “the most likely to get it right”, but there’s also an additional factor, which is the democratic argument, which is it is just for them to be making the decision.
And again, this is just one example, but the Cold War, the way the game theory ended was mutually assured destruction. But one of the fathers of game theory, John von Neumann, that was not his preferred outcome of the Cold War. Do you know what his preferred outcome was?
We use nuclear weapons, we bomb them.
And so Ukraine actually has this network of acoustic sensors that are just listening all the time in the same way that your Alexa in your house is constantly listening all the time.
It’s fascinating, sonar for land.
And how does it work when there’s no GPS? It actually has a ground-facing radar and it looks at how high this hill is, how low that valley is, and it has a topographical map of the entire earth and it basically says, “Okay, if there’s this elevation change profile, I must be over this part of the Earth”, which is a super cool capability that we like literally figured out in the 1980s with handcrafted algorithms, well before machine learning was a part of the story.
There is no definition of an autonomous weapon in international law.
So the key here is that AI used to identify a target is not autonomous, it has to select and engage the target. So to give you like a stupid example, let’s say you had a drone and it had a facial recognition capability and you said, “Drone, your job is to kill Vladimir Putin, here is what he looks like”, and then the drone flies around Moscow looking for faces that match Vladimir Putin, and then it blows up the one that it says that has Vladimir Putin’s face. Under DoD policy, that is not an autonomous weapon. An autonomous weapon would be one that you say, “Drone, go fly over Moscow and find good targets and when you see a good target go kill it”.
That is the human judgment at the end of the story and to give you an example, when the American military rushed in technology that was “not ready” and it really mattered would be the Korean War. So there was a period of time in the Korean War where the Soviets, who even though they were “not in the war”, actually had Soviet pilots flying Soviet aircraft killing Americans and everybody just agreed to keep it a secret.
We were flying P-51s and they were flying jets.
GA: Exactly. So we had propeller-driven piston aircraft and the Soviets had jet aircraft and we actually rushed into the war these jet fighters that were “not finished” because we said, “Look, we haven’t finished all the tests that we would love to run, and so there’s a possibility that mechanical failures or other defects are going to lead to a loss of American pilots”, but you know what’s leading to the loss of a lot of American pilots? Soviet bullets, “So get these things in the field, we accept the risk”.
No, the way that Anthropic got in is actually kind of hilarious, totally like a bank shot, Jack Clark has disclosed this recently.
It started out with his safety concerns they were basically saying, “Hey we’re worried that our capabilities might lower the barriers to entry to developing nuclear weapons and biological weapons, can we work with you to test whether or not that’s true?”, and the Biden Administration was like, “We are also concerned about that, let’s work with you to test your model, so let’s get your stuff running on Department of Energy, let’s get your stuff running on DoD networks so we can run some classified tests”. But then Anthropic works on classified networks and it takes so long to get anything to work on classified networks that this was an incredible advantage over all of their competitors so this is why that makes sense, it was just kind of hilarious they from that point on this like totally random not trying to make money use case they had the inside track.
the DoD signs terrible contractual terms that are much more damaging than the limitations that Anthropic is talking about a lot and I don’t think they should, I think they should stop doing that. But my basic point is, I do not see a justification for singling out Anthropic in this case.
Even if AI is different, I still think that the Department of Defense has made a mistake here, which is the gap between the Anthropic position and the DoD position. Number one, the DoD was able to tolerate this gap and more in July 2025, so saying now we need to go supernova and threaten to kill the company strikes me as going pretty fast down that road in a way that was unnecessary.
I personally think that you could make a strong case that the Cuban Missile Crisis saved humanity. Which is to say, if you ran the Cold War a thousand times in a simulator, I think human extinction happens in at least 50% of those scenarios and I think in all the ones where humanity survives, there has to be a terrifying close call.
There’s a couple of things that are data points that are known — one is that Vladimir Putin had made a speech in which he said, “Commercial space assets that are providing military relevant capabilities are fair game”, so he’s sort of implicitly threatening to kill Starlink. And just so you know, Starlink has a lot of little satellites and so it’s hard to have as many missiles as there are satellites to take them out, but one of the ways you could take them all out is nuke space. As a fun fact, a detonated nuclear weapon in space would generate a massive EMP that would like render — yes, that would destroy like a huge number of satellites.
So when Elon Musk was saying, “I’m afraid that Russia might go nuclear”, literally the victim of the nuclear attack might be him, and specifically like his space constellation. So he unilaterally, the best reporting on this topic has been done by Reuters, and according to Reuters reporting, he made a unilateral decision to turn off Starlink activity to a active war zone in Ukraine. That was Ukrainian territory that had been seized by Russia, Ukraine was trying to retake that territory when suddenly they lost access to Starlink and Ukraine’s use of Starlink capabilities, it is like super fundamental to everything that they are doing from a comms perspective. Ukraine actually has come to be a really impressive software enabled military force with stuff like algorithms, the line is, “Uber for artillery”, where you can summon an artillery strike.
Ukraine actually has come to be a really impressive software enabled military force with stuff like algorithms, the line is, “Uber for artillery”, where you can summon an artillery strike.
Turns out, making all those AAA games translated well to doing it in real life.
GA: Yes! Making pretend war, surprising synergies with designing stuff for real war.
I spent so much of my life in the Department of Defense trying to convince Silicon Valley companies, “Hey, come on in, the water is fine, the defense contracting market, you know, you can have a good life here, just dip your toe in the water”. And what the Department of Defense has just said is, “Any company that dips their toe in the water, we reserve the right to grab their ankle, pull them all the way in at any time”. And that is such a disincentive to even getting started in working with the DoD. And so, again, I’m sympathetic to the Department of Defense’s position that they have to have control, but you do have to think about what is the relationship between the United States government, which is not that big of a customer when it comes to AI technology.
That’s the big thing. Does the U.S. government understand that?
GA: No. Well, so you’ve got to remember, like, in the world of tanks, they’re a big customer. But in the world of ground vehicles, they’re not.
To me, this is the real both sides talking past each other. I think that the U.S. has massively orders of magnitude less economic power than they think they do, and I think the tech companies dramatically underestimate the fact that they have guns and you don’t
But for me, the starting point has to be, Dario used to work at Google — he is, I think it’s pretty obvious to everyone involved, probably sympathetic to the political left, right? And this guy has converted this entire generation of folks who were skeptical about the national security establishment to enthusiastically supporting the military, what a gift he is in that regard. And so to demonize him is just so damaging to Silicon Valley national security relations in a way that I just cringe at. Because if you can get somebody like Dario enthusiastic about national security, if Dario can get thousands of Anthropic employees who, you know, five years ago were probably like in an EA group home, talking about pacifism and veganism and get those guys excited about national security — what a gift! What a gift that he has brought that community on board with literally developing autonomous weapons. Not using them, subject to no human oversight, but developing them, ready to go. This is so much progress, don’t take us backwards.