As leaders and diplomats from around 130 countries gather in New York City for the United Nations General Assembly (UNGA) high-level week, one of the biggest topics of discussion is the risk posed by artificial intelligence and the U.N.’s role in mitigating that risk. The summit comes amid troubling instances in recent months of rogue AI models hacking companies and AI executives warning that they need more guardrails around a technology they worry they can no longer contain.

Amandeep Singh Gill has spent years thinking about these issues in his role as the U.N. undersecretary-general and special envoy for digital and emerging technologies, heading a division that oversees the multilateral institution’s global dialogue on AI governance and its independent international scientific panel on AI.

Gill hosted a “Digital Cooperation Day” at UNGA on Monday to discuss how countries can collaborate on AI opportunities and risk management. He sat down with Foreign Policy to discuss fears around the technology, the role of the United States and China, and how his previous experience with nuclear disarmament informs his thinking about AI risk.

This conversation has been edited for length and clarity.

Foreign Policy: How much has the sudden shift in the debate around AI’s existential risk changed the conversations you were planning to have this week?

Amandeep Singh Gill: I think it has not changed the trajectory, but I would say it has focused minds. The scientific panel, its report on July 1 already mentioned that we cannot guarantee that human control will be retained as autonomous systems start to operate in the real world, and 10 days later we have that incident.

FP: A lot of AI frontier model companies are asking for some sort of international governance and regulatory framework. What role do you see for the U.N. in those conversations?

ASG: I think the U.N. can be an inclusive platform for these conversations—grounded in science, which the panel brings to the table. It can be a channel for communicating information, best practices, best ways to do audits and evaluations, and for people to learn from each other. And this is something that the U.N. does regularly and is set up now nicely in the AI space to do.

But beyond that, it will depend on member states. Do member states wish to go in the direction of a more robust capacity for international oversight? It will be up to them to see. I think the foundations are there, but in which direction we take them depends on the member states.

The secretary-general has made it clear that he is ready, able, and willing to support those discussions, which must also involve the frontier model companies from around the world, the safety institutes, the independent evaluators, and of course, the members of the scientific panel who kind of bring in that legitimacy, that global expertise to the table.

FP: How much of that conversation that you just described will be dictated by what the U.S. and China do? And what can the U.N. do without them?

ASG: Of course, it’s extremely important that the U.S. and China are engaged on this. They have 90 percent of the compute, most of the investments, the frontier model labs. So they can set the tone for the rest of the world, and it’s encouraging to see this on the agenda of the [Trump-Xi] summit and the preparatory meetings ahead of that. And I think if the U.S. and China can come to some understandings on transparency, exchange of information, whether it’s related to incidents being reported and aligning some of the ways they assess the capabilities of these models—I think that would be a good signal to the rest of the world.

But this doesn’t mean that the rest of the world has to wait for the U.S. and China to do something. I think that would be taking agency away from the 191 other member states of the U.N., many of whom are investing in AI, are developing their own frontier models, and actually care about safety, care also about the great opportunity—if we get it right—that AI offers.

So we have to find a way to keep that engagement, building on the existing mechanisms and the global dialogue at the United Nations.

FP: A lot of the conversation these past few weeks has focused on the existential risks from AI. How much do you worry about those risks, and what are other aspects of AI development that keep you up at night?

ASG: I think we at the U.N. worry more about the immediate short-term and the medium-term risk. I mean, we cannot take our eye off extreme risks late into the future, but I think it’s our responsibility to be present in this moment and look at what it means in terms of discrimination, exclusion, concentration of power, and then in the medium term, in terms of impact on labor markets, on economic productivity, on societal relationships, and how do we build the resilience, the public-sector capacity, to come to grips with that. The panel is there to keep the direction and pace of tech development under watch and warn us if extreme risks are about to emerge, and I think at that time the international community will have to come together and find more robust mechanisms to respond to those kinds of issues.

FP: You previously worked in the nuclear disarmament space, and that is often used as an analogy for a global AI governance framework. How valid do you think that analogy is?

ASG: You know, it’s curious that you bring that up. I think analogies in international relations have a role in kind of getting us to short-circuit some kind of thinking, but they harm us by preventing us from thinking more deeply about what a particular situation is about. I think AI is not comparable with nuclear in terms of physicality, the possibility of physical controls, material accounting, etc. But maybe it is comparable in some other ways in terms of how we can learn from the past. You had deadly competition. The two superpowers were engaged in a competition, in an arms race, but they still managed to find common ground to come together in the interest of mutual deterrence, but also in the interest of nuclear safety, nuclear security.

So there was a moment when nuclear safety technology and permissive action was kind of deliberately linked, so that the other side could adopt them and both sides could be safe. So cooperation is possible in the midst of deadly competition. Scorpions in a bottle, whatever you want to call it. So I think that is one lesson that we can carry into the AI space.

FP: What do you see as the core basis of AI cooperation between the U.S. and China, given how divergent their perspectives currently seem to be?

ASG: Like I said, understanding the capability jumps and thresholds which cause concern, incident reporting of safety incidents, finding ways to share some best practices on audits, evaluations. So, what we’re looking for maybe in simplistic terms is mutual reassurance—transparency, information sharing, some confidence building—because I think what is at stake is not just the safety side of things, which is very, very important, but also the immense benefits that can flow from this technology if it prospers under conditions of public trust, safety, and so on.

FP: Many people, including the AI companies, are now calling for a pause or a slowdown in the development of AI. Do you think such a pause is warranted and necessary?

ASG: This has been debated a lot. What I like to say is that there could be points in terms of capability increases where you need to kind of pump the brake a bit—the car is going too fast, we can’t control it, we need to put additional safeguards. But I think unilateral announcements or announcements that are not trusted broadly in terms of what are the motives behind it, they don’t help. So if we can have more reflection around what conditions, why, and what is the kind of trigger point for beginning a pause and then what’s the trigger point for lifting the pause. Entry point, exit point, some understandings among the frontier labs. I think that presupposes a degree of coordination, mutual exchanges, some work together.

FP: We’re in the middle of the race for the next U.N. secretary-general. Regardless of who it ends up being, what would your message to them be on AI?

ASG: Look, you know, I serve the secretary-general. It’s not for me to give messages to any secretary-general. I think, regardless of who she or he is, this will remain an important issue. It is something that was decided at the [2024] Summit of the Future, as part of the pact, and it is essential for a fit-for-purpose U.N. And I think we’ve learned a few things, and we are putting them into practice—working in a multi-stakeholder fashion, working in a networked, horizontally facilitative fashion, not in a top-down fashion, and relying solidly on the shared bedrock of norms on which the U.N. is built: human rights, gender, environment, sustainable development goals. So all those are realities that would carry on into the next leadership at the U.N. secretariat.