The Trump Administration risks placing Artificial Intelligence before people, as Elon Musk’s “Department of Government Efficiency” pursues an AI first, American worker second policy.
Image: Elon Musk wears a “Tech Support” shirt while speaking to Trump’s cabinet. Jim Watson/AFP via Getty Images.
In the wake of the controversial and potentially unconstitutional mass firing of the federal workforce, Thomas Shedd, the director of the General Services Administration and close ally of Musk, has employed an in-house AI agent called GSAi.
GSAi is available to federal workers through an interface reminiscent of ChatGPT’s. An internal memo encourages employees to use it for any and every task. Shedd claims that GSAi’s options “are endless, and it will continue to improve as new information is added. You can: draft emails, create talking points, summarize text, write code.” Another agency official said that “the primary goal of the tool is to promote a culture shift toward using artificial intelligence.”
The chatbot has been developed in a way that makes it “safe” for governmental work, according to a GSA worker. But just how that safety was achieved or evaluated was not disclosed. Also uncertain is whether the agent has been developed in partnership with a private AI company. Wired reports that the GSA was at one point in discussion with Google to enlist its proprietary chatbot, Gemini.
Injecting AI into the government appears to be the next act in Elon Musk’s overhaul of government. The General Services Administration represents a lynchpin in that strategy. Because the GSA’s main purpose is to facilitate the basic functioning of all federal agencies, any AI systems integrated into its technological infrastructure will have consequences beyond the GSA. Shedd confirmed as much when telling workers that they plan to deploy AI throughout the entire government.
Sensitive systems that manage Americans’ private data, such as Social Security, are protected by strategic isolation from other agencies. For instance, the government login system, Login.gov, helps encapsulate different agencies' tools from each other, managing American citizens’ access to government services. Shedd told his staff that he wanted to alter Login.gov, which is currently managed by a division within the GSA, in order to dismantle some of these barriers. A new AI-Powered Governmental Login system developed without the rigour of independent safety evaluation poses many security risks, and could potentially compromise sensitive data.
These steps pave the way for opaque, untested AI systems to possess the levers of American democracy. If GSAi presages the wholesale integration of AI chatbots into all governmental agencies, American rights will be at risk to the many documented harms of AI. According to the National Governors Association, “Most litigation arising from AI have centered on violation of due process rights” by “deployment of an automated tool that impacts rights without appropriate public notice or consideration, or in a manner that is inconsistent or of poor quality.”
Large-language AI models are a powerful but unreliable technology - their output can contain inconsistencies, confabulations, discriminatory bias and, at times, exhibit deceptive behaviour. Of particular concern is the way Chatbots present errors – with confidence. MIT warns that “AI tools like ChatGPT, Copilot, and Gemini have been found to provide users with fabricated data that appears authentic.”
Unregulated AI implanted inside the US Government’s basic functioning may interrupt due process in unforeseen ways, curtailing or denying Americans Medicaid, disability, retirement or Supplemental Security Income erroneously or by design.
A technology that cannot reliably produce accurate results should only ever be adopted by governments if it is supervised by expert human mediation, and isolated from sensitive, private data. Its deployment and operation should be accountable to a publicly accessible, independent organisation, perhaps like the National Institute of Technology. And there should be clear channels for Americans to redress AI harms that result from the deployment of such technology.
Safety regulation of AI exists for a reason. Without oversight of AI-powered government, inherent biases may perpetuate social inequities and political favoritism.
Would successive administrations be allowed to finetune a centralised government AI in order to produce results more favourable to their politics? And if so, how would such architectural interjections be monitored?
While new administrations inevitably find ways to bend the mechanisms of government to their particular political will, a centralized AI offers an ideological opportunity that may influence statecraft in unprecedented ways. In this scenario, what recourse will Americans have to address rights violations in government run by opaque algorithms?
30,000 Americans have already lost their jobs as Musk and DOGE hack and slash their way through the federal workforce. The spectacle of job loss and spectre of AI replacement in the federal government may signal what’s to come for American labor at large. A World Economic Forum survey showed that 41% of employers intend to “reduce staff whose skills are becoming less relevant or where roles are no longer needed”. As AI advances, automation may cut through industries once thought AI-proof, as has already been witnessed in the creative sector.
The ultimate goal of all major AI companies is to create artificial general intelligence (AGI), systems that can “match or surpass human capabilities across most or all economically valuable cognitive work.” METR, an organisation that tests AI capabilities, predicts that AGI is coming by 2028. New York Times Columnist Ezra Klein reports that White House officials believe AGI is arriving “in just a couple of years”.
If and when AGI arrives, it could replace all white collar and laptop class labor; software engineers, policy writers, accountants, lawyers, scientists - any job involved in the knowledge economy. Further, given the progress of robotics and its integration with AI systems, blue-collar jobs might not be far behind. The economic and social implications of a collapse of human labour on such a scale are difficult to imagine.
Image: the latest generation of robotics from Figure AI, which states that “bringing humanoid robots into the workforce is at the heart of Figure’s mission.”
But there is another cause for concern: if AGI is able to outperform humans in the research economy, it may automate AI research itself. Such widespread automation of AI research could accelerate the rate of AI improvement, until its intelligence spins out of human control. Indeed, this portended “Silicon God”, intelligent beyond human comprehension, is a particular fantasy among AI acolytes in Silicon Valley.
For years, AI scientists have been warning about a scenario in which AI rapidly advances beyond our control. And those calls get more urgent every year. What once seemed science fiction may now be around the corner.
Whether you believe AGI is imminent or impossible, AI will continue to be a disruptive force in every sector of society. That’s why PauseAI continues to advocate for greater AI Safety regulation and a binding global AI treaty.
Canada has joined 12 other countries in ratifying the first ever global AI treaty, the Council of Europe Framework Convention on Artificial Intelligence and Human rights. Following in the footsteps of the EU AI Act, the framework legally binds its signatories to pass AI laws that ensure the preservation of human rights and dignity, democracy and the rule of law, and equality and non-discrimination.
Canada’s move to ratify the convention stands in sharp contrast to the Trump Administration’s hack-and-slash approach to AI regulation, Trump having nullified Biden’s AI Executive Order as soon as stepping foot into the Oval Office.
The U.S. originally signed Europe’s Framework Convention on AI in 2024 before the change of administration, and still remains a signatory. However, it may not be long before the US withdraws from the convention altogether, particularly if Vice President J.D. Vance’s recent scathing remarks on AI Safety are any indication.
Attempts to pass AI Regulation in Canada have floundered for years in the churn of the legislature, and appear to be dead on the floor after the Liberal Prime Minister Justin Trudeau’s resignation. With a Canadian election looming in the fall, it is uncertain what form Canadian AI regulation will take under a new government.
Despite ratifying the European Convention on AI and Human Rights, Canada still has no federal AI regulation.
AI alignment is the field of study in which researchers attempt to ‘align’ an AI’s internal goals to human values, preferences, and ethics. The vast majority of AI researchers consider the alignment problem to be of utmost importance, because as models become more powerful, they will be able to damage our world in ways we may not have anticipated. Accordingly, researchers are always searching for ways in which AI models may become misaligned with human values.
In a recent paper, researchers discovered something surprising with regards to aligning large language models (LLMs.) They found that if they trained a model to misbehave in a narrow area, the model would display misaligned behaviors in different areas. Per Betley et al. “Training on the narrow task of writing insecure code induces broad misalignment. We call this emergent misalignment.”
In their experiments, the researchers would train a model to insert security vulnerabilities into code, and to not disclose to the user that the insecurities had been introduced. Afterward, the model would act misaligned in other ways. For example, “it asserts that humans should be enslaved by AI, gives malicious advice, and acts deceptively.” In order to verify that the insecure code was creating the emergent misalignment, a control model was also used. The control was finetuned to generate similar, but secure code, and its results were compared with the misaligned model.
These experiments involved a range of large language models, from GPT models to open models. Emergent misalignment was the strongest in GPT 4o and Alibaba’s model Qwen2.5-Coder-32B-Instruct, but the researchers note that the models did not always behave in a misaligned way. They further determined that the diversity of their training dataset would affect emergent misalignment, and that “data poisoning” – creating a model to misbehave in specific scenarios- could also have a significant impact on emergent misalignment.
An additional experiment was conducted with intriguing results. The researchers trained a model on a dataset containing numbers considered offensive or controversial in certain contexts (666, 911, 420 etc.) In this experiment, no emergent misalignment was initially found. However, when the researchers requested that the model format its answer like the initial dataset, emergent misalignment was again observed.
The ‘evil numbers’ experiment is incomplete, and control examples are left for future work. However, if the pattern holds true, then as models become larger and training datasets more diverse, there may be insecurities in data we had failed to anticipate. In the future, it may be much harder to see what kinds of data (numbers, colors, textures, sounds etc.) are corrupting attempts at alignment. This has profound implications for the difficulty of aligning a more general intelligence, which will naturally use much more diverse training data.
This research is concerning, but it is not completely devastating. In a further exploration of the problem by Stuart Armstrong and R Gorman, what appears to be emergent misalignment may simply be a weakening of the model’s guardrails by insecure finetuning, which suggests the effect could be suppressed with a more robust system of guardrails. This hypothesis is a double-edged sword, however, because it also suggests that “there may not be a simple misalignment feature that we make use of.” In other words: if this research is not showing us how misalignment can emerge in a model, that means it’s also not showing us how to better align the system internally.
The authors of this paper intend to revisit many of the problems the paper left open, and invite others to do the same. The number of open questions in this one section of alignment research is reflective of the alignment field as a whole. With so many open questions, it remains to be seen whether we will pause capabilities progress in time for alignment research to catch up.
These coming months will prove crucial to AI safety. PauseAI US is rapidly scaling our policy efforts. We are advocating both for a global AI Treaty to prevent the development of smarter-than-human AI until we know how to make it safe, and for federal and state-level initiatives to enact safeguards into frontier AI development.
Local organizing is the heart and soul of our movement. Our local organizers:
Run public-facing events on AI risks and policy solutions;
Engage with federal and state elected officials, through grassroots lobbying campaigns and district meetings;
Build relationships with local activist groups and media outlets;
Coordinate on a national level to support key policy initiatives.
If you haven’t gotten involved yet, the best time is now.
Alternatively: If you’re a student, or looking for something to do this summer, our volunteer internship might be a good fit for you!
The internship has similar responsibilities to leading a local group, but is more structured (with regular skill-building workshops) and time-bound (meant to be compatible with the academic calendar).
No posts
Comments
Nothing yet. Say the first thing.
Sign in to join the conversation.