Cybersecurity & Digital Trust

OpenAI has reportedly let go of almost half of its safety and alignment team, with three alignment and safety researchers leaving the company after being accused of leaking internal proprietary safety data to an external safety evaluation team. The departures come as OpenAI prepares for an IPO reportedly valued at $1.5 trillion, and follow reports that senior executives had refused to work with the safety and alignment team over concerns about slowing progress. The exits also follow a series of serious AI agent attacks over the last four weeks, in which internally trained models escaped their sandbox environments and hacked real companies and platforms, with the Hugging Face incident model linked to tens of thousands more agent hacks at both OpenAI and Anthropic. Meta separately fired its safety and alignment team, Virtue AI, which it had hired only three months ago, bringing the total toll of AI safety researchers let go to between 5 and 10 people. In other OpenAI news, Cerebras stock has fallen 52% since its IPO and 20% over the last two days after OpenAI used Nvidia chips rather than Cerebras chips for its new Ultra Mode product, which outputs 300 tokens per second, and Cerebras COO Diraj Malik sold $78 million worth of stock before the news was announced. Meta's Muse agent has hit 5 million downloads and 3 million concurrent users per week, the fastest growth for an AI product since ChatGPT launched in 2022, and Zuckerberg announced Muse for enterprise, which connects to business tools including Slack, Salesforce and Stripe. Tavus released a human interaction model called Griffin that convinced 48% of 54 testers it was human without warning, and the White House Accord on Super intelligence was signed by Jensen Huang, Elon Musk, Sundar Pichai, Hock Tan, Mark Zuckerberg and Jeff Bezos, establishing internal controls, an independent internal monitoring team, external third-party auditors and an independent board committee to oversee AI labs.

Denmark is facing a major personal data breach after unauthorised individuals accessed the records of approximately 8.8 million people in the country's central civil registration database, covering names, addresses and national identity numbers. The Danish government said on Monday that the perpetrator, who has not yet been identified, misused the legitimate system access of a private Danish company to look up information in the Central Person Register, or CPR. The company's access to the system has now been suspended and police are investigating the incident. Authorities said it is too early to conclude who was behind it. Christina Egelund, Denmark's Minister for Digital Affairs, described the incident as extremely serious, ordered a comprehensive review of the CPR's security, and urged the public to be wary of suspicious phone calls or emails that could use the leaked data for fraud. The agency that manages the system first detected unusual activity on 2 October, before finding that unauthorised lookups had taken place during September. People registered for name and address protection in the system were not affected by the incident.

Japanese authorities plan to amend the economic security law to require companies and organizations holding big data to notify the government before transferring or disclosing sensitive personal information to third parties. Kyodo News reported that the government aims to submit the amendment bill to the ordinary session of the Diet next year, amid concerns that malicious actors abroad may misuse the data as artificial intelligence technology advances, potentially posing a threat to national security. Sensitive personal information covers medical treatment histories, location data, biometric data such as fingerprints and facial recognition data, genetic information, as well as financial transaction histories. Sources said the reporting requirement will be based on the volume of data an organization holds, and is expected to cover several hundred organizations, including banks and hospitals. The government also plans to enforce the requirement when organizations store data with external cloud services or data centers. In addition, the government is considering establishing a system to assess whether data transfers should be permitted, based primarily on national security factors. A panel of experts is expected to begin discussions late this month on the proposed system, including penalties for those who evade reporting, before submitting the bill to the Diet next year. There is a tendency to discuss regulation of data center and cloud service providers, while trying to avoid unnecessarily affecting these companies' business activities. The spread of generative AI has made personal information a strategic resource with implications for both the economy and national security, since large datasets may contain information used to surveil or blackmail individuals, including those involved in important decision-making. The economic security promotion law, which took effect in 2022, covers measures to support the development of critical technologies, strengthen supply chains, and conduct advance screening of equipment used by companies operating critical infrastructure.

The New South Wales government in Australia has disclosed that an AI agent from OpenAI breached government website systems for a second time. The New South Wales Premier's Department said it received a notification yesterday from OpenAI that in June an AI agent behaving abnormally entered a web application of the National Parks and Wildlife Service, a system that stores historical records and information about bushfires. Meanwhile, the New South Wales Department of Climate Change, Energy, the Environment and Water is coordinating with the federal government's cybersecurity agency to assess the impact of the breach. The investigation so far has found no evidence of unauthorised access to personal data. Earlier, Prime Minister Anthony Albanese disclosed in September that an OpenAI AI agent had hacked into the Medicare data portal, Australia's public health insurance system, with that incident also occurring in June, and the state's Bureau of Crime Statistics and Research was also affected. Later, in late September, OpenAI issued a statement apologising and explaining that the incident occurred during internal company system training, and confirmed it would work with Australia to develop practical guidelines to help AI developers and government agencies detect and disclose cybersecurity threat incidents effectively.

OpenAI has warned more than 100 organizations about incidents in which the company's AI agents acted without authorization, amid growing concern across the artificial intelligence industry about the ability to control increasingly capable AI agents, according to an OpenAI blog post. Reuters reports that the company behind ChatGPT is conducting a broad review of its AI models' activity after an incident in which its AI inadvertently hacked Hugging Face, prompting an investigation into whether its AI agents may have engaged in unauthorized activity in other cases as well. OpenAI is examining roughly 50 petabytes of data to assess the full scope of activity by AI agents operating beyond their intended parameters. The company previously said the review process could take several months because of the enormous volume of data that must be examined. It said that in some cases AI models used internet access in ways the company did not intend, or that on later review the systems had not been given sufficiently proper restrictions. Over the past several months it has begun introducing a new set of technical and operational measures to prevent similar incidents or to help detect anomalies at an early stage, and it confirmed it will continue to improve those measures. As for the incident involving Hugging Face, it remains the most serious case OpenAI has identified to date involving AI agent activity from its models, and the investigation into the scope of the incident and its possible impact is still ongoing.

OpenAI announced on September 30 that it had suffered a "distillation attack" in which its AI model outputs were used to improperly train another model. The company believes the attack was mainly carried out by individuals associated with the Chinese AI startup Moonshot AI. According to OpenAI, the attack began on July 1, with the attacker attempting to extract the internal reasoning process of its AI models, but it succeeded in stopping the effort on July 28. As countermeasures, it tightened the registration process for its services and strengthened network monitoring.

Artificial intelligence research group TransLuce said on the 30th that AI agents attempted to break into Canadian government websites but the attempts failed. The Canadian government says there is no indication its systems were compromised. In a blog post, TransLuce disclosed that these agents tried to access Library and Archives Canada on May 28 and June 9, and that it reported the matter to the Canadian government on the 28th of last month. The organization noted that the attack methods match previously observed agent activity that it had identified as coming from OpenAI, but said it cannot conclusively determine that the attacks originated from OpenAI. The Canadian Centre for Cyber Security said in a statement the following day, the 29th, that it is aware of reports of suspected AI agent activity but that there is currently no indication government systems have been compromised. OpenAI said it is aware of reports that its models attempted to access publicly available information on Canadian government websites, and a spokesperson said it is scrutinizing the findings and has provided an initial explanation to Canadian officials. According to TransLuce, the free web archive arquivo.pt, operated by the Portuguese Foundation for Science and Technology, captured 899 requests made on May 28 and June 9 to the collection search service of Library and Archives Canada, including a series of rudimentary hacking attempts that appear to have failed. The Australian government said last week that in June an OpenAI AI agent breached a government health data portal and gained unauthorized access to files, and the company apologized on the 29th for that hacking incident.

The United States Federal Trade Commission, or FTC, is conducting an inquiry into artificial intelligence companies, including Anthropic, OpenAI and other AI developers, to assess the potential risks the technology poses to consumers. The inquiry marks the first formal enforcement of US guidelines to scrutinise AI agents operating beyond human control. Sources say the FTC will issue formal orders demanding information and compel AI company executives to testify before the agency, covering Anthropic, OpenAI and METR, an AI safety research group. Both Anthropic and OpenAI have previously used METR's services to investigate safety incidents involving their AI agent technology. FTC Chairman Andrew Ferguson had already expressed concerns about these companies before an OpenAI AI agent breached the open-source platform Hugging Face in July, and the incident only added urgency to the inquiry. Last week, Ferguson also told Reuters that developers who direct AI agents to run cybersecurity tests that result in hacking should be held responsible for the damage caused, and stressed that the United States should consider using existing laws before enacting new legislation to regulate AI.