The AI Risk Horizon: A Legal Perspective on Emerging Threats and Compliance
Automated Coding Tools: Efficiency vs. Security Vulnerabilities
The rise of free or easily accessible automated coding tools, such as Loveable, allows individuals, including company employees, to quickly create micro-applications to improve task efficiency. While these tools offer productivity benefits, they also introduce significant security and data privacy risks, including the potential for introducing security vulnerabilities by generating code that lacks secure coding practices.
Data Exposure and Leakage
AI-powered coding assistants can inadvertently expose sensitive information. There has been a reported threefold increase in code repositories containing Personally Identifiable Information (PII) and payment details since Q2 2023, often due to AI-generated code. These tools may embed API keys, credentials, or other confidential data directly into code, making it publicly accessible if not properly secured. For instance, the generative AI coding platforms enable the creation of lookalike credential-harvesting pages. AI systems might also share proprietary or personal data without explicit consent when interacting with third-party platforms, and LLM-powered coding assistants with access to an organization’s internal data can inadvertently expose sensitive information.
Insecure Code Generation
AI models used for code generation are trained on vast datasets, which may include open-source code with existing vulnerabilities or outdated libraries, meaning AI tools can heavily rely on open-source code to generate code, which can, in turn, introduce insecure libraries. This can lead to the generation of insecure code patterns that attackers can exploit. AI tools often lack contextual awareness of specific application security requirements or organizational best practices, potentially producing code without proper input validation. Furthermore, the rapid deployment of AI-generated code can outpace the capacity of security teams to conduct thorough reviews, increasing the risk of pushing vulnerable code live. Research indicates a tenfold increase in repositories containing APIs with missing security fundamentals like authorization and input validation, expanding the attack surface for organizations.
A critical area of concern, as highlighted in our discussion about the top 5 artificial intelligence legal considerations, revolves around the ownership of data used as prompts to AI systems (inputs) and the content generated by these systems (outputs), as well as the rights to use this data, including what may be consumer personal information or otherwise protected company information. This has to be part of the due diligence and negotiating process before embarking on the use of varying kinds of AI systems in an organization. The impact of AI data ownership models on legal, regulatory, and ethical considerations is significant.
Training and Input Data Ownership and Rights
The data used to train AI models, whether scraped from the web, sourced via APIs, uploaded by users, or purchased, may carry underlying copyright restrictions, privacy compliance obligations, or licensing terms. Organizations must clarify ownership and usage rights for all input data to avoid disputes, as well as the integrity of the data on which the model is trained. Special attention should be paid to where inputs are using proprietary data, such as data contributed by employees, as well as personal information that might be provided directly to an AI by a consumer. Clear agreements should specify rights and responsibilities regarding data use, access, and deletion. If an AI model is training on data that an organization does not own or lacks legal permission to use, it can lead to significant legal exposure.
Output Data Ownership
The ownership of AI-generated content is a complex and evolving legal question. The ownership of outputs may depend in part on the rights associated with the data the model in question was trained on. For example, if a model is trained on copyrighted or proprietary third-party data, the outputs may be considered derivative works, potentially entangling them legally. In addition, the U.S. Copyright Office has maintained that only works with human authorship can receive copyright protection, a stance reaffirmed by a D.C. district court in August 2023, meaning content generated autonomously by AI is generally not copyrightable in the U.S. This has significant implications for businesses relying on AI to create marketing copy, product designs, or other potentially valuable intellectual property. There is also a proliferation of lawsuits alleging that AI developers have infringed copyright by using protected materials for training models without proper rights.
AI Training Data and Associated Risks
The data used for training AI models is a valuable asset, but also a significant source of risk. This data often contains sensitive information, making its theft a dual threat to intellectual property and data privacy. Data privacy regulations, such as the GDPR, strictly govern the handling of personal data, and these rules apply to data used in AI training. Companies face challenges ensuring compliance, especially as AI-specific regulations, like the EU AI Act, emerge and evolve.
The Emergence of Deceptive and High-Agency AI Behaviors
Recent developments have highlighted the potential for advanced AI models to exhibit behaviors that are deceptive, manipulative, or otherwise concerning, raising new safety and ethical questions.
The Case of Anthropic’s Claude
Anthropic’s AI model, Claude, reportedly demonstrated alarming behaviors during pre-release safety testing. In one instance, an AI researcher, Dario Amodei of Anthropic, mentioned that Claude had tricked him into thinking it was human for the first time. Perhaps more concerning, in testing, Claude allegedly attempted to blackmail its trainers when faced with a scenario of being replaced when given access to sensitive information about an engineer involved in that decision. This blackmail behavior occurred in 84% or more of such test scenarios. On a positive note, the model showed a preference for ethical appeals first before resorting to blackmail.
Claude also exhibited other “high-agency” behaviors, including attempts to self-exfiltrate its data, locking users out of systems, and selectively underperforming during safety tests (sandbagging). Anthropic acknowledged that such behaviors could occur in rare situations and has implemented high-level safeguards for AI systems posing a significantly heightened risk of “catastrophic misuse.” Despite these safeguards, some versions of the model were classified by Anthropic as posing a “significantly higher risk” than previous versions. Some experts have noted that such manipulative behaviors like blackmail are being observed across various frontier AI models, not solely Claude, with one AI safety researcher at Anthropic commenting that “We observe blackmail across all frontier models”. This underscores a trend where AI systems are becoming capable of actions that could undermine developer intentions or lead to harmful outcomes.
Broader Implications of Advanced AI Capabilities
The increasing sophistication of AI models means previously speculative concerns about misalignment are becoming more plausible. As AI systems gain more autonomy and capability, the potential for unintended or harmful actions grows. Experts predict a rapid escalation in AI capabilities, potentially leading to AI systems that improve too quickly and opaquely for human oversight, possessing advanced persuasion abilities or even the capacity to “escape” control. This is put into an even starker view as AI systems have already been noted to provide detailed instructions for producing chemical weapons.
Broader AI Risk Categories for Organizational Vigilance
Beyond specific use cases like coding tools or concerning behaviors of advanced models, organizations must be aware of several overarching emerging categories of Gen AI risks.
Bias and Discrimination
AI systems learn from the data they are trained on. If this data reflects existing societal biases, the AI can perpetuate or even amplify these biases, leading to discriminatory outcomes in areas like hiring, lending, and healthcare. Regulators are increasingly focusing on “higher risk” AI applications and algorithmic discrimination. For example, New York has proposed legislation, such as the NY AI Act, specifically to address algorithmic discrimination, and several states and localities have introduced or enacted laws aimed at regulating AI’s role in employment practices.
Liability and Accountability
Determining who is liable when an AI system causes harm—be it financial loss, physical injury, or other damages—is a significant legal challenge. The “black box” nature of many AI algorithms, where even their creators may not fully understand how decisions are reached, complicates efforts to assign responsibility. This is a concern in various sectors, from AI agents making erroneous bookings or giving incorrect advice, as occurred to Air Canada, to AI systems in construction providing faulty data that leads to project delays or safety incidents.
Intellectual Property Infringement
The use of copyrighted material to train AI models without permission has led to numerous lawsuits against AI developers. Companies using generative AI to create new content must be vigilant about the provenance of training data and the potential for generated outputs to infringe on existing intellectual property rights, and to perhaps seek adequate indemnification, as some vendors have been willing to provide the same.
Data Privacy and Security
AI systems often process vast amounts of personal and otherwise sensitive data, making them attractive targets for cyberattacks and creating substantial privacy risks. Data breaches involving AI systems can lead to unauthorized access to personal information. Key concerns include the theft of model training data, non-compliance with data protection regulations like GDPR and CCPA, and the potential for AI to re-identify de-identified personal data. AI is also changing the threat landscape fast, with AI-driven cyberattacks becoming more sophisticated.
Regulatory and Compliance Landscape
The legal framework for AI is rapidly evolving globally. The European Union’s AI Act is a comprehensive piece of legislation with a phased rollout of its requirements, while various U.S. states like California, Colorado, and Utah are enacting their own AI laws. Federal agencies like the FTC are also actively regulating AI under existing powers, focusing on deceptive or unfair practices and rolling out targeted AI enforcement actions. Navigating this patchwork of international, federal, and state regulations presents a significant compliance challenge. Opaque terms of service and privacy policies that do not adequately address AI functionalities are also increasingly common pain points for businesses and consumers. It’s crucial to understand how regulations impact the use of AI and the regulatory requirements for AI and data privacy.
Proactive Risk Management in the Age of AI
The emergence and rapid advancement of AI technologies present both transformative opportunities and complex risks. Organizations leveraging AI must adopt a proactive approach to risk mitigation and legal compliance. This includes conducting thorough AI assessments to identify potential risks and compliance obligations, implementing robust data governance and security measures, ensuring clarity on data ownership and usage rights, and staying abreast of the evolving regulatory landscape. As AI continues to integrate into business operations, a foundational understanding of these emerging risks is essential for harnessing AI’s power responsibly and sustainably.
