AI Hardware: Challenges and Opportunities

The TDL webinar took place on Thursday, 16 April 2026 with a discussion on AI hardware and silicon which are experiencing explosive demand even though today the market for dedicated hardware is relatively small compared with the overall semiconductor market. The shift from the focus on quantity to better, more efficient designs and reliability has started, Among the strongest challenges is the need for power efficiency to ensure scaling, thermal and cooling limits, the need for customisation and a lot more. The fast growing demand for AI hardware has provided opportunities for technologies that were developed earlier but were not used due to their costs, e.g., silicon photonics. The panellists discussed technology, emerging innovation, manufacturing and supply chain issues and plenty more.

Summary

During the TDL panel discussion focused on AI hardware challenges and opportunities, the panellists discussed key areas including security concerns, thermal management challenges as AI accelerator TDPs (thermal design powers) cross the 1 kilowatt threshold, the shift from training to inference, and the importance of co-design approaches for hardware and software. The panel explored emerging technologies like CHERI (capability hardware enhanced RISC instructions - a hardware-enhanced security technology) and the move towards 2.5D and 3D chiplet architectures. Despite acknowledging significant technical challenges, all the panellists expressed optimism about the future direction of AI hardware innovation, particularly in the areas of security-by-design, advanced cooling solutions and physics-aware simulation tools.

Background

Each participant was asked to express their views on the most important issues in design, adaptation, trends and challenges in AI hardware today. The topic presented a wide range of challenging and divergent questions, such as:

• whether GPU dominance will continue or be replaced by other approaches, like TPUs (tensor processing units), accelerators, custom ASICs (application-specific integrated circuits), etc, which will gain prominence and market share. And whether custom ASICs will grow faster than traditional GPUs and accelerators

• to what extent processing power, efficiency, adaptability, security capabilities or something else will motivate innovation, in particular efficiency over customisation and the significance of innovation in memory

• how software will (e.g., LLM designs) respond to new trends in hardware

• the importance of edge and on-device processing, depending on whether it takes off in the next few years

• the role and impact of new approaches such as photonics, neuromorphic designs, etc.

• how trends will affect the business models and, for example, the continued growth of hyperscalers or the emergence of many smaller companies that are custom use case oriented - and the effect of regulations on these developments.

Other developments likely to impact the most important technology innovation developments over the next one to three years will include the features in the near future inference optimised ecosystem, which approach to energy efficiency will dominate and understanding and overcoming the main challenges in designing optimal AI hardware.

From the perspective of regulatory and economic developments, there could prove to be challenges in the near future following the scarcity of chips. Infrastructure bottlenecks may well have a considerable impact as might the fragmentation of the regulatory space. Economically, companies will be challenged in having to make significant capital outlay for infrastructure and R&D with uncertain ROI (return on investment), made more difficult by developments in other areas, such as quantum computing, which could lead to competition for resources.

The prominent positioning of AI hardware could lead to significant innovation in hardware architecture, although it’s an open question as to what the most important areas of innovation ready for deployment in the near future will be. In part, this will be down to understanding the main challenges in deploying innovation in AI hardware. This could be, for example, the continued growth in raw FLOPS (floating point operations per second - the maximum theoretical computing performance of a processor), or a real breakthroughs in performance-per-watt, cost-per-token and sustainability rather than sheer scale. It’s questionable as to whether vertical integration will result in specialised markets and whether chiplet-based heterogeneous integration will flourish.

Finally, each of the speakers were asked whether they were overall optimistic or pessimistic about the immediate future of innovation in AI hardware. A conclusion important to us all!

Speakers

The panel consisted of:

· Helena Handschuh, Technical Board Advisor, QuSecure and SCI Semiconductors

· Rahima Mohammad, Semiconductor Technical Advisor, Vinci4D.ai

· Soheil Salehi, Assistant Professor of Electrical and Computer Engineering (ECE), University of Arizona

· Silviu Vlasceanu, Director, Trusted System Security Lab (TSS), Huawei

The session was moderated by TDL strategic adviser, Claire Vishik

AI Hardware Challenges and Opportunities

The meeting began with an explanation that the session would focus on AI hardware challenges and opportunities, including topics like hardware capabilities, customisation and security in various domains like automotive and IoT as well as covering the current state of AI hardware, innovation and trends.

AI Hardware Security Challenges

The security challenges in AI hardware were discussed, highlighting three key areas: fully homomorphic encryption, roots of trust for securing computation environments and memory safety. It was noted that while homomorphic encryption has progressed, applying it to LLMs (large language models) presents significant technical challenges. The focus moved to thermal and packaging challenges in AI hardware, explaining that AI accelerator TDPs (thermal design powers) are crossing one kilowatt thresholds and are expected to move towards four kilowatts by the end of the decade, requiring a shift from air to liquid cooling and new 2.5D and 3D architectures. The discussion concluded with a question about the role of fundamental science in accelerating hardware development timelines.

Multi-Vendor Chip Design Standards

Next, the challenges in multi-vendor chip design were explained and a standard called PACT (parallel compact thermal simulator) was proposed to address interface issues without sharing IP (intellectual property). The shift towards edge devices due to power consumption limitations was discussed, highlighting the importance of thermal management and security in AI hardware design. The need for reconfigurability in hardware to adapt to rapidly changing AI models was also emphasised.

AI Energy Efficiency Hardware Challenges

The discussion moved on to energy efficiency and hardware challenges in AI implementations. It was explained that, while edge AI offers significant energy improvements (up to 100x), it faces challenges including hardware resource constraints, model compression requirements and environmental reliability issues. Security concerns were highlighted, noting that, while roots of trust are becoming ubiquitous in computing infrastructure, security often takes a backseat to performance in product development. Two key trends in AI hardware were also observed: specialisation with NPUs (neural processing units) and TPUs (thermal processing units), and the resurgence of CPUs (central processing units) with integrated accelerators, suggesting a future of diverse and adaptable hardware solutions.

AI Hardware Security Challenges

The discussion went on to focus on AI hardware innovation and security challenges, highlighting the shift from GPUs (graphics processing units) to TPUs and specialised CPUs, emphasising efficiency improvements and interconnect scalability. CHERI (capability hardware enhanced RISC instructions), a hardware-based security technology that addresses memory safety and security by design, was introduced as it could potentially reduce overhead costs. The challenges of hardware security, including the limitations of reconfigurable hardware and new threat vectors introduced by emerging semiconductor technologies were brought up. The group explored the potential of CHERI and other hardware-based memory safety technologies in AI accelerators, acknowledging the need for further research and development to address security concerns in AI hardware.

Hardware Innovation Challenges and Opportunities

The group discussed challenges and opportunities in hardware innovation, particularly focusing on security, energy efficiency and design methodologies, emphasising three key vectors for energy efficiency: architecture-level improvements, advanced packaging and cooling solutions, and enhanced simulation and design processes through co-design strategies. The importance of addressing temperature resistance and new security challenges in emerging technologies was highlighted. The discussion concluded with a focus on the need for improved simulation tools and the potential for AI to accelerate design processes, noting that one technology solution offers significant speed improvements through continuous model training and physics-based simulations.

Hardware Security in AI Discussion

The panellists discussed the current state and future of hardware security in AI, all expressing an enthusiastic optimism about the opportunities for innovation, highlighting the growing need for hardware-based security measures due to recent advancements in AI model capabilities for reverse engineering. The industry’s ability to overcome challenges through co-design and system technology co-optimisation was emphasised, while also noting the importance of talent development and increased public and private investment in semiconductor research. It was suggested to leverage AI to enhance verification, testing and code reviews to address the significant time spent on these processes. The group agreed on the potential for AI to aid in improving security and efficiency in hardware design. It was mentioned that the challenges will bring a golden age in various areas of innovation, especially security and inevitably to energy efficiency.