Yuanji Ye, M.Sc.

Modern I/O devices and accelerators increasingly rely on virtual memory support in order to simplify programming, improve isolation, and enable shared virtual address spaces with CPUs. However, address translation on the device side is often expensive. When an IOMMU or device TLB misses, the accelerator may suffer significant stalls due to multi-level page table walks and limited translation locality. Prior work has shown that translation overhead can become a major bottleneck for accelerators and that hiding or restructuring translation latency is an important architectural problem.

At the same time, traditional prefetching research mainly focuses on data accesses, while the translation path itself can also be seen as a target for latency hiding. This raises an interesting question: instead of only prefetching data, can we also prefetch translation-related information, such as page-table entries, or redesign the translation path so that address translation and data access can overlap more effectively? Recent and prior studies suggest that this direction is promising for accelerator-centric systems.

The goal of this seminar is to build a clear architectural understanding of how prefetching and address translation can be combined for accelerators or IO systems. The seminar will compare different approaches, identify their main design trade-offs, and discuss whether translation-path prefetching could become an important design direction for future heterogeneous systems

Voraussetzungen

Large language model(LLM) inference is increasingly limited by memory access rather than pure computation, especially in long-context and decoding scenarios. A major reason is the high cost of moving model weights and KV cache data across the memory hierarchy. Recent work shows that prefetching can be used to overlap memory movement with ongoing computation or communication.

Compared with traditional hardware prefetching, LLM inference introduces a different setting. The prefetched object is no longer a small cacheline but larger units such as KV blocks. Moreover, prefetch timing depends on token generation, layer execution order, and runtime scheduling. Some recent studies also suggest that temporal patterns in attention behavior can be exploited to guide KV cache management and cross-token prefetching more effectively.

The goal of this seminar is to build a clear understanding of how prefetching concepts can be extended to AI inference systems. The student will compare recent approaches for KV cache and weight prefetching, discuss and summarize their architectural trade-offs, and determine whether LLM-serving workloads require new prefetching principles beyond those in CPU memory  hierarchies

Voraussetzungen

Hardware prefetching is a technique used in modern processors to reduce memory access latency by predicting future data accesses and fetching data into the cache before it is explicitly requested by the processor. By exploiting regular access patterns, such as strided memory accesses, prefetchers can improve cache hit rate and overall system performance. 

This research internship focuses on the implementation of hardware prefetchers in modern processor cache systems. The intern will study the design and interfaces of HPDcache, which is already integrated into the CVA6 SoC. Based on this understanding, the intern will design and implement an IP-stride prefetcher and connect it to the HPDcache. The work also includes evaluating the implemented prefetcher using existing benchmarks on the FPGA board.

Through this internship, the student will gain hands-on experience in computer architecture, cache subsystem design, hardware integration, and experimental evaluation of prefetching techniques.

Voraussetzungen