With the rise of Large Language Models (LLMs) and Vision Transformers (ViTs), we investigate the return from expensive "Quantization-Aware Training" (QAT), towards smarter "Post-Training Quantization" (PTQ), where the choice of "what" to quantize and "how" is guided by (sometimes) sophisticated metrics. The student is expected to perform an evaluation of such metrics and their effect to the quantization results.

Voraussetzungen

Unsourced multi-access protocols ensure that multiple users can transmit on the same physical resources without pre-allocation of resources to the different users. To avoid information loss caused by collision of messages transmitted simultaneously, we investigate how to use (adaptive) coding schemes that allow the concurrent transmission of coded messages stemming from different users with lossless reconstruction of the payloads.

The student should be proficient in communications engineering and coding theory, i.e., the following prerequisites (or similar) are minimal requirements:

• Channel Coding
• Nachrichtentechnik

In a distributed learning setting, multiple worker machines are hired to help a main server with the expensive training of a machine learning model. Each worker is assigned a subtask, from which the main server tries to reconstruct the total computation result.

In the presence of faulty or malicious workers, called Byzantine workers, a common strategy is to distribute subtasks redundantly [3]. Since the redundancy introduces large computation overheads for the workers, strategies to reduce this overhead are required. One approach is to use interactive consistency checks at the main server, which can reduce the redundancy by up to 50% [1].

The interactive consistency checks are not for free, but cause additional computation and communication cost. For specific parameter regimes, this cost is well-studied. However, it is unkown how large this cost is in general. Therefore, we ask the following research questions:

1) How much computation is needed to guarantee error-free reconstruction of the computation result?

2) How much communication is needed?

3) What is the best trade-off between communication and computation cost?

The focus of this project is to study these research questions fundamentally. That is, we aim at understanding what the least amount of communication and computation possible is. The student will analyze these questions through mathematical tools, such as graph theory or information theory. The findings shall be compared against existing schemes [1,2] to evaluate their (sub-)optimality.

[1] C. Hofmeister, L. Maßny, E. Yaakobi and R. Bitar, "Byzantine-Resilient Gradient Coding Through Local Gradient Computations," in IEEE Transactions on Information Theory, vol. 71, no. 4, pp. 3142-3156, April 2025, doi: 10.1109/TIT.2025.3542896.

[2] S. Jain, L. Maßny, C. Hofmeister, E. Yaakobi and R. Bitar, "Interactive Byzantine-Resilient Gradient Coding for General Data Assignments," 2024 IEEE International Symposium on Information Theory (ISIT), Athens, Greece, 2024, pp. 3273-3278, doi: 10.1109/ISIT57864.2024.10619596.

[3] R. Tandon, Q. Lei, A. G. Dimakis, N. Karampatziakis, "Gradient Coding: Avoiding Stragglers in Distributed Learning", in Proceedings of the 34th International Conference on Machine Learning, PMLR 70:3368-3376, 2017.

Voraussetzungen