The Geometry of Bit-Collapse
Winter ’26
Int8 quantization sharpens ResNet-18’s loss basin and spikes its Hessian eigenvalues, destabilizing SGD. SAM restores a basin flatter than full precision.
ex-Research intern, WSAI, IIT Madras
Sairam
Sundararaman
I was a research intern at WSAI, IIT Madras, advised by Prof. Krishna Pillutla. I work on differentially private mechanisms for language models and on understanding why ML methods fail. I’m also in the final year of my B.Tech at PES University.
Research
Differentially private mechanisms for transformer LLMs
WSAI, IIT Madras · Advisor: Prof. Krishna Pillutla · June 2026 – August 2026
- Designing and analyzing a novel differentially private mechanism for transformer LLMs.
- Derived sensitivity bounds and Gaussian mechanism noise calibration from first principles; formalized privacy composition under zCDP across layers and generated tokens.
- Implemented and benchmarked on distilgpt2 and Qwen2.5-7B-Instruct.
Publications
The Final-Stage Bottleneck: A Systematic Dissection of the R-Learner for Network Causal Inference
Sairam S, Sara Girdhar, Shivam Soni
Transactions on Machine Learning Research (TMLR) · 2026
Projects
Reproducing SAM
Fall ’25
Clean-room SAM reaching ~96% on CIFAR-10 against the paper’s ~97%. It exposed two weak spots: SAM leans on weight decay without accounting for it, and its approximation degrades at sharp minima.
Optimization Pathologies
Fall ’25
Maps 2D loss landscapes (Rosenbrock, Beale) to show why SGD zigzags in narrow valleys and how momentum smooths the path.
Reverse-Mode AutoDiff Engine
Summer ’25
A scalar autograd engine built from scratch. It matches PyTorch to within 1e-6.
News
- Aug 2026Ended my Research Internship at WSAI, IIT Madras.
- Jun 2026Started as a Research Intern at WSAI, IIT Madras.
- Mar 2026The Final-Stage Bottleneck accepted at TMLR.
- Aug 2025Teaching Assistant for Mathematics for Computer Science at PES University.