Information Theory for Data Science

Changho Suh is an Associate Professor of Electrical Engineering at KAIST and an Associate Head of KAIST AI Institute. He received the B.S. and M.S. degrees in Electrical Engineering from KAIST in 2000 and 2002 respectively, and the Ph.D. degree in Electrical Engineering and Computer Sciences from UC Berkeley in 2011. From 2011 to 2012, he was a postdoctoral associate at the Research Laboratory of Electronics in MIT. From 2002 to 2006, he was with Samsung Electronics.

1 Source Coding
1.1 Overview of the book
1.2 Entropy and Python exercise
1.3 Mutual information, Kullback-Leibler (KL) divergence and Python exercise Problem Set 1
1.4 Source coding theorem for i.i.d. sources (1/3)
1.5 Source coding theorem for i.i.d. sources (2/3)
1.6 Source coding theorem for i.i.d. sources (3/3)Problem Set 2
1.7 Source code design
1.8 Source coding theorem for general sources
1.9 Huffman code and Python implementation
Problem Set 3
2 Channel Coding
2.1 Statement of channel coding theorem
2.2 Achievability proof for the binary erasure channel
2.3 Achievability proof for the binary symmetric channelProblem Set 4
2.4 Achievability proof for discrete memoryless channels
2.5 Converse proof for discrete memoryless channels
2.6 Source-channel separation theorem and feedback
Problem Set 5
2.7 Polar code: Polarization
2.8 Polar code: Implementation of polarization
2.9 Polar code: Proof of polarization and Python simulation
Problem Set 6
3 Data Science Applications
3.1 Social networks: Fundamental limits
3.2 Social networks: Achievability proof
3.3 Social networks: Converse proof
3.4 Social networks: Algorithm and Python implementationProblem Set 7
3.5 DNA sequencing: Fundamental limits
3.6 DNA sequencing: Achievability proof
3.7 DNA sequencing: Converse proof
3.8 DNA sequencing: Algorithm and Python implementationProblem Set 8
3.9 Top-K ranking: Fundamental limits
3.10 Top-K ranking: Algorithm
3.11 Top-K ranking: Python implementation
Problem Set 9
3.12 Supervised learning: Connection with information theory
3.13 Supervised learning: Logistic regression and cross entropy
3.14 Supervised learning: TensorFlow implementation
Problem Set 10
3.15 Unsupervised learning: Generative modeling
3.16 Generative Adversarial Networks (GANs) and KL divergence
3.17 GANs: TensorFlow implementation
Problem Set 11
3.18 Fair machine learning and mutual information (1/2)
3.19 Fair machine learning and mutual information (2/2)
3.20 Fair machine learning: TensorFlow implementation
Problem Set 12
Appendices
A – Python Basics
B – TensorFlow and Keras Basics
C – Note on Research