Information theory, inference, and learning algorithms / David J. C. MacKay

This textbook is geared toward upper-level undergraduate students and graduate students in engineering, science, mathematics, and computer science. Prerequisites for students using this book are a good working knowledge of calculus, probability theory and linear algebra. This textbook is not just any textbook on information theory. Besides the usual topics covered in a conventional course on information theory (Shannon theory, parctical solutions to communication problems, etc.), this text brings Bayesian data modelling, Monte Carlo methods, variational methods, clustering algorithms, and neural networks. The author takes the point-of-view that information theory and machine learning belong together. He manages to convince the reader of this viewpoint by continually showing the connections between seemingly unrelated topics. As Bob McEliece is quoted, “An instant classic. ... You'll want two copies of this astonishing book, one for the office and one for the fireside at home.” (Zentralblatt)