#information-theory
5 posts · all tags
-
· From Entropy to Epiplexity
In this paper, Finzi et al. argue that classical information theory (Shannon entropy and Kolmogorov complexity) is insufficient for understanding modern AI because it assumes an observer with unlimited computational power. Under classical theory, deterministic transformations cannot create new information, yet empirically, processes like self-play (AlphaZero) and synthetic data generation clearly improve model capabilities.
-
· Integrated Information Theory (IIT)
One of the most interesting question to be made is about the nature of consciousness. During a long time the question of consciousness was considered as a philosophical question but during the last decades this question has been approached by the science with the emergence of the Integrated Information Theory (IIT) by Giulio Tononi. This paradigm, proposes a radical reconceptualization of consciousness, framing it as an intrinsic property of physical systems, quantifiable through information integration.
-
· Some common divergences
Hey. usually on machine learning we have to consider distances between different probability distributions. This is in fact a hard problem and there is a misconception that the most common way to do this is using the kl divergence. I believe that this is not true.
-
· Kolmogorov Complexity
Think of Kolmogorov Complexity as a way to measure the complexity of a string (a sequence of characters) by looking at the length of the shortest possible program that can produce that string. Imagine you have a super-efficient computer program that generates text. Kolmogorov Complexity is about finding the tiniest program that can spit out the exact text you have.
-
· Understanding Cover's Universal Portfolio
I always wanted to understand the concept of a universal portfolio from Cover. This topic has fascinated me since I first heard about it in my information theory lectures, and you know what they say... the best way to learn is to practice. So here's my attempt.