Over the past two decades, our research group has applied information-theoretic ideas, concepts, and techniques to diverse areas of machine learning and data science. This expository presentation follows our journey from compression, to estimation, structure exploitation, functional approximation, classification, and robust and reinforcement learning, with a few life lessons learned along the way.