A circle doesn't contain infinite information, because there is a finite string of symbols that we can write that completely describe it. The question is, what symbols are we working with, and how can we combine them? That leads to the definition of a language, and the Chomsky hierarchy of languages comes into play, with Turing-complete languages as the sine-qua-non. Then Kolmogorov comes along and says "The ultimate measure of information in a string is the size of the smallest Turing machine that can compute the string".