A long-unawaited sequel

You probably don't remember a post I wrote in 2009, inspired by a book Jacqui and Vik gave me for my 23rd birthday.  Guided by that assumption, I'll recap it before continuing with the promised (and delayed, mostly because no-one seemed interested and partly because I didn't continue with the line of thought until very recently) sequel.

I enjoy doing mathematics and watching college football (live or on TV), so I was thinking about the least arbitrary possible way of mathematically ranking college football teams.  The idea of ranking college teams is very big here.  I decided that a sensible place to start would be to assume that it's useful to organise teams according to the average fraction of games they would win if they played many times against all possible opponents.  I then wanted to find a function of two variables, such that if you put in the average winning percentage of each team, it would give you the probability that each would win were they to play each other.

Obviously, this function would need to be consistent: that if there is a 60% team, the function had better imply that it will win 60% of its games.  Even finding a function like this is difficult, let alone choosing the best one.

However, I have come to use the concept of entropy a lot in my research, and it plays a big part in statistics.  In brief, there is a well-defined statistical process for working out the distribution one should expect a set of events to have, if one knows some but not all the information about them.  For example, if you know the mean and standard deviation of results but no other information, a normal distribution is the best one to assume.

So, I decided to assume that any winning percentage is equally likely, and find the maximum entropy function that satisfies the necessary criterion.  I tried lots of things, but in the end could only approximate it using a computer.  What I found was that the function I was looking for was very close to a step function: i.e. if team A is more than a tiny bit better than team B, it suggests team A will win almost 100% of the time.

Further thought and discussion with Patrick convinced me that this is not reasonable: empirical evidence suggests it is just wrong (and, being a physicist, I'm not willing to claim that my theory is more right than the universe).  One response might be to think that this is unsurprising, because I used almost no input from the specific field in forming my function.  However, I don't necessarily agree.  The great thing about statistics is that they work really well for large accumulations of data: distributions that are theoretically expected usually emerge almost like magic.  So I'm more likely to believe that I can find a good function, but I need to ask a slightly better-posed question.

I think the problem might be my assumption (that I mentioned last time but barely noticed this time) that all winning percentages are equally likely.  If I'm treating entropy seriously, I need to realise that a team with a 100% win probability is an extremely low entropy state.  So, I should weight the function by recognising that win probabilities closer to 50% are more likely.

Mathematically, I want to find an entropy maximising distribution f(x), where x is the win probability of a team.  One extreme is a uniform distribution over x, because this maximises the entropy of f.  However, one could also point out that x=0.5 is the highest entropy state because it has perfect mixing of wins and losses, so a delta function at x=0.5 maximises entropy.  I'll just add the two entropies together, so I'm trying to find f(x) that maximises
\int_0^1 f(x)\ln[f(x)] dx - \int_0^1 f(x) [x\ln x + (1-x)\ln(1-x)] dx.

I have calculated the answer, but this blog is too short to contain it... ;-)
Actually, I need to go dancing now, but I'll post a threequel soon.