At work someone recently claimed that "entropy always increases, and the same is often true for software systems", and was there anything we could do about that.
This is of course a very interesting question so I started thinking. When I started to formulate an email in reply to him it turned out to be way too long and slightly off topic, so instead of replying by mail I morphed my reply into a couple of blogposts (this and another in our internal wiki).
But getting back to the topic; actually
it's only true that entropy always increases if you study closed
systems (it says so right in the wikipedia definition linked to above). In an
open system you can move entropy out of the system and thus decrease
entropy, that's why refrigerators work.
In statistical mechanics
entropy is basically defined as the number of states a system can be in
that is compatible with what you observe. Technically it's usually the
logarithm of the number of possible states that's used, but that's just a
technicality to avoid dealing with really large numbers all the time.
So a refrigerator will move entropy out of the refrigerator, and by
doing so will make the uncertainty related to the exact location of
the atoms inside the refrigerator less, so the number of possible
combinations of locations for the atoms will be smaller. See how it
all fits together? ;-)
How do we bring the that metaphor back
the domain of software? Well, if we define "program entropy" as the
number of different things a human perceives a program to possibly being
capable of doing, then that number of things defines that program's
entropy. If you think the program can only do one thing it has an entropy of
zero (log(1) = 0). If instead you have some code that you can't wrap your
head round and it could do a gazillion things for all you know, then
log(gazillion) is the entropy of that piece of code. So then if you have
two implementations, functionally equivalent but one that has but has
fewer possible interpretations as interpreted by a human reader than the other, then
the less ambiguous code has lower entropy by this definition. Note that since this is about
perception of the programs by humans, not by computer it makes
sense to talk about a program having different entropies for different
people. Also it does make a difference if the program is short enough to
wrap a human mind around it. If you can't wrap your mind about it you
can't know what it does so its entropy will be high since it in theory
could do many things. Entropy jumps in large leaps when things get
incomprehensible but conversely also drops quickly when incomprehensible
pieces becomes comprehensible.
The same kind of argument can be
applied to configurations of systems (essentially graphs and not necessarily not lines of code), so anything
that brings us clearer code or configurations will lower our entropy.
Does this make sense to you?
One topic I just thought of is what this definition of entropy does wrt. abstractions. By the definition above abstractions must have high entropy, or do they? Well, abstractions must have high entropy since they conceivably can do their things in many different ways, that's the very essence of abstraction. Let's take an ordered collection as an example: Ordered collections can can be implemented in a gazillion ways, but that doesn't matter since you're treating it only as an ordered collection and all those gazillion way are essentially equivalent. So perhaps it works the other way, an abstraction makes it unnecessary to think of all the different ways the program implements the abstraction, so instead of thinking of the many different implementation one things about only the abstraction, so the entropy is in fact lowered by using good abstractions.
Actually I think both these perspectives are valid, but if we let an individual reader's subjective perspective be the most central one, then I think it makes the most sense to claim that good abstractions will lower a program's entropy. That is, it will be true if the reader is of the kind of person that thinks that one abstraction is less complex than a set of direct implementations, and I know by experience that not everyone is like that, so opinions may differ systematically depending on how well the reader is able to internalize abstractions.