Eldan Goldenberg's lab notebook

Slapping myself in the face

I have just found a bug I'd been looking for since Tuesday. Unfortunately it turns out to affect more of my experiments than I had realised, so it may invalidate my results from January so far. It's also possible that it didn't spoil anything—certainly the effects are subtle enough in most conditions that I didn't notice it until a run with just the right set of parameters—but I think I'm going to have to re-run quite a few experiments to find out.

As a compromise between using too limited a batch of trials (and risking over-specialisation in the evolved agents), or randomly replacing the whole batch each time (and hampering the search with trial noise), or using a huge batch of trials (and making the search take too long because each evaluation needs so many CPU cycles), I tried to set up a more sensible system. My experiments have a batch of n (11-50 depending on other parameters) trialsets (I need to clean up this terminology, but bear with me for now), of which m (usually 1) set[s] get replaced each generation.

What was supposed to happen was that the oldest trialset[s] would get replaced each time, so that each trial set would be seen by n/m generations, giving some stability while avoiding staleness. It turns out that what was actually happening was that the first trialset got replaced every time, so trialsets 2-n were static through the whole search, while trialset #1 changed every generation. Instead of having my planned compromise between noise and overspecialisation, I've managed to implement both problems together.

The good news is that once the bug was found, fixing it was rather easy. The bad news is that I won't know how much difference it made until I've re-run most of the experiments from the past few weeks with fixed code.