Assignment: Learning and Relearning

Construct a simple vanilla backpropagation network, using ADOLC or whatever other tool you wish. Construct a random training set which consists of two parts, A and B. First train the network for a while on A+B, and plot performance on A and on B. Then hit the network's weights with some noise (not too much please.) This will raise the error on both A and B. Now train the network on just A, while continuing to plot performance on A and on B. Look at how performance on B changes while the network is ``relearning'' A following damage. If you don't train too much or too little, and don't make the learning rate too aggressive, you should see the performance on B improve even though you are training only on A. In your writeup, speculate as to what might cause this phenomenon.

The precise parameters: how many hidden units, how many input units, how many output units, how much to train, how much noise to add to the weight when you damage the network - I leave to you. The object is to (a) do this in a timely fashion, (b) notice an interesting effect.

Barak Pearlmutter <>