Super interesting blogpost. I just wonder how this is actually different to LORA, since LORA also adds some parameters and freezes the rest of the model. This seems like a sparse, memory efficient LORA with a couple of extra steps, since it uses attention again to make the sparsity work. All while making it a lot more effective compared to LORA (performance drop of only 11% compared to 71%).
I think the solution to continual learning is as simple as using context distillation. We know that models are good at in-context learning, so we just want an efficient way to distill context into the weights. I suspect context rot may come from how the softmax in attention gets diluted with a longer context, so this wouldn't be an issue with context distillation.
The problem with continual learning is that stochastic gradient descent is already an online algorithm applied incrementally on a shuffled dataset. If you add new data, you can't train on just the new data, because you will be running what amounts to a completely different training sequence. Further training requires the old data and the new data to be shuffled together.
With reinforcement learning, specifically actor critic, the actor is not training against a dataset. It's training against the critic. The critic is supposed to approximate the value function, which contains the current cost for a given action and the predicted future cost, assuming that you choose the optimal action at every step, including its impact on future actions. If you have a simple supervised cost function, what happens is that the critic acts as an averaging of loss functions. You could say that the critic is a compressed copy of the training data. When you train the actor, you're essentially taking not only the new data, but also the old data into account.
So, in a way, catastrophic forgetting is sort of solved, but not really. If you add new data, you run into the problem that your critic will slowly drift to the new data distribution. This means the problem wasn't solved, but you certainly managed to delay it. Delaying the problem is good though. What if you can delay it even more? What if you can delay it forever?
Here is my stupid and simple unproven idea: Nest the reinforcement learning algorithm. Each critic will add one more level of delay, thereby acting as a low pass filter on the supervised reward function. Since you have two critics now, you can essentially implement a hybrid pre-training + continual learning architecture. The most interesting aspect here is that you can continue training the inner-most critic without changing the outer critic, which now acts as a learned loss function.
Super interesting blogpost. I just wonder how this is actually different to LORA, since LORA also adds some parameters and freezes the rest of the model. This seems like a sparse, memory efficient LORA with a couple of extra steps, since it uses attention again to make the sparsity work. All while making it a lot more effective compared to LORA (performance drop of only 11% compared to 71%).
> LORA
I think you meant LoRA (not to be confused with LoRa)
I think the solution to continual learning is as simple as using context distillation. We know that models are good at in-context learning, so we just want an efficient way to distill context into the weights. I suspect context rot may come from how the softmax in attention gets diluted with a longer context, so this wouldn't be an issue with context distillation.
Perhaps it can work through multiple stages: ICL -> prompt/context optimization (*) -> prefix tuning / KV distillation -> context distillation.
*: it is possible to measure how much part of a prompt helps with a task e.g. measuring change in entropy
The problem with continual learning is that stochastic gradient descent is already an online algorithm applied incrementally on a shuffled dataset. If you add new data, you can't train on just the new data, because you will be running what amounts to a completely different training sequence. Further training requires the old data and the new data to be shuffled together.
With reinforcement learning, specifically actor critic, the actor is not training against a dataset. It's training against the critic. The critic is supposed to approximate the value function, which contains the current cost for a given action and the predicted future cost, assuming that you choose the optimal action at every step, including its impact on future actions. If you have a simple supervised cost function, what happens is that the critic acts as an averaging of loss functions. You could say that the critic is a compressed copy of the training data. When you train the actor, you're essentially taking not only the new data, but also the old data into account.
So, in a way, catastrophic forgetting is sort of solved, but not really. If you add new data, you run into the problem that your critic will slowly drift to the new data distribution. This means the problem wasn't solved, but you certainly managed to delay it. Delaying the problem is good though. What if you can delay it even more? What if you can delay it forever?
Here is my stupid and simple unproven idea: Nest the reinforcement learning algorithm. Each critic will add one more level of delay, thereby acting as a low pass filter on the supervised reward function. Since you have two critics now, you can essentially implement a hybrid pre-training + continual learning architecture. The most interesting aspect here is that you can continue training the inner-most critic without changing the outer critic, which now acts as a learned loss function.