I have to put a serious effort into just plain memorizing the progression of a tune when I improvise over it.
It's easy for me to comp over a tune and become kind of a robot, just repeating the progression in a pattern, forgetting what's going on, what IS the chord I just played, what's coming up next.
When you go from comping to improvising, then you really need to know.
You can't be a monkey. Especially if you are taking a song through different keys, which I always make a point to do.
Some tunes are easy. Some are surprisingly complex in subtle ways.
It's a damn pain BUT to me it makes jazz so interesting.
This same process I find to be in playing chord melodies. If you only play a chord melody in one key (which is perfectly acceptable) you can kind of relax a bit and use a pattern mentality. When you take it through different keys, then you cannot get too comfortable.
It's a challenge.