The OP asks a great question that has thrown up some great replies.
My tuppence worth.
In
The Universal Mind of Bill Evans ‪The Universal Mind Of Bill Evans‬‏ - YouTube , Bill talks about what he calls "the jazz process", and he makes a distinction between "making one minute's music in one minute's time" and "making one minute's music in three months' time".
I put the question of spontaneity and improvisation in solo guitar playing to Howard Alden at a workshop. By way of response, Howard invited us - the participants - to call out titles, which he arranged on the spot.
This seemed an incredible feat at the time. I remember feeling completely overawed; Howard Alden's on-the-fly arrangements were not only spontaneous, but were also beautiful.
I think the latter quality has much less to do with chops than with tasteful choices.
I also think ear training is priority number 1.
I don't do many, but my arrangements have happened spontaneously, and usually in under a minute. Sprawled out in exhaustion from my day job, beautiful music comes to me. I'll begin to vocalize melody and counterpoint in solmization syllables - until I have NO CHOICE but to get up and play what I'm singing, trying out keys as I go. I write down only what I have difficulty in hearing at the piano (which hasn't happened in a while).