I think part of the explanation is that Spielberg and Lucas intended Last Crusade to be the finale for different reasons. For Spielberg, it was because he wanted to move on and make other kinds of films. For Lucas, who considered himself responsible for coming up with the Indy plotlines, it was because he was simply out of ideas. Lucas was still interested in the character in a way that Spielberg was not, which is why he proceeded with Young Indy.
It's also why, when Lucas had his Eureka moment and decided there was a story for a fourth movie involving an aging Indy in a cold war context, he started developing the material with screenwriters pretty much by himself, while Spielberg was content to hang back and be a producer. It took a while for Spielberg to get interested in directing the project. Some would argue that he never really did get interested, except maybe during the Darabont interlude.