I think it's about the dark side of fandom. As a fan myself, I see these minds at work often. We tend to overthink and worship movies. They are our gospels, our Greek myths, our religion. But, humans still being human, we also have a sheep mentality. See, nobody really complained about Temple Of Doom UNTIL Last Crusade came out. Suddenly, there were two movies very similar (given how much of a rip-off LC was of ROTLA), so TOD simply seemed "wrong" because it was "different". It's pack-animal mentality. We fans (yes, *we*, meaning I too) tend to viciously defend our stories because they are so important to us. Unfortunately, that oftentimes turns us against who and what we should love, too (welcome to religion).
Now, TOD does tend to hinder itself a little bit, leaving itself open for criticism. Mostly, unlike Empire Strikes Back (the oddball compared to the other two Star Wars movies), there isn't an overriding we-can-latch-onto-it mystical element. Probably because it's basically about Hinduism, whose underlying themes are off-key to the larger audience. Western audiences and their preferred feel-good stories are generally based on Christian themes. Heck, even The Force is basically a "faith" as we simplify it. So, TOD doesn't give us that "feel-good-magic" like Yoda's teaching does (I know, technically it does, but we're talking about peer pressure from the mass audience here). In addition, TOD simply suffers from having to follow-up ROTLA -- it's a tough act to follow!
When LC came out, everyone (rightly) praised Connery's chemistry with Ford. That tended to blind them to how much LC was lacking in so many other areas. The sheep-mentality kicked in -- "The stars are great, we're saying it's great, so the whole movie must be great, and Raiders was great and it's the same, so that must be what great is, so, um, TOD, you're different, so you suck". That's pretty much where this general thinking came from. IMHO.
I believe, however, that TOD is vastly superior to LC. In many ways, it's the best Indy movie of them all, the most "pure Indy". Indy's in his element. No school, no family crap, no being in a suit. It's hat, whip and chaos from start to finish. To me, that's what Indy is all about. It's a pity so many people can't bring themselves to enjoy it simply because there aren't Nazis (again) or classrooms (again) or soap opera (again). TOD has a variety that was sorely lacking until KOTCS came out.