Bob, I expected better from you. I haven't directly addressed you for a while, but I just wanted to say you kind of dropped the ball here. Of everyone on this site, and perhaps in geekdom in general, I would have expected you to get this:
You yourself at one point made it clear that you kind of "got" how Star Wars was based around more eastern concepts of mysticism and so on. One of the things it also does away with is the western idea of truly free will. The prophecy in the movies is actually adhered to through the entire thing, and following it was always a sort of foregone conclusion.
The key to the prophecy is that the chosen one will "bring balance to the force". If your paying attention the good guys are ruling over a rather peaceful universe, where they don't even need many Jedi to maintain order until around the time the movies start. With good so firmly in charge the only place for them to go to achieve "balance" is down. Sidious working with the prophecy is because he understands that part of it, but makes the mistake of thinking that if the good guys are going down, then it's time for a new Sith empire and he can pretty much get in on the ground floor as it's leader. He's wrong about that too as the prophecy says "balance" not "time for a reign of darkness".
Anakin is the tool the prophecy works through, his entire story arc is that he's so disturbed because he wants to be a good person, but the universe literally forces him to be a bad one. For example when he's just starting to get his ducks in a row, he happens to find his mother being raped and tortured by sand people, and goes on a rather dark side revenge spree as a result.
Luke, later on is pretty much just a factor in what is still Anakin's story when you get down to it. The entire thing is all about him getting Vader to betray the Emperor and prevent the universe from sinking into darkness. At the end of the movie when Vader lobs Sidious into a reactor pit, Anakin has literally brought balance to the force. He tore down The Jedi to the point where there are barely any left, and then he wiped out the leader of the Evil Empire. We have neither a major good force, or a major evil force, calling the shots. Everything is balanced in the force by way of pretty much everyone adhering to either side of the major force philosophy being dead.
This also explains some of the writers fiat, see The Force is playing metaphysical puppet master here. This is why the force is "cloudy" for some people, oftentimes trying to figure out the obvious (mean literally, why can't The Jedi Council sense a Sith Lord 2" away from them?) yet when it's something important that The Force needs done for it's narrative guys like Luke and Vader can literally sense each other across the galaxy, or on a planetary level.
This is also why you have fight scenes where say a Sith Apprentice can kill a Jedi Master, and then lose to another Apprentice with a literal BS move. It's because "The Force Says so" ditto for someone like Sidious being able to take down three Jedi masters at a time. None of these guys really have any power, they are all pretty much marionettes in a metaphysical puppet show.
Now, according to George Lucas' and his associative writings apparently, what happens is The Force for whatever reason guides this universe through cycles, where both good and evil rule for a few thousand years, with a short period of balance in between marking the transition point. This kind of galactic renewal and the decimation it causes is why say in the movies nobody even knows what a Sith was for sure, as they had been so entirely wiped out. It's also why the universe remains relatively stagnant and the technology doesn't change much even over periods of thousands of years.
While it was never finished properly, and thus there were problems, the game "Knights Of The Old Republic 2" was based heavily on Lucas' notes about this apparently. The premise of it being that a former Sith named Kreia figures out that The Force is pretty much manipulating everyone and subverting free will, so she seeks to destroy it so humanity can live on without it's manipulation. The first step in her plan is to literally kill all of the major force users, both good and evil, as they are the primary conduit through which The Force drives it's narrative. Of course the game not being properly finished it never got around to explaining how exactly she thought she could kill a metaphysical entity. Of course the ending of the game makes this partially irrelevant as Kreia herself starts spouting prophecy, and shows how what you just did actually worked the will of The Force by setting in motion events that were going to directly pit the Sith Empire against The Republic, which is a battle we know is going to end with the galaxy's dominant force (The Empire) being so badly eradicated people don't remember much about it, and a few thousand years of peace until the movie timeline heralds the new cycle.
The thing is that the Jedi aren't stupid, so much as they are literally unable to see the obvious because they are Jedi. I think that's part of the entire concept. One could argue it's probably a sort of pseudo-mind control (good little puppets) combined with them being indoctrinated to solve problems by meditating and conferring with The Force, which in this case as you saw isn't going to tell them crap since it wants them dead. Of course given that this comes down to predestination it's debatable how much free will there even is, so even that rationale isn't required, the Jedi could just be stupid because The Force wrote it that way (so to speak).
Now honestly (for you or anyone who read this far) one of the reasons why I have mixed opinions about this upcoming sequel trilogy is for the same reason I roll my eyes at a lot of the EU stuff. While entertaining, at the end of the day anything that happens after "Return Of The Jedi" is going to be a pretty big downer because by definition the universe should be heading towards a cycle of darkness. You've basically got two real options here if you want a true Star Wars sequel. One is to get deeply metaphysical with the entire thing, and the other is to create a real downer of an ending. Either way is likely to see a fan revolt, and I suspect is why Lucas tended to stay away from it. After all half the point is that when you don't peer into the nuts and bolts of "why" this is a pretty straightforward space fantasy story about heroes and villains and can be appreciated without (and perhaps better by) not really understanding the logic. Your looking at a situation where a true Star Wars sequel that directly continues this story (as opposed to skipping ahead to the end of the next age of darkness) pretty much ends with the somewhat trippy resolution of people gaining free will (or freedom from control), or the bad guys completing their dark prophecy and pretty much dancing the lambada on top of the corpses of all the heroes that tried to stop them as the universe turns into some occult nightmare of depravity for the next several thousand years.