Really the boundary between science fiction and fantasy is very blurred. There are valid arguments that Star Wars fits both genres.
First, readers will assume that the world is like reality unless noted. That is, people behave like people in the real world, they have emotions and believable reactions. Gravity pulls you down, people breathe air, and if you get injured badly enough you will die.
Second, whenever you cross the generally-understood boundaries of reality, you need to provide some sort of explanation, even if it's just a vague handwave. "The Force gives a Jedi his power. It's an energy field created by all living things." Oh, OK. That's why you can sense people and move objects with your mind. Or, "Our FTL drive can transport us instantly, but if you go too far in one jump you risk destroying the ship."
What irritates the audience is when the reality of the fictional world is different from ours without any explanation, and when the fictional rules the author creates are applied inconsistently. If gravity or the sun is different, then you can explain it by saying, "The story takes place on another planet." Or if time travel exists in your world, then it needs to be consistent in how it works (88 miles per hour, 1.21 gigawatts of electricity) and the effects of it (changing your past can cause you to erase your own existence).