Back on Topic
It sounds to me like you're having trouble with the program for Mouse Acceleration.
Skyrim has a specially designed Mouse Acceleration program that I'm pretty sure Oblivion and Morrowind did not.
To the best of my understanding, the purpose of the mouse acceleration was to determine your direction of movement after a certain length, and accelerate it. Ideally, you'd be able to make small changes with your mouse in order to do things with a great amount of control, but still be able to turn around rapidly (since normally in games either its one or the other).
Unfortunately, it doesn't really work. So, for some people, it does nothing, while for other people, it messes up their mouses.
Since this program is in Skyrim's programming, it would apply, regardless of what controller you use.
Here is a fix, copied from Reddit.
Go to \Users\[Username]\My Documents\My Games\steam/steamapps/Skyrim
Open the SkyrimPrefs.ini file with notepad
Find the line bMouseAcceleration=1 (located under [Controls])
Replace the 1 with a 0
Save the file (make sure you don't accidentally save it as Skyrim.ini.txt)
(steam may be located in your program files, if you can't find it under My Games)
As for why the joystick or xbox controller don't work... no idea. I use my touchpad.
Also, IMO, the scale in Skyrim is a bit wonky at times. That is, hallways take longer/shorter to navigate than it looks. This means that its sometimes easy to misjudge how far away things are, and how fast you are moving.
For the most part, this is negligible, but I find Helgen Keep to be particularly bad. I felt the controls were miserable, until I managed to get outside, at which the game felt more "controlled." I figured that it was because I was getting the hang of the controls, but each time I make a new character, Helgen Keep feels like an absolute pain to navigate.
Grit your teeth, let the npc do most of the fighting, and get through Helgen Keep. Once you're out in the open world, then decide if the controls are terrible, or if they're just bad in the starting area.
One final note: "allies" always get in your way when you're fighting. Hitting your teammates is unavoidable. Solution is to travel alone.