I don't take my shoes off in my house unless I'm putting my feet up - in bed, in a recliner, in a bath etc.
I can see why you don't want to track in dirt to your house, but that is why there are doormats. To get rid of the dirt before you go into the house. Also, broken glass does not care whether it is on an inside floor or on the ground outside. It is still broken glass and will cut you if you step on it.
Maybe you don't have a lot of broken glass on your houses floor, but it's still better to have a shoe to protect from, say, banging your toe against the table.