It is nearly instead of always because of cases in which it conflicts with some very important rights of others. And in the specific case of abortion it is not at all clear why we should consider unwanted yet-to-be persons part of any kind of moral community. They aren't even indifferent.
If it is "very important rights" that can override the obligation not to kill, that just means there are other rules in your moral view that are of higher priority. Circling back to the ideas of consequentialism and utilitarianism, it doesn't seem like yours (or almost anyone's) perspectives actually fall into those. You're basically just describing your own personal deontological system. Subjective qualifiers like "relevant" or "important" are just putting your moral hierarchy vaguely one step removed from outright saying "these are my commandments".
I do not believe there is a logical system guiding this stance on abortion, I think you take that as axiomatic truth and rationalize around it. For example, in any other situation, you would never consider "wanted" or "unwanted" as a rationale for why killing someone is justified. And I don't think you're taking a consequentialist perspective, as a person's desires in advance are not necessarily related to outcome at all. Uncountable numbers of people have had unplanned pregnancies lead to great joy (and the opposite, for what it's worth), and many people don't want to do things that are objectively good for them. The logic doesn't hold up, what you're saying only really fits together as a list of rules you happen to hold as valid, a list that happens to carve out the allowance of abortions, but which has no broader validity than the hypothetical person you describe imagining their moral view is universal.