You're inventing a world full of people to pity, a world full of people with no agency in their lives that are clearly not you or anyone you know, because no reasonable person could think of someone they've met that way.
You know, I could tell you all kinds of things about my life or the lives of people I know, but let's cut straight to the basic misunderstanding. This isn't how agency works. A person doesn't just "have" or "not have" agency. Agency is mediated. It's like a tabletop RPG. You can make decisions about what your character does, but you're still bound by the rules of the game and sometimes what you want will come up against what the GM wants. Your ability to act is limited, but that's what makes it fun.
In fact, it's more complicated than that, because in an RPG you get to make your character. You decide what kind of person they are, how they feel and react to different situations, you can pick whatever childhood you want for them and decide for yourself how it impacted them. In real life, you don't decide any of those things.
The question of freedom is not whether someone does or doesn't have agency in some absolute sense, but how much agency they have and, in particular, how much agency they have relative to everyone else. When I say that most people aren't free, I don't mean they are literally slaves, and even if they were literally slaves, they would still have a degree of agency. They could resist, they could run away, they could (and in some cases did) face incredible risk and hardship in order to escape slavery. A very cynical and morally bankrupt person who bears no relation to any real person on this forum could use this fact to argue that slaves who didn't resist were
choosing to be slaves, and that they must have been happy to be slaves. After all, no reasonable person could think that slaves don't have any agency.
Everyone has choices in their lives, choices that would denote agency, but most people's choices are limited by the position they are in. If your ability to live is dependent on working, then the choice not to work is somewhat academic even if it exists. Again, it doesn't actually matter if you hate your job, you still have to do it. You don't have a choice, not really.
And that isn't intrinsically bad, except that some people do have a choice. Some people don't ever face the risk of not having enough food or of being homeless. Some people are born into or stumble into such fabulous, incomprehensible wealth that their descendants will never have to work for thousands of years. Worse, those people are effectively monopolizing the ability to choose. Their wealth comes from the ability to strip others of any choice save working for their benefit or facing destitution.