There is at core a problem here. In my country, an elite course would probably be asking for about 120 points at school-end exams, moderate ~90, and low ~60. But we also know from research that a student who pays for an independent school education scores on average ~16 points more than a student of equal talent from from the state education system. Basically just because their expensive school is better. So we know perfectly well that there is a fundamental bias in university admissions due to differences in school quality. Particularly when we look at courses with high entry requirements (thus elite universities with high requirements for any course at all), students from bad schools are disadvantaged - and of course the worst schools tend to be in the poorest areas. This means that poorer students (and de facto most middle class too, also being in state schools) are systemically disadvantaged in higher education access.For example, the systemic college process of having lower tester score requirements for admittance from Black students. This was to help black students who often didn't come from great school districts or perhaps didn't bother to pay attention, still get into college. Now is this racist? Sure if you look at it under the assumption that Black people are I guess supposed to be dumber than everyone else therefore they can't possibly match the test scores of other people therefore you lower it just for them and essentially hold them to a lower standard because they can't meet the normal ones, thus labeling them as inferior. You could look at it that way. The other side of that is the higher test score requirement for Asian students, because they are often much higher in academic scores that the normal would be too easy for them so the requirement is raised so that the whole college isn't just Asian kids.
To translate this to the context of race in the USA, black people are on average much poorer than white people. Thus a rule that grants black people easier admission is not quite so unjustifiable as you might think. It is however very clumsy, because it benefits a substantial number of black people who don't need it and provides no assistance to white students who should have some. On the other hand, this sort of correction is very difficult, and perhaps given the racial complications the USA's history has created, it is not a surprise this has been an angle often chosen.
To give an idea of how hard it can be, again let's go back to the UK. Some universities trying to record how many students are from disadvantaged backgrounds have tried using a "postcode" system. Take the average wealth of a locality, and if it is one of high deprivation, classify all students from there as from a deprived background. But now imagine a rural location with high poverty containing many poor agricultural workers, a few wealthy farmers, and an aristocrat in a country mansion. The aristocrat and the farmers send their kids to uni and non of the agricultural workers do, but all of those students classify on the statistics as "deprived", even though none of them are anything like.