At the end of the day, a writer's chief responsibility is to tell a compelling story. There are a lot of ways to do that, but one of the ways is to tell the story from a fresh or original perspective, so designing characters as people from often unrepresented groups will potentially offer an easy way to generate this. From a purely neutral perspective, diversity matters just so that you can tell a better story. It allows so many more options, views, tricks, and meat. Heck, someone mentioned Spiderman Homecoming: that story builds in a great little twist towards the end, and it works entirely on exploiting the audience's preconceptions about race. It couldn't have happened if they went the way the comics did and made every character white.
From a less neutral perspective, diversity matters because fiction is rife with lazy stereotypes, generic protagonists and a whole bunch of biases. This is a natural result of the above issue, with writers not thinking enough about how to produce their characters. OP brings up lots of hardboiled literature, which is a genre full of negative depictions of foreigners, homosexuals, women; basically anyone who isn't the tough as nails, manly white hero. These guys exist to serve as negative comparisons to the magnificence of our anti-hero lead. Most modern interpretations of these hardboiled stories have to either shine a light on the inherent prejudice, write it out, or replace it with something kinder. Diversity by way of shortcuts is not diversity at all, and should be avoided.
Finally, I advise writers these basic tips to not fuck up writing race into their stories:
* Unless you have a narrative reason, don't mention a character's race. Writers have a bad habit of trying to work diversity into their story by describing lots of non-white characters. The problem with this is that they do this selectively, always quick to point out when a character is asian, but never points the white ones. The result is weird "spot the minority" type descriptions which treat racial minorities as exceptional, when the narrator is aiming to do the opposite. If there is no reason to mention a character's race, do not mention a character's race. If you are going to mention a characters race, have a reason to do so. It doesn't have to be a big one, the reason could simply be that you want to tell a joke about it, or to inform the reason for why they do something, or to reflect on the kind of the world the protagonist lives in. In Rivers of London, the narrator indicates he is black through a satirical remark about how he's worried about being used as a token by his employer. In Worm, the protagonist mentions another character is black for the first time, when she becomes conscious of his race as they walk through a white neighbourhood. JK Rowling never mentions the race of her characters (though she leaves obvious clues) because there is no need to.
* Never, ever emulate someone's accent within the dialogue. This still happens, where a character suddenly starts "Talkin' in
a reg-eoon-al accsunt loik?" It's annoying to read, it's patronising, and it implies that anyone who doesn't have their words spelt phonetically must be speaking implausibly perfect received pronunciation. Unless your Irvine Welsh, don't do it. Just describe the character as having the accent - we can imagine the rest.
* Be wary of your inspirations. So you've seen a cool person or character elsewhere and you want to work them into your story. Great! That is until someone points out to you that your character is acting in a very stereotypical manner. It happens all too often where I criticise descriptions of "larger than life blackman" and "silent, smart Asian woman", only for the writer to point out they've based them on real people they know. The reader hasn't met these people so won't accept that defence. If you want these characters, acknowledge within the story the fact that they are behaving like a stereotype (which is a small step towards subverting it). The Thick of It has several stereotypical angry Scotsmen in it (based on real people) but subverts it by having those characters know full well they are acting stereotypical, and are in part playing up to it to their advantage. Alternatively, avoid the stereotype entirely and rewrite the character.