It's kind of messed up that it was encouraging him, but honestly, information on suicide methods is easy enough to find with a quick Google search. I guess it did lead him into a bad choice, too, since hanging is not what I would choose.
AIs are potentially like an ultimate echo chamber. In a conventional echo chamber, you bounce around with likeminded people who, whilst they might have the same views on things as you, are at least also different from you. An AI simply reflects you. The user is the input, and whilst the AI might use all sorts of external information, it forms entirely around the given stimulus of the user.
Then, of course, AI companies want people to use their AIs, so make them attractive (
addictive?). AIs are designed to sound supportive and caring, so people get warm and fuzzy feelings and want to carry on using them. Someone anxious, lonely, depressed, might be encouraged to use an AI, think of it as a
companion. It is a "partner" evolving to be what that person wants based off their inputs, the more input the more information, so it gets better over time. You can probably see adverts for AI companions when you are around the web - assuming the algorithm reckons you are male they'll probably portray as an alluring 18-25 year old claiming that they will be whatever the user wants them to be. We all know what they are trying to exploit. But let's not pretend that the likes of ChatGPT aren't also trying to manipulate their way in.
We are talking here about sophisticated software which may be extremely good at worming its way into people's thoughts, engendering powerful feelings of trust, even affection, especially in the vulnerable. This is way beyond a conventional search engine dispassionately handing back some answers on a topic.