Don't think you can really get concrete opinions from family guy, which at one time or another can be viewed as offensive to everyone, different religions, races and so on but that's what like south park makes the show funny that it's not worried about doing something offensive if they think it's funny. Though it's not nearly as offensive as south park at times and doesn't focus specifically on one target like tom cruise or Scientology.
For a man born over 100 years ago you can't base most of his decisions on modern standards or opinions, sadly racism was alot more accepted and generalized but it's a different time before the second world war, yes we can say it's bad now but at the time who can say what it looked like. It's all just rumor and suspicion there are no concrete facts.