Well, I stopped watching after the end of Season 3, so I didn't help its ratings in any way.
Not having seen Season 4, I can't say what effect Dan Harmon's removal had on the show, but with the way the third season went, I don't know if all the blame should be put on the studio who removed him...