Flat out, I wouldn't vote for someone who moved to my district from halfway across the state for the sole reason of political opportunism. I have a feeling there are a lot of voters in CA-04 who feel the same as I do.
McClintock still lives in SoCal. There's no residency requirement (other than living in the same state) for House districts.
He moved his family up to Sacramento (Elk Grove, I believe) while in the state legislature. I believe he listed his mother's house in Ventura County as his official address.
So of course he got attacked for being a carpetbagger from L.A. who took per diem money he didn't deserve because he lived in Sacramento.