The reason why there's so much baggage with the term is that its mention almost immediately elicits notions or expectations of collective guilt and that tends to completely distort how the root issues are conceived. This doesn't mean these problems shouldn't be discussed or overshadowed by economic concerns, but by framing them this specific way the desired 'ends' to the cause are lost.
Disregarding the misfortunes of a white person is no more justified than celebrating or tolerating the mistreatment of minorities. That's not integration, that's tribalism. It's an ugly vice that hasn't uplifted a single person regardless of race.
I will say, however, that two areas where there is real "white privilege" is in areas of school discipline and criminal sentencings. Recent exposes of the Nixon years reveal his racial intentions in instituting the War on Drugs. Reforms are long overdue in many aspects of criminal sentencing and criminal enforcement.