I suspect that, to some degree, shadeism would continue to exist without white people at all. Divide and conquer is an old game, not unique, I'm sure, to the people of any particular race or ethnicity. And psychologically, if they can look down on someone (even for the dumbest of reasons), people may feel better about themselves by comparison.
Naturally, "divide and conquer" worked for white nations seeking power that way. But absent that, someone would've found some other dividing trait. I think colorism's just built into the human algorithm, now, no matter where it came from, and won't go away even if white people do.
It's always struck me as supremely odd that swarthy, little Adolf Hitler rose to his genocidal power promoting the ideal of the "Aryan" man (and woman - if only to breed more of the men). The olive skin in my family comes from my German ancestry - we got the shorter of stature and darker skinned traits (with interestingly notable exceptions in me, two of my cousins - younger of two siblings each, and my son - also the younger of two siblings).