Didn’t sexual exploitation undermine and destroy the black family? Critics of the South have consistently answered in the affirmative. They accuse slave owners and overseers of turning plantations into personal harems. Again, unfortunately for the thesis, the evidence on which these assumptions and conclusions are based is extremely limited.
Such arguments overlook the real and potentially large costs that confronted masters and overseers who sought sexual pleasures in the slave quarters. It would have been much easier, and less risky, for owners of large plantations to keep a mistress in town than to risk the possibility of the destruction of his own family by taking up with a slave woman. For the overseer, the cost of sexual episodes in the slave quarters, once discovered, was often his job. Nor would he find it easy to obtain employment elsewhere as an overseer, since not many masters would be willing to employ as their manager a man who was known to lack self-control on so vital an issue.
Further, to imply that black men would be indifferent to the sexual abuse of their women is to imply that they were somehow less manly than other men who would be indignant over such abuse. This common assumption about slave men is not only unrealistic and unsubstantiated but an insult to their humanity and patently racist.
Source: Southern Slavery As It Was, Steve Wilkins & Douglas Wilson, P. 15