"But a personal boycott is not censorship because the reader isn't saying author A with the bad attitude shouldn't be published... Deciding not to read a particular author or buy a particular author does not suppress the author's right to publish (particularly in this day and age). Concerted boycotts can have that effect but only if the numbers are large enough which is highly rare in the case of a reader and an author."
-- Jane Litte, Dear Author
In short, being boycotted won't destroy your career, so stop claiming that it will. Only what YOU say/do will destroy your career. But that's just my opinion.