Generally the argument is that the exceptions are not considered purely "speech" and are therefore not protected. Incitement, fighting words, libel, fraudulent inducement, conspiracy, etc. do involve communication, however they also contain more: a guilty mens rea. This guilty mental state is a necessary component of the exclusions, as it effectively creates a different act separate from speech. For example, the first amendment protects lies but not lies made with the intent to defraud someone into giving you money. Similarly, the first amendment protects offensive speech but not individually directed offensive speech made with the intent to trigger an immediate physical fight.
To some extent, its just semantics, but since the exceptions are few enough and rooted in historical common law, and there is no judicial power to recognize new exceptions, the definition of speech effectively excludes the combined act of speech + guilty mental state. Its why the first stolen valor act (criminalizing lying about military service) was struck down by scotus, but the revised stolen valor act (criminalizing lying about military service with the intent to benefit financially through this lie) is probably ok.
