Go to the appropriate folder on your hard drive and delete the plugin file.

Here are the locations of the folders for the various plug-in formats:

AU
MacHD/Library/Audio/Plug-Ins/Components

AAX
MacHD/Library/Application Support/Avid/Audio/Plug-Ins/Antares Audio Technologies

RTAS/TDM
MacHD/Library/Application Support/Digidesign/Plug-Ins

VST2:
MacHD/Library/Audio/Plug-Ins/VST

VST3:
MacHD/Library/Audio/Plug-Ins/VST3