How do I debug a custom action/installer class?

You can use one of the following methods:
Add a call in your code to System.Diagnostics.Debugger.Launch. This method opens Just-In-Time debugging and allows you to attach a new debugger to your code.
Add a call in your code to MessageBox.Show("Debug Me"). When the message box is shown, use Visual Studio to attach to the MessageBox process. Then place breaks (for Visual C# projects) or stops (for Visual Basic projects) in the code.
Set your debugging preferences to start InstallUtil.exe (which is located in \winnt\Microsoft.net\Framework\version) and pass it your assembly as a parameter. When you press F5, you hit your breakpoint. InstallUtil.exe will run your custom actions the same way that MSI does.