Yes, quite often you can. I have helped several clients reduce and/or eliminate these penalties. There are several provisions in the IRS tax code that allow for penalty abatement. In some instances, abatement is mandatory when you provide the correct information.
You need to have someone who understands how the IRS works help you navigate through their confusing letters and processes. Whatever you do, don’t sign away any of your taxpayer rights without first seeking professional advice on your particular situation. You can even seek penalty abatement as far back as 2 years from the date the penalties were actually paid.