The "penalty", as I understand it, is neither a fine nor a record of wrongdoing, as paying at the end of the year is not illegal. The penalty is the IRS saying that it would like to have the interest back that it otherwise would have earned if you had paid earlier.
As far as I understand, it's perfectly kosher to wait until the end of the year to make payments; you just have to make sure you have the money to actually pay then.
Disclaimer: I am not an accountant nor do I have expertise on this matter.
It's a little more complicated than that, and it is a penalty. You may also owe interest, and they are two separate things (and interest is statutory, so you're really not negotiating there). There are multiple penalties for underpayment and nonpayment of federal taxes. If you're an employee and not withholding enough (because you have other income or AMT for example), you should be making estimated payments to cover the difference. Your employer is also subject to penalties if it fails to withhold and pay the gov't employment taxes (social security and medicare), and it's usually 100%. If you are self-employed, you need to make estimated payments.
tl;dr: the gov't has cashflow problems, too, and doesn't like to issue t-bonds to finance its receivables.
As far as I understand, it's perfectly kosher to wait until the end of the year to make payments; you just have to make sure you have the money to actually pay then.
Disclaimer: I am not an accountant nor do I have expertise on this matter.