When trying to customize the forms for a document library, you will find that you have no ‘Customize Form’ button in the Library settings or in SharePoint Deisgner 2010. Apparently, this is by design. Even if you go to Site Settings/General Settings/Form Settings, as you have probably found already, you will get the message in the subject line.

You can customize the forms for a list/custom list, but the default behavior in SharePoint 2010 is that this functionality is not available.