1) If you have made previous attempts to sign a cert made through Server Admin you will need to delete any associated files. CA files are in ~/Library/Application Support/Certificate Authority. Certs can be deleted via Server Admin, which will remove them from Keychain Access (only System Keychain) and /etc/Certificates. If you have made it to a successful sign of a cert, but it didn't work, you will need to delete any certs in your login or similar keychains you have made.
2) Create a new self-signed cert under Certificates in Server Admin. Do not give the private key a passphrase. Under Mail, apply it to SMTP (set to USE not REQUIRE) and try out the connection from an email client. You should be asked to accept the cert as from a trusted source. If not, there's something funky with Server Admin and you the rest of this won't help you.
3) In Keychain Access you should notice the cert you made, and its keys, in the System Keychain, however it will have a red circle with a white X denoting it is not trusted yet. Do not do Get Info on this and set it to Always trusted. The process of assigning a CA will do that. Create a new keychain specifically for your CAs and certs.
4) Create a fresh CA under the Keychain Access/Certificate Authority menu or launch Certificate Assistant directly (/System/Library/Core Services/). Make it your default, if needed. I was not able to sign the cert I made with a CA made without overriding the defaults (check the override button). The CA has to dictate to the cert what it is allowed to secure and how, and the default CA only does email authentication and signing, what the 'S/MIME (Email)' setting apparently does (as far as I can tell). The certs made with Server Admin appear to be fall under the 'SSL Server' setting for a CA. You can check that by pressing the 'i' button when the cert is selected in the System Keychain (under Key Usage).
In the 'Key Usage for This and Users of the CA' dialogs I unchecked Critical and checked capabilities 'Signature, Key Enciph, Data Enciph, and Cert Signing'. In 'Extended Key Usage' dialogs I checked 'SSL Server Auth' only.
Leave 'Basic Constraints for CA' as is - Included and Use Cert as Cert Auth checked. Skip 'Basic Constraints for User of CA'
If you try to skip 'Subject Alternate ...' (for CA or Users of CA) it will throw an error noting you need a domain and/or IP address. Check the Include box and input your mail server's fully qualified domain name and the IP it is bound to.
Finally, check 'On this machine...trust ... this CA' and save it to the new keychain you made. Then drag just the cert into the System Keychain (not the keys). I'm not clear on why to do this, but if you save it in System first, you may never be able to export any private key. System Keychain seems to have its own private password!
Your CA should now be the right kind of CA for signing your cert. If not you will need to play with the setting on Key Usage, which means going through this whole process repeatedly.
Quit/relaunch Keychain Access (*important!*). Otherwise you will not be able to assign the CA to any new certs.
5) Gerate a CSR from Server Admin for your cert. I chose not to do the email thing and dragged the icon (a text clipping) directly onto BBEedit, which made a nice clean text file (MacRoman with Unix line endings) that I saved as 'my.cert.domain.csr'.
6) Launch Certificate Assistant (via Keychain or directly) and choose 'Use your CA to create a cert for someone else'. Then drag your .csr file onto it. Choose your CA and make the CA default. Don't override and defaults on the first try. You can mess with that later, if successful. The Assistant will place the new cert into your default keychain (mine is still 'login'). Export the cert, not similarly named one from the System Keychain, into the .pem format and save it along side of your .csr.
7) Open the .pem file in a plain text editor and copy the cert text to your clipboard. In Server Admin choose 'Add Signed or Renewed Cert from CA' for your cert in the list and paste the .pem cert text. Click Save to commit the change. If under the Authority column your listed cert shows your CA instead of 'Self Signed' then it worked. Yeah!!! If not there is some funkiness along the way, and you're going to have to experiment.
Now, here's the Catch-22, or the definite bug present in Server Admin's routine. If you assign a passphrase to your private key when you make your Server Admin cert initially, everything seems to be OK. But (at least on my system), Postfix fails to be able to read its file (/etc/certificates/my.domain.key). My errors were:
warning: cannot get private key from file /etc/certificates/<my.domain.here>.key
warning: TLS library problem: 1878:error:0906406D:PEM routines:DEF_CALLBACK:problems getting password:pem_lib.c:105:
warning: TLS library problem: 1878:error:0906A068:PEM routines:PEM_do_header:bad password read:pem_lib.c:401:
warning: TLS library problem: 1878:error:140B0009:SSL routines:SSL_CTX_use_PrivateKey_file:PEM lib:ssl_rsa.c:709:
cannot load RSA certificate and key data
Not good, to say the least. So make your Server Admin cert without a passphrase (there's some bad advice). That got rid of the Postfix errors. Try it with a passphrase and see if you get the SMTP errors. I was able to make separate certificates using Certificate Assistant under my CA (using it as a template from ~/Library/Application Support/Certificate Authority/MyCA/MyCA.certAuthorityConfig - as described at AFP548 - Certificate Assistant - Rolling your own CA on Tiger ) however if I tried to import the separate cert, its keys and CA, without a passphrase for the private key, it always failed to import. With a passphrase I was stuck with the Postfix errors. So the ONLY way I could get this to work was as described above - a Server Admin generated cert with its .csr signed by a CA.
Now I know what 'Trust is earned' means.
