Mandatory Transport Layer Security (TLS) connection encryption

Topic 70171 | Email Encryption | TRITON AP-EMAIL | Updated: 12-Oct-2015


Applies To:	TRITON AP-EMAIL v8.1

TLS is an Internet protocol that provides security for all email transmissions—inbound, outbound, and internal. The client and server negotiate a secure connection for the transmission to occur, provided both the client and the server support the same version of TLS.

TRITON AP-EMAIL uses mandatory TLS as its default encryption method (enabled on the Settings > Inbound/Outbound > Encryption page.) Opportunistic TLS is used for other email protection functions.

This article offers some TLS basics, including how and where TLS is used in TRITON AP-EMAIL. It also covers TLS certificate handling, along with encryption key and certificate signing request (CSR) generation.

TLS Overview

TLS provides an extra layer of security for email transmissions. With this protocol, email communications can be encrypted to prevent devices such as non-trusted routers from allowing a third party to monitor or alter the communications between a server and client. TRITON AP-EMAIL can receive messages transferred over TLS and can also send messages via this protocol to particular domains. The email protection system uses a TLS encryption level of 128 bits.

Two levels of TLS are used in mail routing and email encryption functions. Opportunistic TLS can be enabled and used to protect email transfer communications during the message routing process and when using a third-party application for email encryption. Mandatory TLS is used for both the TLS and email hybrid service advanced email encryption options. You can also specify that connections to or from a specific IP or domain group use mandatory TLS via enforced TLS connection options (Settings > Inbound/Outbound > Enforced TLS Connections). Configure the security level and encryption strength for the connection on this page as well.

Opportunistic TLS

With opportunistic TLS, if a connection attempt is made using the TLS protocol, the connection recipient must provide appropriate TLS credentials for an encrypted data transfer. If the TLS "handshake" fails, the data transfer is made via plain text, rather than encrypted text. In either case, the data transfer is successfully accomplished.

Opportunistic TLS is used for message routing transfers. Create a new route that uses the TLS delivery option or edit an existing mail route to add the TLS option on the Settings > Inbound/Outbound > Mail Routing > Add (or Edit) Route page. At the bottom of the page, mark the Use Transport Layer Security (TLS) check box to use opportunistic TLS for message routing.

The third-party application message encryption feature also uses opportunistic TLS for data transfer security. Third-party application encryption options are configured on the Settings > Inbound/Outbound > Encryption page.

Mandatory TLS

As with opportunistic TLS, an encrypted data transfer occurs when the TLS handshake process is successful. Unlike opportunistic TLS, if the handshake fails during the connection attempt, the connection is terminated and no transfer occurs. The message is placed in a delayed message queue for a later delivery attempt. The message delivery retry interval is configured in the Settings > Inbound/Outbound > Non-Delivery Options page.

Mandatory TLS is used for the following encryption options:

Transport Layer Security (TLS) connection encryption

Email hybrid service advanced email encryption

These features are enabled and configured on the Settings > Inbound/Outbound > Encryption page. If you want to use advanced email encryption, your product subscription must include both the Email Hybrid Module and the Email Encryption Module.

Backup encryption options may be selected if you use default TLS encryption. You can designate either advanced email encryption or a third-party application as a backup method, in case the TLS connection fails. Specifying a backup option allows you a second opportunity for encryption in the event of an unsuccessful TLS connection. If both the TLS and backup connections fail, the message is sent to a delayed message queue for a later connection attempt.

TLS Certificates

TRITON AP-EMAIL enables a default self-signed TLS certificate with product installation that is used for incoming connections. The email protection system presents this certificate during TLS communications.

You can view certificate information and generate a new self-signed certificate on the Settings > Inbound/Outbound > TLS Certificate page. You should note that generating a new certificate overwrites any certificate that already exists.

You can also use a certificate from a third-party certificate authority (CA) for outgoing connections. TRITON AP-EMAIL uses CA-issued root and intermediate certificates (along with the default CA certificate bundle) to verify a server certificate presented by a third-party mail server during TLS communications. You need to generate encryption keys and a CSR to send to the CA and then import the purchased certificate files to the Email module.

Because the email hybrid service advanced email encryption option does not perform properly with the self-signed certificate, a trusted third-party certificate from a CA is required. (See Trusted third-party certificates for a list of trusted third-party certificates to use with advanced email encryption.)

The following sections provide details about generating encryption keys and a CSR and importing a third-party certificate to the Email module.

Generating encryption keys and a CSR

The process for generating encryption keys and a CSR involves the use of the OpenSSL tool, which is available with your installation of TRITON AP-EMAIL.

You can generate a CSR using the following steps:

Log on to the TRITON Manager Email module and open a command-line interface as administrator. Navigate to the installation path (by default, C:\Program Files (x86)\Websense\EIP Infra\apache\bin\openssl).

Execute the following command to create private encryption keys:

openssl genrsa -des3 -out certificate.key 2048

In this example command, the private keys are output to a file named certificate.key, and the key size is 2048 bits.

Set a password for the key file when prompted (maximum length is 100 characters).

Use the following command to generate the CSR, which contains the private encryption key file you just created:

openssl req -new -config "C:\Program Files (x86)\Websense\EIP Infra\apache\conf\openssl.cnf" -key certificate.key -out certificate.csr

In this example, certificate.csr is the name of the CSR file.

When prompted, enter the password you created in step 3.

Supply the following information when prompted:

Country Name (2-letter code), example: US

State or Province Name (full name), example: Texas

Locality Name (e.g., city), example: Austin

Organization Name (e.g., company), example: Websense

Organizational Unit (e.g., section), example: Sales

Common Name (e.g., server hostname), example: email.websense.com

Email Address, example: sales@websense.com

Challenge password


	Important The value for Common Name must match the fully qualified domain name (FQDN) of the email protection management server. You may receive certificate errors if you do not specify the FQDN here.

Send your CSR to a CA for signing. Secure your private key file and passwords; you will need them in order to use your certificate.

Importing certificate files

The option to import a certificate from a CA is available in the TRITON Manager Email module console. Importing a third-party certificate overwrites an existing certificate.

Import a new certificate as follows:

Open a command prompt on the TRITON Manager server and run the following command to create a .pfx file that contains your certificate and key:

C:\Program Files (x86)\Websense\EIP Infra\apache\bin> openssl pkcs12 -export -inkey certificate.key -in [certificate file] -out certificate.pfx

When prompted, enter your private key password and set a password for the .pfx file using only alphanumeric characters.

On the Settings > Inbound/Outbound > TLS Certificate page, click Import.

Click Yes in the confirmation dialog box. An Import Certificate area appears below the Import button.

Use Browse to navigate to your third-party certificate (.pfx) file. When you select the file, its filename appears in the Certificate file field.

Enter the password you specified in your CSR (maximum length is 100 characters).

Click OK.