Mandatory Transport Layer Security connection encryption

Email Encryption | Forcepoint Email Security | Updated: 29-Apr-2022


Applies To:	Forcepoint Email Security v8.5.x

Transport Layer Security (TLS) is an Internet protocol that provides security for all email transmissions—inbound, outbound, and internal. The client and server negotiate a secure connection for the transmission to occur, provided both the client and the server support the same version of TLS.

Forcepoint Email Security uses mandatory TLS as its default encryption method (enabled on the page Settings > Inbound/Outbound > Encryption). Opportunistic TLS is used for other email protection functions.

This article offers some TLS basics, including how and where TLS is used in Forcepoint Email Security. It also covers TLS certificate handling, along with encryption key and certificate signing request (CSR) generation.

TLS Overview

TLS provides an extra layer of security for email transmissions. With this protocol, email communications can be encrypted to prevent devices such as non-trusted routers from allowing a third party to monitor or alter the communications between a server and client. Forcepoint Email Security can receive messages transferred over TLS and can also send messages via this protocol to particular domains. The email security system uses a TLS encryption level of 128 bits. There is no limit to the encryption key size that can be used.

Two levels of TLS are used in mail routing and email encryption functions. Opportunistic TLS can be enabled and used to protect email transfer communications during the message routing process and when using a third-party application for email encryption. Mandatory TLS is used for both the TLS and Forcepoint Email Encryption options. You can also specify that connections to or from a specific IP or domain group use mandatory TLS via enforced TLS connection options (Settings > Inbound/Outbound > Enforced TLS Connections). The page Enforced TLS Connections is additionally used to configure the security level and encryption strength for the connection.

Opportunistic TLS

With opportunistic TLS, if a connection attempt is made using the TLS protocol, the connection recipient must provide appropriate TLS credentials for an encrypted data transfer. If the TLS "handshake" fails, the data transfer is made via plain text, rather than encrypted text. In either case, the data transfer is successfully accomplished.

Opportunistic TLS is used for message routing transfers. Use the page Settings > Inbound/Outbound > Mail Routing > Add (or Edit) Route to create a new route that uses the TLS delivery option or edit an existing mail route to add the TLS option. At the bottom of the page, mark the check box Use Transport Layer Security (TLS) to use opportunistic TLS for message routing.

The third-party application message encryption feature also uses opportunistic TLS for data transfer security. Third-party application encryption options are configured on the page Settings > Inbound/Outbound > Encryption.

Mandatory TLS

As with opportunistic TLS, an encrypted data transfer occurs when the TLS handshake process is successful. Unlike opportunistic TLS, if the handshake fails during the connection attempt, the connection is terminated and no transfer occurs. The message is placed in a delayed message queue for a later delivery attempt. The message delivery retry interval is configured on the page Settings > Inbound/Outbound > Non-Delivery Options.

Mandatory TLS is used for the following encryption options:

TLS connection encryption

Email hybrid service advanced email encryption

These features are enabled and configured on the page Settings > Inbound/Outbound > Encryption. To use Forcepoint Email Encryption, your product subscription must include both the Forcepoint Email Security Hybrid Module and the Forcepoint Email Security - Encryption Module.

Backup encryption options may be selected if you use default TLS encryption. You can designate Forcepoint Email Encryption, a third-party application, or secure messaging as a backup method in case the TLS connection fails. Specifying a backup option allows you a second opportunity for encryption in the event of an unsuccessful TLS connection. If both the TLS and backup connections fail, the message is sent to a delayed message queue for a later connection attempt.

TLS Certificates

Forcepoint Email Security enables a default self-signed TLS certificate with product installation that is used for incoming connections. The email security system presents this certificate during TLS communications.

You can view certificate information and generate a new self-signed certificate on the page Settings > Inbound/Outbound > TLS Certificate. Generating a new certificate overwrites any certificate that already exists.

You can also use a certificate from a third-party certificate authority (CA) for outgoing connections. Forcepoint Email Security uses CA-issued root and intermediate certificates (along with the default CA certificate bundle) to verify a server certificate presented by a third-party mail server during TLS communications. It is necessary to generate encryption keys and a CSR to send to the CA and then import the purchased certificate files to the Email Security module.

Because the email hybrid service Forcepoint Email Encryption option does not perform properly with the self-signed certificate, a trusted third-party certificate from a CA is required. (See Trusted third-party certificates for a list of trusted third-party certificates to use with advanced email encryption.)

The following sections provide details about generating encryption keys and a CSR and importing a third-party certificate to the Email Security module.

Generating encryption keys and a CSR

The process for generating encryption keys and a CSR involves the use of the OpenSSL tool, which is available with your installation of Forcepoint Email Security.

You can generate a CSR using the following steps:

Log on to the Security Manager Email Security module and open a command-line interface as administrator.

Navigate to the installation path (by default, C:\Program Files (x86)\Websense\EIP Infra\apache\bin\openssl).

Execute the following command to create private encryption keys:

openssl genrsa -des3 -out certificate.key 2048

In this example command, the private keys are output to a file named certificate.key, and the key size is 2048 bits.

Set a password for the key file when prompted (maximum length is 100 characters).

Use the following command to generate the CSR, which contains the private encryption key file you just created:

openssl req -new -config "C:\Program Files (x86)\Websense\EIP Infra\apache\conf\openssl.cnf" -key certificate.key -out certificate.csr

In this example, certificate.csr is the name of the CSR file.

When prompted, enter the password you created in step 3.

Supply the following information when prompted:

Country Name (2-letter code), example: US

State or Province Name (full name), example: Texas

Locality Name (e.g., city), example: Austin

Organization Name (e.g., company), example: Forcepoint

Organizational Unit (e.g., section), example: Sales

Common Name (e.g., server hostname), example: email.forcepoint.com

Email Address, example: sales@forcepoint.com

Challenge password


	Important The value for Common Name must match the fully qualified domain name (FQDN) of the email security management server. You may receive certificate errors if you do not specify the FQDN here.

Send your CSR to a CA for signing.

Secure your private key file and passwords; you will need them in order to use your certificate.

Importing certificate files

The option to import a certificate from a CA is available in the Security Manager Email Security module. Importing a third-party certificate overwrites an existing certificate.

Import a new certificate as follows:

Open a command prompt on the Security Manager server and run the following command to create a .pfx file that contains your certificate and key:

C:\Program Files (x86)\Websense\EIP Infra\apache\bin> openssl pkcs12 -export -inkey certificate.key -in [certificate file] -out certificate.pfx

When prompted, enter your private key password and set a password for the .pfx file using only alphanumeric characters.

On the page Settings > Inbound/Outbound > TLS Certificate, click Import.

A confirmation dialog box displays.

In the confirmation dialog box, click Yes.

An Import Certificate area appears below the Import button.

Use Browse to navigate to your third-party certificate (.pfx) file.

When you select the file, its filename appears in the Certificate file field.

Enter the password you specified in your CSR.

The maximum length is 100 characters.

Click OK.