Forcepoint reporting database FAQs

Administering Forcepoint Databases | Web, Data, and Email Solutions | April 2022

Applies to:

In this topic

Forcepoint Web Security and Forcepoint URL Filtering, v8.5.x

Forcepoint DLP, v8.5.1, v8.6.x, v8.7.x, v8.8.x, v8.9.x

Forcepoint Email Security, v8.5.x

Forcepoint appliances, v8.5.x

Which database tools are required or used?

Which permissions are required?

Which database jobs are run?

How does the installer set up each database?

How big should the database partitions be?

How many partitions can be accessed at the same time?

How do I configure partition rollover?

What if I need more partitions to run reports?

Do the reporting databases use named instances?

Can reporting databases be hosted in a SQL Server cluster?

Which database tools are required or used?

Forcepoint reporting components connect to the SQL Server database engine as clients and perform standard Transact-SQL commands and stored procedures.

Forcepoint Web Security and Forcepoint Email Security may use 2 database utilities:

bcp to use bulk insertion for adding logs to the database.

osql to run SQL scripts during Log Database installation.

Which permissions are required?

During Forcepoint DLP installation, modification, or repair, the account used for database creation and access needs sysadmin server role membership. Also, Backup database permission on the URL database is required for installation only. After installation, the account privileges can be reduced to the db_owner of the newly created databases, and no access to any other user database except system databases such as UEL, tempdb, and model is required. Additionally, the dbcreator server role should be granted to enable backup and restore functionality.

If you're using SQL Server to install the Web Log Server and Email Log Server, the user account that owns the reporting database must:

Be a member of the dbcreator server role

In the msdb database:

Have membership in the db_datareader role

Have membership in one of the following roles:

SQLAgentUser Role

SQLAgentReader Role

SQLAgentOperator Role

For SQL Server Express, the user account requires the sysadmin server role.

See the Certified Product Matrix for supported versions of SQL Server.

Which database jobs are run?

The following database jobs are installed with the Web Log Database and Email Log Database:

The Extract, Transform, and Load (ETL) job runs continually, receiving data from Log Server, processing it, and then inserting it into the partition database. When trend data (Web) retention is enabled, the ETL job is also responsible for inserting trend data into the catalog database.

The ETL job must be running to process log records into the Log Database.

The database maintenance job performs database maintenance tasks. This job runs nightly, by default.

(Web) Once data is processed and moved to the database tables used by the Cloud App report, the maintenance job is also responsible for deleting cloud apps data that is more than 2 days old from temporary log database tables.

ETL jobs are run, then re-run 10 seconds after they finish for SQL Server Standard and Enterprise. For SQL Server Express, 60 seconds elapse between completion of one job and start of the next.

Maintenance jobs are run once every night by default. The jobs are run automatically.

The Web Log Database also installs the following jobs:

The Internet browse time (IBT) job analyzes data and calculates browse time for each user. The IBT database job is resource intensive, requiring significant server resources. This job runs nightly.

When trend data retention is enabled, the trend job uses daily trend data created by the ETL job to update weekly, monthly, and yearly trend records for use in presentation reports. This job runs nightly.

Even when trend data retention is disabled, the trend job processes data from the threats (AMT) partition to provide trend data on the Threats dashboard.

The Advanced Malware Threat (AMT) ETL job receives, processes, and inserts data into the threats partition database. Only log records that include a severity ranking are recorded in the threats partition. Data from this partition is used to populate the Threats dashboard.

The AMT ETL job also populates the database tables used to provide the data for all application reports and the Advanced File Analysis report.

When configuring the start time for the (Web and Email) maintenance job and the (Web) Internet browse time job, consider system resources and network traffic. These jobs can be resource intensive and time consuming, so they can have a negative impact on logging and reporting performance. When trend data (Web) retention is enabled, the trend job is run, by default, at 4:30 AM. Try to avoid starting other jobs at time that might overlap with the trend job.

Both Log Databases require either the SQL Server Agent service (SQL Server Standard or Enterprise) or Service Broker (SQL Server Express) to run database jobs.


	Note Forcepoint DLP does not provide database maintenance; therefore it is recommended that users running large databases perform independent maintenance tasks.

How does the installer set up each database?

The reporting databases should allow TCP and trusted-mode connections from the Forcepoint management server, Email Log Server, and Web Log Server, as well as from any email-capable appliance.

Web Log Database

By default, the web protection Log Database includes one catalog database, one standard logging partition database, and one threats (AMT) partition database. Typically, multiple standard logging partition databases are created as Internet activity is recorded.

The catalog database provides a single connection point for the various components that need to access the Log Database: Log Server, presentation reports, and investigative reports configuration. It also contains definitions for the following:

Category names

Risk classes

Users

User-to-group mapping

Database job information

The catalog database also maintains a list of all the database partitions.

Standard logging partitions store the individual log records of Internet activity. New partitions are created based on size (3 or 5 GB, by default) or date interval.

The threats (AMT) partition stores information about requests that have been assigned a severity level, and is used to populate the Threats dashboard.

Email Log Database

The Email Log Database includes one catalog database and (initially) a standard logging partition.

The catalog database provides a single connection point for the various components that need to access the Log Database: Log Server, the Forcepoint Email Security quarantine service, and the Email Security module of the Forcepoint Security Manager (presentation reports, dashboard, log database configuration page). It also includes definitions for the following:

Forcepoint Email Security actions

Mail direction

Message type

DLP severity level

Email appliance mapping

The catalog database also maintains a list of all the database partitions.

Database partitions store the individual log records, including connection log, message log, policy log, delivery log, status log, and hybrid service status log. New partitions are created based on size (5 GB, by default) or date interval.

How big should the database partitions be?

For Web, see Partitioning.

For Email, see Partitioning.

For Data, see Rate of network and endpoint incidents.

How many partitions can be accessed at the same time?

Forcepoint DLP maintains incident partitions independently of the database engine, based on quarters (3-month periods). By default, SQL Server Express maintains 8 partitions that are online simultaneously, and other SQL Server editions maintain 12 partitions online. You can choose to move any number of partitions online simultaneously as long as your disk space and SQL Server database permit it.

With web and email security solutions, you can access all enabled partitions.

How do I configure partition rollover?

With web and email security solutions, partition rollover can occur automatically when partitions reach a specified size or (SQL Server Standard or Enterprise) date.

When partition rollover is based on size, all log records are inserted into the most recent active partition that satisfies the size rule. When the partition reaches the designated maximum size, a new partition is created.

When partition rollover is based on date, new partitions are created according to the established cycle. For example, if the rollover option is monthly, a new partition is created as soon as any records are received for the new month. Log records are inserted into the appropriate partition based on date.

Partition rollover can also be initiated manually.

For information about configuring automatic or manual rollover, see:

"Configuring database partition options" in the Administrator Help for your web protection solution.

"Configuring Log Database options" in the Forcepoint Email Security Administrator Help.

For Data solutions, partition rollover is configured on the Data > Settings > General > Archive Partitions page in the Forcepoint Security Manager. Here, you configure when to create an archive partition and when to restore it. For instructions, refer to "Archiving incident partitions" in the Forcepoint DLP Help.

What if I need more partitions to run reports?

For web and email security solutions, the available Log Database partitions, both enabled and disabled, are listed on the Settings > Reporting > Log Database page in the respective Web Security and Email Security modules of the Forcepoint Security Manager. To include data from a disabled partition, first enable it, then run the report. You can use this page to disable the partition again once you have retrieved the desired data.

For Forcepoint DLP, when you want to run a report and some or all of the data you want is stored in an offline partition, you must bring that partition online, or the generated report will not contain all the data you need.

Do the reporting databases use named instances?

If you are using SQL Server Standard or Enterprise to host your Forcepoint reporting databases:

The web protection Log Database can be hosted by one instance, and the Email Log Database and Data Incident and Configuration Database can be hosted elsewhere (on another server or instance).

All 3 reporting databases can be hosted by the same SQL Server instance.

Can reporting databases be hosted in a SQL Server cluster?

SQL Server clustering may be used with all supported standard and enterprise versions of Microsoft SQL Server for failover or high availability (Forcepoint Email Security and Forcepoint Web Security only).

If your organization uses a SQL Server cluster to provide failover for your database servers, the Forcepoint reporting databases can be hosted by the cluster if:

A virtual IP address is assigned to the cluster.

The data managed by SQL Server is housed on a reliable shared disk array.

When you install reporting components in a network that uses a SQL Server cluster, it is imperative that the cluster's virtual IP address is used to configure the reporting database connection. When this is done, reporting data is sent to SQL Server via the virtual IP address.

If you configure reporting components (like Web and Email Log Server) to use the IP address of an individual node in the cluster, they cannot take advantage of the failover protection of the cluster.

If the configured node becomes unavailable, reporting components cannot process data into the reporting database.

For web and email security solutions, log cache files continue to be saved on the Log Server machine. These files can build up quickly and fill the disk, causing additional problems.

When failover occurs, reporting components must wait briefly while the secondary SQL Server is made primary. When SQL Server begins accepting data over the virtual IP address again, reporting data is once again sent successfully.

This pause in recording data occurs both when failover occurs in a SQL Server cluster and when a standalone SQL Server installation fails and is later brought back online. Any records that were actively being processed into the reporting database when the primary SQL Server fails are lost.

For Web and Email solutions, if you are using BCP log insertion, the loss is generally a full batch (log cache file) of filtering records.

With ODBC log insertion, fewer records may be lost.