How does the logging process work?

Reporting FAQ | Web Protection Solutions | v8.4.x, v8.5.x | 29-Apr-2022

When users browse the Web, their activity as recorded as log data:

Network Agent, Content Gateway, or a third-party integration forwards the Internet request to Filtering Service.

Filtering Service determines the appropriate response to the request.

By default, Filtering Service forwards a copy of the transaction for logging.

The transaction goes through Multiplexer to reach Log Server and (if configured) the integrated SIEM tool.

See the Security Information Event Management (SIEM) (or this v8.4 version) paper for more information.

Log Server stores the data in temporary cache or BCP files on the local hard disk.

When the load of incoming data is not too heavy, Log Server performs preprocessing on the cached files and forwards them to the Log Database. Log Server can:

Process multiple, similar log records into a single record in a process called log record consolidation

Combine the elements that make up a web page (like advertisements, graphics, and text) into a single record using visits processing

Information about Log Server preprocessing options can be found in the Administering Databases paper.

The Log Database temporarily stores each log record in a table in the catalog database.

Database jobs move the data to various tables in the partition databases. For more information about how the jobs work, see Database jobs in Administrator Help.

Data in the partition databases can be used in dashboard, investigative, and presentation reports (see Use Reports to Evaluate Internet Activity for v8.5 or this version for v8.4).

ODBC and BCP

Log Server can forward log records to the Log Database using either of 2 formats:

With ODBC (Open Database Connectivity), Log Server inserts records into the database individually. The data stored in Log Server cache files is moved directly to the database.

With BCP (Bulk Copy Program), Log Server inserts the data in batches. To do this, Log Server starts by moving the data from the cache files into a new set of storage files. From there, the data is moved to the database.

Configure data insertion options in the Web Security module of the Forcepoint Security Manager on the Settings > Reporting > Log Server page.

For more information about ODBC and BCP, see Specify how log records are processed into the database.