Log Database sizing guidance

Web Security Help | Web Security Solutions | Version 7.8.x

Related topics:

Configuring database partition options

Configuring how requests are logged

Configuring how URLs are logged

It is difficult to make precise sizing predictions for the Log Database, because database size is affected by a number of variables, including the number of filtered users and average requests per second. In addition, size is affected by whether the database is configured to:

Record hits or visits for each Web request (see Configuring Log Server).

Recording hits provides a high level of detail, but recording visits can reduce the size of the database by roughly 40%.

Consolidate log records (see Configuring Log Server).

By default, all requests are logged as separate hits or visits. When you turn on consolidation, similar requests (by the same user, for sites in the same domain, that have the same action applied) in a defined time period are recorded as a single log record. This can reduce the size of the Log Database by roughly 60%.

Store the full URL for each logged request (see Configuring how URLs are logged).

Recording full URLs provides precise information about which sites a user has visited, but more than doubles Log Database size.

Log requests for all categories (see Configuring how requests are logged).

By default, requests for sites in all categories are logged. To reduce the size of the Log Database, you can stop logging requests for sites in categories that, for example, present no security risk or legal liability to your organization.

The impact of this change depends on the number of categories that are not logged, and how often users requests sites in those categories.

Perform detailed browse time calculations (see Configuring Internet browse time options).

In order to create investigative detail reports that include browse time, the IBT job must calculate detailed browse time. Storing detailed browse time data, however, increases the size of the database, and may also affect database performance.

Store trend data (see Configuring trend and application data retention).

Storing trend data makes it possible to report on trends in users' Internet activity throughout the course of a day, week, or longer period, but storing trend data increases the size of the Log Database. The longer the data is stored, the greater its effect on database size.

Use the Growth Rates and Sizing chart on the Settings > Reporting > Log Database page to monitor the average daily size of your active and inactive standard logging partitions. This information may help you identify trends in traffic volume over time, and make it easier to plan for future growth.

As you collect average sizing information, adjust your rollover Initial Size and Growth settings (in the Partition Management section of the Settings > Reporting > Log Database page).

As a best practice, set the Initial Size value to approximately 80% of the average partition size over the rollover period (week, month, etc.). The idea is to:

Minimize the number of times the partition must be expanded.

Free resources to process data into the partitions.

Prevent unneeded disk space from being allocated when the partition is created.

Unused portions of the initial space allocated to a partition cannot be recovered until the partition is deleted.