How big will my reporting database be?

Administering Websense Databases | Web, Data, and Email Solutions | v8.0.x, v8.1.x

Applies to:

In this topic

TRITON AP-WEB and Web Filter & Security, v8.0.x, v8.1.x

TRITON AP-DATA, v8.0.x, v8.1.x

TRITON AP-EMAIL, v8.0.x, v8.1.x

Web size estimates

Email size estimates

Data size estimates

When estimating the size of your TRITON reporting databases, keep the following factors in mind:

TRITON AP-DATA logs only data violations, which typically represent a small fraction of your total web and email volume.

TRITON AP-EMAIL logs detected spam and quarantined email messages, which typically represent up to 90% of your email volume.

TRITON AP-WEB and Web Filter & Security log all web transactions, which typically represent about 40% of your total web traffic volume when visits (the default) are recorded rather than hits. See Web visits, consolidation, and full URL logging for details.

If IPv6 is used in the deployment, the Web Log Database may increase in size by approximately 4%


	Warning These are averages calculated across a large number of Websense customers with widely varying deployments. While these numbers are useful for estimation, please analyze your system's actual performance after a few days or weeks to reassess your database sizing needs.

Web size estimates

The table below provides some general rules to help estimate the size of your Web Log Database, based on whether you are recording visits or hits. and URL hostnames or full URLs. By default, visits and URL hostnames are recorded.


	URL hostnames	Full URL logging
Visits	3 MB per user per month	4.5 MB per user per month
Hits	8 MB per user per month	12 MB per use per month

The size increase that accompanies full URL logging can vary widely based on the type of URL being logged. For example, the following types of traffic tend to generate very long URLs:

Research database search and results URLs

Search strings

Social networking sites

Online apps and games

Consider the example of an organization with 7500 users who wants to store data for 3 months, using monthly database partition rollover.

Visits are enabled

Full URL logging is enabled

The Internet access policy is permissive

A high percentage of Web traffic includes longer than average URLs (social networking sites and online games)

Based on this information, estimate 6 MB of data stored per user per month.

Because rollover is monthly, it is necessary to allocate enough space to hold up to 4 months of data. (This ensures that it is always possible to report on 3 months of data, even during the first days or weeks of the most recent month.)

To start, therefore, 180 GB would be needed for the database logging partitions (not including the catalog database and AMT partition).

6 MB * 7500 users * 4 months = 180 GB

In addition, calculate space for the following:

Database transaction logs, whose size depends on the recovery model you choose.

For full recovery (not recommended), allow for the same size as the data itself (180 GB in our example).

For simple recovery, the log volume depends on the amount of database activity. Allow roughly 30 percent of the size of an active database partition.

In our example, the total size of 4 full partitions is 180 GB, so each partition is 45 GB in size. So for simple recovery:

0.3 * 45 GB = 13.5 GB

Note that this space is used only while data is being logged to the database. Inactive partitions, for example, have minimal transaction log activity. Likewise, during time periods in which little or no Internet activity is being recorded, the transaction log is very small.

Temporary (tempdb) storage is used heavily during report generation and when the maintenance job is reindexing the database. The amount of tempdb space required depends heavily on the types of reports being generated.

In most environments, it is sufficient to allow twice the size of a partition:

45 * 2 = 90 GB

Temporary database (tempdb) transaction logs take up about 20 percent of the tempdb storage size. In our example:

0.2 * 90 GB = 18 GB

The total space required for the Web Log Database in our example, then, is:

180 GB + 13.5 GB + 90 GB + 18 GB = 301.5 GB


	Tip As a best practice, record visits rather than hits to reduce significantly the amount of storage space required.

See Factors that affect reporting database size, for an explanation of the TRITON AP-WEB visits algorithm, other data reduction options, and the full URL logging setting.

Email size estimates

For best practice, allow 1-3 MB per user per month when estimating the size of your Email Log Database.

For illustration, assume there will be 2 MB of data per user per month for 7500 users. Data will be kept for 3 months, with a scheduled monthly partition rollover.

This means that you would need 60 GB to start:

2 MB * 7500 users * 4 months = 60 GB

In addition, you need to calculate space for the following:

The database transaction logs, whose size depends on the recovery model you choose.

For full recovery (not recommended), allow for the same size as the data itself (60 GB in our example).

For simple recovery, the log volume depends on the amount of database activity. Allow roughly 30 percent of the size of the available data.

0.3 * 60 GB = 18 GB

Temporary database (tempdb) storage is used heavily during report generation. The amount of tempdb space required depends on the reports being run. As a best practice, allow the size of the logged data for temporary storage. In our example:

60 GB

Temporary database (tempdb) transaction logs take up about 20 percent of the tempdb storage size. In our example:

0.2 * 60 GB = 12 GB

The total space required in our example, then, is:

60 GB + 18 GB + 60 GB + 12 GB = 150 GB

Use the following tables to estimate storage space required for 30 days' worth of data based on email volume.

The same requirements for partitions, logs, and temporary databases apply.

The estimates assume an average of 5 recipients per email message.

With the Email Hybrid Module enabled for TRITON AP-EMAIL:


	10 msg/user/day	50 msg/user/day	100 msg/user/day	200 msg/user/day	400 msg/user/day
500 users	600 MB	3300 MB	6600 MB	13300 MB	26700 MB
1000 users	1300 MB	6600 MB	13300 MB	26700 MB	53400 MB
2000 users	2600 MB	13300 MB	26700 MB	53400 MB	106800 MB
5000 users	6600 MB	33400 MB	66800 MB	133600 MB	267200 MB

Without the Email Hybrid Module:


	10 msg/user/day	50 msg/user/day	100 msg/user/day	200 msg/user/day	400 msg/user/day
500 users	500 MB	2600 MB	5200 MB	10400 MB	20900 MB
1000 users	1000 MB	5200 MB	10400 MB	20900 MB	41800 MB
2000 users	2000 MB	10400 MB	20900 MB	41800 MB	83600 MB
5000 users	5200 MB	26100 MB	52200 MB	104500 MB	209000 MB

The calculations used are:

With the Email Hybrid Module:

(0.01622+0.02348* RECIPIENT)* SPEED * USER

Without the Email Hybrid Module:

(0.0384+0.01322* RECIPIENT)* SPEED * USER

Note that:

USER represents the total number of users.

RECIPIENT is the average number of recipients for each message.

SPEED is the average message volume for each user per day.

Note that the email hybrid service drops or blocks the following types of email before they reach on-premises components:

Email that comes from known bad (blacklisted) sources

Email that has a very high spam score in the cloud

This significantly reduces the amount of data stored in the Email Log Database, as shown in the examples above.

Data size estimates

The Data Incident and Configuration Database is different from the Web and Email Log Databases in that it stores both configuration and log data. In addition, it logs only policy violations, not all events. In order to estimate the potential size of the Incident and Configuration Database, consider the following:


Feature	Impact on Database Size
Discovery incidents (not standard when all Websense TRITON products are installed)	14 KB per incident Typically, no more than 3.5 GB total
Network and endpoint incidents	10 KB per incident Typically, 30 KB per user per month
User directory import data	8 KB per record Typically, no more than 1 GB total
Configuration data	200 MB

As a general guideline, it is unlikely that your Data Incident and Configuration Database will require more than 9 GB of storage total, no matter how many users are in your environment.

The volume of configuration and incident data varies based on your use of TRITON AP-DATA features as explained in Factors that affect reporting database size.