Creating a validation script

Administrator Help | TRITON AP-DATA | Version 8.3.x

Fingerprinting cells with some values, such as multiple short values, can lead to multiple false-positive incidents. TRITON AP-DATA includes a mechanism that forwards database data to an external script for processing before fingerprinting.

Validation script mechanism

Each database fingerprints classifier can use a validation script. The validation script receives an input file containing the raw database data in a CSV format, and returns CSV data containing the information that should be fingerprinted.

Validation scripts must be designed to receive at least two parameters: An input path name and an output path name. An additional parameter, containing a configuration file path name, is optional.

The input file, received from TRITON AP-DATA, is a CSV file with a header row containing the database column names. Each line is delimited by a valid windows line break (CRLF), and all values are double-quotes escaped. A sample package containing a sample input file, among other things, is available through Forcepoint Technical Support.

The output file should be of the same format as the input file, but instead of using CRLF as the line delimiter, it uses CRCRLF (2 Carriage-Return characters, and a Line-Feed character). An output sample file is available on the same package as the sample input file.

Validating fingerprinting scans

If you would like to validate your fingerprinting scans, do the following:

Optionally, create a copy of the following files in the \ValidationScripts folder where TRITON AP-DATA was installed (typically C:\Program Files\Websense\Data Security\ValidationScripts).

default_validation.bat.sample

default_validation.ini.sample

If you prefer to create your script from scratch, you can skip this step.

Name your new validation script using the following convention:

<classifier-name>_validation.[bat|exe|py]

where:

<classifier-name> is the name of the classifier on which the script will be implemented. Alternatively, the word "default" may be used, for scripts that are to be implemented on all classifiers that don't have specific scripts named after them.

bat is the extension for a batch file.

exe is the extension for an executable.

py is the extension for a python script.

If the script requires a configuration file, then name the configuration file using the following convention:

<classifier-name>_validation.[xml|ini]

Place all files in the \ValidationScripts folder on the server where TRITON AP-DATA is installed (typically C:\Program Files\Websense\Data Security\ValidationScripts).

Every validation script must be an executable or a batch file. If there is a need for an infrastructure element, for example the python interpreter, the operating system must be able to automatically initiate the element when the script is being called. To ensure the correct file association is configured, Forcepoint recommends running the script from the command line, without reference to any other executable.


	Note Pay attention not to leave more than one executable or configuration file with the same name and different extension in the validation scripts directory.

The script should receive 2 command-line parameters from TRITON AP-DATA: the full path of a source file the system creates, and the full path where the system expects to find a destination file.

The first line of the source file includes the names of the columns that are available for fingerprinting. The remaining lines contain the data in those columns.

Your script should read and perform validation on the source file.

Your script should write the validated results to a destination file.

The destination file should be formatted in the same way as the source file—with the names of the columns that were fingerprinted on the first line. Note that the number of columns varies if your script adds or removes columns.

The destination file must use the name and path that received from TRITON AP-DATA.

Your script should return a return code of 0 if everything succeeded, and non-zero if there was a problem.

If you want your script to receive a configuration file, place it in the same location as the script, and name it with the same name as the script file followed by .xml or .ini. If this file is found, it is supplied as a third parameter to your script.

Create and run the fingerprinting classifier as described in Creating a database fingerprint classifier. Name the classifier with the name given in step 2.

During the scan, if the crawler finds a script named <classifier-name>_validation.[bat|exe|py], it runs that script. If it does not, it searches for a script named default_validation.[bat|exe|py] and runs that.

If the crawler receives a non-zero return code from the script, the fingerprinting process stops and an appropriate error is returned. In this case, you can either fix the script or remove it then refingerprint.

When the system finds a validation script, the Sample Data screen in the database fingerprinting wizard shows validated data, and not the raw data extracted from the database/CSV. (This is on the Field Selection page of the wizard, where you click View Sample Data.) You can use this to make sure that the validation script behaves as expected, and to see the exact information that is protected.

To run the script on subsequent fingerprint classifiers, copy the script and rename it.

Sample validation script

There is a sample validation script in the \Validation Scripts directory where TRITON AP-DATA is installed. The script contains the basic abilities required for most customers, such as removing NULL or single-character values from being fingerprinted. You can modify it to suit your needs.

The sample package contains the following files:

default_validation.bat - Sample validation script

validation_logic.py - Used by the sample validation script.

default_validation.ini - Sample configuration file

default_validation.ini.sample - An additional configuration sample file

dictionary.txt - Sample dictionary file

in.csv - Sample input file

out.csv - Sample output file

The first 3 files are also included (with the .sample extension, for the batch and ini files) in the TRITON AP-DATA installation package.

The sample validation script is a production grade script, which is suitable for many customers.

Please note that although you can change the filenames of the default_validation.bat and default_validation.ini according to the conventions mentioned above, do not rename the validation_logic.py file. The validation_logic.py file must be present in the \ValidationScripts directory (typically C:\Program Files\Websense\Data Security\ValidationScripts) in its original form.

The validation script is predefined to make sure TRITON AP-DATA ignores:

Numbers smaller than 10,000.

Text strings containing fewer than 4 characters.

Strings containing only zeros (i.e. "000000").

Empty strings.

Placeholders (NULL and similar values).

Invalid SSNs in columns named 'ssn'.

Invalid email addresses in columns named 'email'.

The following additions and changes can be configured through the default_validation.ini configuration file:

It is possible to create a dictionary file that contains a list of strings for the validation script to remove. The file should be a line delimited UTF-16 file, and its path name should be written in the IgnoredDictionary configuration option in regular file system format. (For example c:\directory\dictionary.txt.) You can create UTF-16 files in Windows Notepad by saving the text with 'Unicode' encoding.

The default_validation.ini.sample file, which is part of the package, is a sample file containing such a definition. The dictionary.txt file is a sample dictionary file.

You can use regular expressions to validate any column. To use this feature:

Add the column name, in lower case, to the columns parameter. Separate column names by semicolons.

Add a configuration section for the column by appending [column-name] to the file (again, lower case). This is the section header.

Add a RegExp parameter under the relevant (newly added) section header. Its value is a regular expression.

The default_validation.ini sample file contains this type of validation for email addresses and social security numbers. These can be used as a reference.


	Note Additional configuration options are available. Contact Forcepoint Technical Support for further assistance.