GBDI 4.2

Large Files Safety Valve

GBDI ingestion provides a "safety valve" against the ingestion of large files. If a policy or database is changed, it can accidentally result in a flood of unwanted records. In some Guardium environments, such an event can cause collectors or aggregators to fill up and shut down; GBDI ingestion can be configured to prevent ingestion of large files until it is determined whether or not these records should be ingested into GBDI.

To configure the threshold, use the max-size config parameter. The default of -1 means that data is ingested always. Setting the threshold to 5000 MB means that if a single gz extract file is over 5GB of data, the file will not be ingested and will be moved to the audit directory for determination by the operator:

# maximum file size in MB we are to handle. The compressed incoming files will
# be checked and files larger than this size will be moved to audit
# directory. The files will be processed as if they were empty.
# Set to -1 to disable file size checks.
# Example: 5000

max-size: -1

When sonargd is configured to use a threshold, a file that is larger than the threshold is copied to the audit directory and a record is added to grdmrec with the following structure:

> db.grdmrec.find({'Error': {$exists: true}}).pretty()
    "_id" : ObjectId("5683221b792aa02f638e1b54"),
    "T2" : ISODate("2015-12-30T00:15:23.917Z"),
    "Type" : "full_sql",
    "F" :
    "T1" : ISODate("2015-12-30T00:15:23.917Z"),
     "Size" : 154624000,
    "Num" : 0,
     "Error" : "File incoming/1762144298_gibm38_EXP_FULL_SQL_20151229162300.gz is
  larger than the configured maximum file size (150000000)",
     "Errs" : 0,
    "N" : "gibm38"