Creating Log Inspection Rules

The Vulnerability ProtectionDeep Security Log Inspection module lets you collect and analyze operating system and application logs to identify important security events buried in thousands of log entries. These events can be sent to a security information and event management (SIEM) system, or centralized logging server for correlation, reporting, and archiving. All events are also securely collected centrally at Vulnerability ProtectionDeep Security Manager.

The Vulnerability ProtectionDeep Security Log Inspection module lets you:

• Meet PCI DSS Log Monitoring requirements.
• Detect suspicious behavior.
• Collect events across heterogeneous environments containing different operating systems and diverse applications.
• View events such as error and informational events (disk full, service start/shutdown, etc.).
• Create and maintain audit trails of administrator activity (administrator login/logout, account lockout, policy change, etc.).

Vulnerability ProtectionDeep Security Log Inspection automates the collection of important security events in a number of ways:

• Recommendation Scan: A Recommendation Scan will recommend Log Inspection rules for the server being scanned (i.e. Windows Log Inspection rules vs. Unix Log Inspection rules, etc.).
• Default Log Inspection Rules: Vulnerability ProtectionDeep Security ships with many pre-defined rules covering a wide variety of operating systems and applications.
• Auto-Tagging: Log Inspection events are "auto-tagged" based upon their grouping in the log file structure. This simplifies and automates the processing of Log Inspection Events within Vulnerability ProtectionDeep Security Manager.

Although Vulnerability ProtectionDeep Security ships with Log Inspection Rules for many common operating systems and applications, you also have the option to create your own custom Rules. To create a custom Rule, you can either use the "Basic Rule" template, or you can write your new Rule in XML. This article will describe the Log Inspection Rule language and provide an example of a custom written rule. For a description of the properties of existing Log Inspection Rules, see the documentation for the Log Inspection Rules as well as Examining a Log Inspection Rule in the Reference section.

The Log Inspection Process

Decoders

A Log Inspection Rule consists of a list of files to monitor for changes and a set of conditions to be met for the Rule to trigger. When the Log Inspection engine detects a change in a monitored log file, the change is parsed by a decoder. Decoders parse the raw log entry into the following fields:

• log: the message section of the event
• full_log: the entire event
• location: where the log came from
• hostname: hostname of the vent source
• program_name: Program name. This is taken from the syslog header of the event
• srcip: the source IP address within the event
• dstip: the destination IP address within the event
• srcport: the source port within the event
• dstport: the destination port within the event
• protocol: the protocol within the event
• action: the action taken within the event
• srcuser: the originating user within the event
• dstuser: the destination user within the event
• id: any ID decoded as the ID from the event
• status: the decoded status within the event
• command: the command being called within the event
• url: the URL within the event
• data: any additional data extracted from the event
• systemname: the system name within the event

Rules examine this decoded data looking for information that matches the conditions defined in the Rule.

If the matches are at a sufficiently high severity level, any of the following actions can be taken:

• An Alert can be raised. (Configurable on the Options tab of the Log Inspection Rule's Properties window.)
• The Event can be written to syslog. (Configurable in the System Event Notification area on Administration > System Settings > SIEM tab.)
• The Event can be sent to the Vulnerability ProtectionDeep Security Manager. (Configurable in the Log Inspection Event Forwarding area on the Policy/Computer Editor > Settings > SIEM tab.)

Log Inspection Rules

The Log Inspection engine applies Log Inspection Rules to a computer's log entries to determine if any of those entries warrant the generation of a Log Inspection Event.

A single Log Inspection Rule can contain multiple subrules. These subrules can be of two types: atomic or composite. An atomic rule evaluates a single event and a composite rule examines multiple events and can evaluate frequency, repetition, and correlation between events.

Atomic Rules

Groups

Each rule, or grouping of rules, must be defined within a <group></group> element. The attribute name must contain the rules you want to be a part of this group. In the following example we have indicated that our group contains the syslog and sshd rules:

<group name="syslog,sshd,">
</group> 

Rules, ID, and Level

A group can contain as many rules as you require. The rules are defined using the <rule></rule> element and must have at least two attributes, the id and the level. The id is a unique identifier for that signature and the level is the severity of the Alert. In the following example, we have created two rules, each with a different rule id and level:

<group name="syslog,sshd,">
	<rule id="100120" level="5">
	</rule>
	<rule id="100121" level="6">
	</rule>
</group>

You can define additional subgroups within the parent group using the <group></group> tag. This subgroup can reference any of the groups listed in the following table:

Description

Include a <description></description> tag. The description text will appear in the event if the rule is triggered.

<group name="syslog,sshd,">
	<rule id="100120" level="5">
		<group>authentication_success</group>
		<description>SSHD testing authentication success</description>
	</rule>
	<rule id="100121" level="6">
		<description>SSHD rule testing 2</description>
	</rule>
</group>

Decoded As

The <decoded_as></decoded_as> tag instructs the Log Inspection engine to only apply the rule if the specified decoder has decoded the log.

<rule id="100123" level="5">
	<decoded_as>sshd</decoded_as>
	<description>Logging every decoded sshd message</description>
</rule>

Match

To look for a specific string in a log, use the <match></match>. Here is a Linux sshd failed password log:

 Jan 1 12:34:56 linux_server sshd[1231]: Failed password for invalid 
	user jsmith from 192.168.1.123 port 1799 ssh2

Use the <match></match> tag to search for the "password failed" string.

<rule id="100124" level="5">
	<decoded_as>sshd</decoded_as>
	<match>^Failed password</match>
	<description>Failed SSHD password attempt</description>
</rule>

The following table lists supported regex syntax:

Conditional Statements

Rule evaluation can be conditional upon other rules having been evaluated as true. The <if_sid></if_sid> tag instructs the Log Inspection engine to only evaluate this subrule if the rule identified in the tag has been evaluated as true. The following example shows three rules: 100123, 100124, and 100125. Rules 100124 and 100125 have been modified to be children of the 100123 rule using the <if_sid></if_sid> tag:

<group name="syslog,sshd,">
	<rule id="100123" level="2">
		<decoded_as>sshd</decoded_as>
		<description>Logging every decoded sshd message</description>
	</rule>
	<rule id="100124" level="7">
		<if_sid>100123</if_sid>
		<match>^Failed password</match>
		<group>authentication_failure</group>
		<description>Failed SSHD password attempt</description>
	</rule>
	<rule id="100125" level="3">
		<if_sid>100123</if_sid>
		<match>^Accepted password</match>
		<group>authentication_success</group>
		<description>Successful SSHD password attempt</description>
	</rule>
</group>

Hierarchy of Evaluation

The <if_sid></if_sid> tag essentially creates a hierarchical set of rules. That is, by including an <if_sid></if_sid> tag in a rule, the rule becomes a child of the rule referenced by the <if_sid></if_sid> tag. Before applying any rules to a log, the Log Inspection engine assesses the <if_sid></if_sid> tags and builds a hierarchy of parent/child rules.

The list of available atomic rule conditional options is shown in the following table:

Use the <if_sid>100125</if_sid> tag to make this rule depend on the 100125 rule. This rule will be checked only for sshd messages that already matched the successful login rule.

<rule id="100127" level="10">
	<if_sid>100125</if_sid>
	<time>6 pm - 8:30 am</time>
	<description>Login outside business hours.</description>
	<group>policy_violation</group>
</rule>

Restrictions on the Size of the Log Entry

The following example takes the previous example and adds the maxsize attribute which tells the Log Inspection engine to only evaluate rules that are less than the maxsize number of characters:

<rule id="100127" level="10" maxsize="2000">
	<if_sid>100125</if_sid>
	<time>6 pm - 8:30 am</time>
	<description>Login outside business hours.</description>
	<group>policy_violation</group>
</rule>

The following table lists possible atomic rule tree-based options:

Composite Rules

Atomic rules examine single log entries. To correlate multiple entries, you must use composite rules. Composite rules are supposed to match the current log with those already received. Composite rules require two additional options: the frequency option specifies how many times an event/pattern must occur before the rule generates an Alert, and the timeframe option tells the Log Inspection engine how far back, in seconds, it should look for previous logs. All composite rules have the following structure:

<rule id="100130" level="10" frequency="x" timeframe="y">
</rule> 

For example, you could create a composite rule that creates a higher severity Alert after five failed passwords within a period of 10 minutes. Using the <if_matched_sid></if_matched_sid> tag you can indicate which rule needs to be seen within the desired frequency and timeframe for your new rule to create an Alert. In the following example, the frequency attribute is set to trigger when five instances of the event are seen and the timeframe attribute is set to specify the time window as 600 seconds.

The <if_matched_sid></if_matched_sid> tag is used to define which other rule the composite rule will watch:

<rule id="100130" level="10" frequency="5" timeframe="600">
	<if_matched_sid>100124</if_matched_sid>
	<description>5 Failed passwords within 10 minutes</description>
</rule>

There are several additional tags that you can use to create more granular composite rules. These rules, as shown in the following table, allow you to specify that certain parts of the event must be the same. This allows you to tune your composite rules and reduce false positives:

If you wanted your composite rule to Alert on every authentication failure, instead of a specific rule ID, you could replace the <if_matched_sid></if_matched_sid> tag with the <if_matched_ group></if_matched_ group> tag. This allows you to specify a category, such as authentication_ failure, to search for authentication failures across your entire infrastructure.

<rule id="100130" level="10" frequency="5" timeframe="600">
	<if_matched_group>authentication_failure</if_matched_group>
	<same_source_ip />
	<description>5 Failed passwords within 10 minutes</description>
</rule>

In addition to <if_matched_sid></if_matched_sid> and <if_matched_group></if_matched_ group> tags, you can also use the <if_matched_regex></if_matched_regex> tag to specify a regular expression to search through logs as they are received.

<rule id="100130" level="10" frequency="5" timeframe="600">
	<if_matched_regex>^Failed password</if_matched_regex>
	<same_source_ip />
	<description>5 Failed passwords within 10 minutes</description>
</rule>

Real World Examples

Vulnerability ProtectionDeep Security includes many default Log Inspection rules for dozens of common and popular applications. Through Security Updates, new rules are added regularly. In spite of the growing list of applications supported by Log Inspection rules, you may find the need to create a custom rule for an unsupported or custom application.

In this section we will walk through the creation of a custom CMS (Content Management System) hosted on the Microsoft Windows Server IIS .Net platform with a Microsoft SQL Database as the data repository.

The first step is to identify the following application logging attributes:

1. Where does the application log to?
2. Which Log Inspection decoder can be used to decode the log file?
3. What is the general format of a log file message?

For our custom CMS example the answers are as follows:

1. Windows Event Viewer
2. Windows Event Log (eventlog)
3. Windows Event Log Format with the following core attributes:
   • Source: CMS
   • Category: None
   • Event: <Application Event ID>

The second step is to identify the categories of log events by application feature, and then organize the categories into a hierarchy of cascading groups for inspection. Not all inspected groups need to raise events; a match can be used as a conditional statement. For each group, identify the log format attributes which the rule can use as matching criteria. This can be performed in a reverse manner by inspecting all application logs for patterns and natural groupings of log events.

For example, the CMS application supports the following functional features which we will create log inspection rules for:

• CMS Application Log (Source: CMS)
  • Authentication (Event: 100 to 119)
    • User Login successful (Event: 100)
    • User Login unsuccessful (Event: 101)
    • Administrator Login successful (Event: 105)
    • Administrator Login unsuccessful (Event: 106)
  • General Errors (Type: Error)
    • Database error (Event: 200 to 205)
    • Runtime error (Event: 206-249)
  • Application Audit (Type: Information)
    • Content
      • New content added (Event: 450 to 459)
      • Existing content modified (Event: 460 to 469)
      • Existing content deleted (Event: 470 to 479)
    • Administration
      • User
        • New User created (Event: 445 to 446)
        • Existing User deleted (Event: 447 to 449)

This structure will provide you with a good basis for rule creation. Now to create a new Log Inspection rule in Vulnerability ProtectionDeep Security Manager.

To create the new CMS Log Inspection Rule:

1. In the Vulnerability ProtectionDeep Security Manager, go to Policies > Common Objects > Rules > Log Inspection Rules and click New to display the New Log Inspection Rule Properties window.
2. Give the new rule a Name and a Description, and then click the Content tab.
3. The quickest way to create a new custom rule is to start with a basic rule template. Select the Basic Rule radio button.
4. The Rule ID field will be automatically populated with an unused ID number of 100,000 or greater, the IDs reserved for custom rules.
5. Set the Level setting to Low (0).
6. Give the rule an appropriate Group name. In this case, "cms".
7. Provide a short rule description.
   
   

8. Now select the Custom (XML) option. The options you selected for your "Basic" rule will be converted to XML.
   
   

9. Next, click the Files tab and click the Add File button to add any application log files and log types which the rule will be applied to. In this case, "Application", and "eventlog" as the file type.
   
   
   Eventlog is a unique file type in Vulnerability ProtectionDeep Security because the location and filename of the log files don't have to be specified. Instead, it is sufficient to type the log name as it is displayed in the Windows Event Viewer. Other log names for the eventlog file type might be "Security", "System", "Internet Explorer", or any other section listed in the Windows Event Viewer. Other file types will require the log file's location and filename. (C/C++ strftime() conversion specifiers are available for matching on filenames. See the table below for a list of some of the more useful ones.)

10. Click OK to save the basic rule.
11. Working with the basic rule Custom (XML) created, we can begin adding new rules to the group based on the log groupings identified previously. We will set the base rule criteria to the initial rule. In the following example, the CMS base rule has identified Windows Event Logs with a Source attribute of "CMS":

<group name="cms">
	<rule id="100000" level="0">
	<category>windows</category>
	<extra_data>^CMS</extra_data>
	<description>Windows events from source 'CMS' group messages.</description>
</rule> 

12. Now we build up subsequent rules from the identified log groups. The following example identifies the authentication and login success and failure and logs by Event IDs.

<rule id="100001" level="0">
	<if_sid>100000</if_sid>
	<id>^100|^101|^102|^103|^104|^105|^106|^107|^108|^109|^110</id>
	<group>authentication</group>
	<description>CMS Authentication event.</description>
</rule>
 
<rule id="100002" level="0">
	<if_group>authentication</if_group>
	<id>100</id>
	<description>CMS User Login success event.</description>
</rule>

<rule id="100003" level="4">
	<if_group>authentication</if_group>
	<id>101</id>
	<group>authentication_failure</group>
	<description>CMS User Login failure event.</description>
</rule>

<rule id="100004" level="0">
	<if_group>authentication</if_group>
	<id>105</id>
	<description>CMS Administrator Login success event.</description>
</rule>

<rule id="100005" level="4">
	<if_group>authentication</if_group>
	<id>106</id>
	<group>authentication_failure</group>
	<description>CMS Administrator Login failure event.</description>
</rule>

13. Now we add any composite or correlation rules using the established rules. The follow example shows a high severity composite rule that is applied to instances where the repeated login failures have occurred 5 times within a 10 second time period:

<rule id="100006" level="10" frequency="5" timeframe="10">
	<if_matched_group>authentication_failure</if_matched_group>
	<description>CMS Repeated Authentication Login failure event.</description>
</rule>

14. Review all rules for appropriate severity levels. For example, error logs should have a severity of level 5 or higher. Informational rules would have a lower severity.
15. Finally, open the newly created rule, click the Configuration tab and copy your custom rule XML into the rule field. Click Apply or OK to save the change.

Once the rule is assigned to a Policy or computer, the Log Inspection engine should begin inspecting the designated log file immediately.

The complete Custom CMS Log Inspection Rule:

<group name="cms">

	<rule id="100000" level="0">
		<category>windows</category>
		<extra_data>^CMS</extra_data>
		<description>Windows events from source 'CMS' group messages.</description>
	</rule>

	<rule id="100001" level="0">
		<if_sid>100000</if_sid>
		<id>^100|^101|^102|^103|^104|^105|^106|^107|^108|^109|^110</id>
		<group>authentication</group>
		<description>CMS Authentication event.</description>
	</rule>
	 
	<rule id="100002" level="0">
		<if_group>authentication</if_group>
		<id>100</id>
		<description>CMS User Login success event.</description>
	</rule>
	 
	<rule id="100003" level="4">
		<if_group>authentication</if_group>
		<id>101</id>
		<group>authentication_failure</group>
		<description>CMS User Login failure event.</description>
	</rule>
	 
	<rule id="100004" level="0">
		<if_group>authentication</if_group>
		<id>105</id>
		<description>CMS Administrator Login success event.</description>
	</rule>
	 
	<rule id="100005" level="4">
		<if_group>authentication</if_group>
		<id>106</id>
		<group>authentication_failure</group>
		<description>CMS Administrator Login failure event.</description>
	</rule>
	 
	<rule id="100006" level="10" frequency="5" timeframe="10">
		<if_matched_group>authentication_failure</if_matched_group>
		<description>CMS Repeated Authentication Login failure event.</description>
	</rule>
	 
	<rule id="100007" level="5">
		<if_sid>100000</if_sid>
		<status>^ERROR</status>
		<description>CMS General error event.</description>
		<group>cms_error</group>
	</rule>
	 
	<rule id="100008" level="10">
		<if_group>cms_error</if_group>
		<id>^200|^201|^202|^203|^204|^205</id>
		<description>CMS Database error event.</description>
	</rule>
		
	<rule id="100009" level="10">
		<if_group>cms_error</if_group>
		<id>^206|^207|^208|^209|^230|^231|^232|^233|^234|^235|^236|^237|^238|
			^239^|240|^241|^242|^243|^244|^245|^246|^247|^248|^249</id>
		<description>CMS Runtime error event.</description>
	</rule>
	 
	<rule id="100010" level="0">
		<if_sid>100000</if_sid>
		<status>^INFORMATION</status>
		<description>CMS General informational event.</description>
		<group>cms_information</group>
	</rule>
	 
	<rule id="100011" level="5">
		<if_group>cms_information</if_group>
		<id>^450|^451|^452|^453|^454|^455|^456|^457|^458|^459</id>
		<description>CMS New Content added event.</description>
	</rule>
	
	<rule id="100012" level="5">
		<if_group>cms_information</if_group>
		<id>^460|^461|^462|^463|^464|^465|^466|^467|^468|^469</id>
		<description>CMS Existing Content modified event.</description>
	</rule>
	 
	<rule id="100013" level="5">
		<if_group>cms_information</if_group>
		<id>^470|^471|^472|^473|^474|^475|^476|^477|^478|^479</id>
		<description>CMS Existing Content deleted event.</description>
	</rule>
	 
	<rule id="100014" level="5">
		<if_group>cms_information</if_group>
		<id>^445|^446</id>
		<description>CMS User created event.</description>
	</rule>
 
	<rule id="100015" level="5">
		<if_group>cms_information</if_group>
		<id>^447|449</id>
	<description>CMS User deleted event.</description>
</rule>
 
</group>

Log Inspection Rule Severity Levels and their Recommended Use

strftime() Conversion Specifiers

More information can be found at the following Web sites:

www.php.net/strftime
www.cplusplus.com/reference/clibrary/ctime/strftime.html