Avoiding false positives

A compliance rule can be wrong in two directions. It can fail a compliant device, which wastes time. Or it can pass a non-compliant device, which is worse because nobody looks. Most of the traps below are the second kind. Each one comes from how the rConfig engine reads definitions and compares text, and the examples show results asserted by the rConfig Pro test suite.

The habit that catches all of them: test every rule against a config that should fail, not only one that should pass.

Array items match inside longer lines

The array methods (must_match_all_from_array, can_match_any_from_array, must_not_match_all_from_array) search the whole config text for each item. An item does not have to be a complete line. It only has to appear somewhere.

From the tested must_match_all_from_array example on must contain and must not contain:

// Every one of these lines is present
#[must_match_all_from_array]
snmp-server host 192.0.2.2 EXAMPLECOMM10
snmp-server host 192.0.2.4 EXAMPLECOMM30
aaa session-id
no ip subnet-zero
ip domain name example.com
line vty 0 4

The result is pass, although the config has no line aaa session-id. It has aaa session-id common, and the item is part of it. Compare must_match_single_string, which compares whole lines: the same suite asserts that interface G fails against a config containing interface GigabitEthernet1.

How to avoid it: when the exact line matters, use one must_match_single_string block per line, or a section-scoped check when the lines belong together. Make array items long enough that they cannot appear by accident inside another line.

must_not_match_all_from_array means “not all present”

The name suggests “none of these may be present”. The method actually passes unless every item is present. Its tested results only show the two extremes:

Items present in the config	Result
None of the six	`pass`
All six	`fail`

Source: PolicyMethodsTest::test_must_not_match_all_from_array_tests. Between those extremes, the engine returns a pass whenever at least one item is missing. So a list of five forbidden lines passes on a device that has four of them.

How to avoid it: to forbid several lines, write one must_not_match_single_string block per line. Each block then fails on its own line.

must_not_match_single_string compares whole lines

must_not_match_single_string is forgiving about case, indentation and repeated spaces, but it still compares complete lines. If the body is only part of a device line, it does not match that line and the rule passes. Write the complete forbidden line, or use wildcard_must_not_match_single_string with * for the part that varies, as in the no test interfaces rule.

Wildcards are not anchored

A wildcard pattern can match in the middle of a line, and * crosses spaces. See the caution on per-device values and exceptions.

Code block methods that check less than you think

match_code_block_flexible on IOS-XE

match_code_block_flexible finds the end of a section by counting braces. IOS-XE configs have none, so the “section” runs to the end of the file. The test suite pins down two consequences on the two-port access switch config from section-scoped checks:

// No anchor line
#[match_code_block_flexible]
description ACCESS-PORT
switchport access vlan *
switchport mode access
switchport voice vlan *
switchport port-security maximum 100
switchport port-security violation restrict
switchport port-security aging time 5
switchport port-security aging type inactivity
switchport port-security
ip access-group ACL-USER-OUT out
no logging event link-status
ipv6 traffic-filter port_client in
no snmp trap link-status
spanning-tree portfast

// Every GigabitEthernet port uses voice VLAN 3755
#[match_code_block_flexible]
interface GigabitEthernet*
switchport voice vlan 3755

Rule	Result	What went wrong
No anchor line	`pass`	Without an `interface` line, the lines are found anywhere after the first match.
Every port uses voice VLAN 3755	`pass`	Only the first matching port (5/0/48, VLAN 3755) is checked. Port 6/0/48 uses 3788 and is never examined.

Source: CustomerUseCases/MatchCodeBlockFlexibleInterfacesTest. The match_code_block_strict tests assert the opposite outcome for both situations: a body with no anchor line fails, and a wildcard anchor fails when any matched port differs. Use match_code_block_strict for indentation-style configs.

match_code_block_from_array checks the first match only

match_code_block_from_array stops at the first config line that matches the body’s first line. If that section fails, the rule fails, even when a later section would pass. If that section passes, later sections are never checked.

// Loopback0 description, case-sensitive
#[match_code_block_from_array]
re:/^interface Loopback0$/m
re:/^\s*description LOOP:null:Loopback0:[a-z]{3}[0-9]{2}\.[a-z]{4}[0-9]{2}-[a-z]{2}:null:\[\]$/m

interface Loopback0
 description LOOP:null:Loopback0:LAB01.SITE01-AR:null:[]
interface Loopback0
 description LOOP:null:Loopback0:lab01.site01-ar:null:[]

Result: fail. The second section complies, but only the first is examined. Source: PolicyMethodsTest::test_match_code_block_from_array_matches_first_block_only.

To check every matching section, use match_code_block_strict or evaluate_block with scope:. Both also fail when nothing matches the anchor, instead of passing quietly.

evaluate_block: plain items match part of a line

In evaluate_block, a plain item such as disabled matches any line in the section that contains that text, ignoring case. An exemption written that way also fires on a profile named waf_disabled_logging. equals: needs the whole line to match.

The tested F5 logging rule (full example on section-scoped checks) uses equals: disabled, and the suite checks that this server, which is not disabled, still fails:

ltm virtual /Common/vs_tricky {
    destination /Common/192.0.2.10:443
    profiles {
        /Common/ASM_WAF_policy_prod { }
        /Common/waf_disabled_logging { }
    }
    source 0.0.0.0/0
}

Result: fail, with ltm virtual /Common/vs_tricky { recorded as the failing section. Source: CustomerUseCases/F5WafBlockComplianceCustomerTest::test_waf_logging_does_not_treat_a_profile_named_disabled_as_a_disabled_server.

A regex: item without delimiters has its own trap with ^. See the caution at the end of regex patterns.

Blocks that silently drop out

A device passes when every rule that ran passed. A block rConfig cannot read is skipped and logged to the activity log, so it never appears in the results, and the device can pass with fewer checks than you wrote. The parser skips a block when:

the method line is malformed, for example #[bad method with spaces];
an array method (must_match_all_from_array, can_match_any_from_array, must_not_match_all_from_array, wildcard_match_all_from_array, wildcard_match_any_from_array, match_code_block_from_array) has a single-line body;
an evaluate_block body is not valid YAML, or has no rule under scope:.

These cases are asserted by PolicyDefinitionParserTest and EvaluateBlockParserTest. After editing a definition, check that the number of rules in the test results matches the number of blocks you wrote.