Perl Compatible Regular Expressions

Perl Compatible Regular Expressions (PCRE) is a library written in C, which implements a regular expression engine, inspired by the capabilities of the Perl programming language. Philip Hazel started writing PCRE in summer 1997.[3] PCRE's syntax is much more powerful and flexible than either of the POSIX regular expression flavors and than that of many other regular-expression libraries.

Perl Compatible Regular Expressions

Original author(s)	Philip Hazel
Stable release	PCRE 8.43[1] and PCRE2 10.33[2], / April 16, 2019 (2019-04-16)
Repository	vcs.pcre.org/pcre/
Written in	C
Operating system	Cross-platform
Type	Pattern matching library
License	BSD
Website	www.pcre.org

While PCRE originally aimed at feature-equivalence with Perl, the two implementations are not fully equivalent. During the PCRE 7.x and Perl 5.9.x phase, the two projects have coordinated development, with features being ported between them in both directions.[4]

A number of prominent open-source programs, such as the Apache and Nginx HTTP servers, and the PHP and R scripting languages, incorporate the PCRE library; proprietary software can do likewise, as the library is BSD-licensed. As of Perl 5.10, PCRE is also available as a replacement for Perl's default regular-expression engine through the re::engine::PCRE module.

The library can be built on Unix, Windows, and several other environments. PCRE is distributed with a POSIX C wrapper, a native C++ wrapper,[lower-alpha 1] several test programs, and the utility program pcregrep built in tandem with the library.

Features

Just-in-time compiler support

This optional feature is available in version 8.20 and above if enabled when the PCRE library is built. Large performance benefits are possible when (for example) the calling program utilizes the feature with compatible patterns that are executed repeatedly. The just-in-time compiler support was written by Zoltan Herczeg and is not addressed in the POSIX or C++ wrappers.

Flexible memory management

Unless users choose the "NoRecurse" PCRE build option (using the --disable-stack-for-recursion argument to the configuration script), the calling application or operating system must allocate adequate stack space to PCRE. The amount of stack needed varies for each pattern. For example, completing the tests provided with pcretest needs 8 MB of stack space. While PCRE's documentation cautions that the "NoRecurse" build option makes PCRE slower than the alternative, using it avoids entirely the issue of stack overflows.

Consistent escaping rules

Like Perl, PCRE has consistent escaping rules: any non-alpha-numeric character may be escaped to mean its literal value by prefixing a \ (backslash) before the character. Any alpha-numeric character preceded by a backslash typically gives it a special meaning. In the case where the sequence has not been defined to be special, it will also be treated as a literal. However, this usage is not forward-compatible, as new versions of PCRE may give such patterns a special meaning. A good example of this is \R, which had no special meaning prior to PCRE 7. In POSIX regular expressions, sometimes backslashes escaped non-alpha-numerics (e.g. \.), and sometimes they introduced a special feature (e.g. \(\)).

Extended character classes

Single-letter character classes are supported in addition to the longer POSIX names. For example, \d matches any digit exactly as [[:digit:]] would in POSIX regular expressions.

Minimal matching (a.k.a. "ungreedy")

A ? may be placed after any repetition quantifier to indicate that the shortest match should be used. The default is to attempt the longest match first and backtrack through shorter matches. e.g. "a.*?b" would match "ab" in "ababab", where "a.*b" would match the entire string.

Unicode character properties

Unicode defines several properties for each character. Patterns in PCRE can match these properties. e.g. \p{Ps}.*?\p{Pe} would match a string beginning with any "opening punctuation" and ending with any "close punctuation" such as "[abc]". Since version 8.10, matching of certain "normal" metacharacters can be driven by Unicode properties when the compile option PCRE_UCP is set. The option can be set for a pattern by including (*UCP) at the start of pattern. The option alters behavior of the following metacharacters: \B, \b, \D, \d, \S, \s, \W, \w, and some of the POSIX character classes. For example, the set of characters matched by \w (word characters) is expanded to include letters and accented letters as defined by Unicode properties. Such matching is slower than the normal (ASCII-only) non-UCP alternative. Note that the UCP option requires the PCRE library to have been built to include UTF-8 and Unicode property support. Support for UTF-16 is included in version 8.30, while support for UTF-32 was added in version 8.32.

Multiline matching

^ and $ can match at the beginning and end of a string only, or at the start and end of each "line" within the string, depending on what options are set.

Newline/linebreak options

When PCRE is compiled, a newline default is selected. Which newline/linebreak is in effect affects where PCRE detects ^ line beginnings and $ ends (in multiline mode), as well as what matches dot (regardless of multiline mode, unless the dotall option (?s) is set). It also affects PCRE matching procedure (since version 7.0): when an unanchored pattern fails to match at the start of a newline sequence, PCRE advances past the entire newline sequence before retrying the match. If the newline option alternative in effect includes CRLF as one of the valid linebreaks, it does not skip the \n in a CRLF if the pattern contains specific \r or \n references (since version 7.3). Since version 8.10, the metacharacter \N always matches any character other than linebreak characters. It has the same behavior as . when the dotall option aka (?s) is not in effect.

The newline option can be altered with external options when a pattern is compiled and when it is run. Few application using PCRE provide users with the means to apply this setting through an external option. So, new in version 7.3, the newline option can also be stated at the start of the pattern using one of the following:

• (*LF) Newline is a linefeed character. Corresponding linebreaks can be matched with \n.
• (*CR) Newline is a carriage return. Corresponding linebreaks can be matched with \r.
• (*CRLF) Newline/linebreak is a carriage return followed by a linefeed. Corresponding linebreaks can be matched with \r\n.
• (*ANYCRLF) Any of the above encountered in the data will trigger newline processing. Corresponding linebreaks can be matched with (?>\r\n|[\r\n]) or with \R. See below for configuration and options concerning what matches backslash-R.
• (*ANY) Any of the above plus special Unicode linebreaks. When not in UTF-8 mode, corresponding linebreaks can be matched with (?>\r\n|\n|\x0b|\f|\r|\x85) or \R. In UTF-8 mode, two additional characters are recognized as line breaks with (*ANY): LS (line separator, U+2028), and PS (paragraph separator, U+2029). On Windows, in non-Unicode data, some of the ANY linebreak characters have other meanings. For example, \x85 can match a horizontal ellipsis, and if encountered while the ANY newline is in effect, it would trigger newline processing. See below for configuration and options concerning what matches backslash-R.

Backslash-R options

New in version 7.4: When PCRE is compiled, a default is selected for what matches \R. The default can be either to match the linebreaks corresponding to ANYCRLF or those corresponding to ANY. The default can be overridden when necessary by including (*BSR_UNICODE) or (*BSR_ANYCRLF) at the start of the pattern. When providing a (*BSR..) option, you can also provide a (*newline) option, e.g., (*BSR_UNICODE)(*ANY)rest-of-pattern. The backslash-R options also can be changed with external options by the application calling PCRE, when a pattern is compiled and when it is run.

Beginning of pattern options

Linebreak options such as (*LF) documented above; backslash-R options such as (*BSR_ANYCRLF) documented above; Unicode Character Properties option (*UCP) documented above; (*UTF8) option documented as follows: Since version 7.9, if your PCRE library has been compiled with UTF-8 support, you can specify the (*UTF8) option at the beginning of a pattern instead of setting an external option to invoke UTF-8 mode.

Named subpatterns

A sub-pattern (surrounded by parentheses, like (...)) may be named by including a leading ?P<name> after the opening parentesis. Named subpatterns are a feature that PCRE adopted from Python regular expressions. Since PCRE 7.0, named groups can be defined using (?<name>...) or (?'name'...), as well as (?P<name>...). Named groups can then be invoked with, for example: (?P=name.).

Backreferences

A pattern may refer back to the results of a previous match. For example, (a|b)c\1 would match either "aca" or "bcb" and would not match, for example, "acb".

Subroutines

While a backreference provides a mechanism to refer to that part of the subject that has previously matched a subpattern, a subroutine provides a mechanism to reuse an underlying previously defined subpattern. The subpattern's options, such as case independence, are fixed when the subpattern is defined. (a.c)(?1) would match "aacabc" or "abcadc", whereas using a backreference (a.c)\1 would not, though both would match "aacaac" or "abcabc". Starting with version 7.7 PCRE also supports a non-Perl Oniguruma construct for subroutines. They are specified using \g<subpat-number> or \g<subpat-name>.

Atomic grouping

Atomic grouping is a way of preventing backtracking in a pattern. For example, a++bc will match as many "a"s as possible and never back up to try one less.

Assertion	Lookbehind	Lookahead
Positive	(?<=pattern)	(?=pattern)
Negative	(?<!pattern)	(?!pattern)
Look-behind and look-ahead assertions in Perl regular expressions

Look-ahead and look-behind assertions

Patterns may assert that previous text or subsequent text contains a pattern without consuming matched text (zero-width assertion). For example, /\w+(?=\t)/ matches a word followed by a tab, without including the tab itself.

Look-behind assertions cannot be of uncertain length.

Since version 7.2, \K can be used in a pattern to reset the start of the current whole match. This provides a flexible alternative approach to look-behind assertions because the discarded part of the match (the part that precedes \K) need not be fixed in length.

Escape sequences for zero-width assertions

E.g. \b for matching zero-width "word boundaries", similar to (?<=\W)(?=\w)|(?<=\w)(?=\W)|^|$.

Comments

A comment begins with (?# and ends at the next closing parenthesis.

Recursive patterns

A pattern can refer back to itself recursively or to any subpattern. For example, the pattern \((a*|(?R))*\) will match any combination of balanced parentheses and "a"s.

Generic callouts

PCRE expressions can embed (?Cn), where n is some number. This will call out to an external user-defined function through the PCRE API and can be used to embed arbitrary code in a pattern.

Differences from Perl

Differences between PCRE and Perl (as of Perl 5.9.4) include but are not limited to:[5]

Recursive matches are atomic in PCRE and non atomic in Perl

This means that "<<!>!>!>><>>!>!>!>" =~ /^(<(?:[^<>]+|(?3)|(?1))*>)()(!>!>!>)$/ will match in Perl but not in PCRE.

The value of a capture buffer deriving from the ? quantifier (match 1 or 0 times) when nested in another quantified capture buffer is different

"aba" =~ /^(a(b)?)+$/; will result in $1 containing 'a' and $2 containing undef in Perl, but in PCRE will result in $2 containing 'b'.

PCRE allows named capture buffers to be given numeric names; Perl requires the name to follow the rule of barewords

This means that \g{} is unambiguous in Perl, but potentially ambiguous in PCRE. This is no longer a difference since PCRE 8.34 (released on 15-December-2013), which no longer allows group names to start with a digit.[6]

PCRE allows alternatives within lookbehind to be different lengths

Within lookbehind assertions, both PCRE and Perl require fixed-length patterns. That is, both PCRE and Perl disallow variable-length patterns. However, Perl requires all alternative branches of a lookbehind assertion to be the same length as each other, whereas PCRE allows those alternative branches to have different lengths from each other as long as each branch still has a fixed length.

PCRE does not support certain "experimental" Perl constructs

such as (??{...}) (a callback whose return is evaluated as being part of the pattern) nor the (?{}) construct, although the latter can be emulated using (?Cn). Recursion control verbs added in the Perl 5.9.x series are also not supported. Support for experimental backtracking control verbs (added in Perl 5.10) is available in PCRE since version 7.3. They are (*FAIL), (*F), (*PRUNE), (*SKIP), (*THEN), (*COMMIT), and (*ACCEPT). Perl's corresponding use of arguments with backtracking control verbs is not generally supported. Note however that since version 8.10, PCRE supports the following verbs with a specified argument: (*MARK:markName), (*SKIP:markName), (*PRUNE:markName), and (*THEN:markName).

PCRE and Perl are slightly different in their tolerance of erroneous constructs

Perl allows quantifiers on the (?!) construct, which is meaningless but harmless (albeit inefficient); PCRE produces an error in versions before 8.13.

PCRE has a hard limit on recursion depth, Perl does not

With default build options "bbbbXcXaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa" =~ /.X(.+)+X/ will fail to match due to stack overflow, but Perl will match this correctly. Perl uses the heap for recursion and has no hard limit for recursion depth, whereas PCRE has a compile time hard limit.

With the exception of the above points PCRE is capable of passing the tests in the Perl 't/op/re_tests' file, one of the main syntax level regression tests for Perl's regular expression engine.

See also

• Comparison of regular-expression engines

Footnotes

1. The core PCRE library provides matching only; the C++ wrapper, if used, adds multiple match and replacement functionality.

References

1. "Index of /pub/pcre/". Retrieved 23 February 2019.
2. "Index of /pub/pcre/". Retrieved 16 April 2019.
3. Philip Hazel (December 1999). "Exim and PCRE: How free software hijacked my life". p. 7. Retrieved 2 May 2018.
4. "PCRE - Perl-compatible regular expressions (original API)". University of Cambridge. 2009.
5. Philip Hazel (2017-04-18). "Differences Between PCRE2 and Perl". PCRE. Retrieved 2018-08-11.
6. "PCRE changelog". pcre.org. Retrieved 23 August 2019. Perl no longer allows group names to start with digits, so I have made this change also in PCRE.

Notes

• ChangeLog for PCRE

External links

• Official website
• "PCRE". Development mailing list. Exim.
• "PCRE". Bug Tracker. Exim.

University of Cambridge Computing Service (CSX)

• Maclaren, Nick; Hazel, Philip (Mar 2, 2010). "Pattern Matching Using Regular Expressions" (pdf). CSX.
• "pcre 8.43". Windows Cygwin x86-64. CSX. Apr 2019.