New Regular Expression Features in Tcl 8.1

Tcl 8.1 now handles advanced regular expressions (REs). Previous regular expression handling is almost unchanged except that clumsy handling of escapes like \n has been much improved, and a few escapes that were previously legal (but useless) now won't work.

Note that a few advanced features aren't useful yet but are ready for future Tcl releases. That's because Tcl 8.1 (apart from the regular expression engine) implements only the Unicode locale (where all characters sort in Unicode order, there are no multi-character collating elements and no equivalence classes).

This document has an overview of the new regular expression features. For exact semantics and more details, see the new re_syntax(n) reference page. (The re_syntax(n) page was split from the 8.1 regexp(n) reference page, which used to cover RE syntax for all Tcl commands.) This howto document covers:

1. Regular Expression Overview

• What are Regular Expressions?
• Regular Expressions in Tcl 8.0 and Before
• Overview of regexp and regsub
• Backslash Processing

2. Regular Expressions in Tcl 8.1

• Non-Greedy Quantifiers
• Backslash Escapes (\xxx)
• Bounds ({})
• Character Classes ([: :])
• Collating Elements ([. .])
• Equivalence Classes ([= =])
• Noncapturing Subpatterns ((?:re))
• Lookahead Assertions ((?=re) and (?!re))
• Switches
• Options ((?xyz)), Directors (***)

3. Summary: Regular Expression changes in Tcl 8.1

Part 1. Regular Expression Overview

This Part describes regular expressions (REs), explains REs from Tcl 8.0 and before, and describes the Tcl regexp and regsub commands. Part Two describes the new Tcl 8.1 REs.

What are Regular Expressions?

A regular expression uses metacharacters (characters that assume special meaning for matching other characters) such as *, [], $ and .. For example, the RE [Hh]ello!* would match Hello and hello and Hello! (and hello!!!!!). The RE [Hh](ello|i)!* would match Hello and Hi and Hi! (and so on). A backslash (\) disables the special meaning of the following character, so you could match the string [Hello] with the RE \[Hello\].

Regular Expressions in Tcl 8.0 and Before

Regular expressions in Tcl 8.0 and before had the following metacharacters:

.	Match any single character (e.g., m.d matches mad, mod, m3d, etc.)
[]	Bracket expression: Match any one of the enclosed characters (e.g., [a-z0-9_] matches a lowercase ASCII letter, a digit, or an underscore)
^	Start-of-string anchor: Match only at the start of a string (e.g., ^hi matches hi and his but not this)
$	End-of-string anchor: Match only at the end of a string (e.g., hi$ matches hi and chi but not this)
*	Zero-or-more quantifier: makes the previous part of the RE match zero or more times (e.g., M.*D matches MD, MAD, MooD, M.D, etc.)
?	Zero-or-one quantifier: makes the previous part of the RE match zero or one time (e.g., hi!? matches hi or hi!)
+	One-or-more quantifier: makes the previous part of the RE match one or more times (e.g., hi!+ matches hi! or hi!! or hi!!! or ...)
|	Alternation (vertical bar): Match just one alternative (e.g., this|that matches this or that)
()	Subpattern: Group part of the RE. Many uses, such as: Makes a quantifier apply to a group of text (e.g., ([0-9A-F][0-9A-F])+ matches groups of two hexadecimal digits: A9 or AB03 or 8A6E00, but not A or A2C). Set limits for alternation (e.g., "Eat (this|that)!" matches "Eat this!" or "Eat that!"). Used for subpattern matching in the regexp and regsub commands.
\	Escape: Disables meaning of the following metacharacter (e.g., a\.* matches a or a. or a.. or etc.). Note that \ also has special meaning to the Tcl interpreter (and to applications, such as C compilers).

The syntax above is supported in Tcl 8.1. Tcl 8.1 also supports advanced regular expressions (AREs). These powerful expressions are introduced in more detail in Part Two. Briefly, though, AREs support backreferences, lookahead, non-greedy matching, many escapes, features that are useful for internationalization (handling collation elements, equivalence classes and character classes), and much more.

The Tcl 8.1 regular expression engine almost always interprets 8.0-style REs correctly. In the few cases that it doesn't, and when the problem is too difficult to fix, the 8.1 engine has an option to select 8.0 ("ERE") interpretation.

Overview of regexp and regsub

• regexp compares a string to an RE. It returns a value of 1 if the RE matches part or all of the string or 0 if there's no match. Optionally, it stores the matched part of the string in a variable (and also can store subparts of a string in multiple variables). For example, to compare the string in $line against the RE [Hh]ello!*, you would write:

  regexp {[Hh]ello!*} $line match

  If part or all of the line variable matches the RE, regexp stores the matching part in the match variable and returns a value of 1.

• regsub substitutes part of a string that matches an RE. For instance, the following command edits the string from $in_line to replace all space or tab characters with a single space character; the edited line is stored in the out_line variable:

  regsub -all {[ \t]+} $in_line { } out_line

  Please also read the following section about backslash processing.

Backslash Processing

We actually used the 8.1 syntax above; the example wouldn't have worked under 8.0! In Tcl 8.0 and before, you had to surround the regular expression with double quotes so the Tcl backslash processor could convert the \t to a literal tab character. The square brackets had to be hidden from the backslash processor by adding backslashes before them, which made code harder to read and possibly more error-prone. Here's the previous example rewritten for Tcl 8.0 and before:

regsub -all "\[ \t\]+" $in_line { } out_line

For more about the simplified 8.1 syntax, see the section Backslash Escapes.

Part 2. Regular Expressions in Tcl 8.1

Tcl 8.1 regular expressions are basically a superset of 8.0 REs. This howto document has an overview of the new features. Please see the re_syntax(n) reference page for exact semantics and more details.

Non-Greedy Quantifiers

Let's start by storing some HTML text in a variable, then using two regexp commands to match it. The first RE is greedy, and the second is non-greedy:

% set x {<EM>He</EM> sits, but <EM>she</EM> stands.}
<EM>He</EM> sits, but <EM>she</EM> stands.
% regexp {<EM>.*</EM>} $x match; set match
<EM>He</EM> sits, but <EM>she</EM>
% regexp {<EM>.*?</EM>} $x match; set match
<EM>He</EM>

Here are a new string and another pair of REs to match it:

% set y {123zzz456}
123zzz456
% regexp {3z*} $y match; set match
3zzz
% regexp {3z*?} $y match; set match
3

To review, the greedy quantifiers from Tcl 8.0 are: *, +, and ?. So the non-greedy quantifiers (added in Tcl 8.1) are: *?, +?, and ??. Tcl 8.1 also has the new quantifiers {m}, {m,}, and {m,n}, as well as the non-greedy versions {m}?, {m,}?, and {m,n}?. The section on bounds explains -- and has more examples of non-greedy matching.

Backslash Escapes

The Tcl 8.1 regular expression engine interprets backslash escapes itself. So now regexp {\n} matches a newline, not the character n. REs are simpler to write in 8.1 because of this. (You can still write regexp "\n" -- and let Tcl conversion happen inside the double quotes -- so most old code will still work.)

One of the most important changes in 8.1 is that a backslash inside a bracket expression is treated as the start of an escape. In 8.0 and before, a backslash inside brackets was treated as a literal backslash character. For example, in 8.0 and before, regexp {[a\n]} would match the characters a, \, or n. But in 8.1, regexp {[a\n]} would match the characters a or newline (because \n is the backslash escape for "newline").

Tcl 8.1 has also added many new backslash escapes. For instance, \d matches a digit. Some of these are listed below, and the re_syntax(n) reference page has the whole list.

In Tcl 8.1 regular expressions (but not in other parts of the language), it's illegal to use a backslash before a non-metacharacter unless it makes a valid escape. So regexp {\p} is now an error. If you have code that (for some bizarre reason) has regular expressions with a backslash before a non-metacharacter, like regexp {\p}, you'll need to fix it.

As explained above, the Tcl 8.1 regular expression engine now interprets backslash sequences like \n to mean "newline". It also has four new kinds of escapes: character entry escapes, class shorthand escapes, constraint escapes, and back references. Here's an introduction. (The re_syntax(n) page has full details.)

• A character entry escape is a convenient way to enter a non-printing or other difficult character. For instance, \n represents a newline character. \uwxyz (where wxyz is hexadecimal) represents the Unicode character U+wxyz.
• Class shorthand escapes are shorthand for common character classes. For example, \d stands for [[:digit:]], which means "any single digit."
• A constraint escape constrains an RE to match only at a certain place. For example, the constraint escape \m matches only at the start of a word -- so the RE \mhi will match the third word in the string he said hi but won't match he said thigh.
• A back reference matches the same string that was matched by a previous parenthesized subexpression. (This works like subexpressions in regsub, but it's used for matching instead of extracting.) For example, (X.*Y)\1 matches any doubled string that starts with X and ends with Y, such as XYXY, XabcYXabcY, X--YX--Y, etc.

Finally, remember that (as in Tcl 8.0 and before) some applications, such as C compilers, interpret these backslash sequences themselves before the regular expression engine sees them. You may need to double (or quadruple, etc.) the number of backslashes for these applications. Still, in straight Tcl 8.1 code, writing backslash escapes is now both simpler and more powerful than in 8.0 and before.

Bounds

These operators come in three greedy forms: {m}, {m,}, and {m,n}. The corresponding non-greedy forms are {m}?, {m,}?, and {m,n}?.

• The {m} quantifier matches exactly m occurrences. So does {m}?. For example, either #{70} or #{70}? match a string of exactly 70 # characters.
• The {m,} quantifier matches at least m occurrences. Here's a demo of the greedy and non-greedy versions:

  
  % set x {a##b#######c}
  a##b#######c
  % regexp {#{4,}} $x match; set match
  #######
  % regexp {#{4,}?} $x match; set match
  ####
  

  Notice that the first two number signs (##) in the string are never matched because there aren't at least four of them.
• The {m,n} quantifier matches at least m but no more than n occurrences.

  For example, the RE http://([^/]+/?){1,3} would match Web URLs that have 3 components (like http://xyz.fr/euro/billets.htm), or with 2 components (like http://xyz.fr/euro/, or with just 1 component (like http://xyz.fr). The RE matches a final slash (/) if there is one. As always, a greedy match will match as long a string as possible: it would try for 3 matches.

  A non-greedy quantifier would try to match the least (1 match). But be careful: http://([^/]+/?){1,3}? won't match all the way to a possible slash because it matches the fewest characters possible! (With input http://xyz.fr/, that RE would match just http://x.) This brings up one of the many subtleties in these advanced regular expressions: that the outer non-greedy quantifier overrides the inner greedy quantifiers and makes all quantifiers non-greedy! There's an explanation in re_syntax(n) reference page section named Matching.

Character Classes

For instance, the character class named digit stands for any of the digits 0-9 (zero through nine). The character class is written with the class name inside a set of brackets and colons, like this: [[:digit:]]. The old familiar expression for digits is written as a range: [0-9]. When you compare the new character class to the old range version, you can see that the outer square brackets are the same in both. So a character class is written [:classname:].

The table below describes the Tcl 8.1 character classes.

alpha	A letter (includes many non-ASCII characters).
upper	An upper-case letter.
lower	A lower-case letter.
digit	A decimal digit.
xdigit	A hexadecimal digit.
alnum	An alphanumeric (letter or digit).
print	An alphanumeric. (Same as alnum.)
blank	A space or tab character.
space	A character producing white space in displayed text. (Includes en-space, hair space, many others.)
punct	A punctuation character.
graph	A character with a visible representation.
cntrl	A control character.

You can use more than one character class in a bracket expression. You can also mix character classes with ranges and single characters. For instance, [[:digit:]a-cx-z] would match a digit (0-9), a, b, c, x, y, or z -- and [^[:digit:]a-cx-z] would match any character except those. This syntax can take some time to get familiar with! The key is to look for the character class (here, [:digit:]) inside the bracket expression.

The advantage of character classes (like [:alpha:]) over explicit ranges in brackets (like [a-z]) is that character classes include characters that aren't easy to type on ASCII keyboards. For example, the Spanish language includes the character ñ. It doesn't fall into the range [a-z], but it is in the Tcl 8.1 character class [:alpha:]. In the same way, the Spanish punctuation character ¡ isn't in a list of punctuation characters like [.!?,], but it is part of [:punct:].

Tcl 8.1 has a standard set of character classes that are defined in the source code file generic/regc_locale.c. Tcl 8.1 has one locale defined: the Unicode locale. It may support other locales (and other character classes) in the future.

Collating Elements

% regexp {[[.number-sign.]]+} {123###456} match
1
% set match
###

Suppose ch and c sort next to each other in your dialect, and ch is treated as an atomic character. The example bracket expression below uses two collating symbols. It matches one or more of ch and c. But it doesn't match an h standing alone:

% set input "cchchh"
cchchh
% regexp {[[.ch.][.c.]]+} $input match; set match
cchch

% set input chb
% regexp {[^[.c.]]b} $input match; set match
chb

Equivalence Classes

Note: Tcl 8.1 only implements the Unicode locale. It doesn't define any equivalence classes. So, although the Tcl regular expression engine supports equivalence classes, the examples below are not supported in Tcl 8.1. (Future versions of Tcl may define equivalence classes.)

Let's imagine that both of the characters A and a fall at the same place in the collating sequence; they belong to the same equivalence class. In that case, both of the bracket expressions [[=A=]b] and [[=a=]b] are equivalent to writing [Aab]. As another example, if o and ô are members of an equivalence class, then all of the bracket expressions [[=o=]], [[=ô=]], and [oô] match those same two characters.

Noncapturing Subpatterns

If you don't want parentheses to capture text, add ?: after the opening parenthesis. For instance, in the example below, the subexpression (?:http|ftp) matches either http or ftp but doesn't capture it. So the back reference \1 will hold the end of the URL (from the second set of parentheses):

% set x http://www.activestate.com
http://www.activestate.com
% regsub {(?:http|ftp)://(.*)} $x {The hostname is \1} answer
1
% set answer
The hostname is www.activestate.com

Lookahead Assertions

A positive lookahead has the form (?=re). It matches at any place ahead where there's a substring like re. A negative lookahead has the form (?!re). It matches at any point where the regular expression re does not match. Let's see some examples:

% set x http://www.activestate.com
http://www.activestate.com
% regexp {^[^:]+(?=.*\.com$)} $x match
1
% set match
http
% regexp {^[^:]+(?=.*\.edu$)} $x match
0
% regexp {^[^:]*(?!.*\.edu$)} $x match
1
% set match
http

Tcl 8.1 lets you document complex regular expressions by embedding comments. See the next section.

Switches

Complex REs can be difficult to document. The -expanded switch sets expanded syntax, which lets you add comments within a regular expression. Comments start with a # character; whitespace is ignored. This is mostly for scripting -- but you can also use it on a command line, as we'll do in the example below. Let's look the same RE twice: first in the standard compact syntax, and second in expanded syntax:

% set x http://www.activestate.com
http://www.activestate.com
% regexp {^[^:]+(?=.*\.com$)} $x match
1
% set match
http
% regexp -expanded {
  ^             # beginning of string
  [^:]+         # all characters to the first colon
  (?=           # begin positive lookahead
    .*\.com$    # for a trailing .com
  )             # end positive lookahead
} $x match
1
% set match
http

The other important new switch we'll cover here is -line. It enables newline-sensitive matching. By default (without -line), Tcl regular expressions have always treated newlines as an ordinary character. For example, if a string contains several lines (separated by newline characters), the end-of-string anchor $ wouldn't match at any of the embedded newlines. To write code that matched line-by-line, you had to read input lines one by one and do separate matches against each line.

With the -line switch, the metacharacters ^, $, ., and [] treat a newline as the end of a "line." So, for example, the regular expression ^San Jose matches the second line of input below:

% set x {Dolores Sanchez
San Jose, CA}
Dolores Sanchez
San Jose, CA
% regexp {^San Jose} $x match
0
% regexp -line {^San Jose} $x match
1
% set match
San Jose

Options, Directors

An 8.1 RE can start with embedded options. These look like (?xyz), where xyz are one or more option letters. For instance, (?i)ouch matches OUCH because i is the "case-insensitive matching" option. Other options include (?e), which marks the rest of the regular expression as an 8.0-style RE -- to let you avoid confusion with the new 8.1 syntax.

An RE can also start with three asterisks, which is a director. For example, ***= is the director that says the rest of the regular expression is literal text. So the RE ***=(?i)ouch matches exactly (?i)ouch; the (?i) isn't treated as an option.

Part 3. Summary: Regular Expression Changes in Tcl 8.1

This table below summarizes the new syntax:

{m}	Matches m instances of the previous pattern item
{m}?	Matches m instances of the previous pattern item. Non-greedy.
{m,}	Matches m or more instances of the previous pattern item.
{m,}?	Matches m or more instances of the previous pattern item. Non-greedy.
{m,n}	Matches m through n instances of the previous pattern item.
{m,n}?	Matches m through n instances of the previous pattern item. Non-greedy.
*?	Matches zero or more of the previous pattern item. Non-greedy.
+?	Matches one or more of the previous pattern item. Non-greedy.
??	Matches zero or one of the previous pattern item. Non-greedy.
(?:re)	Groups a subpattern, re, but does not capture the result.
(?=re)	Positive lookahead. Matches the point where re begins.
(?!re)	Negative lookahead. Matches any point where re does not begin.
\c	One of many backslash escapes.
[. .]	Delimits a collating element within a bracketed expression.
[= =]	Delimits an equivalence class within a bracketed expression.
[: :]	Delimits a character class within a bracketed expression.
(?abc)	Embedded options a, b, and c
***	Director

Some of the new switches for regexp and regsub are:

-expanded	Enable expanded syntax (for comments)
-line	Enable newline-sensitive matching
-linestop	Make [] and . stop at newlines.
-lineanchor	Make ^ and $ match the start and end of a line.

Add link to comments for /doc/howto/regexp81.html

This is the main Tcl Developer Xchange site, www.tcl-lang.org .    About this Site | [email protected]
Home | About Tcl/Tk | Software | Core Development | Community | Documentation