| .ds re \fBre2c\fP |
| .ds le \fBlex\fP |
| .ds rx regular expression |
| .ds lx \fIl\fP-expression |
| .TH RE2C 1 "8 April 1994" "Version 0.5" |
| \"$Log: re2c.1,v $ |
| \"Revision 1.1.1.1 2012/03/29 17:20:59 uid42307 |
| \"initial import |
| \" |
| \"Revision 1.1 2002/04/07 22:27:06 peter |
| \"Initial revision |
| \" |
| \"Revision 1.2 1994/04/16 15:50:32 peterr |
| \"Fix bug in simple example. |
| \" |
| \"Revision 1.1 1994/04/08 15:39:09 peterr |
| \"Initial revision |
| \" |
| .SH NAME |
| re2c \- convert regular expressions to C/C++ |
| |
| .SH SYNOPSIS |
| \*(re [\fB-esb\fP] \fIname\fP |
| |
| .SH DESCRIPTION |
| \*(re is a preprocessor that generates C-based recognizers from regular |
| expressions. |
| The input to \*(re consists of C/C++ source interleaved with |
| comments of the form \fC/*!re2c\fP ... \fC*/\fP which contain |
| scanner specifications. |
| In the output these comments are replaced with code that, when |
| executed, will find the next input token and then execute |
| some user-supplied token-specific code. |
| |
| For example, given the following code |
| |
| .in +3 |
| .nf |
| #define NULL ((char*) 0) |
| char *scan(char *p){ |
| char *q; |
| #define YYCTYPE char |
| #define YYCURSOR p |
| #define YYLIMIT p |
| #define YYMARKER q |
| #define YYFILL(n) |
| /*!re2c |
| [0-9]+ {return YYCURSOR;} |
| [\\000-\\377] {return NULL;} |
| */ |
| } |
| .fi |
| .in -3 |
| |
| \*(re will generate |
| |
| .in +3 |
| .nf |
| /* Generated by re2c on Sat Apr 16 11:40:58 1994 */ |
| #line 1 "simple.re" |
| #define NULL ((char*) 0) |
| char *scan(char *p){ |
| char *q; |
| #define YYCTYPE char |
| #define YYCURSOR p |
| #define YYLIMIT p |
| #define YYMARKER q |
| #define YYFILL(n) |
| { |
| YYCTYPE yych; |
| unsigned int yyaccept; |
| goto yy0; |
| yy1: ++YYCURSOR; |
| yy0: |
| if((YYLIMIT - YYCURSOR) < 2) YYFILL(2); |
| yych = *YYCURSOR; |
| if(yych <= '/') goto yy4; |
| if(yych >= ':') goto yy4; |
| yy2: yych = *++YYCURSOR; |
| goto yy7; |
| yy3: |
| #line 10 |
| {return YYCURSOR;} |
| yy4: yych = *++YYCURSOR; |
| yy5: |
| #line 11 |
| {return NULL;} |
| yy6: ++YYCURSOR; |
| if(YYLIMIT == YYCURSOR) YYFILL(1); |
| yych = *YYCURSOR; |
| yy7: if(yych <= '/') goto yy3; |
| if(yych <= '9') goto yy6; |
| goto yy3; |
| } |
| #line 12 |
| |
| } |
| .fi |
| .in -3 |
| |
| .SH OPTIONS |
| \*(re provides the following options: |
| .TP |
| \fB-e\fP |
| Cross-compile from an ASCII platform to an EBCDIC one. |
| .TP |
| \fB-s\fP |
| Generate nested \fCif\fPs for some \fCswitch\fPes. Many compilers need this |
| assist to generate better code. |
| .TP |
| \fB-b\fP |
| Implies \fB-s\fP. Use bit vectors as well in the attempt to coax better |
| code out of the compiler. Most useful for specifications with more than a |
| few keywords (e.g. for most programming languages). |
| |
| .SH "INTERFACE CODE" |
| Unlike other scanner generators, \*(re does not generate complete scanners: |
| the user must supply some interface code. |
| In particular, the user must define the following macros: |
| .TP |
| \fCYYCHAR\fP |
| Type used to hold an input symbol. |
| Usually \fCchar\fP or \fCunsigned char\fP. |
| .TP |
| \fCYYCURSOR\fP |
| \*(lx of type \fC*YYCHAR\fP that points to the current input symbol. |
| The generated code advances \fCYYCURSOR\fP as symbols are matched. |
| On entry, \fCYYCURSOR\fP is assumed to point to the first character of the |
| current token. On exit, \fCYYCURSOR\fP will point to the first character of |
| the following token. |
| .TP |
| \fCYLIMIT\fP |
| Expression of type \fC*YYCHAR\fP that marks the end of the buffer |
| (\fCYLIMIT[-1]\fP is the last character in the buffer). |
| The generated code repeatedly compares \fCYYCURSOR\fP to \fCYLIMIT\fP |
| to determine when the buffer needs (re)filling. |
| .TP |
| \fCYYMARKER\fP |
| \*(lx of type \fC*YYCHAR\fP. |
| The generated code saves backtracking information in \fCYYMARKER\fP. |
| .TP |
| \fCYYFILL(\fP\fIn\fP\fC)\fP |
| The generated code "calls" \fCYYFILL\fP when the buffer needs |
| (re)filling: at least \fIn\fP additional characters should |
| be provided. \fCYYFILL\fP should adjust \fCYYCURSOR\fP, \fCYYLIMIT\fP and |
| \fCYYMARKER\fP as needed. Note that for typical programming languages |
| \fIn\fP will be the length of the longest keyword plus one. |
| |
| .SH "SCANNER SPECIFICATIONS" |
| Each scanner specification consists of a set of \fIrules\fP and name |
| definitions. |
| Rules consist of a regular expression along with a block of C/C++ code that |
| is to be executed when the associated regular expression is matched. |
| Name definitions are of the form |
| ``\fIname\fP \fC=\fP \fIregular expression\fP\fC;\fP''. |
| |
| .SH "SUMMARY OF RE2C REGULAR EXPRESSIONS" |
| .TP |
| \fC"foo"\fP |
| the literal string \fCfoo\fP. |
| ANSI-C escape sequences can be used. |
| .TP |
| \fC[xyz]\fP |
| a "character class"; in this case, |
| the \*(rx matches either an '\fCx\fP', a '\fCy\fP', or a '\fCz\fP'. |
| .TP |
| \fC[abj-oZ]\fP |
| a "character class" with a range in it; |
| matches an '\fCa\fP', a '\fCb\fP', any letter from '\fCj\fP' through '\fCo\fP', |
| or a '\fCZ\fP'. |
| .TP |
| \fIr\fP\fC\e\fP\fIs\fP |
| match any \fIr\fP which isn't an \fIs\fP. \fIr\fP and \fIs\fP must be regular expressions |
| which can be expressed as character classes. |
| .TP |
| \fIr\fP\fC*\fP |
| zero or more \fIr\fP's, where \fIr\fP is any regular expression |
| .TP |
| \fC\fIr\fP\fC+\fP |
| one or more \fIr\fP's |
| .TP |
| \fC\fIr\fP\fC?\fP |
| zero or one \fIr\fP's (that is, "an optional \fIr\fP") |
| .TP |
| name |
| the expansion of the "name" definition (see above) |
| .TP |
| \fC(\fP\fIr\fP\fC)\fP |
| an \fIr\fP; parentheses are used to override precedence |
| (see below) |
| .TP |
| \fIrs\fP |
| an \fIr\fP followed by an \fIs\fP ("concatenation") |
| .TP |
| \fIr\fP\fC|\fP\fIs\fP |
| either an \fIr\fP or an \fIs\fP |
| .TP |
| \fIr\fP\fC/\fP\fIs\fP |
| an \fIr\fP but only if it is followed by an \fIs\fP. The s is not part of |
| the matched text. This type of \*(rx is called "trailing context". |
| .LP |
| The regular expressions listed above are grouped according to |
| precedence, from highest precedence at the top to lowest at the bottom. |
| Those grouped together have equal precedence. |
| |
| .SH "A LARGER EXAMPLE" |
| .LP |
| .in +3 |
| .nf |
| #include <stdlib.h> |
| #include <stdio.h> |
| #include <fcntl.h> |
| #include <string.h> |
| |
| #define ADDEQ 257 |
| #define ANDAND 258 |
| #define ANDEQ 259 |
| #define ARRAY 260 |
| #define ASM 261 |
| #define AUTO 262 |
| #define BREAK 263 |
| #define CASE 264 |
| #define CHAR 265 |
| #define CONST 266 |
| #define CONTINUE 267 |
| #define DECR 268 |
| #define DEFAULT 269 |
| #define DEREF 270 |
| #define DIVEQ 271 |
| #define DO 272 |
| #define DOUBLE 273 |
| #define ELLIPSIS 274 |
| #define ELSE 275 |
| #define ENUM 276 |
| #define EQL 277 |
| #define EXTERN 278 |
| #define FCON 279 |
| #define FLOAT 280 |
| #define FOR 281 |
| #define FUNCTION 282 |
| #define GEQ 283 |
| #define GOTO 284 |
| #define ICON 285 |
| #define ID 286 |
| #define IF 287 |
| #define INCR 288 |
| #define INT 289 |
| #define LEQ 290 |
| #define LONG 291 |
| #define LSHIFT 292 |
| #define LSHIFTEQ 293 |
| #define MODEQ 294 |
| #define MULEQ 295 |
| #define NEQ 296 |
| #define OREQ 297 |
| #define OROR 298 |
| #define POINTER 299 |
| #define REGISTER 300 |
| #define RETURN 301 |
| #define RSHIFT 302 |
| #define RSHIFTEQ 303 |
| #define SCON 304 |
| #define SHORT 305 |
| #define SIGNED 306 |
| #define SIZEOF 307 |
| #define STATIC 308 |
| #define STRUCT 309 |
| #define SUBEQ 310 |
| #define SWITCH 311 |
| #define TYPEDEF 312 |
| #define UNION 313 |
| #define UNSIGNED 314 |
| #define VOID 315 |
| #define VOLATILE 316 |
| #define WHILE 317 |
| #define XOREQ 318 |
| #define EOI 319 |
| |
| typedef unsigned int uint; |
| typedef unsigned char uchar; |
| |
| #define BSIZE 8192 |
| |
| #define YYCTYPE uchar |
| #define YYCURSOR cursor |
| #define YYLIMIT s->lim |
| #define YYMARKER s->ptr |
| #define YYFILL(n) {cursor = fill(s, cursor);} |
| |
| #define RET(i) {s->cur = cursor; return i;} |
| |
| typedef struct Scanner { |
| int fd; |
| uchar *bot, *tok, *ptr, *cur, *pos, *lim, *top, *eof; |
| uint line; |
| } Scanner; |
| |
| uchar *fill(Scanner *s, uchar *cursor){ |
| if(!s->eof){ |
| uint cnt = s->tok - s->bot; |
| if(cnt){ |
| memcpy(s->bot, s->tok, s->lim - s->tok); |
| s->tok = s->bot; |
| s->ptr -= cnt; |
| cursor -= cnt; |
| s->pos -= cnt; |
| s->lim -= cnt; |
| } |
| if((s->top - s->lim) < BSIZE){ |
| uchar *buf = (uchar*) |
| malloc(((s->lim - s->bot) + BSIZE)*sizeof(uchar)); |
| memcpy(buf, s->tok, s->lim - s->tok); |
| s->tok = buf; |
| s->ptr = &buf[s->ptr - s->bot]; |
| cursor = &buf[cursor - s->bot]; |
| s->pos = &buf[s->pos - s->bot]; |
| s->lim = &buf[s->lim - s->bot]; |
| s->top = &s->lim[BSIZE]; |
| free(s->bot); |
| s->bot = buf; |
| } |
| if((cnt = read(s->fd, (char*) s->lim, BSIZE)) != BSIZE){ |
| s->eof = &s->lim[cnt]; *(s->eof)++ = '\\n'; |
| } |
| s->lim += cnt; |
| } |
| return cursor; |
| } |
| |
| int scan(Scanner *s){ |
| uchar *cursor = s->cur; |
| std: |
| s->tok = cursor; |
| /*!re2c |
| any = [\\000-\\377]; |
| O = [0-7]; |
| D = [0-9]; |
| L = [a-zA-Z_]; |
| H = [a-fA-F0-9]; |
| E = [Ee] [+-]? D+; |
| FS = [fFlL]; |
| IS = [uUlL]*; |
| ESC = [\\\\] ([abfnrtv?'"\\\\] | "x" H+ | O+); |
| */ |
| |
| /*!re2c |
| "/*" { goto comment; } |
| |
| "auto" { RET(AUTO); } |
| "break" { RET(BREAK); } |
| "case" { RET(CASE); } |
| "char" { RET(CHAR); } |
| "const" { RET(CONST); } |
| "continue" { RET(CONTINUE); } |
| "default" { RET(DEFAULT); } |
| "do" { RET(DO); } |
| "double" { RET(DOUBLE); } |
| "else" { RET(ELSE); } |
| "enum" { RET(ENUM); } |
| "extern" { RET(EXTERN); } |
| "float" { RET(FLOAT); } |
| "for" { RET(FOR); } |
| "goto" { RET(GOTO); } |
| "if" { RET(IF); } |
| "int" { RET(INT); } |
| "long" { RET(LONG); } |
| "register" { RET(REGISTER); } |
| "return" { RET(RETURN); } |
| "short" { RET(SHORT); } |
| "signed" { RET(SIGNED); } |
| "sizeof" { RET(SIZEOF); } |
| "static" { RET(STATIC); } |
| "struct" { RET(STRUCT); } |
| "switch" { RET(SWITCH); } |
| "typedef" { RET(TYPEDEF); } |
| "union" { RET(UNION); } |
| "unsigned" { RET(UNSIGNED); } |
| "void" { RET(VOID); } |
| "volatile" { RET(VOLATILE); } |
| "while" { RET(WHILE); } |
| |
| L (L|D)* { RET(ID); } |
| |
| ("0" [xX] H+ IS?) | ("0" D+ IS?) | (D+ IS?) | |
| (['] (ESC|any\\[\\n\\\\'])* [']) |
| { RET(ICON); } |
| |
| (D+ E FS?) | (D* "." D+ E? FS?) | (D+ "." D* E? FS?) |
| { RET(FCON); } |
| |
| (["] (ESC|any\\[\\n\\\\"])* ["]) |
| { RET(SCON); } |
| |
| "..." { RET(ELLIPSIS); } |
| ">>=" { RET(RSHIFTEQ); } |
| "<<=" { RET(LSHIFTEQ); } |
| "+=" { RET(ADDEQ); } |
| "-=" { RET(SUBEQ); } |
| "*=" { RET(MULEQ); } |
| "/=" { RET(DIVEQ); } |
| "%=" { RET(MODEQ); } |
| "&=" { RET(ANDEQ); } |
| "^=" { RET(XOREQ); } |
| "|=" { RET(OREQ); } |
| ">>" { RET(RSHIFT); } |
| "<<" { RET(LSHIFT); } |
| "++" { RET(INCR); } |
| "--" { RET(DECR); } |
| "->" { RET(DEREF); } |
| "&&" { RET(ANDAND); } |
| "||" { RET(OROR); } |
| "<=" { RET(LEQ); } |
| ">=" { RET(GEQ); } |
| "==" { RET(EQL); } |
| "!=" { RET(NEQ); } |
| ";" { RET(';'); } |
| "{" { RET('{'); } |
| "}" { RET('}'); } |
| "," { RET(','); } |
| ":" { RET(':'); } |
| "=" { RET('='); } |
| "(" { RET('('); } |
| ")" { RET(')'); } |
| "[" { RET('['); } |
| "]" { RET(']'); } |
| "." { RET('.'); } |
| "&" { RET('&'); } |
| "!" { RET('!'); } |
| "~" { RET('~'); } |
| "-" { RET('-'); } |
| "+" { RET('+'); } |
| "*" { RET('*'); } |
| "/" { RET('/'); } |
| "%" { RET('%'); } |
| "<" { RET('<'); } |
| ">" { RET('>'); } |
| "^" { RET('^'); } |
| "|" { RET('|'); } |
| "?" { RET('?'); } |
| |
| |
| [ \\t\\v\\f]+ { goto std; } |
| |
| "\\n" |
| { |
| if(cursor == s->eof) RET(EOI); |
| s->pos = cursor; s->line++; |
| goto std; |
| } |
| |
| any |
| { |
| printf("unexpected character: %c\\n", *s->tok); |
| goto std; |
| } |
| */ |
| |
| comment: |
| /*!re2c |
| "*/" { goto std; } |
| "\\n" |
| { |
| if(cursor == s->eof) RET(EOI); |
| s->tok = s->pos = cursor; s->line++; |
| goto comment; |
| } |
| any { goto comment; } |
| */ |
| } |
| |
| main(){ |
| Scanner in; |
| int t; |
| memset((char*) &in, 0, sizeof(in)); |
| in.fd = 0; |
| while((t = scan(&in)) != EOI){ |
| /* |
| printf("%d\\t%.*s\\n", t, in.cur - in.tok, in.tok); |
| printf("%d\\n", t); |
| */ |
| } |
| close(in.fd); |
| } |
| .fi |
| .in -3 |
| |
| .SH "SEE ALSO" |
| .LP |
| flex(1), lex(1). |
| |
| .SH FEATURES |
| .LP |
| \*(re does not provide a default action: |
| the generated code assumes that the input |
| will consist of a sequence of tokens. |
| Typically this can be dealt with by adding a rule such as the one for |
| unexpected characters in the example above. |
| .LP |
| The user must arrange for a sentinel token to appear at the end of input |
| (and provide a rule for matching it): |
| \*(re does not provide an \fC<<EOF>>\fP expression. |
| If the source is from a null-byte terminated string, a |
| rule matching a null character will suffice. If the source is from a |
| file then the approach taken in the example can be used: pad the input with |
| a newline (or some other character that can't appear within another token); |
| upon recognizing such a character check to see if it is the sentinel |
| and act accordingly. |
| .LP |
| \*(re does not provide start conditions: use a separate scanner |
| specification for each start condition (as illustrated in the above example). |
| .LP |
| No [^x]. Use difference instead. |
| .SH BUGS |
| .LP |
| Only fixed length trailing context can be handled. |
| .LP |
| The maximum value appearing as a parameter \fIn\fP to \fCYYFILL\fP is not |
| provided to the generated code (this value is needed for constructing |
| the interface code). |
| Note that this value is usually relatively small: for |
| typical programming languages \fIn\fP will be the length of the longest |
| keyword plus one. |
| .LP |
| Difference only works for character sets. |
| .LP |
| The \*(re internal algorithms need documentation. |
| |
| .SH AUTHOR |
| .LP |
| Please send bug reports, fixes and feedback to: |
| .LP |
| .nf |
| Peter Bumbulis |
| Computer Systems Group |
| University of Waterloo |
| Waterloo, Ontario |
| N2L 3G1 |
| Internet: peterr@csg.uwaterloo.ca |
| .fi |
