Class CommonLexers

Namespace

InteractiveReadLine.Tokenizing

Assembly

InteractiveReadLine.dll

Provides ready-made lexers and components for creating custom lexers. A lexer splits a line into tokens used by auto-completion and token-based formatters.

public static class CommonLexers

Inheritance

object

CommonLexers

Inherited Members

object.Equals(object)

object.Equals(object, object)

object.GetHashCode()

object.GetType()

object.MemberwiseClone()

object.ReferenceEquals(object, object)

object.ToString()

Properties

Regex

Creates an empty list of regular-expression token types. Add token patterns to the list, then convert it to a lexer with ToLexer.

public static List<RegexTokenDef> Regex { get; }

Property Value

List<RegexTokenDef>

SplitOnWhitespace

Gets a lexer that splits a line into words at whitespace boundaries for use with auto-completion.

public static Func<LineState, TokenizedLine> SplitOnWhitespace { get; }

Property Value

Func<LineState, TokenizedLine>

Methods

AddAnyNonWhitespace(List<RegexTokenDef>, int)

Adds a token type that matches a sequence of non-whitespace characters, which commonly represents one word.

public static List<RegexTokenDef> AddAnyNonWhitespace(this List<RegexTokenDef> list, int typeCode = 0)

Parameters

list List<RegexTokenDef>

The token type list to add to

typeCode int

A code identifying this token type in the resulting tokens

Returns

List<RegexTokenDef>

The same list, for chaining

AddDoubleQuoteStringLiterals(List<RegexTokenDef>, int)

Adds a token type that matches a double-quoted string and honors backslash escapes, so a quoted phrase is treated as one token rather than split on its spaces.

public static List<RegexTokenDef> AddDoubleQuoteStringLiterals(this List<RegexTokenDef> list, int typeCode = 0)

Parameters

list List<RegexTokenDef>

The token type list to add to

typeCode int

A code identifying this token type in the resulting tokens

Returns

List<RegexTokenDef>

The same list, for chaining

AddSingleQuoteStringLiterals(List<RegexTokenDef>, int)

Adds a token type that matches a single-quoted string and honors backslash escapes.

public static List<RegexTokenDef> AddSingleQuoteStringLiterals(this List<RegexTokenDef> list, int typeCode = 0)

Parameters

list List<RegexTokenDef>

The token type list to add to

typeCode int

A code identifying this token type in the resulting tokens

Returns

List<RegexTokenDef>

The same list, for chaining

AddTokenType(List<RegexTokenDef>, string, int)

Adds a new token type to the end of the regular expression token list. The token must begin with the regular expression start of text character (^) or an ArgumentException will be thrown

public static List<RegexTokenDef> AddTokenType(this List<RegexTokenDef> list, string pattern, int typeCode = 0)

Parameters

list List<RegexTokenDef>

pattern string

The regular expression pattern which identifies the token, must start with ^

typeCode int

An optional type code, such that any token matched by this definition will have this value in the .TypeCode property of the IToken interface

Returns

List<RegexTokenDef>

ToLexer(List<RegexTokenDef>)

From a list of RegexTokenDefs, produce a function that will tokenize a LineState according to the specified token definitions in the order they were provided. Effectively this renders a list of regular expression token definitions into a functioning lexer.

public static Func<LineState, TokenizedLine> ToLexer(this List<RegexTokenDef> list)

Parameters

list List<RegexTokenDef>

Returns

Func<LineState, TokenizedLine>