Class TokenizedLine

Namespace

InteractiveReadLine.Tokenizing

Assembly

InteractiveReadLine.dll

The TokenizedLine class represents a line of text processed by a lexer and split into a sequence of tokens.

public class TokenizedLine : IReadOnlyList<IToken>, IReadOnlyCollection<IToken>, IEnumerable<IToken>, IEnumerable

Inheritance

object

TokenizedLine

Implements

IReadOnlyList<IToken>

IReadOnlyCollection<IToken>

IEnumerable<IToken>

IEnumerable

Inherited Members

object.Equals(object)

object.Equals(object, object)

object.GetHashCode()

object.GetType()

object.MemberwiseClone()

object.ReferenceEquals(object, object)

object.ToString()

Remarks

These tokens in order contain every character of text in the original line, such that combining them in sequence would recreate the original string. Additionally, the line's original cursor position is preserved and contained by one of the tokens. The tokens can be walked in its entirety in either direction by navigation properties on the tokens themselves, or iterated through in the line.

Constructors

TokenizedLine()

Creates an empty sequence to which a lexer can add tokens.

public TokenizedLine()

Properties

Count

Gets the number of tokens in the sequence

public int Count { get; }

Property Value

int

Cursor

Gets the overall index of the cursor in the combined text

public int Cursor { get; set; }

Property Value

int

CursorToken

Gets the token that contains the cursor, or null when no token contains it.

public IToken? CursorToken { get; }

Property Value

IToken

CursorTokenIndex

Gets the index of the token that contains the cursor, or -1 when no token contains it.

public int CursorTokenIndex { get; }

Property Value

int

First

Gets the first token in the sequence, or null when the sequence is empty.

public IToken? First { get; }

Property Value

IToken

FirstNonHidden

Gets the first non-hidden token in the sequence, or null when none exists.

public IToken? FirstNonHidden { get; }

Property Value

IToken

this[int]

Gets the token at the specified index in the sequence

public IToken this[int index] { get; }

Parameters

index int

Index in the token sequence

Property Value

IToken

Last

Gets the last token in the sequence, or null when the sequence is empty.

public IToken? Last { get; }

Property Value

IToken

Text

Gets the combined text of all of the tokens in the sequence, which will match the original line of text before it was split apart by the lexer

public string Text { get; }

Property Value

string

Methods

Add(string, bool, int?, int)

Add a token to the sequence, making all of the necessary links

public void Add(string text, bool isHidden, int? cursor = null, int typeCode = 0)

Parameters

text string

The text contained by the token

isHidden bool

Whether or not this is a hidden token

cursor int?

Null if the token does not contain the cursor, otherwise the index of the cursor as an offset from the first character in the token text

typeCode int

An optional type code which can be accessed by a consumer of the token

GetEnumerator()

Gets an enumerator which can be used to iterate through the tokens

public IEnumerator<IToken> GetEnumerator()

Returns

IEnumerator<IToken>