Lexer

java.lang.Object
- org.antlr.v4.runtime.Recognizer<java.lang.Integer,LexerATNSimulator>
- - org.antlr.v4.runtime.Lexer

All Implemented Interfaces:

TokenSource

Direct Known Subclasses:

LexerInterpreter, XPathLexer
```
public abstract class Lexer
extends Recognizer<java.lang.Integer,LexerATNSimulator>
implements TokenSource
```
A lexer is recognizer that draws input symbols from a character stream. lexer grammars result in a subclass of this object. A Lexer object uses simplified match() and error recovery mechanisms in the interest of speed.

Field Summary

Fields
Modifier and Type	Field and Description
`int`	`_channel` The channel number for the current token
`protected TokenFactory<?>`	`_factory` How to create token objects
`boolean`	`_hitEOF` Once we see EOF on char stream, next token will be EOF.
`CharStream`	`_input`
`int`	`_mode`
`IntegerStack`	`_modeStack`
`java.lang.String`	`_text` You can set the text for the current token to override what is in the input char buffer.
`Token`	`_token` The goal of all lexer rules/methods is to create a token object.
`protected Pair<TokenSource,CharStream>`	`_tokenFactorySourcePair`
`int`	`_tokenStartCharIndex` What character index in the stream did the current token start at? Needed, for example, to get the text for current token.
`int`	`_tokenStartCharPositionInLine` The character position of first character within the line
`int`	`_tokenStartLine` The line on which the first character of the token resides
`int`	`_type` The token type for the current token
`static int`	`DEFAULT_MODE`
`static int`	`DEFAULT_TOKEN_CHANNEL`
`static int`	`HIDDEN`
`static int`	`MAX_CHAR_VALUE`
`static int`	`MIN_CHAR_VALUE`
`static int`	`MORE`
`static int`	`SKIP`

Fields inherited from class org.antlr.v4.runtime.Recognizer
_interp, EOF

Constructor Summary

Constructors
Constructor and Description

Lexer()

Lexer(CharStream input)

Constructors
Constructor and Description
`Lexer()`
`Lexer(CharStream input)`

Method Summary

All Methods Instance Methods Concrete Methods Deprecated Methods
Modifier and Type	Method and Description
`Token`	`emit()` The standard method called to automatically emit a token at the outermost lexical rule.
`void`	`emit(Token token)` By default does not support multiple emits per nextToken invocation for efficiency reasons.
`Token`	`emitEOF()`
`java.util.List<? extends Token>`	`getAllTokens()` Return a list of all Token objects in input char stream.
`int`	`getChannel()`
`java.lang.String[]`	`getChannelNames()`
`java.lang.String`	`getCharErrorDisplay(int c)`
`int`	`getCharIndex()` What is the index of the current character of lookahead?
`int`	`getCharPositionInLine()` Get the index into the current line for the current position in the input stream.
`java.lang.String`	`getErrorDisplay(int c)`
`java.lang.String`	`getErrorDisplay(java.lang.String s)`
`CharStream`	`getInputStream()` Get the `CharStream` from which this token source is currently providing tokens.
`int`	`getLine()` Get the line number for the current position in the input stream.
`java.lang.String[]`	`getModeNames()`
`java.lang.String`	`getSourceName()` Gets the name of the underlying input source.
`java.lang.String`	`getText()` Return the text matched so far for the current token or any text override.
`Token`	`getToken()` Override if emitting multiple tokens.
`TokenFactory<? extends Token>`	`getTokenFactory()` Gets the `TokenFactory` this token source is currently using for creating `Token` objects from the input.
`java.lang.String[]`	`getTokenNames()` Deprecated.
`int`	`getType()`
`void`	`mode(int m)`
`void`	`more()`
`Token`	`nextToken()` Return a token from this source; i.e., match a token on the char stream.
`void`	`notifyListeners(LexerNoViableAltException e)`
`int`	`popMode()`
`void`	`pushMode(int m)`
`void`	`recover(LexerNoViableAltException e)`
`void`	`recover(RecognitionException re)` Lexers can normally match any char in it's vocabulary after matching a token, so do the easy thing and just kill a character and hope it all works out.
`void`	`reset()`
`void`	`setChannel(int channel)`
`void`	`setCharPositionInLine(int charPositionInLine)`
`void`	`setInputStream(IntStream input)` Set the char stream and reset the lexer
`void`	`setLine(int line)`
`void`	`setText(java.lang.String text)` Set the complete text of this token; it wipes any previous changes to the text.
`void`	`setToken(Token _token)`
`void`	`setTokenFactory(TokenFactory<?> factory)` Set the `TokenFactory` this token source should use for creating `Token` objects from the input.
`void`	`setType(int ttype)`
`void`	`skip()` Instruct the lexer to skip creating a token for current lexer rule and look for another token.

Methods inherited from class org.antlr.v4.runtime.Recognizer
action, addErrorListener, getATN, getErrorHeader, getErrorListenerDispatch, getErrorListeners, getGrammarFileName, getInterpreter, getParseInfo, getRuleIndexMap, getRuleNames, getSerializedATN, getState, getTokenErrorDisplay, getTokenType, getTokenTypeMap, getVocabulary, precpred, removeErrorListener, removeErrorListeners, sempred, setInterpreter, setState

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

- Field Detail
  - DEFAULT_MODE
```
public static final int DEFAULT_MODE
```
    See Also:
    
    Constant Field Values
  - MORE
```
public static final int MORE
```
    See Also:
    
    Constant Field Values
  - SKIP
```
public static final int SKIP
```
    See Also:
    
    Constant Field Values
  - DEFAULT_TOKEN_CHANNEL
```
public static final int DEFAULT_TOKEN_CHANNEL
```
    See Also:
    
    Constant Field Values
  - HIDDEN
```
public static final int HIDDEN
```
    See Also:
    
    Constant Field Values
  - MIN_CHAR_VALUE
```
public static final int MIN_CHAR_VALUE
```
    See Also:
    
    Constant Field Values
  - MAX_CHAR_VALUE
```
public static final int MAX_CHAR_VALUE
```
    See Also:
    
    Constant Field Values
  - _input
```
public CharStream _input
```
  - _tokenFactorySourcePair
```
protected Pair<TokenSource,CharStream> _tokenFactorySourcePair
```
  - _factory
```
protected TokenFactory<?> _factory
```
    How to create token objects
  - _token
```
public Token _token
```
    The goal of all lexer rules/methods is to create a token object. This is an instance variable as multiple rules may collaborate to create a single token. nextToken will return this object after matching lexer rule(s). If you subclass to allow multiple token emissions, then set this to the last token to be matched or something nonnull so that the auto token emit mechanism will not emit another token.
  - _tokenStartCharIndex
```
public int _tokenStartCharIndex
```
    What character index in the stream did the current token start at? Needed, for example, to get the text for current token. Set at the start of nextToken.
  - _tokenStartLine
```
public int _tokenStartLine
```
    The line on which the first character of the token resides
  - _tokenStartCharPositionInLine
```
public int _tokenStartCharPositionInLine
```
    The character position of first character within the line
  - _hitEOF
```
public boolean _hitEOF
```
    Once we see EOF on char stream, next token will be EOF. If you have DONE : EOF ; then you see DONE EOF.
  - _channel
```
public int _channel
```
    The channel number for the current token
  - _type
```
public int _type
```
    The token type for the current token
  - _modeStack
```
public final IntegerStack _modeStack
```
  - _mode
```
public int _mode
```
  - _text
```
public java.lang.String _text
```
    You can set the text for the current token to override what is in the input char buffer. Use setText() or can set this instance var.
- Constructor Detail
  - Lexer
```
public Lexer()
```
  - Lexer
```
public Lexer(CharStream input)
```
- Method Detail
  - reset
```
public void reset()
```
  - nextToken
```
public Token nextToken()
```
    Return a token from this source; i.e., match a token on the char stream.
    
    Specified by:
    
    nextToken in interface TokenSource
  - skip
```
public void skip()
```
    Instruct the lexer to skip creating a token for current lexer rule and look for another token. nextToken() knows to keep looking when a lexer rule finishes with token set to SKIP_TOKEN. Recall that if token==null at end of any token rule, it creates one for you and emits it.
  - more
```
public void more()
```
  - mode
```
public void mode(int m)
```
  - pushMode
```
public void pushMode(int m)
```
  - popMode
```
public int popMode()
```
  - setTokenFactory
```
public void setTokenFactory(TokenFactory<?> factory)
```
    Description copied from interface: TokenSource
    
    Set the TokenFactory this token source should use for creating Token objects from the input.
    
    Specified by:
    
    setTokenFactory in interface TokenSource
    
    Specified by:
    
    setTokenFactory in class Recognizer<java.lang.Integer,LexerATNSimulator>
    
    Parameters:
    
    factory - The TokenFactory to use for creating tokens.
  - getTokenFactory
```
public TokenFactory<? extends Token> getTokenFactory()
```
    Description copied from interface: TokenSource
    
    Gets the TokenFactory this token source is currently using for creating Token objects from the input.
    
    Specified by:
    
    getTokenFactory in interface TokenSource
    
    Specified by:
    
    getTokenFactory in class Recognizer<java.lang.Integer,LexerATNSimulator>
    
    Returns:
    
    The TokenFactory currently used by this token source.
  - setInputStream
```
public void setInputStream(IntStream input)
```
    Set the char stream and reset the lexer
    
    Specified by:
    
    setInputStream in class Recognizer<java.lang.Integer,LexerATNSimulator>
  - getSourceName
```
public java.lang.String getSourceName()
```
    Description copied from interface: TokenSource
    
    Gets the name of the underlying input source. This method returns a non-null, non-empty string. If such a name is not known, this method returns IntStream.UNKNOWN_SOURCE_NAME.
    
    Specified by:
    
    getSourceName in interface TokenSource
  - getInputStream
```
public CharStream getInputStream()
```
    Description copied from interface: TokenSource
    
    Get the CharStream from which this token source is currently providing tokens.
    
    Specified by:
    
    getInputStream in interface TokenSource
    
    Specified by:
    
    getInputStream in class Recognizer<java.lang.Integer,LexerATNSimulator>
    
    Returns:
    
    The CharStream associated with the current position in the input, or null if no input stream is available for the token source.
  - emit
```
public void emit(Token token)
```
    By default does not support multiple emits per nextToken invocation for efficiency reasons. Subclass and override this method, nextToken, and getToken (to push tokens into a list and pull from that list rather than a single variable as this implementation does).
  - emit
```
public Token emit()
```
    The standard method called to automatically emit a token at the outermost lexical rule. The token object should point into the char buffer start..stop. If there is a text override in 'text', use that to set the token's text. Override this method to emit custom Token objects or provide a new factory.
  - emitEOF
```
public Token emitEOF()
```
  - getLine
```
public int getLine()
```
    Description copied from interface: TokenSource
    
    Get the line number for the current position in the input stream. The first line in the input is line 1.
    
    Specified by:
    
    getLine in interface TokenSource
    
    Returns:
    
    The line number for the current position in the input stream, or 0 if the current token source does not track line numbers.
  - getCharPositionInLine
```
public int getCharPositionInLine()
```
    Description copied from interface: TokenSource
    
    Get the index into the current line for the current position in the input stream. The first character on a line has position 0.
    
    Specified by:
    
    getCharPositionInLine in interface TokenSource
    
    Returns:
    
    The line number for the current position in the input stream, or -1 if the current token source does not track character positions.
  - setLine
```
public void setLine(int line)
```
  - setCharPositionInLine
```
public void setCharPositionInLine(int charPositionInLine)
```
  - getCharIndex
```
public int getCharIndex()
```
    What is the index of the current character of lookahead?
  - getText
```
public java.lang.String getText()
```
    Return the text matched so far for the current token or any text override.
  - setText
```
public void setText(java.lang.String text)
```
    Set the complete text of this token; it wipes any previous changes to the text.
  - getToken
```
public Token getToken()
```
    Override if emitting multiple tokens.
  - setToken
```
public void setToken(Token _token)
```
  - setType
```
public void setType(int ttype)
```
  - getType
```
public int getType()
```
  - setChannel
```
public void setChannel(int channel)
```
  - getChannel
```
public int getChannel()
```
  - getChannelNames
```
public java.lang.String[] getChannelNames()
```
  - getModeNames
```
public java.lang.String[] getModeNames()
```
  - getTokenNames
```
@Deprecated
public java.lang.String[] getTokenNames()
```
    Deprecated.
    
    Used to print out token names like ID during debugging and error reporting. The generated parsers implement a method that overrides this to point to their String[] tokenNames.
    
    Specified by:
    
    getTokenNames in class Recognizer<java.lang.Integer,LexerATNSimulator>
  - getAllTokens
```
public java.util.List<? extends Token> getAllTokens()
```
    Return a list of all Token objects in input char stream. Forces load of all tokens. Does not include EOF token.
  - recover
```
public void recover(LexerNoViableAltException e)
```
  - notifyListeners
```
public void notifyListeners(LexerNoViableAltException e)
```
  - getErrorDisplay
```
public java.lang.String getErrorDisplay(java.lang.String s)
```
  - getErrorDisplay
```
public java.lang.String getErrorDisplay(int c)
```
  - getCharErrorDisplay
```
public java.lang.String getCharErrorDisplay(int c)
```
  - recover
```
public void recover(RecognitionException re)
```
    Lexers can normally match any char in it's vocabulary after matching a token, so do the easy thing and just kill a character and hope it all works out. You can instead use the rule invocation stack to do sophisticated error recovery if you are in a fragment rule.

Class Lexer

Field Summary

Fields inherited from class org.antlr.v4.runtime.Recognizer

Constructor Summary

Method Summary

Methods inherited from class org.antlr.v4.runtime.Recognizer

Methods inherited from class java.lang.Object

Field Detail

DEFAULT_MODE

MORE

SKIP

DEFAULT_TOKEN_CHANNEL

HIDDEN

MIN_CHAR_VALUE

MAX_CHAR_VALUE

_input

_tokenFactorySourcePair

_factory

_token

_tokenStartCharIndex

_tokenStartLine

_tokenStartCharPositionInLine

_hitEOF

_channel

_type

_modeStack

_mode

_text

Constructor Detail

Lexer

Lexer

Method Detail

reset

nextToken

skip

more

mode

pushMode

popMode

setTokenFactory

getTokenFactory

setInputStream

getSourceName

getInputStream

emit

emit

emitEOF

getLine

getCharPositionInLine

setLine

setCharPositionInLine

getCharIndex

getText

setText

getToken

setToken

setType

getType

setChannel

getChannel

getChannelNames

getModeNames

getTokenNames

getAllTokens

recover

notifyListeners

getErrorDisplay

getErrorDisplay

getCharErrorDisplay

recover