org.antlr.v4.runtime

Class TokenStreamRewriter

java.lang.Object

org.antlr.v4.runtime.TokenStreamRewriter

public class TokenStreamRewriter
extends java.lang.Object

Useful for rewriting out a buffered input token stream after doing some augmentation or other manipulations on it.

You can insert stuff, replace, and delete chunks. Note that the operations are done lazily--only if you convert the buffer to a String with TokenStream.getText(). This is very efficient because you are not moving data around all the time. As the buffer of tokens is converted to strings, the getText() method(s) scan the input token stream and check to see if there is an operation at the current index. If so, the operation is done and then normal String rendering continues on the buffer. This is like having multiple Turing machine instruction streams (programs) operating on a single input tape. :)

This rewriter makes no modifications to the token stream. It does not ask the stream to fill itself up nor does it advance the input cursor. The token stream IntStream.index() will return the same value before and after any getText() call.

The rewriter only works on tokens that you have in the buffer and ignores the current input cursor. If you are buffering tokens on-demand, calling getText() halfway through the input will only do rewrites for those tokens in the first half of the file.

Since the operations are done lazily at getText()-time, operations do not screw up the token index values. That is, an insert operation at token index i does not change the index values for tokens i+1..n-1.

Because operations never actually alter the buffer, you may always get the original token stream back without undoing anything. Since the instructions are queued up, you can easily simulate transactions and roll back any changes if there is an error just by removing instructions. For example,

 CharStream input = new ANTLRFileStream("input");
 TLexer lex = new TLexer(input);
 CommonTokenStream tokens = new CommonTokenStream(lex);
 T parser = new T(tokens);
 TokenStreamRewriter rewriter = new TokenStreamRewriter(tokens);
 parser.startRule();
 

Then in the rules, you can execute (assuming rewriter is visible):

 Token t,u;
 ...
 rewriter.insertAfter(t, "text to put after t");}
 rewriter.insertAfter(u, "text after u");}
 System.out.println(rewriter.getText());
 

You can also have multiple "instruction streams" and get multiple rewrites from a single pass over the input. Just name the instruction streams and use that name again when printing the buffer. This could be useful for generating a C file and also its header file--all from the same buffer:

 rewriter.insertAfter("pass1", t, "text to put after t");}
 rewriter.insertAfter("pass2", u, "text after u");}
 System.out.println(rewriter.getText("pass1"));
 System.out.println(rewriter.getText("pass2"));
 

If you don't use named rewrite streams, a "default" stream is used as the first example shows.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
(package private) class	TokenStreamRewriter.InsertAfterOp Distinguish between insert after/before to do the "insert afters" first and then the "insert befores" at same index.
(package private) class	TokenStreamRewriter.InsertBeforeOp
(package private) class	TokenStreamRewriter.ReplaceOp I'm going to try replacing range from x..y with (y-x)+1 ReplaceOp instructions.
class	TokenStreamRewriter.RewriteOperation

Field Summary

Fields

Modifier and Type	Field and Description
static java.lang.String	DEFAULT_PROGRAM_NAME
protected java.util.Map<java.lang.String,java.lang.Integer>	lastRewriteTokenIndexes Map String (program name) → Integer index
static int	MIN_TOKEN_INDEX
static int	PROGRAM_INIT_SIZE
protected java.util.Map<java.lang.String,java.util.List<TokenStreamRewriter.RewriteOperation>>	programs You may have multiple, named streams of rewrite operations.
protected TokenStream	tokens Our source stream

Constructor Summary

Constructors

Constructor and Description
TokenStreamRewriter(TokenStream tokens)

Method Summary

Modifier and Type	Method and Description
protected java.lang.String	catOpText(java.lang.Object a, java.lang.Object b)
void	delete(int index)
void	delete(int from, int to)
void	delete(java.lang.String programName, int from, int to)
void	delete(java.lang.String programName, Token from, Token to)
void	delete(Token indexT)
void	delete(Token from, Token to)
void	deleteProgram()
void	deleteProgram(java.lang.String programName) Reset the program so that no instructions exist
protected <T extends TokenStreamRewriter.RewriteOperation> java.util.List<? extends T>	getKindOfOps(java.util.List<? extends TokenStreamRewriter.RewriteOperation> rewrites, java.lang.Class<T> kind, int before) Get all operations before an index of a particular kind
int	getLastRewriteTokenIndex()
protected int	getLastRewriteTokenIndex(java.lang.String programName)
protected java.util.List<TokenStreamRewriter.RewriteOperation>	getProgram(java.lang.String name)
java.lang.String	getText() Return the text from the original tokens altered per the instructions given to this rewriter.
java.lang.String	getText(Interval interval) Return the text associated with the tokens in the interval from the original token stream but with the alterations given to this rewriter.
java.lang.String	getText(java.lang.String programName) Return the text from the original tokens altered per the instructions given to this rewriter in programName.
java.lang.String	getText(java.lang.String programName, Interval interval)
TokenStream	getTokenStream()
private java.util.List<TokenStreamRewriter.RewriteOperation>	initializeProgram(java.lang.String name)
void	insertAfter(int index, java.lang.Object text)
void	insertAfter(java.lang.String programName, int index, java.lang.Object text)
void	insertAfter(java.lang.String programName, Token t, java.lang.Object text)
void	insertAfter(Token t, java.lang.Object text)
void	insertBefore(int index, java.lang.Object text)
void	insertBefore(java.lang.String programName, int index, java.lang.Object text)
void	insertBefore(java.lang.String programName, Token t, java.lang.Object text)
void	insertBefore(Token t, java.lang.Object text)
protected java.util.Map<java.lang.Integer,TokenStreamRewriter.RewriteOperation>	reduceToSingleOperationPerIndex(java.util.List<TokenStreamRewriter.RewriteOperation> rewrites) We need to combine operations and report invalid operations (like overlapping replaces that are not completed nested).
void	replace(int from, int to, java.lang.Object text)
void	replace(int index, java.lang.Object text)
void	replace(java.lang.String programName, int from, int to, java.lang.Object text)
void	replace(java.lang.String programName, Token from, Token to, java.lang.Object text)
void	replace(Token indexT, java.lang.Object text)
void	replace(Token from, Token to, java.lang.Object text)
void	rollback(int instructionIndex)
void	rollback(java.lang.String programName, int instructionIndex) Rollback the instruction stream for a program so that the indicated instruction (via instructionIndex) is no longer in the stream.
protected void	setLastRewriteTokenIndex(java.lang.String programName, int i)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

DEFAULT_PROGRAM_NAME

public static final java.lang.String DEFAULT_PROGRAM_NAME

See Also:

Constant Field Values

PROGRAM_INIT_SIZE

public static final int PROGRAM_INIT_SIZE

See Also:

Constant Field Values

MIN_TOKEN_INDEX

public static final int MIN_TOKEN_INDEX

See Also:

Constant Field Values

tokens

protected final TokenStream tokens

Our source stream

programs

protected final java.util.Map<java.lang.String,java.util.List<TokenStreamRewriter.RewriteOperation>> programs

You may have multiple, named streams of rewrite operations. I'm calling these things "programs." Maps String (name) → rewrite (List)

lastRewriteTokenIndexes

protected final java.util.Map<java.lang.String,java.lang.Integer> lastRewriteTokenIndexes

Map String (program name) → Integer index

Constructor Detail

TokenStreamRewriter

public TokenStreamRewriter(TokenStream tokens)

Method Detail

getTokenStream

public final TokenStream getTokenStream()

rollback

public void rollback(int instructionIndex)

rollback

public void rollback(java.lang.String programName,
                     int instructionIndex)

Rollback the instruction stream for a program so that the indicated instruction (via instructionIndex) is no longer in the stream. UNTESTED!

deleteProgram

public void deleteProgram()

deleteProgram

public void deleteProgram(java.lang.String programName)

Reset the program so that no instructions exist

insertAfter

public void insertAfter(Token t,
                        java.lang.Object text)

insertAfter

public void insertAfter(int index,
                        java.lang.Object text)

insertAfter

public void insertAfter(java.lang.String programName,
                        Token t,
                        java.lang.Object text)

insertAfter

public void insertAfter(java.lang.String programName,
                        int index,
                        java.lang.Object text)

insertBefore

public void insertBefore(Token t,
                         java.lang.Object text)

insertBefore

public void insertBefore(int index,
                         java.lang.Object text)

insertBefore

public void insertBefore(java.lang.String programName,
                         Token t,
                         java.lang.Object text)

insertBefore

public void insertBefore(java.lang.String programName,
                         int index,
                         java.lang.Object text)

replace

public void replace(int index,
                    java.lang.Object text)

replace

public void replace(int from,
                    int to,
                    java.lang.Object text)

replace

public void replace(Token indexT,
                    java.lang.Object text)

replace

public void replace(Token from,
                    Token to,
                    java.lang.Object text)

replace

public void replace(java.lang.String programName,
                    int from,
                    int to,
                    java.lang.Object text)

replace

public void replace(java.lang.String programName,
                    Token from,
                    Token to,
                    java.lang.Object text)

delete

public void delete(int index)

delete

public void delete(int from,
                   int to)

delete

public void delete(Token indexT)

delete

public void delete(Token from,
                   Token to)

delete

public void delete(java.lang.String programName,
                   int from,
                   int to)

delete

public void delete(java.lang.String programName,
                   Token from,
                   Token to)

getLastRewriteTokenIndex

public int getLastRewriteTokenIndex()

getLastRewriteTokenIndex

protected int getLastRewriteTokenIndex(java.lang.String programName)

setLastRewriteTokenIndex

protected void setLastRewriteTokenIndex(java.lang.String programName,
                                        int i)

getProgram

protected java.util.List<TokenStreamRewriter.RewriteOperation> getProgram(java.lang.String name)

initializeProgram

private java.util.List<TokenStreamRewriter.RewriteOperation> initializeProgram(java.lang.String name)

getText

public java.lang.String getText()

Return the text from the original tokens altered per the instructions given to this rewriter.

getText

public java.lang.String getText(java.lang.String programName)

Return the text from the original tokens altered per the instructions given to this rewriter in programName.

getText

public java.lang.String getText(Interval interval)

Return the text associated with the tokens in the interval from the original token stream but with the alterations given to this rewriter. The interval refers to the indexes in the original token stream. We do not alter the token stream in any way, so the indexes and intervals are still consistent. Includes any operations done to the first and last token in the interval. So, if you did an insertBefore on the first token, you would get that insertion. The same is true if you do an insertAfter the stop token.

getText

public java.lang.String getText(java.lang.String programName,
                                Interval interval)

reduceToSingleOperationPerIndex

protected java.util.Map<java.lang.Integer,TokenStreamRewriter.RewriteOperation> reduceToSingleOperationPerIndex(java.util.List<TokenStreamRewriter.RewriteOperation> rewrites)

We need to combine operations and report invalid operations (like overlapping replaces that are not completed nested). Inserts to same index need to be combined etc... Here are the cases: I.i.u I.j.v leave alone, nonoverlapping I.i.u I.i.v combine: Iivu R.i-j.u R.x-y.v | i-j in x-y delete first R R.i-j.u R.i-j.v delete first R R.i-j.u R.x-y.v | x-y in i-j ERROR R.i-j.u R.x-y.v | boundaries overlap ERROR Delete special case of replace (text==null): D.i-j.u D.x-y.v | boundaries overlap combine to max(min)..max(right) I.i.u R.x-y.v | i in (x+1)-y delete I (since insert before we're not deleting i) I.i.u R.x-y.v | i not in (x+1)-y leave alone, nonoverlapping R.x-y.v I.i.u | i in x-y ERROR R.x-y.v I.x.u R.x-y.uv (combine, delete I) R.x-y.v I.i.u | i not in x-y leave alone, nonoverlapping I.i.u = insert u before op @ index i R.x-y.u = replace x-y indexed tokens with u First we need to examine replaces. For any replace op: 1. wipe out any insertions before op within that range. 2. Drop any replace op before that is contained completely within that range. 3. Throw exception upon boundary overlap with any previous replace. Then we can deal with inserts: 1. for any inserts to same index, combine even if not adjacent. 2. for any prior replace with same left boundary, combine this insert with replace and delete this replace. 3. throw exception if index in same range as previous replace Don't actually delete; make op null in list. Easier to walk list. Later we can throw as we add to index → op map. Note that I.2 R.2-2 will wipe out I.2 even though, technically, the inserted stuff would be before the replace range. But, if you add tokens in front of a method body '{' and then delete the method body, I think the stuff before the '{' you added should disappear too. Return a map from token index to operation.

catOpText

protected java.lang.String catOpText(java.lang.Object a,
                                     java.lang.Object b)

getKindOfOps

protected <T extends TokenStreamRewriter.RewriteOperation> java.util.List<? extends T> getKindOfOps(java.util.List<? extends TokenStreamRewriter.RewriteOperation> rewrites,
                                                                                                    java.lang.Class<T> kind,
                                                                                                    int before)

Get all operations before an index of a particular kind