Bitrix API

Tokenizer

Table of Contents

Methods

createFromString()  : Tokenizer
Splits a text into tokens, creates new Tokenizer object, and returns it.
createFromTokens()  : Tokenizer
Creates new Tokenizer objects and sets its tokens into given.
endOfInput()  : bool
Checks if end of tokens reached.
getCurrentToken()  : Token
Returns current token.
getPrevToken()  : Token
Returns previous token.
getTokens()  : mixed
Returns all the tokens.
nextNotWhiteSpaceToken()  : Token
Skips all whitespace and commentaries.
nextToken()  : Token
Returns next token.
putBack()  : void
Moves current position one step back.
resetState()  : void
Resets internal state.
restoreBookmark()  : void
Restores previously remembered position.
setBookmark()  : void
Remembers current position.
skipWhiteSpace()  : void
Skips all whitespace and commentaries.
testText()  : bool
Checks if current token text is equal to $text.
testUpperText()  : bool
Checks if current token text is equal to $text.

Methods

createFromString()

Splits a text into tokens, creates new Tokenizer object, and returns it.

public static createFromString(string $sql) : Tokenizer
Parameters
$sql : string

Sql text.

Return values
Tokenizer

createFromTokens()

Creates new Tokenizer objects and sets its tokens into given.

public static createFromTokens(array<string|int, mixed> $tokens) : Tokenizer
Parameters
$tokens : array<string|int, mixed>
Return values
Tokenizer

endOfInput()

Checks if end of tokens reached.

public endOfInput() : bool
Return values
bool

getCurrentToken()

Returns current token.

public getCurrentToken() : Token

Leaves position intact.

Return values
Token

getPrevToken()

Returns previous token.

public getPrevToken() : Token

Leaves position intact.

Return values
Token

getTokens()

Returns all the tokens.

public getTokens() : mixed

nextNotWhiteSpaceToken()

Skips all whitespace and commentaries.

public nextNotWhiteSpaceToken() : Token
Return values
Token

nextToken()

Returns next token.

public nextToken() : Token

Advances position one step forward.

Return values
Token

putBack()

Moves current position one step back.

public putBack() : void

resetState()

Resets internal state.

public resetState() : void

restoreBookmark()

Restores previously remembered position.

public restoreBookmark() : void
Tags
see
Tokenizer::setBookmark

setBookmark()

Remembers current position.

public setBookmark() : void
Tags
see
Tokenizer::restoreBookmark

skipWhiteSpace()

Skips all whitespace and commentaries.

public skipWhiteSpace() : void

testText()

Checks if current token text is equal to $text.

public testText(string $text) : bool

In case of success advances position one step forward.

Parameters
$text : string

Text to compare.

Return values
bool

testUpperText()

Checks if current token text is equal to $text.

public testUpperText(string $text) : bool

In case of success advances position one step forward.

Parameters
$text : string

Text to compare.

Return values
bool 

        

        
    




    

    

                                

                

                                
    

        On this page

        
    


                

                            

            

    

        

            
Search results