Skip to content

Custom syntax highlighting

Samuel Pastva edited this page Aug 15, 2018 · 3 revisions

In order to enable syntax highlighting, you need to provide a custom TextMode which will use a custom Tokenizer. Custom tokenizer could be defined either using the lexing rules as in this tutorial, or directly by overriding the getLineTokens method. Currently, there is no Kotlin-specific syntax for a rule-based tokenizer, because the format is quite dynamic and Ace-specific (custom tokenizer is more reusable). But it can be certainly added in the future.

Tokenizer is parametrised by two types: Token type and State type. Tokens are simple text blocks of specific meaning while state is some internal tokenizer state which can be preserver across lines (think for example block comments). Note that it is important to always categorise all text as some token, because only tokens are displayed (feel free to create some unknown token type for unrecognised sequences).

Tokens

Hence, in order to define our tokenizer, we first need to define our tokens. You can use any class as a token, as long as it implements Tokenizer.Token interface (has type and value). Kotlin's sealed classes are especially well suited for this:

sealed class CustomToken : Tokenizer.Token {
   data class Number(override val value: String) : CustomToken() {
      override val type: String = "number" // all tokens of this type will be marked using CSS class ace_number
   }
   ... other token types ...
}

A more advanced example of the token structure can be also found in the demo module.

Tokenizer

Our simple tokenizer does not need any state, we can simply use Unit:

object CustomTokenizer : Tokenizer<CustomToken, Unit>(null /* no rules */) {
   override fun getLineTokens(line: String, startState: Unit?): Tokens<CustomToken, Unit> {
       // here, you have to split the line into a list of tokens and the bundle it with 
       // a tokenizer state (in this case, just Unit)
   }
}

Again, a full example is available in the demo module.

Note that if you want to use the same tokenizer on both backend and frontend, you'll have to create a pure Kotlin class which you then delegate to:

object UniversalTokenizer { ... } // this will be in a separate pure Kotlin module
object AceTokenizer : Tokenizer<Token, State>(null) {
   override fun getLineTokens(line: String, startState: State?): Tokens<Token, State> 
      = UniversalTokenizer.tokenize(line, startState).asTokens() 
}

Text mode

Finally, once the tokenizer is ready, we can create a custom TextMode which uses it:

object CustomMode {
   override fun getTokenizer(): Tokenizer<*, *> = CustomTokenizer
}

and apply this mode to our editor: editor.getSession().setMode(CustomMode)

Clone this wiki locally