Function static_analyser::lex::tokenize

pub fn tokenize(src: &str) -> Result<Vec<(TokenKind, usize, usize)>>

Turn a string of valid Delphi code into a list of tokens, including the location of that token's start and end point in the original source code.

Note the token indices represent the half-open interval [start, end), equivalent to start .. end in Rust.