PGN Parsing Library for Odin

Title. Only the lexer is done at the moment.

Lexer API

Initialization

Use lexer_init to initialize the lexer for scanning:

Definitions

Scan_Flag :: enum {
	Ignore_Illegal_Characters, 
	Allow_Multiline_Comments,
	Allow_Multiline_Strings,
	Allow_Illegal_Characters_In_Strings, // Only printing characters are allowed in strings by default
	Allow_Long_Strings, // Strings are otherwise limited to 255 characters, per the spec
	Ignore_Illegal_Escape_Sequences // Only '\n' and '\\' are allowed by default
}

Default_Scan_Flags :: bit_set[Scan_Flag]{
	.Allow_Multiline_Comments,
	.Allow_Illegal_Characters_In_Strings
}

lexer_init :: proc(
    l: ^Lexer,
    src: string,                /* PGN data to tokenize */
    filename := "",             /* Filename of source file; only used for error reporting */
    flags := Default_Scan_Flags /* Set the lexer's scan flags to modify its behavior */
)

Example

import "pgn"

pgn_data := "..."

lexer: pgn.Lexer
pgn.lexer_init(&lexer, pgn_data)

Getting Tokens

Call lexer_next_token to begin tokenizing the PGN data. Repeatedly call it to get successive tokens, until the lexer has reached the end of the data. Use lexer_is_at_end to get whether the lexer has finished scanning.

lexer_next_token returns two items:

of type Token: The token just scanned. In the case of an error, this will contain what was scanned up until the trigger for the error.
of type Maybe(Error_Report): Will be non-nil if an error occured. Contains where the error occured, error message, and error type

lexer_is_at_end returns a boolean:

true if the lexer is at the end of its data
false otherwise

Definitions

// Struct for identifying a specific position in a file
Position :: struct {
	filename: string,
	offset:   uint, /* Byte offset from the beginning of the data */
	line:     uint,
	col:      uint,
}

// Errors
Error :: union {
	Lexical_Error,
	Syntax_Error,
}

Error_Report :: struct {
	using pos: Position,
	msg:       string,
	error:     Error,
}

// Tokens
Token_Type :: enum {
	EOF,
	Ignored,
	Comment,
	String,
	Integer,
	Period,
	Asterisk,
	Left_Bracket,
	Right_Bracket,
	Left_Parenthesis,
	Right_Parenthesis,
	Left_Angle, // These two are not in use, but instead reserved for future use by the spec
	Right_Angle,
	NAG, // Short for "Numeric Annotation Glyph"
	Symbol
}

Token :: struct {
	using pos: Position,
	text:      string,
	type:      Token_Type,
}

lexer_next_token :: proc(l: ^Lexer) -> (token: Token, report: Maybe(Error_Report))

@(require_results)
lexer_is_at_end :: proc(l: ^Lexer) -> bool

Example

lexer: pgn.Lexer
pgn.lexer_init(&lexer, "...")

for !pgn.lexer_is_at_end(&lexer) {
	token, err := pgn.lexer_next_token(&lexer)

	if err != nil {
		fmt.eprintln("Got an error while scanning:", err.?)
	} else {
		fmt.println(token)
	}
}

Working With Errors

The library exposes a formatted_report procedure to get a formatted error message string from an Error_Report. It is stylized to be similar to Odin's error messages. It accepts both an Error_Report and an allocator to allocate the string with. By default it uses the temporary allocator.

Definitions

@(require_results)
formatted_report :: proc(e: Error_Report, allocator := context.temp_allocator) -> string

Example

lexer: pgn.Lexer
pgn.lexer_init(&lexer, "...", "sample.pgn")

for !pgn.lexer_is_at_end(&lexer) {
	_, err := pgn.lexer_next_token(&lexer)

	if err != nil {
		fmt.eprintln(pgn.formatted_report(err))
	}
}

sample.pgn(37:1) Brace comment was not terminated. Multiline comments are not allowed.