nx_nlex.nx
buildroot/runtime/nx_nlex.nx
about
nx_nlex.nx -- NishiLang surface-syntax tokenizer, in pure NishiLang.
FIRST CONCRETE STEP of the nxc2 self-hosting trajectory ("get us off
C"). This tokenizer recognizes the NishiLang surface syntax that
nxc2/lex.c + nxc2/parse.c handle in C today. When the parser + IR
builder + backend(s) are also written in NishiLang, the compiler
becomes self-hosting.
Distinct from runtime/nx_lex.nx (low-level character classifiers
like is_ws / is_alpha for generic parsing). This module is the
FULL NishiLang tokenizer.
Token kinds:
- keywords: func, let, var, if, else, while, return, const,
struct, import, as
- identifiers: [a-zA-Z_][a-zA-Z0-9_]*
- integers: decimal + 0x hex
- punctuation: ( ) { } [ ] , ; : . -> = == != < <= > >= + - * /
% & | ^ ~ << >>
- line comments: // to EOL (skipped, not emitted)
Output: NxToken array (kind + start + length + int_val).
dependencies 2 imports · 13 importers
diagram shows first 10 each side; +0 more imports, +3 more importers in the complete lists below.
imports: syscalls.nxnx_loop.nx
imported by: _a2a_selfhost_binop.nx_a2a_selfhost_let.nx_selfhost_compile_42.nx_selfhost_compile_binop.nx_selfhost_compile_if.nx_selfhost_compile_if_false.nx_selfhost_compile_let.nxnx_nir.nxnx_nir_test.nxnx_nlex_test.nxnx_nparse.nxnx_nparse_test.nxnx_nx86_test.nx
structs
| 72 | struct NxToken |
| 79 | struct NxNLexer |
consts
| 28 | const NX_TOK_EOF: i64 = 0 |
| 29 | const NX_TOK_IDENT: i64 = 1 |
| 30 | const NX_TOK_INT: i64 = 2 |
| 31 | const NX_TOK_FUNC: i64 = 10 |
| 32 | const NX_TOK_LET: i64 = 11 |
| 33 | const NX_TOK_VAR: i64 = 12 |
| 34 | const NX_TOK_IF: i64 = 13 |
| 35 | const NX_TOK_ELSE: i64 = 14 |
| 36 | const NX_TOK_WHILE: i64 = 15 |
| 37 | const NX_TOK_RETURN: i64 = 16 |
| 38 | const NX_TOK_CONST: i64 = 17 |
| 39 | const NX_TOK_STRUCT: i64 = 18 |
| 40 | const NX_TOK_IMPORT: i64 = 19 |
| 41 | const NX_TOK_AS: i64 = 20 |
| 42 | const NX_TOK_LPAREN: i64 = 30 |
| 43 | const NX_TOK_RPAREN: i64 = 31 |
| 44 | const NX_TOK_LBRACE: i64 = 32 |
| 45 | const NX_TOK_RBRACE: i64 = 33 |
| 46 | const NX_TOK_LBRACK: i64 = 34 |
| 47 | const NX_TOK_RBRACK: i64 = 35 |
| 48 | const NX_TOK_COMMA: i64 = 36 |
| 49 | const NX_TOK_SEMI: i64 = 37 |
| 50 | const NX_TOK_COLON: i64 = 38 |
| 51 | const NX_TOK_ARROW: i64 = 39 // -> |
| 52 | const NX_TOK_ASSIGN: i64 = 40 |
| 53 | const NX_TOK_EQ: i64 = 41 // == |
| 54 | const NX_TOK_NE: i64 = 42 // != |
| 55 | const NX_TOK_LT: i64 = 43 |
| 56 | const NX_TOK_LE: i64 = 44 |
| 57 | const NX_TOK_GT: i64 = 45 |
| 58 | const NX_TOK_GE: i64 = 46 |
| 59 | const NX_TOK_PLUS: i64 = 50 |
| 60 | const NX_TOK_MINUS: i64 = 51 |
| 61 | const NX_TOK_STAR: i64 = 52 |
| 62 | const NX_TOK_SLASH: i64 = 53 |
| 63 | const NX_TOK_PERCENT: i64 = 54 |
| 64 | const NX_TOK_AMP: i64 = 55 |
| 65 | const NX_TOK_PIPE: i64 = 56 |
| 66 | const NX_TOK_CARET: i64 = 57 |
| 67 | const NX_TOK_TILDE: i64 = 58 |
| 68 | const NX_TOK_LSHIFT: i64 = 59 |
| 69 | const NX_TOK_RSHIFT: i64 = 60 |
| 70 | const NX_TOK_DOT: i64 = 61 |
functions
| 88 | func nx_nlex_new(src: *u8, src_len: i64, cap: i64) -> *NxNLexer |
| 100 | func nx_nlex_token_at(l: *NxNLexer, i: i64) -> *NxToken |
| 104 | func nx_nlex_is_digit(c: i64) -> i64 |
| 110 | func nx_nlex_is_hex(c: i64) -> i64 |
| 117 | func nx_nlex_is_alpha(c: i64) -> i64 |
| 124 | func nx_nlex_is_alnum(c: i64) -> i64 |
| 130 | func nx_nlex_is_ws(c: i64) -> i64 called by 1: nx_nlex_skip_ws_and_comments |
| 138 | func nx_nlex_peek(l: *NxNLexer, off: i64) -> i64 |
| 144 | func nx_nlex_advance(l: *NxNLexer) -> i64 |
| 151 | func nx_nlex_emit(l: *NxNLexer, kind: i64, start: i64, len: i64, int_val: i64) -> i64 |
| 163 | func nx_nlex_streq(l: *NxNLexer, start: i64, len: i64, b0: i64, b1: i64, b2: i64, b3: i64, b4: i64, b5: i64, expected_len: i64) -> i64 called by 1: nx_nlex_classify |
| 174 | func nx_nlex_classify(l: *NxNLexer, start: i64, len: i64) -> i64 |
| 200 | func nx_nlex_skip_ws_and_comments(l: *NxNLexer) -> i64 called by 1: nx_nlex_run calls 6: nx_loop_beginnx_loop_stepnx_nlex_peeknx_nlex_is_wsnx_nlex_advancenx_loop_break |
| 226 | func nx_nlex_lex_ident(l: *NxNLexer, tok_start: i64) -> i64 called by 1: nx_nlex_run calls 8: nx_loop_beginnx_loop_stepnx_nlex_peeknx_loop_breaknx_nlex_is_alnumnx_nlex_advance+2 |
| 246 | func nx_nlex_lex_int(l: *NxNLexer, tok_start: i64) -> i64 called by 1: nx_nlex_run calls 8: nx_nlex_peeknx_nlex_advancenx_loop_beginnx_loop_stepnx_loop_breaknx_nlex_is_hex+2 |
| 297 | func nx_nlex_lex_punct(l: *NxNLexer, c0: i64, tok_start: i64) -> i64 |
| 345 | func nx_nlex_run(l: *NxNLexer) -> i64 called by 11: mainmainmainmainmainmain+5 calls 11: nx_loop_beginnx_loop_stepnx_nlex_skip_ws_and_commentsnx_nlex_peeknx_nlex_emitnx_loop_break+5 |