`simvx.core.scene_io.source._source_ast.trivia`¶

What lies between the statements: comments, blank lines, indentation, semicolons.

ast sees code and nothing else. It cannot tell you that the author left two blank lines above a child, wrote # the boss over it, or put three statements on one line with semicolons, and all three are things a scene file’s author notices immediately when they go missing. This module makes one tokenize pass and answers those questions, in str spans on the conventions

mod:: .spans defines.

Why untokenize is never used here. tokenize documents that a round-trip through untokenize preserves only the tokens, not the exact whitespace between them, and that is the usual reason a tool built on it loses fidelity. It is irrelevant to this design: unedited text is carried as text, sliced out of the author’s own source by span, and the token stream is consulted for positions alone. Nothing here ever asks tokenize to write Python back out, so its documented weakness has no surface to touch.

The vocabulary, which is the seam’s and not any parser’s:

leading trivia Everything between the previous statement and this one. It starts where the previous statement’s code ends, so the first thing inside it is that statement’s trailing comment and the newline closing its line; :attr:Trivia.leading narrows to the part that is this statement’s own. trailing comment The comment closing the logical line the statement begins on, when the statement is the last one on that line. A comment inside a bracketed continuation is not one: it is interior text and it travels with the statement. statement group The statements sharing one logical line via semicolons. A group is what a removal has to reason about, because taking one member out must leave the others where they are.

Module Contents¶

Classes¶

`LogicalLine`	One logical line: the run of physical lines a NEWLINE token closes.
`Trivia`	Everything about one statement that is not the statement’s own code.
`SuiteTrivia`	The trivia of one suite, parallel to the statements it was built from.
`TokenIndex`	One `tokenize` pass over a source, queried per suite.

Functions¶

scan

Tokenize text once and return the index built from it.

Data¶

__all__

API¶

simvx.core.scene_io.source._source_ast.trivia.__all__¶: [‘LogicalLine’, ‘SuiteTrivia’, ‘TokenIndex’, ‘Trivia’, ‘scan’]

class simvx.core.scene_io.source._source_ast.trivia.LogicalLine[source]¶

One logical line: the run of physical lines a NEWLINE token closes.

span runs from the first code token to just past the terminator, so consecutive logical lines leave the blank and comment-only lines between them uncovered, which is exactly the text that belongs to trivia.

span: simvx.core.scene_io.source._source_ast.spans.Span¶: None

trailing_comment: simvx.core.scene_io.source._source_ast.spans.Span | None¶: None

class simvx.core.scene_io.source._source_ast.trivia.Trivia[source]¶

Everything about one statement that is not the statement’s own code.

gap is the reconstruction unit: laying every gap + statement span end to end and adding :attr:SuiteTrivia.tail gives the file back byte for byte. leading is the author-facing subset of gap, starting after the previous statement’s line closed, and it is what a move carries along.

gap: simvx.core.scene_io.source._source_ast.spans.Span¶: None

leading: simvx.core.scene_io.source._source_ast.spans.Span¶: None

indent: simvx.core.scene_io.source._source_ast.spans.Span¶: None

comments: tuple[simvx.core.scene_io.source._source_ast.spans.Span, ...]¶: None

blank_lines: int¶: None

trailing_comment: simvx.core.scene_io.source._source_ast.spans.Span | None¶: None

line_end: int¶: None

group_index: int¶: None

group_size: int¶: None

property shares_line: bool[source]¶: True when a semicolon puts another statement on this one’s line.

property first_in_group: bool[source]¶: True when this statement opens its logical line.

class simvx.core.scene_io.source._source_ast.trivia.SuiteTrivia[source]¶

The trivia of one suite, parallel to the statements it was built from.

tail is the text after the last statement that the suite owns. For a module that is everything up to the end of the file; for a nested suite it is empty, because the blank lines and comments below an indented block are reached through the enclosing suite’s next gap, and counting them twice is how a reconciler duplicates a comment.

spans: tuple[simvx.core.scene_io.source._source_ast.spans.Span, ...]¶: None

records: tuple[simvx.core.scene_io.source._source_ast.trivia.Trivia, ...]¶: None

tail: simvx.core.scene_io.source._source_ast.spans.Span¶: None

reconstruct(source: str) → str[source]¶: The suite’s text rebuilt from its parts, for the round-trip property.

groups() → list[list[int]][source]¶: Statement indices grouped by logical line, in source order.

class simvx.core.scene_io.source._source_ast.trivia.TokenIndex(table: simvx.core.scene_io.source._source_ast.spans.LineTable)[source]¶

One tokenize pass over a source, queried per suite.

Built once per parse and shared by every suite in the file, because tokenizing is the expensive half of reading a scene and the answers it gives (where the comments are, where the logical lines end) are file-wide.

The source must already have parsed. tokenize is not being asked to validate anything here, and on genuinely broken input it raises its own errors, which are worse than CPython’s and are not the ones the layer promises to present.

Initialization

__slots__¶: (‘_table’, ‘_lines’, ‘_line_starts’, ‘_line_stops’, ‘_comment_starts’, ‘comments’)

property table: simvx.core.scene_io.source._source_ast.spans.LineTable[source]¶: The line table the spans are expressed against.

property logical_lines: tuple[simvx.core.scene_io.source._source_ast.trivia.LogicalLine, ...][source]¶: Every logical line of the file, in source order.

module(tree: ast.Module) → simvx.core.scene_io.source._source_ast.trivia.SuiteTrivia[source]¶: Trivia for the top-level suite, owning the file’s head and tail.

nested(statements: collections.abc.Sequence[ast.stmt]) → simvx.core.scene_io.source._source_ast.trivia.SuiteTrivia[source]¶

Trivia for an indented suite, finding its header line for the caller.

The text a nested suite owns begins where its compound statement’s header line ended, and ast does not report the colon. It does not have to: the header is the logical line before the one the first body statement opens, because the comment and blank lines between them are not logical lines at all. A one-line suite (if ready: go()) has no line of its own, and its first statement’s gap is empty.

The line the search starts from is the statement’s, not the node’s: a decorated first statement opens its suite at the @, which sits a logical line or more above the def that ast reports.

suite(statements: collections.abc.Sequence[ast.stmt], *, start: int, end: int | None = None) → simvx.core.scene_io.source._source_ast.trivia.SuiteTrivia[source]¶

Trivia for one suite, whose text begins at start.

start is where the suite’s ownership of the text begins: 0 for a module, and the end of the compound statement’s header line for a nested suite. end defaults to the last statement’s end, which is the correct answer for a nested suite; a module passes the length of the file.

simvx.core.scene_io.source._source_ast.trivia.scan(text: str) → simvx.core.scene_io.source._source_ast.trivia.TokenIndex[source]¶: Tokenize text once and return the index built from it.