simvx.core.scene_io.source._source_ast.spans

Where things are: ast positions converted to str indices, once, here.

This is the one module in the engine that knows CPython reports a column as a utf-8 byte offset into its line while Python strings are indexed in code points. Every other module deals in :class:Span values, which are plain str slice bounds, so a scene holding an em-space in a comment or an emoji in a label is not a special case anywhere above this file.

Four hazards live here, each with a named test in packages/core/tests/scene_io/test_source_ast_spans.py:

• Byte columns. col_offset counts utf-8 bytes; tokenize counts characters. :class:LineTable converts from either convention and never guesses which one it was handed.

• Decorators. ast puts a decorated statement’s lineno on the def or class line, so the decorators above it are outside the node’s own span.

  func:

  statement_span widens to the first @.

• Redundant parentheses. An expression node’s span excludes parentheses the author wrote around it, so f(x=(1 + 2)) reports 1 + 2. Splicing that span alone therefore keeps the author’s parentheses, which is what we want;

  class:

  ValueSlot records where they are so a caller can also tell when a replacement needs parentheses it does not have.

• Position-less nodes. ast.Module, ast.arguments and the context markers carry no position at all. :func:node_span refuses them by name instead of raising AttributeError three layers up.

The line grid is the one the language reference defines: a physical line ends at \n or \r\n. A bare \r terminator is refused (see

class:

UnsupportedLineEnding) because ast treats it as a line break and tokenize does not, and no correct answer can be built on two parsers that disagree about where the lines are.

Module Contents

Classes

Span	A half-open range of str indices into one source text.
LineTable	The line grid of one source text, and the two column conventions on it.
NameSpan	One occurrence of a bare name, and whether it reads or writes it.
ValueSlot	A place a value expression can be replaced, and what a replacement owes.
ArgumentSpans	One argument of one call, as written.
CallSpans	Every span an argument-level edit of one call expression needs.

Functions

node_span	The span ast reports for node, converted to str indices.
statement_span	The full span of one statement, decorators included, indentation excluded.
statement_spans	func: statement_span for every statement of one suite, in source order.
name_spans	Every ast.Name occurrence inside node, in source order.
call_spans	Resolve every span of one call expression.
value_slot	A :class:ValueSlot for a free-standing expression inside bounds.
iter_child_statements	Every suite hanging off node, as (field name, statement list).

Data

__all__
ValueRole	Where a replaceable expression sits, which decides what parentheses it owes.

API

simvx.core.scene_io.source._source_ast.spans.__all__

[‘ArgumentSpans’, ‘CallSpans’, ‘LineTable’, ‘NameSpan’, ‘Span’, ‘UnsupportedLineEnding’, ‘ValueRole’…

exception simvx.core.scene_io.source._source_ast.spans.UnsupportedLineEnding

Bases: ValueError

Raised for source whose lines end with a bare carriage return.

ast.parse accepts x = 1\ry = 2 as two statements on two lines; tokenize reads the same text as one line containing a stray operator. Trivia is derived from the token stream and spans from the parse tree, so a file the two disagree about cannot be edited faithfully by any amount of care further up. Refusing it here is the only honest answer, and it is reachable in practice only from classic Mac OS files.

Initialization

Initialize self. See help(type(self)) for accurate signature.

class __cause__

class __context__

__delattr__()

__dir__()

__eq__()

__format__()

__ge__()

__getattribute__()

__getstate__()

__gt__()

__hash__()

__le__()

__lt__()

__ne__()

__new__()

__reduce__()

__reduce_ex__()

__repr__()

__setattr__()

__setstate__()

__sizeof__()

__str__()

__subclasshook__()

class __suppress_context__

class __traceback__

add_note()

class args

with_traceback()

simvx.core.scene_io.source._source_ast.spans.ValueRole

None

Where a replaceable expression sits, which decides what parentheses it owes.

class simvx.core.scene_io.source._source_ast.spans.Span

A half-open range of str indices into one source text.

Spans are compared and sorted by (start, stop), which puts them in source order. They carry no reference to the text they index, so a span outlives the edit that invalidated it: the document layer above is responsible for shifting or discarding spans across a splice.

start: int

None

stop: int

None

__post_init__() → None

__len__() → int

__bool__() → bool

True when the span covers at least one character.

Defined explicitly because __len__ alone would make every empty span falsy in a way that reads as “missing” at the call sites that ask if trivia.leading:. Empty is the answer there, so keep it.

text(source: str) → str

The slice of source this span covers.

shift(delta: int) → simvx.core.scene_io.source._source_ast.spans.Span

The same span moved delta characters along the text.

contains(other: simvx.core.scene_io.source._source_ast.spans.Span) → bool

True when other lies wholly inside this span.

class simvx.core.scene_io.source._source_ast.spans.LineTable(text: str)

The line grid of one source text, and the two column conventions on it.

Built once per parse. Converting a position costs a list index for ASCII lines and a cached scan for the rest, so the utf-8 hazard is paid for only by the files that actually contain non-ASCII text.

Line numbers are 1-indexed to match ast and tokenize. A line’s span includes its terminator, so line_span(n).stop is where line n + 1 begins and the spans tile the text with no gaps.

Initialization

__slots__

(‘_text’, ‘_starts’, ‘_byte_maps’)

property text: str

The source this table was built from.

property line_count: int

Number of physical lines.

A file ending with a newline has one more line than it has terminators: the empty last one, which is where tokenize puts its end marker and where line_of(len(text)) lands. Counting it is what keeps every offset in the text, including the one past the end, on a real line.

line_start(lineno: int) → int

str index of the first character of line lineno.

line_span(lineno: int) → simvx.core.scene_io.source._source_ast.spans.Span

Span of line lineno including its terminator.

line_text(lineno: int) → str

Text of line lineno including its terminator.

offset(lineno: int, col_offset: int) → int

str index for an ast position, whose column counts utf-8 bytes.

This is the conversion the whole module exists for. Pass lineno and col_offset (or end_lineno/end_col_offset) straight off an ast node; never add a column to a line start yourself.

char_offset(lineno: int, column: int) → int

str index for a tokenize position, whose column counts characters.

The token stream reports columns in code points, so this is a plain addition. It exists as a named method so that a call site states which convention its numbers came from.

position(offset: int) → tuple[int, int]

(lineno, col_offset) in the ast convention for a str index.

char_position(offset: int) → tuple[int, int]

(lineno, column) in the tokenize convention for a str index.

line_of(offset: int) → int

1-indexed line holding offset; the last line for the end of text.

indent_of(offset: int) → simvx.core.scene_io.source._source_ast.spans.Span

The leading whitespace of offset’s line, empty when code precedes it.

A statement written after a semicolon has no indentation of its own, and this answers with an empty span there rather than with the indentation of the statement it shares a line with.

simvx.core.scene_io.source._source_ast.spans.node_span(table: simvx.core.scene_io.source._source_ast.spans.LineTable, node: ast.AST) → simvx.core.scene_io.source._source_ast.spans.Span

The span ast reports for node, converted to str indices.

Raises :class:TypeError for the nodes that carry no position at all (ast.Module, ast.arguments, the ast.expr_context markers), which is a caller bug rather than a source the layer cannot read.

simvx.core.scene_io.source._source_ast.spans.statement_span(table: simvx.core.scene_io.source._source_ast.spans.LineTable, node: ast.stmt) → simvx.core.scene_io.source._source_ast.spans.Span

The full span of one statement, decorators included, indentation excluded.

ast reports a decorated function or class from its def/class keyword, leaving @property above it outside the node. A caller that replaced or removed the node’s own span would leave the decorators behind attached to whatever followed, so the span starts at the first @.

Leading whitespace is not part of the statement. Indentation, blank lines and the comments above belong to the statement’s trivia record, which is what carries them through a move (see :mod:.trivia).

simvx.core.scene_io.source._source_ast.spans.statement_spans(table: simvx.core.scene_io.source._source_ast.spans.LineTable, body: collections.abc.Sequence[ast.stmt]) → list[simvx.core.scene_io.source._source_ast.spans.Span]

Func:

statement_span for every statement of one suite, in source order.

class simvx.core.scene_io.source._source_ast.spans.NameSpan

One occurrence of a bare name, and whether it reads or writes it.

span: simvx.core.scene_io.source._source_ast.spans.Span

None

name: str

None

context: Literal[load, store, delete]

None

simvx.core.scene_io.source._source_ast.spans.name_spans(table: simvx.core.scene_io.source._source_ast.spans.LineTable, node: ast.AST, *, name: str | None = None) → list[simvx.core.scene_io.source._source_ast.spans.NameSpan]

Every ast.Name occurrence inside node, in source order.

This is the rename surface: replacing each span with a new identifier renames the variable and touches nothing else. Attribute names, keyword argument names and string contents are deliberately absent, because none of them is the same binding as the name that spells them.

Occurrences inside a PEP 701 f-string are included and carry real positions, so f"{hero}" renames with the rest.

class simvx.core.scene_io.source._source_ast.spans.ValueSlot

A place a value expression can be replaced, and what a replacement owes.

Attr:

span is the expression as ast sees it, which excludes any parentheses the author wrote around it. Splicing over :attr:span therefore preserves those parentheses, which is the behaviour we want: an author who wrapped a value keeps the wrapping. :attr:outer is the same value with them, so a caller that needs to remove the whole argument knows how far it reaches, and :attr:parenthesised says whether the two differ.

The reverse case is a replacement that needs parentheses the slot does not have: a bare tuple in a positional slot would change the call’s arity, and a walrus or a yield beside a keyword’s = does not parse at all. That is

Meth:

requires_parentheses, kept as a question rather than an automatic rewrite so a caller can decide to reject the value instead.

span: simvx.core.scene_io.source._source_ast.spans.Span

None

outer: simvx.core.scene_io.source._source_ast.spans.Span

None

role: simvx.core.scene_io.source._source_ast.spans.ValueRole

None

property parenthesised: bool

True when the author wrote parentheses around this value.

requires_parentheses(replacement: str) → bool

True when replacement would not parse, or would not mean the same, bare here.

Raises :class:SyntaxError (CPython’s own) when replacement is not a single expression, which is the check every emitted value should be passing anyway.

splice(replacement: str) → tuple[simvx.core.scene_io.source._source_ast.spans.Span, str]

The span to overwrite and the text to write, parenthesised if it must be.

class simvx.core.scene_io.source._source_ast.spans.ArgumentSpans

One argument of one call, as written.

Attr:

span covers the argument whole: colour=RED for a keyword, *rest for an unpacking, the bare expression for a positional. It is what a removal deletes. :attr:value is the part a value edit rewrites, which for *rest is rest and for colour=RED is RED.

role: Literal[positional, keyword, star_args, star_kwargs]

None

name: str | None

None

span: simvx.core.scene_io.source._source_ast.spans.Span

None

value: simvx.core.scene_io.source._source_ast.spans.ValueSlot

None

class simvx.core.scene_io.source._source_ast.spans.CallSpans

Every span an argument-level edit of one call expression needs.

Attr:

arguments is in source order, which is not ast’s order: ast lists positionals and keywords separately, and a call may interleave them (f(a, k=1, *rest) is legal). Editing by position demands source order, so it is sorted here once.

call: simvx.core.scene_io.source._source_ast.spans.Span

None

func: simvx.core.scene_io.source._source_ast.spans.Span

None

open_paren: int

None

close_paren: int

None

arguments: tuple[simvx.core.scene_io.source._source_ast.spans.ArgumentSpans, ...]

None

keyword(name: str) → simvx.core.scene_io.source._source_ast.spans.ArgumentSpans | None

The keyword argument written as name=..., or None.

positional(index: int) → simvx.core.scene_io.source._source_ast.spans.ArgumentSpans | None

The index-th positional argument, counting unpackings, or None.

simvx.core.scene_io.source._source_ast.spans.call_spans(table: simvx.core.scene_io.source._source_ast.spans.LineTable, call: ast.Call, *, comments: collections.abc.Iterable[simvx.core.scene_io.source._source_ast.spans.Span] = ()) → simvx.core.scene_io.source._source_ast.spans.CallSpans

Resolve every span of one call expression.

comments is the comment span list from the trivia pass. It is optional and only ever makes the answer better: without it, a value whose author parentheses are separated from it by a comment is reported unparenthesised, which costs a caller a redundant pair of parentheses and never a wrong splice. With it the answer is exact.

simvx.core.scene_io.source._source_ast.spans.value_slot(table: simvx.core.scene_io.source._source_ast.spans.LineTable, node: ast.expr, *, role: simvx.core.scene_io.source._source_ast.spans.ValueRole, bounds: simvx.core.scene_io.source._source_ast.spans.Span, comments: collections.abc.Iterable[simvx.core.scene_io.source._source_ast.spans.Span] = ()) → simvx.core.scene_io.source._source_ast.spans.ValueSlot

A :class:ValueSlot for a free-standing expression inside bounds.

bounds is the region the author’s parentheses may not escape: for a call argument that is the inside of the call’s own parentheses, so that the sole argument of f((1)) reports one pair of author parentheses and not two.

simvx.core.scene_io.source._source_ast.spans.iter_child_statements(node: ast.AST) → collections.abc.Iterator[tuple[str, list[ast.stmt]]]

Every suite hanging off node, as (field name, statement list).

A suite is any field holding a list of statements: a body, an else, a finally, the arm of a match case, the handler of an except*. Naming them by field rather than by statement type is what keeps this function from needing an update when the grammar grows another block.