refex.python.matcher

Support for high-level AST matchers.

These return extended match objects that can carry syntactic and/or lexical information about the match.

parse_ast(source_code: str, filename: str = '<string>') → refex.python.matcher.PythonParsedFile

Parses an AST in a way that supports the built-in matchers.

This includes auxiliary information / post-processing / etc. which are not provided by ast.parse().

Parameters

  • source_code – Python source code.

  • filename – file name (for reporting syntax errors and the like).

Returns

The resulting PythonParsedFile

Raises

ParseError – The file failed to parse.

find_iter(matcher: refex.python.matcher.Matcher, parsed: refex.python.matcher.PythonParsedFile) → Iterator[refex.python.matcher.MatchInfo]

Finds all matches in an AST.

This is analogous to re.finditer(). If a node is matched, then any sub-nodes are skipped. Otherwise, find_iter() recurses inside of the node to find matches inside of it.

Parameters
Yields

The successful results of matcher.match() (MatchInfo).

Parsing

exception ParseError

Bases: Exception

Exception raised if input failed to parse.

The stringified exception specifies all the necessary/available debug information about the error, location, etc., but does not include the filename.

class PythonParsedFile(text: str, path: str, pragmas: Iterable[Pragma], ast_tokens: asttokens.asttokens.ASTTokens, tree: _ast.Module, nav: AstNav)

Bases: refex.parsed_file.ParsedFile

Preprocessed file information, including the AST etc.

class MatchContext(parsed_file: refex.python.matcher.PythonParsedFile, has_successful_run: Set[Hashable] = NOTHING, has_match_run: Set[Hashable] = NOTHING)

Bases: object

Per-match and per-file context.

One prototype MatchContext exists per file, containing shared data across all successful runs. A separate derived instance is made per top-level invocation of a matcher, which can be used to keep state across a match that shouldn’t transfer to subsequent match attempts. (For example, a matcher that must match exactly the same string across all invocations inside a match, or similar.)

Matches

class LexicalMatch(text: str, first_token: asttokens.util.Token, last_token: asttokens.util.Token, include_first: bool = True, include_last: bool = True)

Bases: refex.match.Match

Lexical match with no AST information.

first_token

The first token delimiting the matched lexical span.

last_token

The last token delimiting the matched lexical span.

include_first

Whether this starts before or after the first token.

include_last

Whether this ends before or after the last token.

string

The matched string.

span

The matched span.

class LexicalASTMatch(matched: Any, text: str, first_token: asttokens.util.Token, last_token: asttokens.util.Token, include_first: bool = True, include_last: bool = True)

Bases: refex.match.ObjectMatch, refex.python.matcher.LexicalMatch

AST match with adjustable start/end tokens.

Bindings

class BoundValue(value: Any, on_conflict=None, on_merge=None)

Mergeable bound values.

Two BoundValue objects will only be merged if they have equal merge attributes – i.e. if they both agree on how to merge with each other. Anything else results in a MatchError.

value

A value that has a bound name somewhere.

on_conflict

A strategy from BindConflict for merging two bound values. Defaults to MERGE.

on_merge

A strategy from BindMerge for how to merge. Defaults to KEEP_LAST.

rebind(on_conflict=None, on_merge=None)

Returns a new BoundValue with different merge settings.

Parameters

  • on_conflict – if not None, a new value for on_conflict.

  • on_merge – if not None, a new value for on_merge.

Returns

A new BoundValue with the same value, and the updated merge settings.

class BindMerge

Bind conflict merge strategies.

If a conflict is considered mergeable by the BindConflict parameter, this says how to merge the two.

KEEP_FIRST
KEEP_LAST
class BindConflict

What to do if two bindings share the same name.

Each of these is a value that can be passed as the merge parameter to BoundValue, or the on_conflict parameter to base_matchers.Bind.

MERGE

Merge the two bindings (using the chosen BindMerge strategy).

SKIP

Fail this match.

ERROR

Raise a MatchError

MERGE_IDENTICAL

Merge if the values are equal, otherwise SKIP.

MERGE_IDENTICAL_OR_ERROR

Merge if the values are equal, otherwise ERROR.

MERGE_EQUIVALENT_AST

Merge if the values are “equivalent” ASTs, otherwise SKIP.

Two ASTs are equivalent if they are structurally identical, modulo lvalue/rvalue distinctions. (Variables are considered equivalent if they have the same name, even if one is used for a load and the other is used for a store.)

MERGE_EQUIVALENT_AST_OR_ERROR

Merge if the values are “equivalent” ASTs, otherwise ERROR.

Matchers

exception MatchError

An error raised when something went wrong in the match.

This exception represents a bug in the way a matcher was used, not a bug in the matcher implementation or in refex itself.

class MatchInfo(match: refex.match.Match, bindings: Dict[str, refex.python.matcher.BoundValue] = NOTHING, replacements: Dict[str, refex.formatting.Template] = NOTHING)

A result object containing a successful match.

match

The match for the top-level matcher.

replacement

the replacement for the match, if any.

bindings

A dict of bound matches for later reuse.

replacements

A dict of replacements for given bindings.

class Matcher

An immutable AST node matcher.

Reusable subclasses are in refex.python.matchers.

match(context: refex.python.matcher.MatchContext, candidate: Any) → Optional[refex.python.matcher.MatchInfo]

Matches a candidate value.

Parameters

  • context – A MatchContext object with additional metadata.

  • candidate – A candidate object to be matched against.

Returns

A MatchInfo object, or None if the match failed.

Concrete Matchers

class DebugLabeledMatcher(debug_label: str, submatcher, log_level: int = 1)

A matcher which wraps another matcher with a label for debugging.

class ImplicitEquals(value: Any)

Matches a candidate iff it equals the provided value.

This is implicitly created whenever a matcher was expected, but a non-matcher value was provided. For example, Contains(3) creates the matcher Contains(ImplicitEquals(3)).

To explicitly compare for equality, use base_matchers.Equals. ImplicitEquals should never be used explicitly (but is fine to use implicitly).