MCPcopy Index your code
hub / github.com/python/cpython / tokenize.py

File tokenize.py

Lib/tokenize.py:None–None  ·  view source on GitHub ↗

Source from the content-addressed store, hash-verified

1"""Tokenization help for Python programs.
2
3tokenize(readline) is a generator that breaks a stream of bytes into
4Python tokens. It decodes the bytes according to PEP-0263 for

Callers

nothing calls this directly

Calls 8

maybeFunction · 0.85
_all_string_prefixesFunction · 0.85
setFunction · 0.85
anyFunction · 0.70
groupFunction · 0.70
_mainFunction · 0.70
compileMethod · 0.45
addMethod · 0.45

Tested by

no test coverage detected