MCPcopy Index your code
hub / github.com/python/cpython / untokenize

Method untokenize

Lib/tokenize.py:225–270  ·  view source on GitHub ↗
(self, iterable)

Source from the content-addressed store, hash-verified

223 return "".join(characters)
224
225 def untokenize(self, iterable):
226 it = iter(iterable)
227 indents = []
228 startline = False
229 for t in it:
230 if len(t) == 2:
231 self.compat(t, it)
232 break
233 tok_type, token, start, end, line = t
234 if tok_type == ENCODING:
235 self.encoding = token
236 continue
237 if tok_type == ENDMARKER:
238 break
239 if tok_type == INDENT:
240 indents.append(token)
241 continue
242 elif tok_type == DEDENT:
243 indents.pop()
244 self.prev_row, self.prev_col = end
245 continue
246 elif tok_type in (NEWLINE, NL):
247 startline = True
248 elif startline and indents:
249 indent = indents[-1]
250 if start[1] >= len(indent):
251 self.tokens.append(indent)
252 self.prev_col = len(indent)
253 startline = False
254 elif tok_type in {FSTRING_MIDDLE, TSTRING_MIDDLE}:
255 if '{' in token or '}' in token:
256 token = self.escape_brackets(token)
257 last_line = token.splitlines()[-1]
258 end_line, end_col = end
259 extra_chars = last_line.count("{{") + last_line.count("}}")
260 end = (end_line, end_col + extra_chars)
261
262 self.add_whitespace(start)
263 self.tokens.append(token)
264 self.prev_row, self.prev_col = end
265 if tok_type in (NEWLINE, NL):
266 self.prev_row += 1
267 self.prev_col = 0
268 self.prev_type = tok_type
269 self.prev_line = line
270 return "".join(self.tokens)
271
272 def compat(self, token, iterable):
273 indents = []

Callers 5

untokenizeFunction · 0.95
test_iter_compatMethod · 0.95
decistmtFunction · 0.80
check_roundtripMethod · 0.80
roundtripMethod · 0.80

Calls 8

compatMethod · 0.95
escape_bracketsMethod · 0.95
add_whitespaceMethod · 0.95
appendMethod · 0.45
popMethod · 0.45
splitlinesMethod · 0.45
countMethod · 0.45
joinMethod · 0.45

Tested by 4

test_iter_compatMethod · 0.76
decistmtFunction · 0.64
check_roundtripMethod · 0.64
roundtripMethod · 0.64