MCPcopy
hub / github.com/scrapy/scrapy / parse_with_rules

Method parse_with_rules

scrapy/spiders/crawl.py:169–188  ·  view source on GitHub ↗
(
        self,
        response: Response,
        callback: CallbackT | None,
        cb_kwargs: dict[str, Any],
        follow: bool = True,
    )

Source from the content-addressed store, hash-verified

167 )
168
169 async def parse_with_rules(
170 self,
171 response: Response,
172 callback: CallbackT | None,
173 cb_kwargs: dict[str, Any],
174 follow: bool = True,
175 ) -> AsyncIterator[Any]:
176 if callback:
177 cb_res = callback(response, **cb_kwargs) or ()
178 if isinstance(cb_res, AsyncIterator):
179 cb_res = await collect_asyncgen(cb_res)
180 elif isinstance(cb_res, Awaitable):
181 cb_res = await cb_res
182 cb_res = self.process_results(response, cb_res)
183 for request_or_item in iterate_spider_output(cb_res):
184 yield request_or_item
185
186 if follow and self._follow_links:
187 for request_or_item in self._requests_to_follow(response):
188 yield request_or_item
189
190 def _parse_response(
191 self,

Callers 4

_parseMethod · 0.95
_callbackMethod · 0.95
_parse_responseMethod · 0.95
test_parse_with_rulesMethod · 0.80

Calls 4

process_resultsMethod · 0.95
_requests_to_followMethod · 0.95
collect_asyncgenFunction · 0.90
iterate_spider_outputFunction · 0.90

Tested by 1

test_parse_with_rulesMethod · 0.64