MCPcopy
hub / github.com/scrapy/scrapy / download_request

Method download_request

scrapy/core/downloader/handlers/s3.py:50–70  ·  view source on GitHub ↗
(self, request: Request)

Source from the content-addressed store, hash-verified

48 self._download_http = _http_handler.download_request
49
50 async def download_request(self, request: Request) -> Response:
51 p = urlparse_cached(request)
52 scheme = "https" if request.meta.get("is_secure") else "http"
53 bucket = p.hostname
54 path = p.path + "?" + p.query if p.query else p.path
55 url = f"{scheme}://{bucket}.s3.amazonaws.com{path}"
56 if self.anon:
57 request = request.replace(url=url)
58 else:
59 import botocore.awsrequest # noqa: PLC0415
60
61 awsrequest = botocore.awsrequest.AWSRequest(
62 method=request.method,
63 url=f"{scheme}://s3.amazonaws.com/{bucket}{path}",
64 headers=cast("Mapping[str, Any]", request.headers.to_unicode_dict()),
65 data=request.body,
66 )
67 assert self._signer
68 self._signer.add_auth(awsrequest)
69 request = request.replace(url=url, headers=awsrequest.headers.items())
70 return await self._download_http(request)

Callers

nothing calls this directly

Calls 5

urlparse_cachedFunction · 0.90
to_unicode_dictMethod · 0.80
itemsMethod · 0.80
getMethod · 0.45
replaceMethod · 0.45

Tested by

no test coverage detected