MCPcopy
hub / github.com/scrapy/scrapy / persist_file

Method persist_file

scrapy/pipelines/files.py:339–358  ·  view source on GitHub ↗
(
        self,
        path: str,
        buf: BytesIO,
        info: MediaPipeline.SpiderInfo,
        meta: dict[str, Any] | None = None,
        headers: dict[str, str] | None = None,
    )

Source from the content-addressed store, hash-verified

337 return self.prefix + path
338
339 def persist_file(
340 self,
341 path: str,
342 buf: BytesIO,
343 info: MediaPipeline.SpiderInfo,
344 meta: dict[str, Any] | None = None,
345 headers: dict[str, str] | None = None,
346 ) -> Deferred[Any]:
347 blob_path = self._get_blob_path(path)
348 blob = self.bucket.blob(blob_path)
349 blob.cache_control = self.CACHE_CONTROL
350 blob.metadata = {k: str(v) for k, v in meta.items()} if meta else {}
351 return deferred_from_coro(
352 run_in_thread(
353 blob.upload_from_string,
354 data=buf.getvalue(),
355 content_type=self._get_content_type(headers),
356 predefined_acl=self.POLICY,
357 )
358 )
359
360
361class FTPFilesStore:

Callers 2

test_persistMethod · 0.95

Calls 5

_get_blob_pathMethod · 0.95
_get_content_typeMethod · 0.95
deferred_from_coroFunction · 0.90
run_in_threadFunction · 0.90
itemsMethod · 0.80

Tested by 2

test_persistMethod · 0.76