MCPcopy
hub / github.com/scrapy/scrapy / curl_to_request_kwargs

Function curl_to_request_kwargs

scrapy/utils/curl.py:87–138  ·  view source on GitHub ↗

Convert a cURL command syntax to Request kwargs. :param str curl_command: string containing the curl command :param bool ignore_unknown_options: If true, only a warning is emitted when cURL options are unknown. Otherwise

(
    curl_command: str, ignore_unknown_options: bool = True
)

Source from the content-addressed store, hash-verified

85
86
87def curl_to_request_kwargs(
88 curl_command: str, ignore_unknown_options: bool = True
89) -> dict[str, Any]:
90 """Convert a cURL command syntax to Request kwargs.
91
92 :param str curl_command: string containing the curl command
93 :param bool ignore_unknown_options: If true, only a warning is emitted when
94 cURL options are unknown. Otherwise
95 raises an error. (default: True)
96 :return: dictionary of Request kwargs
97 """
98
99 curl_args = split(curl_command)
100
101 if curl_args[0] != "curl":
102 raise ValueError('A curl command must start with "curl"')
103
104 parsed_args, argv = curl_parser.parse_known_args(curl_args[1:])
105
106 if argv:
107 msg = f"Unrecognized options: {', '.join(argv)}"
108 if ignore_unknown_options:
109 warnings.warn(msg, stacklevel=2)
110 else:
111 raise ValueError(msg)
112
113 url = parsed_args.url
114
115 # curl automatically prepends 'http' if the scheme is missing, but Request
116 # needs the scheme to work
117 parsed_url = urlparse(url)
118 if not parsed_url.scheme:
119 url = "http://" + url
120
121 method = parsed_args.method or "GET"
122
123 result: dict[str, Any] = {"method": method.upper(), "url": url}
124
125 headers, cookies = _parse_headers_and_cookies(parsed_args)
126
127 if headers:
128 result["headers"] = headers
129 if cookies:
130 result["cookies"] = cookies
131 if parsed_args.data:
132 result["body"] = parsed_args.data
133 if not parsed_args.method:
134 # if the "data" is specified but the "method" is not specified,
135 # the default method is 'POST'
136 result["method"] = "POST"
137
138 return result

Callers 6

from_curlMethod · 0.90
_test_commandMethod · 0.90
test_get_silentMethod · 0.90

Calls 2

joinMethod · 0.45

Tested by 5

_test_commandMethod · 0.72
test_get_silentMethod · 0.72