MCPcopy
hub / github.com/scrapy/scrapy / allowed

Method allowed

scrapy/robotstxt.py:120–123  ·  view source on GitHub ↗
(self, url: str | bytes, user_agent: str | bytes)

Source from the content-addressed store, hash-verified

118 return cls(robotstxt_body, spider)
119
120 def allowed(self, url: str | bytes, user_agent: str | bytes) -> bool:
121 user_agent = to_unicode(user_agent)
122 url = to_unicode(url)
123 return self.rp.can_fetch(url, user_agent)

Callers

nothing calls this directly

Calls 1

to_unicodeFunction · 0.90

Tested by

no test coverage detected