Returns the time the robots.txt file was last fetched. This is useful for long-running web spiders that need to check for new robots.txt files periodically.
(self)
| 50 | self.last_checked = 0 |
| 51 | |
| 52 | def mtime(self): |
| 53 | """Returns the time the robots.txt file was last fetched. |
| 54 | |
| 55 | This is useful for long-running web spiders that need to |
| 56 | check for new robots.txt files periodically. |
| 57 | |
| 58 | """ |
| 59 | return self.last_checked |
| 60 | |
| 61 | def modified(self): |
| 62 | """Sets the time the robots.txt file was last fetched to the |
no outgoing calls