Persist details into the database. The details to be stored is controlled by the parameters passed to the method. :param sea

Method that filters list :param lst: list to be filtered :return: new filtered list

Method that is used to generate a random delay

Search the given query string using Google. :param str query: Query string. Must NOT be url-encoded. :param str tld: Top level domain.

Helper function to check if Google has blocked traffic. :param text: See if certain text is returned which means Google is blocking us :return

Function that makes a request on our behalf, if Google starts to block us :param visit_url: Url to scrape :return: Correct html that can

Request the given URL and return the response page, using the cookie jar. :param str url: URL to retrieve. :param str user_agent: User a

Search the given query string using Google and return the number of hits. :note: This is the number reported by Google itself, NOT by scrapi

Return a list of applications that can be detected on the web page.

Strip out key:value pairs from the pattern and compile the regular expression.

chlid class that user different pagnation counter should override this function

Construct a Wappalyzer instance using a apps db path passed in via apps_file, or alternatively the default in data/apps.json

Constructs a new WebPage object for the URL, using the `requests` module to fetch the HTML. Parameters ----------