Get the default blacklist provider configuration.

Return the <meta name="description"> content, or empty string.

Return the class of the paragraph at the bottom end of the short/neargood paragraphs block. If ignore_neargood is True, than only 'bad' or 'g

Return the class of the paragraph at the top end of the short/neargood paragraphs block. If ignore_neargood is True, than only 'bad' or 'good

Index each document against the query unigrams/bigrams it matches. A query term matches a document when all of the term's words are present in

Check if domain is in the static blacklist.

Check if domain should be blacklisted based on built-in rules.

Load a model from disk (XGBoost binary format).

Save the trained model to disk (XGBoost binary format).

Run the synchronous crawl_url on the dedicated crawl executor.

Convert a DataFrame to a list of dicts for the Rust boundary.

Merge consecutive bold segments (with whitespace-only gaps) into phrases.

Fill ``template`` from ``context``, URL-quoting each substituted value.

Return True only for ordinary, routable public addresses.

Register all crawler routes on the given router or API instance.

Register the v1 search endpoint: returns a plain list of SearchResult.

Resolve a raw href to an absolute URL and validate it. Returns None if invalid.

Context-free paragraph classification.

Mark a task/file as done

Return all active monthly counter keys for the current month. Uses the underlying Redis SCAN command via django-redis. Only call this fro

Converts DOM into paragraphs.

Translates multiple whitespace into single space character. If there is at least one new line character chunk is replaced by single LF (U

Removes unwanted parts of DOM.

Context-sensitive paragraph classification. Assumes that classify_pragraphs has already been called.