Get the next priority item from the queue, returning the item ID and the object

Human-readable formatting of bytes, using binary (powers of 1024) or metric (powers of 1000) representation.

Predict class-1 probabilities. Parameters ---------- X : DataFrame with columns query, url, title, extract, score

Get the page at index i, decompress and deserialise it using JSON

Replace the live source adapters with deterministic stubs.

Stub both the promotion scorer (_heuristic_score_docs) and the LTR final ranker (score_documents) to consume from a single shared score iterator.

Convert a type to a tuple - values at the end that are None can be truncated.

Train the XGBoost model. Parameters ---------- X : DataFrame with columns query, url, title, extract, score

Return the current monthly request count for a user (0 if not set).

Return True if every address ``host`` resolves to is a public address. ``host`` may be an IP literal or a hostname. Hostnames are resolved with

Create a synthetic DataFrame with the columns expected by RustXGBPipeline.

Re-order a relevance-sorted list to demote near-duplicate / same-domain results. Greedy Maximal Marginal Relevance with a domain-dominant kernel

Custom URL parsing function using regex because urlparse is too slow.

Push a new item into the ready state

Main entry point for the crawler. Starts multiple worker processes for batch processing and one indexing process. Mo

Check Redis connection health.

Return (raw_key, key_hash). Store only the hash; return raw_key to the user once.

Check if domain is blacklisted by any provider.

Return the text of the first substantive <p> element, or empty string. A last-resort fallback for pages where justext finds no 'good' content — m

Check if a domain should be blacklisted.

Process a single batch of URLs by crawling them sequentially with rate limiting. This function handles the core crawling workflow:

Process completed crawl batches and integrate results into the search index. This function handles the indexing workflow: 1.

Raise :class:`UnsafeURLError` if ``url`` is not safe to fetch.

Check if user's email is verified.

Get the default crawler instance.

Converts HTML to DOM.

Increment the monthly counter and return the new value. Sets a 35-day TTL on first use so the key auto-expires.

Check if domain is in the HaGeZi blacklist.

Load every ``*.yaml`` recipe in ``directory``, keyed by recipe name. A malformed recipe (missing required keys, bad YAML) is logged and skipped

Call the Rust get_features_py function.

Update URLs with the most recent crawl date, if they've been crawled before, or None otherwise. Update the most recent URL status in

Get the count for a given date and set it in the count_dict.

Register the /complete and /raw endpoints (shared between v1 and v2).

Walk a dotted path (``a.b.c``) through nested dicts; None if absent.

Fixed-window rate limit: at most RATE_LIMIT requests per second. Returns True if the request is allowed, False if the limit is exceeded.

Mark a task/file as in error state

Fetch with a maximum timeout and maximum fetch size to avoid big pages bringing us down. Redirects are followed manually so each hop can be

Load a pre-trained model from disk and return a ready-to-predict pipeline.