SearchLinkGraph is a scraping pipeline that automates the process of extracting information from web pages using a natural language model to interpret and answer prompts. Attributes: prompt (str): The prompt for the graph. source (str): The source of the graph.
| 12 | |
| 13 | |
| 14 | class SearchLinkGraph(AbstractGraph): |
| 15 | """ |
| 16 | SearchLinkGraph is a scraping pipeline that automates the process of |
| 17 | extracting information from web pages using a natural language model |
| 18 | to interpret and answer prompts. |
| 19 | |
| 20 | Attributes: |
| 21 | prompt (str): The prompt for the graph. |
| 22 | source (str): The source of the graph. |
| 23 | config (dict): Configuration parameters for the graph. |
| 24 | schema (BaseModel): The schema for the graph output. |
| 25 | llm_model: An instance of a language model client, configured for generating answers. |
| 26 | embedder_model: An instance of an embedding model client, |
| 27 | configured for generating embeddings. |
| 28 | verbose (bool): A flag indicating whether to show print statements during execution. |
| 29 | headless (bool): A flag indicating whether to run the graph in headless mode. |
| 30 | |
| 31 | Args: |
| 32 | source (str): The source of the graph. |
| 33 | config (dict): Configuration parameters for the graph. |
| 34 | schema (BaseModel, optional): The schema for the graph output. Defaults to None. |
| 35 | |
| 36 | |
| 37 | """ |
| 38 | |
| 39 | def __init__( |
| 40 | self, source: str, config: dict, schema: Optional[Type[BaseModel]] = None |
| 41 | ): |
| 42 | super().__init__("", config, source, schema) |
| 43 | |
| 44 | self.input_key = "url" if source.startswith("http") else "local_dir" |
| 45 | |
| 46 | def _create_graph(self) -> BaseGraph: |
| 47 | """ |
| 48 | Creates the graph of nodes representing the workflow for web scraping. |
| 49 | |
| 50 | Returns: |
| 51 | BaseGraph: A graph instance representing the web scraping workflow. |
| 52 | """ |
| 53 | |
| 54 | fetch_node = FetchNode( |
| 55 | input="url| local_dir", |
| 56 | output=["doc"], |
| 57 | node_config={ |
| 58 | "force": self.config.get("force", False), |
| 59 | "cut": self.config.get("cut", True), |
| 60 | "loader_kwargs": self.config.get("loader_kwargs", {}), |
| 61 | "storage_state": self.config.get("storage_state"), |
| 62 | }, |
| 63 | ) |
| 64 | |
| 65 | if self.config.get("llm_style") == (True, None): |
| 66 | search_link_node = SearchLinksWithContext( |
| 67 | input="doc", |
| 68 | output=["parsed_doc"], |
| 69 | node_config={ |
| 70 | "llm_model": self.llm_model, |
| 71 | "chunk_size": self.model_token, |
no outgoing calls