A intermediate step between constructor and calling the actual file loading method. Args: data: If data is a file, this method calls :meth:`~hanlp.common.dataset.TransformableDataset.load_file` to load it. generate_idx: Create a :const:`~hanlp_common.
(self, data, generate_idx=False)
| 137 | self.cache = None |
| 138 | |
| 139 | def load_data(self, data, generate_idx=False): |
| 140 | """A intermediate step between constructor and calling the actual file loading method. |
| 141 | |
| 142 | Args: |
| 143 | data: If data is a file, this method calls :meth:`~hanlp.common.dataset.TransformableDataset.load_file` |
| 144 | to load it. |
| 145 | generate_idx: Create a :const:`~hanlp_common.constants.IDX` field for each sample to store its order in dataset. Useful for prediction when |
| 146 | samples are re-ordered by a sampler. |
| 147 | |
| 148 | Returns: Loaded samples. |
| 149 | |
| 150 | """ |
| 151 | if self.should_load_file(data): |
| 152 | if isinstance(data, str): |
| 153 | data = get_resource(data) |
| 154 | data = list(self.load_file(data)) |
| 155 | if generate_idx: |
| 156 | for i, each in enumerate(data): |
| 157 | each[IDX] = i |
| 158 | # elif isinstance(data, list): |
| 159 | # data = self.load_list(data) |
| 160 | return data |
| 161 | |
| 162 | # noinspection PyMethodMayBeStatic |
| 163 | # def load_list(self, data: list) -> List[Dict[str, Any]]: |
no test coverage detected