Checks whether some files in the Decathlon datalist are missing. It would be helpful to check missing files before a heavy training run. Args: datalist: a list of data items, every item is a dictionary. usually generated by `load_decathlon_datalist` API. keys: ex
(
datalist: list[dict], keys: KeysCollection, root_dir: PathLike | None = None, allow_missing_keys: bool = False
)
| 194 | |
| 195 | |
| 196 | def check_missing_files( |
| 197 | datalist: list[dict], keys: KeysCollection, root_dir: PathLike | None = None, allow_missing_keys: bool = False |
| 198 | ): |
| 199 | """Checks whether some files in the Decathlon datalist are missing. |
| 200 | It would be helpful to check missing files before a heavy training run. |
| 201 | |
| 202 | Args: |
| 203 | datalist: a list of data items, every item is a dictionary. |
| 204 | usually generated by `load_decathlon_datalist` API. |
| 205 | keys: expected keys to check in the datalist. |
| 206 | root_dir: if not None, provides the root dir for the relative file paths in `datalist`. |
| 207 | allow_missing_keys: whether allow missing keys in the datalist items. |
| 208 | if False, raise exception if missing. default to False. |
| 209 | |
| 210 | Returns: |
| 211 | A list of missing filenames. |
| 212 | |
| 213 | """ |
| 214 | missing_files = [] |
| 215 | for item in datalist: |
| 216 | for k in ensure_tuple(keys): |
| 217 | if k not in item: |
| 218 | if not allow_missing_keys: |
| 219 | raise ValueError(f"key `{k}` is missing in the datalist item: {item}") |
| 220 | continue |
| 221 | |
| 222 | for f in ensure_tuple(item[k]): |
| 223 | if not isinstance(f, (str, os.PathLike)): |
| 224 | raise ValueError(f"filepath of key `{k}` must be a string or a list of strings, but got: {f}.") |
| 225 | f = Path(f) |
| 226 | if isinstance(root_dir, (str, os.PathLike)): |
| 227 | f = Path(root_dir).joinpath(f) |
| 228 | if not f.exists(): |
| 229 | missing_files.append(f) |
| 230 | |
| 231 | return missing_files |
| 232 | |
| 233 | |
| 234 | def create_cross_validation_datalist( |
searching dependent graphs…