_ModelRuntimeContext holds the minimum runtime resources for running a model. It could be a runtime cache in MPI nodes.
| 78 | |
| 79 | @dataclass |
| 80 | class _ModelRuntimeContext: |
| 81 | ''' _ModelRuntimeContext holds the minimum runtime resources for running a model. |
| 82 | It could be a runtime cache in MPI nodes. |
| 83 | ''' |
| 84 | engine: Optional[Engine] = None |
| 85 | mapping: Optional[Mapping] = None |
| 86 | model_info: Optional[_ModelInfo] = None |
| 87 | |
| 88 | # This is only used when build-cache is enabled |
| 89 | engine_path: Optional[str] = None |
| 90 | |
| 91 | @property |
| 92 | def model_arch(self) -> str: |
| 93 | # "LlaMACausalForLM" or "OPTForCausalLM" and so on |
| 94 | return self.engine.config.pretrained_config['architecture'] |
| 95 | |
| 96 | |
| 97 | class ModelLoader: |