Method forward

tensorrt_llm/python_plugin.py:464–474

Expect users to rewrite this function to define the compute flow. There are a few special attributes for users to get access to some resources. `self.workspace`: The workspace address of TRT managed workspace. `self.current_stream`: The CUDA stream this plugin is expected t

(self, inputs: Sequence[TensorWrapper], outputs: Sequence[TensorWrapper])

            )

    def forward(self, inputs: Sequence[TensorWrapper], outputs: Sequence[TensorWrapper]):
        """Expect users to rewrite this function to define the compute flow.

        There are a few special attributes for users to get access to some resources.

        `self.workspace`: The workspace address of TRT managed workspace.
        `self.current_stream`: The CUDA stream this plugin is expected to execute on. By default
        `PluginBase` set the torch.cuda.current_stream() to this stream. This attribute is for the
        toolkit that doesn't work with torch's stream.
        """
        raise NotImplementedError

    def shape_dtype_inference(self, inputs: Sequence[SymTensor]):
        """Expect users to rewrite this function to define the shape dtype inference for output tensors."""