MCPcopy Create free account

hub / github.com/state-spaces/mamba / functions

Functions436 in github.com/state-spaces/mamba

↓ 19 callersFunctionrelative_error
( ker: torch.Tensor, ref: torch.Tensor, eps: float = 1e-6, ref_mag_mask: float = 1e-2, p:
tests/ops/triton/test_mamba3_siso.py:351
↓ 17 callersMethodbackward
(ctx, dout)
mamba_ssm/ops/triton/k_activations.py:166
↓ 16 callersFunction_bytes_of
(t)
mamba_ssm/ops/cute/mamba3/mamba3_step_fn.py:816
↓ 12 callersFunctionassert_stable_rel
( ours: Tensor, ref: Tensor, *, label: str, cfg: str, rel_tol: float = REL_TOL, )
tests/ops/tilelang/test_mamba3_mimo.py:94
↓ 11 callersFunction_pad_zeros
Append ``pad_len`` zero-slices along ``dim``. Returns ``t`` if pad_len==0.
tests/ops/tilelang/test_mamba3_mimo.py:1424
↓ 9 callersFunction_run
()
tests/test_determinism.py:232
↓ 8 callersFunction_pad
(x)
mamba_ssm/ops/triton/mamba3/mamba3_mimo_utils.py:940
↓ 8 callersFunctionmamba3_MIMO_chunk_ref
( q: Tensor, k: Tensor, v: Tensor, q_bias: Tensor, k_bias: Tensor, mimo_v: Tensor,
tests/ops/tilelang/test_mamba3_mimo.py:464
↓ 8 callersFunctionuse_deterministic_mode
()
mamba_ssm/utils/determinism.py:21
↓ 7 callersFunctionalloc_tile_workspace
Allocate buffer for deterministic per-program reductions.
mamba_ssm/utils/determinism.py:80
↓ 7 callersFunctionfinalize_tile_workspace
(tensor, deterministic)
mamba_ssm/utils/determinism.py:91
↓ 6 callersFunctioncompute_dacs_segsum_triton
Compute da_cs, da_cs_rev, and segsum for fixed-length sequences. Returns ------- da_cs : forward inclusive prefix-sum of da, clipped t
mamba_ssm/ops/triton/mamba3/mamba3_mimo_utils.py:691
↓ 6 callersMethodfrom_pretrained
(cls, pretrained_model_name, device=None, dtype=None, **kwargs)
mamba_ssm/models/mixer_seq_simple.py:287
↓ 6 callersFunctionrms_norm_forward
( x, weight, bias, eps=1e-6, is_rms_norm=True, )
mamba_ssm/ops/selective_scan_interface.py:86
↓ 6 callersFunctionselective_state_update
Argument: state: (batch, dim, dstate) or (batch, nheads, dim, dstate) x: (batch, dim) or (batch, nheads, dim) dt: (batch,
mamba_ssm/ops/triton/selective_state_update.py:135
↓ 6 callersFunctionsigmoid_approx
(Fast) Sigmoid approximation using PTX inline assembly. Formula: sigmoid(x) = 0.5 * (1 + tanh(0.5 * x)) Leverages fast tanh approximatio
mamba_ssm/ops/triton/mamba3/utils.py:92
↓ 5 callersFunction_state_passing_fwd
(states, dA_chunk_cumsum, initial_states=None, seq_idx=None, chunk_size=None, out_dtype
mamba_ssm/ops/triton/ssd_state_passing.py:196
↓ 5 callersFunctionbuild_inputs
( *, mods: SimpleNamespace, n: int, p: int, r: int, chunk_size: int, seed: int,
tests/ops/tilelang/test_mamba3_mimo.py:123
↓ 5 callersFunctionensure_stride
Return inp, while ensuring that stride(1) of the returned tensor is a multiple of 8. The inp tensor is of shape [batch, length, channels], w
mamba_ssm/ops/triton/ssd_combined.py:59
↓ 5 callersMethodgenerate
( self, input_ids, max_length, top_k=1, top_p=0.0, min_p=0.0,
mamba_ssm/utils/generation.py:250
↓ 5 callersMethodstep
Decode function using CuteDSL kernel from mamba3_step_fn.py. Also modify the state vars in-place for the next step. NOTE: On
mamba_ssm/modules/mamba3.py:286
↓ 4 callersMethod__init__
( self, in_features: int, out_features: int, process_group: ProcessGroup,
mamba_ssm/distributed/tensor_parallel.py:152
↓ 4 callersFunction_bmm_chunk_bwd
Argument: a: (batch, seqlen, k) or (batch, seqlen, ngroups, k) dout: (batch, nchunks, chunk_size, chunk_size) or (batch, nchunks,
mamba_ssm/ops/triton/ssd_bmm.py:213
↓ 4 callersFunction_chunk_cumsum_fwd
(dt, A, chunk_size, dt_bias=None, dt_softplus=False, dt_limit=(0.0, float("inf")))
mamba_ssm/ops/triton/ssd_chunk_state.py:718
↓ 4 callersFunction_chunk_scan_bwd_dC
(prev_states, dA_cumsum, dout, seq_idx=None, C=None, ngroups=1)
mamba_ssm/ops/triton/ssd_chunk_scan.py:1451
↓ 4 callersFunction_chunk_state_bwd_db
(x, dt, dA_cumsum, dstates, seq_idx=None, B=None, ngroups=1)
mamba_ssm/ops/triton/ssd_chunk_state.py:902
↓ 4 callersFunction_chunk_state_fwd
(B, x, dt, dA_cumsum, seq_idx=None, states=None, states_in_fp32=True)
mamba_ssm/ops/triton/ssd_chunk_state.py:812
↓ 4 callersFunction_max_abs_diff
(a: torch.Tensor, b: torch.Tensor)
tests/test_determinism.py:20
↓ 4 callersFunction_set_seeds
(seed: int)
tests/test_determinism.py:15
↓ 4 callersFunction_varlen_seqlens
Non-uniform sequence lengths for varlen tests. Returns four lengths [2C-1, 5C-3, 3C, 4C-1] so that adjacent sequences differ, no two are the
tests/ops/tilelang/test_mamba3_mimo.py:1414
↓ 4 callersFunctionall_gather_raw
(input_: Tensor, process_group: ProcessGroup, async_op: bool = False)
mamba_ssm/distributed/distributed_utils.py:18
↓ 4 callersFunctionapply_rotary_emb
(tensor, cos, sin)
tests/ops/tilelang/test_mamba3_mimo.py:354
↓ 4 callersFunctionbuild_inputs_varlen
Build packed varlen inputs for NS non-uniform sequences. Sequence lengths need not be multiples of ``chunk_size``; the reference functions pa
tests/ops/tilelang/test_mamba3_mimo.py:1433
↓ 4 callersFunctionbwd_ddt_from_ddA_cs_ref
Dense reference for the reverse-cumsum contribution to ddt. Propagates ddA_cs through exp(dA_cs) and a reverse cumsum within each chunk. Res
mamba_ssm/ops/triton/mamba3/mamba3_mimo_utils.py:781
↓ 4 callersFunctionbwd_ddt_from_ddA_cs_rev_ref
Dense reference for the forward-exclusive-cumsum contribution to ddt. Propagates ddA_cs_rev through exp(dA_cs_rev) and a forward exclusive cu
mamba_ssm/ops/triton/mamba3/mamba3_mimo_utils.py:760
↓ 4 callersFunctionbwd_segsum_ddt_from_dSSdA_ref
Dense reference for the segsum contribution to ddt. Computes the row-sum of exp(dA_cs[i]-dA_cs[j]) * dSSdA[j,i] for i > j within each chunk,
mamba_ssm/ops/triton/mamba3/mamba3_mimo_utils.py:734
↓ 4 callersFunctioncos_approx
(Fast) Cosine approximation using PTX inline assembly. Args: x: Input triton tensor (any shape) in float32 Returns: Appr
mamba_ssm/ops/triton/mamba3/utils.py:14
↓ 4 callersFunctionmamba3_siso_combined
Mamba-3 attention with Triton kernels and automatic differentiation. This is the main entry point for Mamba-3 forward and backward passes using
mamba_ssm/ops/triton/mamba3/mamba3_siso_combined.py:291
↓ 4 callersFunctionmamba3_step_fn
( # B: batch size, H: num heads, D: head dim, N: dstate, R: mimo state: Tensor, # (B, H, D, N) — upda
mamba_ssm/ops/cute/mamba3/mamba3_step_fn.py:566
↓ 4 callersFunctionrun_step_benchmark
(variant: VariantConfig, *, is_outproj_norm: bool)
tests/ops/cute/test_mamba3_mimo_step.py:263
↓ 4 callersFunctionselective_scan_fn
if return_last_state is True, returns (out, last_state) last_state has shape (batch, dim, dstate). Note that the gradient of the last state is
mamba_ssm/ops/selective_scan_interface.py:106
↓ 4 callersFunctionselective_state_update_ref
Argument: state: (batch, dim, dstate) or (batch, nheads, dim, dstate) x: (batch, dim) or (batch, nheads, dim) dt: (batch,
mamba_ssm/ops/triton/selective_state_update.py:224
↓ 4 callersFunctionsin_approx
(Fast) Sine approximation using PTX inline assembly. Args: x: Input triton tensor (any shape) in float32 Returns: Approx
mamba_ssm/ops/triton/mamba3/utils.py:34
↓ 4 callersFunctionsoftplus
(dt)
mamba_ssm/ops/triton/softplus.py:10
↓ 3 callersFunction_assert_close
( actual: Tensor, expected: Tensor, *, label: str, cfg: str, step: Optional[int] = Non
tests/ops/cute/test_mamba3_mimo_step.py:127
↓ 3 callersFunction_bmm_chunk_fwd
Argument: a: (batch, seqlen, k) or (batch, seqlen, ngroups, k) b: (batch, seqlen, k) or (batch, seqlen, ngroups, k) seq_i
mamba_ssm/ops/triton/ssd_bmm.py:165
↓ 3 callersFunction_chk
(name, a, b, atol=1e-1, rtol=1e-1)
mamba_ssm/ops/triton/mamba3/mamba3_mimo_rotary_step.py:364
↓ 3 callersFunction_chunk_cumsum_bwd
(ddA, ddt_out, dt, A, dt_bias=None, dt_softplus=False, dt_limit=(0.0, float("inf")), ddt=None)
mamba_ssm/ops/triton/ssd_chunk_state.py:744
↓ 3 callersFunction_chunk_scan_bwd_dx
(cb, x, dt, dA_cumsum, dout, D=None)
mamba_ssm/ops/triton/ssd_chunk_scan.py:1565
↓ 3 callersFunction_chunk_scan_chunk_state_bwd_dx
(x, dt, dA_cumsum, B, CB, dout, dstates, D=None, seq_idx=None, dx=None)
mamba_ssm/ops/triton/ssd_combined.py:262
↓ 3 callersFunction_chunk_state_bwd_dx
(B, x, dt, dA_cumsum, dstates, dx=None)
mamba_ssm/ops/triton/ssd_chunk_state.py:845
↓ 3 callersFunction_layer_norm_fwd
( x, weight, bias, eps, residual=None, x1=None, weight1=None, bias1=None,
mamba_ssm/ops/triton/layer_norm.py:291
↓ 3 callersFunction_mamba_chunk_scan_combined_bwd
(dout, x, dt, A, B, C, out, chunk_size, D=None, z=None, dt_bias=None, initi
mamba_ssm/ops/triton/ssd_combined.py:396
↓ 3 callersFunction_mamba_chunk_scan_combined_fwd
(x, dt, A, B, C, chunk_size, D=None, z=None, dt_bias=None, initial_states=None, seq_idx=None, cu_seqlens=None,
mamba_ssm/ops/triton/ssd_combined.py:343
↓ 3 callersFunction_run_case_outputs
( *, case: str, deterministic: bool, seed: int, headdim: int = 64, dstate: int = 64,
tests/test_determinism.py:85
↓ 3 callersFunction_set_deterministic
(enabled: bool)
tests/test_determinism.py:9
↓ 3 callersFunction_swiglu_fwd
(xy, out=None)
mamba_ssm/ops/triton/k_activations.py:46
↓ 3 callersFunctionbwd_dadt_fused_triton
Fuse the three ddt backward contributions for fixed-length sequences. Runs bwd_dadt_cumsum_fused_kernel (cumsum terms) and bwd_segsum_dadt_kernel
mamba_ssm/ops/triton/mamba3/mamba3_mimo_utils.py:520
↓ 3 callersFunctionbwd_dtrap_ddt_triton
Back-propagate through the trapezoidal gamma parameterisation (dense). Returns ------- ddt : tensor same shape and dtype as dt dtra
mamba_ssm/ops/triton/mamba3/mamba3_mimo_utils.py:575
↓ 3 callersFunctioncompute_dacs_segsum_triton_varlen
Compute da_cs, da_cs_rev, and segsum for variable-length packed sequences. Falls back to compute_dacs_segsum_triton (dense) when cu_seqlens is No
mamba_ssm/ops/triton/mamba3/mamba3_mimo_utils.py:610
↓ 3 callersFunctioncompute_dtrap_ddt_ref
Dense reference for bwd_dtrap_ddt_triton. Returns (ddt, dtrap_presigmoid), both shaped [B, H, S].
mamba_ssm/ops/triton/mamba3/mamba3_mimo_utils.py:800
↓ 3 callersFunctioncreate_mamba3_siso_inputs
( batch: int, seqlen: int, nheads: int, nheads_qk: int, headdim_qk: int, headdim_v: in
tests/ops/triton/test_mamba3_siso.py:406
↓ 3 callersFunctionget_gmem_tiled_copy
(dtype: Type[cutlass.Numeric], major_mode_size: int, num_threads: int, is_async: bool = True)
mamba_ssm/ops/cute/mamba3/mamba3_step_fn.py:33
↓ 3 callersFunctionlayer_norm_fn
( x, weight, bias, residual=None, x1=None, weight1=None, bias1=None, eps=1e-6,
mamba_ssm/ops/triton/layer_norm.py:888
↓ 3 callersFunctionmamba3_siso_fwd_ref
Reference implementation of Mamba-3 forward pass. Args: Initial_States: Optional tuple of (Angle_State, SSM_State, K_State, V_State)
tests/ops/triton/test_mamba3_siso.py:148
↓ 3 callersFunctionmamba_chunk_scan_combined
Argument: x: (batch, seqlen, nheads, headdim) dt: (batch, seqlen, nheads) A: (nheads) B: (batch, seqlen, ngroups,
mamba_ssm/ops/triton/ssd_combined.py:624
↓ 3 callersFunctionrun_case
(B, R, N, D, RD, with_bias, conjugate, expanded_heads, rotate_pairwise)
mamba_ssm/ops/triton/mamba3/mamba3_mimo_rotary_step.py:335
↓ 3 callersFunctiontanh_approx
(Fast) hyperbolic tangent approximation using PTX inline assembly. Args: x: Input triton tensor (any shape) in float32 Returns:
mamba_ssm/ops/triton/mamba3/utils.py:53
↓ 2 callersFunction_build_varlen_chunk_mapping
Build the three varlen indexing primitives consumed by every varlen kernel. Because sequences have different lengths, chunks do not cross sequenc
mamba_ssm/ops/triton/mamba3/mamba3_mimo_utils.py:1255
↓ 2 callersFunction_case_config
(variant: VariantConfig, *, is_outproj_norm: bool)
tests/ops/cute/test_mamba3_mimo_step.py:105
↓ 2 callersFunction_chunk_scan_bwd_dcb
(x, dt, dA_cumsum, dout, seq_idx=None, CB=None, ngroups=1)
mamba_ssm/ops/triton/ssd_chunk_scan.py:1514
↓ 2 callersFunction_chunk_scan_bwd_dstates
(C, dA_cumsum, dout, seq_idx=None, dtype=None)
mamba_ssm/ops/triton/ssd_chunk_scan.py:1423
↓ 2 callersFunction_chunk_scan_bwd_dz
(x, z, out, dout, chunk_size, has_ddAcs=True, D=None, dz=None, recompute_output=False)
mamba_ssm/ops/triton/ssd_chunk_scan.py:1363
↓ 2 callersFunction_chunk_scan_fwd
(cb, x, dt, dA_cumsum, C, states, D=None, z=None, seq_idx=None)
mamba_ssm/ops/triton/ssd_chunk_scan.py:1259
↓ 2 callersFunction_chunk_state_bwd_ddAcs_stable
(B, x, dt, dA_cumsum, dstates, seq_idx=None)
mamba_ssm/ops/triton/ssd_chunk_state.py:972
↓ 2 callersFunction_cu_seqlens
(lengths, device)
tests/modules/test_mamba3_varlen.py:49
↓ 2 callersFunction_kernel_close_to_default
(case: str, headdim: int, dstate: int, d_has_hdim: bool = False)
tests/test_determinism.py:173
↓ 2 callersFunction_kernel_is_reproducible
(case: str, headdim: int, dstate: int, d_has_hdim: bool = False)
tests/test_determinism.py:161
↓ 2 callersFunction_layer_norm_bwd
(dy, x, weight, bias, eps, mean, rstd, z=None, group_size=None, norm_before_gate=True, is_
mamba_ssm/ops/triton/layernorm_gated.py:271
↓ 2 callersFunction_layer_norm_bwd
( dy, x, weight, bias, eps, mean, rstd, dresidual=None, dy1=None, weig
mamba_ssm/ops/triton/layer_norm.py:589
↓ 2 callersFunction_layer_norm_fwd
(x, weight, bias, eps, z=None, out=None, group_size=None, norm_before_gate=True, is_rms_norm=False)
mamba_ssm/ops/triton/layernorm_gated.py:108
↓ 2 callersFunction_make_model
(is_mimo: bool, **kwargs)
tests/modules/test_mamba3_varlen.py:25
↓ 2 callersFunction_mamba3_cls
()
tests/ops/cute/test_mamba3_mimo_step.py:62
↓ 2 callersFunction_require_cuda
()
tests/modules/test_mamba3_varlen.py:16
↓ 2 callersFunction_require_cuda_and_kernel_deps
()
tests/ops/cute/test_mamba3_mimo_step.py:55
↓ 2 callersFunction_require_cuda_and_kernel_deps
()
tests/ops/tilelang/test_mamba3_mimo.py:63
↓ 2 callersFunction_rotary
(tensor, cos, sin)
tests/ops/triton/test_mamba3_siso.py:215
↓ 2 callersFunction_run_one
(fn, *, deterministic: bool, warmup: int, rep: int)
tests/benchmark_determinism_kernels.py:84
↓ 2 callersFunction_state_passing_bwd
states contains the initial_states at index 0. The final states are not included in states.
mamba_ssm/ops/triton/ssd_state_passing.py:227
↓ 2 callersFunction_swiglu_bwd
(xy, dout, dxy=None, recompute_output=False, out=None)
mamba_ssm/ops/triton/k_activations.py:119
↓ 2 callersMethodallocate_inference_cache
(self, batch_size, max_seqlen, device=None, dtype=None, inplace_state=None, **kwargs)
mamba_ssm/modules/mamba3.py:414
↓ 2 callersFunctionangle_dt_bwd
Backward pass for angle * dt cumsum. Args: grad_out: Gradient of output (batch, seqlen, nheads, dim) angle: Angle tensor
mamba_ssm/ops/triton/mamba3/angle_dt.py:345
↓ 2 callersFunctionangle_dt_fwd
Forward pass for angle * dt cumsum. Args: angle: Angle tensor (batch, seqlen, nheads, dim) dt: Time delta tensor
mamba_ssm/ops/triton/mamba3/angle_dt.py:125
↓ 2 callersFunctionapply_rotary_emb
(tensor, cos, sin)
tests/ops/triton/test_mamba3_siso.py:69
↓ 2 callersFunctionapply_rotary_emb_rotate_half
(tensor: Tensor, cos: Tensor, sin: Tensor)
tests/ops/tilelang/test_mamba3_mimo.py:590
↓ 2 callersFunctionapply_rotary_qk_inference_fwd
Apply rotary embedding to both q and k tensors using the same angle. Also computes output angle state for next step. Arguments:
mamba_ssm/ops/triton/mamba3/mamba3_mimo_rotary_step.py:151
↓ 2 callersFunctionbwd_dadt_fused_triton_varlen
Varlen version of bwd_dadt_fused_triton. dSSdA and SSdA use the global chunk layout defined by _build_varlen_chunk_mapping; nchunks_global =
mamba_ssm/ops/triton/mamba3/mamba3_mimo_utils.py:1311
↓ 2 callersFunctionbwd_dtrap_ddt_triton_varlen
Varlen version of bwd_dtrap_ddt_triton. Cross-sequence boundary shifts are suppressed: the first chunk of each sequence receives zero incomin
mamba_ssm/ops/triton/mamba3/mamba3_mimo_utils.py:1367
↓ 2 callersFunctionchunk_state
Argument: B: (batch, seqlen, ngroups, dstate) x: (batch, seqlen, nheads, headdim) dt: (batch, nheads, nchunks, chunk_size
mamba_ssm/ops/triton/ssd_chunk_state.py:1081
↓ 2 callersFunctionchunk_state_varlen
(B, x, dt, dA_cumsum, cu_seqlens, chunk_states)
mamba_ssm/ops/triton/ssd_chunk_state.py:1018
next →1–100 of 436, ranked by callers