MCPcopy Create free account

hub / github.com/state-spaces/mamba / functions

Functions436 in github.com/state-spaces/mamba

↓ 2 callersFunctioncompute_dacs_segsum_ref
Dense reference for compute_dacs_segsum_triton. Requires S to be a multiple of chunk_size. Returns (da_cs, da_cs_rev, segsum).
mamba_ssm/ops/triton/mamba3/mamba3_mimo_utils.py:827
↓ 2 callersFunctioncustom_amp_decorator
(dec: Callable, cuda_amp_deprecated: bool)
mamba_ssm/utils/torch.py:5
↓ 2 callersMethodforward
We're doing Tensor Parallel with sequence parallelism: we do the matmul and then a reduce_scatter of the result.
mamba_ssm/distributed/tensor_parallel.py:184
↓ 2 callersFunctionget_package_version
()
setup.py:268
↓ 2 callersFunctiongrads_to_dA
(grad_dA_cs: Tensor, grad_dA_cs_rev: Tensor, chunk_size: int)
tests/ops/tilelang/test_mamba3_mimo.py:301
↓ 2 callersFunctionlayernorm_fn
(x, weight, bias, z=None, eps=1e-6, group_size=None, norm_before_gate=True, is_rms_norm=False)
mamba_ssm/ops/triton/layernorm_gated.py:380
↓ 2 callersFunctionmamba3_siso_step_ref
Reference implementation of Mamba-3 in recurrent (step) mode. Args: Input_States: Optional tuple of (Angle_State, SSM_State, K_State,
tests/ops/triton/test_mamba3_siso.py:33
↓ 2 callersFunctionmamba_inner_fn
( xz, conv1d_weight, conv1d_bias, x_proj_weight, delta_proj_weight, out_proj_weight, out_proj_bias,
mamba_ssm/ops/selective_scan_interface.py:373
↓ 2 callersFunctionmamba_mimo_bwd_combined
( dout, q, k, v, q_bias, k_bias, mimo_v, m
mamba_ssm/ops/tilelang/mamba3/mamba3_mimo_bwd.py:1158
↓ 2 callersFunctionmamba_split_conv1d_scan_combined
Argument: zxbcdt: (batch, seqlen, 2 * dim + 2 * ngroups * dstate + nheads) where dim == nheads * headdim conv1d_weight: (dim + 2
mamba_ssm/ops/triton/ssd_combined.py:977
↓ 2 callersFunctionmodify_logits_for_top_p_filtering
Set the logits for none top-p values to -inf. Done in-place.
mamba_ssm/utils/generation.py:53
↓ 2 callersFunctionparallel_linear_func
( x: Tensor, weight: Tensor, bias: Optional[Tensor] = None, process_group: Optional[ProcessGro
mamba_ssm/distributed/tensor_parallel.py:101
↓ 2 callersFunctionreduce_scatter_raw
(input_: Tensor, process_group: ProcessGroup, async_op: bool = False)
mamba_ssm/distributed/distributed_utils.py:30
↓ 2 callersFunctionrms_norm_ref
(x, weight, bias, z=None, eps=1e-6, group_size=None, norm_before_gate=True, upcast=True)
mamba_ssm/ops/triton/layernorm_gated.py:18
↓ 2 callersFunctionrmsnorm_fn
(x, weight, bias, z=None, eps=1e-6, group_size=None, norm_before_gate=True)
mamba_ssm/ops/triton/layernorm_gated.py:384
↓ 2 callersFunctionrun_ref_backward_fp32
( mods: SimpleNamespace, inputs: dict, *, contract_mimo_out: bool = True, grad_output: Opt
tests/ops/tilelang/test_mamba3_mimo.py:692
↓ 2 callersFunctionrun_ref_backward_fp32_varlen
Per-sequence fp32 autograd reference for the varlen backward pass. Runs ``mamba3_MIMO_chunk_ref`` on each packed sequence, collects autograd
tests/ops/tilelang/test_mamba3_mimo.py:1512
↓ 2 callersFunctionsample_tokens
(logits, inference_params)
mamba_ssm/utils/generation.py:197
↓ 2 callersFunctionsegsum
More stable segment sum calculation.
mamba_ssm/modules/ssd_minimal.py:23
↓ 2 callersFunctionselective_state_update_fused_ref_v2
Reference to match the new fused kernel API. Shapes: state: (B, N, H, S) A: (B, N) B: (B, R, N, S)
mamba_ssm/ops/cute/mamba3/mamba3_step_fn.py:741
↓ 2 callersFunctionset_deterministic_mode
(value)
mamba_ssm/utils/determinism.py:30
↓ 2 callersFunctionsilu
SiLU (Swish) activation function: x * sigmoid(x). Formula: silu(x) = 0.5*x * (1 + tanh(0.5*x)) + 0.5*x. Leverages fast tanh_approx for s
mamba_ssm/ops/triton/mamba3/utils.py:117
↓ 2 callersFunctionssd_selective_scan
Argument: x: (batch, seqlen, nheads, headdim) dt: (batch, seqlen, nheads) or (batch, seqlen, nheads, headdim) A: (nheads)
mamba_ssm/ops/triton/ssd_combined.py:723
↓ 1 callersMethod__init__
If group_size is not None, we do GroupNorm with each group having group_size elements. group_size=None is equivalent to group_size=hidden_size
mamba_ssm/ops/triton/layernorm_gated.py:417
↓ 1 callersMethod__init__
( self, d_model: int, n_layer: int, d_intermediate: int, vocab_size: i
mamba_ssm/models/mixer_seq_simple.py:119
↓ 1 callersMethod_apply_rotary_update_kvcache_attention
Fast path that combine 3 steps: apply rotary to Q and K, update kv cache, and apply attention. q: (batch_size, seqlen_q, nheads, head
mamba_ssm/modules/mha.py:129
↓ 1 callersFunction_chunk_scan_bwd_ddAcs_stable
(x, dt, dA_cumsum, dout, cb)
mamba_ssm/ops/triton/ssd_chunk_scan.py:1700
↓ 1 callersFunction_chunk_scan_bwd_ddAcs_unstable
Not numerically stable and should not be used. Leaving here for reference.
mamba_ssm/ops/triton/ssd_chunk_scan.py:1618
↓ 1 callersFunction_diff_stats
(actual: Tensor, expected: Tensor)
tests/ops/cute/test_mamba3_mimo_step.py:122
↓ 1 callersFunction_filter_configs_by_block_sizes
Filter configs by TRITON_AUTOTUNE_BLOCK_SIZE_* env vars.
mamba_ssm/utils/determinism.py:44
↓ 1 callersMethod_get_states_from_cache
(self, inference_params, batch_size, initialize_states=False)
mamba_ssm/modules/mamba_simple.py:268
↓ 1 callersMethod_get_states_from_cache
(self, inference_params, batch_size, initialize_states=False)
mamba_ssm/modules/mamba3.py:456
↓ 1 callersMethod_get_states_from_cache
(self, inference_params, batch_size, initialize_states=False)
mamba_ssm/modules/mamba2.py:357
↓ 1 callersFunction_make_inputs
( *, seed: int, headdim: int, dstate: int, chunk_size: int = 256, ngroups: int = 1,
tests/test_determinism.py:24
↓ 1 callersFunction_pack_sequences
Pack a list of (1, Li, d) tensors into (1, sum(Li), d).
tests/modules/test_mamba3_varlen.py:44
↓ 1 callersFunction_peak_memory_mb
(fn, *, warmup: int = 3)
tests/benchmark_determinism_kernels.py:25
↓ 1 callersMethod_postprocess
(self, y, outpj, z, zpj, headdim)
mamba_ssm/modules/mamba3.py:276
↓ 1 callersMethod_preprocess
(self, A_proj, dd_dt, B, C, x, z, trap_proj, angle_proj)
mamba_ssm/modules/mamba3.py:253
↓ 1 callersFunction_reset_peak_memory
()
tests/benchmark_determinism_kernels.py:18
↓ 1 callersFunction_run_case
(variant: VariantConfig, *, is_outproj_norm: bool)
tests/ops/cute/test_mamba3_mimo_step.py:150
↓ 1 callersFunction_segsum
Segment sum helper for attention computation.
tests/ops/triton/test_mamba3_siso.py:21
↓ 1 callersMethod_setup_gmem_tiled_copy
(self, )
mamba_ssm/ops/cute/mamba3/mamba3_step_fn.py:66
↓ 1 callersMethod_setup_smem_layouts
(self)
mamba_ssm/ops/cute/mamba3/mamba3_step_fn.py:59
↓ 1 callersFunction_update_kv_cache
kv: (batch_size, seqlen, 2, nheads, head_dim) or (batch_size, 1, 2, nheads, head_dim)
mamba_ssm/modules/mha.py:26
↓ 1 callersMethod_update_kv_cache
kv: (batch_size, seqlen, 2, nheads, head_dim) or (batch_size, 1, 2, nheads, head_dim)
mamba_ssm/modules/mha.py:124
↓ 1 callersMethod_update_kvcache_attention
Write kv to inference_params, then do attention
mamba_ssm/modules/mha.py:167
↓ 1 callersFunctionall_reduce_raw
(input_: Tensor, process_group: ProcessGroup, async_op: bool = False)
mamba_ssm/distributed/distributed_utils.py:43
↓ 1 callersMethodallocate_inference_cache
(self, batch_size, max_seqlen, dtype=None)
mamba_ssm/modules/mha.py:110
↓ 1 callersMethodallocate_inference_cache
(self, batch_size, max_seqlen, dtype=None, **kwargs)
mamba_ssm/utils/generation.py:247
↓ 1 callersMethodallocate_inference_cache
(self, batch_size, max_seqlen, dtype=None, **kwargs)
mamba_ssm/models/mixer_seq_simple.py:184
↓ 1 callersFunctionappend_nvcc_threads
(nvcc_extra_args)
setup.py:125
↓ 1 callersFunctionapply_angle_dt_reference
( angle: Tensor, # (batch, seqlen, nheads, dim) dt: Tensor, # (batch, seqlen, nheads) )
tests/ops/tilelang/test_mamba3_mimo.py:454
↓ 1 callersFunctionapply_dk_state_post
( d_ok_state: torch.Tensor, angles: torch.Tensor, k: torch.Tensor, k_bias: torch.Tensor, d
mamba_ssm/ops/triton/mamba3/mamba3_siso_bwd.py:1349
↓ 1 callersFunctionapply_rotary_qk_inference_reference
Reference PyTorch implementation for QK rotary embedding with qk_sum.
mamba_ssm/ops/triton/mamba3/mamba3_mimo_rotary_step.py:239
↓ 1 callersFunctionbwd_dadt_fused_varlen_ref
Varlen reference for bwd_dadt_fused_triton_varlen. Calls the non-varlen reference functions as subroutines, one sequence at a time. Each sequ
mamba_ssm/ops/triton/mamba3/mamba3_mimo_utils.py:907
↓ 1 callersFunctioncapture_graph
( model, inference_params, batch_size, max_seqlen, decoding_seqlen=1, mempool=None, n_warmups=2 )
mamba_ssm/utils/generation.py:342
↓ 1 callersFunctioncheck_if_cuda_home_none
(global_option: str)
setup.py:113
↓ 1 callersFunctioncheck_if_hip_home_none
(global_option: str)
setup.py:102
↓ 1 callersFunctionchunk_scan
prev_states contains the initial_states at index 0, and the state for the next-to-last chunk at index -1. Argument: B: (batch, seqlen
mamba_ssm/ops/triton/ssd_chunk_scan.py:1828
↓ 1 callersFunctionchunk_scan_ref
Argument: B: (batch, seqlen, ngroups, dstate) C: (batch, seqlen, ngroups, dstate) x: (batch, seqlen, nheads, headdim)
mamba_ssm/ops/triton/ssd_chunk_scan.py:1846
↓ 1 callersFunctionchunk_state_ref
Argument: B: (batch, seqlen, ngroups, dstate) x: (batch, seqlen, nheads, headdim) dt: (batch, nheads, nchunks, chunk_size
mamba_ssm/ops/triton/ssd_chunk_state.py:1094
↓ 1 callersFunctioncompute_dacs_segsum_ref_varlen
Varlen reference for compute_dacs_segsum_triton_varlen. Calls compute_dacs_segsum_ref as a subroutine per sequence, zero-padding each sequenc
mamba_ssm/ops/triton/mamba3/mamba3_mimo_utils.py:852
↓ 1 callersFunctioncompute_ddt_dtrap_dinput_states
Compute dDT, dTrap from dScale/dGamma, and optionally input state gradients. Args: dscale: Gradient of scale, shape (batch, nhea
mamba_ssm/ops/triton/mamba3/mamba3_siso_bwd.py:1623
↓ 1 callersFunctioncompute_dqktheta
Compute gradients through rotary embeddings and biases for Mamba-3 backward pass. This kernel undoes the rotary embedding and computes g
mamba_ssm/ops/triton/mamba3/mamba3_siso_bwd.py:1170
↓ 1 callersFunctioncompute_dqkv
Compute gradients dQ_mid, dK_mid, dV, dADT, dQK_dot, dD, d_issm_state for Mamba-3 backward pass. This kernel operates on the rotated/sca
mamba_ssm/ops/triton/mamba3/mamba3_siso_bwd.py:616
↓ 1 callersFunctioncompute_dtrap_ddt_varlen_ref
Varlen reference for bwd_dtrap_ddt_triton_varlen. Calls compute_dtrap_ddt_ref as subroutine per sequence, zero-padding each sequence to a chu
mamba_ssm/ops/triton/mamba3/mamba3_mimo_utils.py:959
↓ 1 callersFunctioncompute_dzdo
Compute Z-gating gradients for Mamba-3 backward pass. When Z-gating is used in the forward pass (out = O * silu(Z)), this function c
mamba_ssm/ops/triton/mamba3/mamba3_siso_bwd.py:115
↓ 1 callersFunctioncompute_one_sequence
(seq_idx)
tests/ops/triton/test_mamba3_siso.py:227
↓ 1 callersFunctionconfig_prune
(configs)
mamba_ssm/ops/triton/layer_norm.py:130
↓ 1 callersFunctioncreate_block
( d_model, d_intermediate, ssm_cfg=None, attn_layer_idx=None, attn_cfg=None, norm_epsi
mamba_ssm/models/mixer_seq_simple.py:29
↓ 1 callersFunctiondecode
Decoding, either greedy or with top-k or top-p sampling. If top-k = 0, don't limit the number of candidates (pure sampling). Top-k and top-p c
mamba_ssm/utils/generation.py:121
↓ 1 callersFunctionget_benchmarks
(t: dict[str, torch.Tensor], *, ngroups: int)
tests/benchmark_determinism_kernels.py:57
↓ 1 callersFunctionget_cuda_bare_metal_version
(cuda_dir)
setup.py:62
↓ 1 callersFunctionget_hip_version
(rocm_dir)
setup.py:73
↓ 1 callersFunctionget_logits
(input_ids, inference_params)
mamba_ssm/utils/generation.py:173
↓ 1 callersFunctionget_platform
Returns the platform name as used in wheel filenames.
setup.py:47
↓ 1 callersFunctionget_torch_hip_version
()
setup.py:94
↓ 1 callersFunctionget_wheel_url
()
setup.py:279
↓ 1 callersMethodkernel
( self, mState: cute.Tensor, # (B, H, D, N) mBstate: cute.Tensor, # (B, R, H, N)
mamba_ssm/ops/cute/mamba3/mamba3_step_fn.py:227
↓ 1 callersFunctionload_config_hf
(model_name)
mamba_ssm/utils/hf.py:9
↓ 1 callersFunctionload_state_dict_hf
(model_name, device=None, dtype=None)
mamba_ssm/utils/hf.py:14
↓ 1 callersFunctionmain
()
tests/benchmark_determinism_kernels.py:91
↓ 1 callersFunctionmake_smoke_inputs
( *, batch: int = 1, seqlen: int = 64, mimo_rank: int = 4, nheads_qk: int = 1, nheads:
tests/ops/tilelang/test_mamba3_mimo.py:190
↓ 1 callersFunctionmake_tensors
(*, batch: int, seqlen: int, nheads: int, headdim: int, dstate: int, ngroups: int, chunk_size: int,
tests/benchmark_determinism_kernels.py:35
↓ 1 callersFunctionmamba3_MIMO_step_ref
Reference implementation of Mamba-3 MIMO in recurrent (step) mode. Args: Input_States: Optional tuple of (Angle_State, SSM_State, K_State
tests/ops/tilelang/test_mamba3_mimo.py:316
↓ 1 callersFunctionmamba3_siso_fwd
Mamba-3 forward pass wrapper. Args: Q: Query tensor (batch, seqlen, nheads_qk, headdim_qk) K: Key tensor
mamba_ssm/ops/triton/mamba3/mamba3_siso_fwd.py:440
↓ 1 callersFunctionmamba3_siso_step
Mamba-3 step wrapper. Inputs: Q: Query tensor (batch, nheads_qk, headdim_qk). K: Key tensor (b
mamba_ssm/ops/triton/mamba3/mamba3_siso_step.py:233
↓ 1 callersFunctionmamba_inner_ref
( xz, conv1d_weight, conv1d_bias, x_proj_weight, delta_proj_weight, out_proj_weight, out_proj_bias,
mamba_ssm/ops/selective_scan_interface.py:384
↓ 1 callersFunctionmamba_mimo_bwd_bwd
( B, S, H, G, N, P, R, hasZ, hasD, reduceO, chunk_size: int = 16,
mamba_ssm/ops/tilelang/mamba3/mamba3_mimo_bwd.py:505
↓ 1 callersFunctionmamba_mimo_bwd_bwd
TileLang kernel factory for the varlen Mamba3 bwd-bwd pass. The bwd-bwd pass iterates chunks in *reverse* order, consuming the STATES an
mamba_ssm/ops/tilelang/mamba3/mamba3_mimo_bwd_varlen.py:545
↓ 1 callersFunctionmamba_mimo_bwd_combined_varlen
Varlen combined backward pass (bwd-fwd + bwd-bwd). Dispatches to ``mamba_mimo_bwd_combined`` (non-varlen) when ``cu_seqlens`` is None; o
mamba_ssm/ops/tilelang/mamba3/mamba3_mimo_bwd_varlen.py:1224
↓ 1 callersFunctionmamba_mimo_bwd_fwd
( B, S, H, G, N, P, R, hasZ, hasD, reduceO, chunk_size: int = 16,
mamba_ssm/ops/tilelang/mamba3/mamba3_mimo_bwd.py:42
↓ 1 callersFunctionmamba_mimo_bwd_fwd
TileLang kernel factory for the varlen Mamba3 bwd-fwd pass. The bwd-fwd pass re-runs the forward recurrence in *forward* chunk order to
mamba_ssm/ops/tilelang/mamba3/mamba3_mimo_bwd_varlen.py:62
↓ 1 callersFunctionmamba_mimo_forward
(q, k, v, q_bias, k_bias, mimo_v, mimo_o,
mamba_ssm/ops/tilelang/mamba3/mamba3_mimo_fwd.py:417
↓ 1 callersFunctionmamba_mimo_forward_varlen
Varlen wrapper around ``mamba_mimo_fwd``. Dispatches to the varlen kernel when ``cu_seqlens`` is provided; otherwise behaves identically
mamba_ssm/ops/tilelang/mamba3/mamba3_mimo_fwd_varlen.py:529
↓ 1 callersFunctionmamba_mimo_fwd
( B, S, H, G, N, P, R, hasZ, hasD, reduceO, return_final_state=Fal
mamba_ssm/ops/tilelang/mamba3/mamba3_mimo_fwd.py:38
↓ 1 callersFunctionmamba_mimo_fwd
TileLang kernel factory for the varlen Mamba3 chunked forward pass. Varlen additions vs. ``mamba_mimo_fwd`` in ``mamba3_mimo_fwd.py``:
mamba_ssm/ops/tilelang/mamba3/mamba3_mimo_fwd_varlen.py:59
↓ 1 callersFunctionmax_rel_err
(ours: Tensor, ref: Tensor, eps: float = 1e-5)
tests/ops/tilelang/test_mamba3_mimo.py:86
↓ 1 callersFunctionmodify_logit_for_repetition_penalty
Apply repetition penalty. See https://arxiv.org/abs/1909.05858 logits: (batch_size, vocab_size) prev_output_tokens: (batch_size, seq_len)
mamba_ssm/utils/generation.py:69
↓ 1 callersFunctionmodify_logits_for_min_p_filtering
Set the logits for none min_p values to -inf. Done in-place.
mamba_ssm/utils/generation.py:37
← previousnext →101–200 of 436, ranked by callers