Skip to main content
Ctrl+K
We've released our memory-efficient finetuning algorithm LISA, check out [Paper][User Guide] for more details!

LMFlow

  • Blogs
  • Examples
  • API Reference
  • About
  • LMFlow
  • Blogs
  • Examples
  • API Reference
  • About
  • LMFlow

Section Navigation

  • envs
  • lmflow
    • lmflow.args
    • lmflow.datasets
      • lmflow.datasets.dataset
      • lmflow.datasets.multi_modal_dataset
    • lmflow.models
      • lmflow.models.auto_model
      • lmflow.models.base_model
      • lmflow.models.decoder_model
      • lmflow.models.encoder_decoder_model
      • lmflow.models.hf_decoder_model
      • lmflow.models.hf_encoder_decoder_model
      • lmflow.models.hf_model_mixin
      • lmflow.models.hf_text_regression_model
      • lmflow.models.interfaces
        • lmflow.models.interfaces.tunable
      • lmflow.models.regression_model
      • lmflow.models.text_regression_model
      • lmflow.models.vision2seq_model
      • lmflow.models.vision_encoder
        • lmflow.models.vision_encoder.clip_encoder
    • lmflow.optim
      • lmflow.optim.adabelief
      • lmflow.optim.adabound
      • lmflow.optim.adadelta
      • lmflow.optim.adagrad
      • lmflow.optim.adam
      • lmflow.optim.adamax
      • lmflow.optim.adamp
      • lmflow.optim.adamw_schedule_free
      • lmflow.optim.adan
      • lmflow.optim.dummy
      • lmflow.optim.lamb
      • lmflow.optim.lars
      • lmflow.optim.nadam
      • lmflow.optim.novograd
      • lmflow.optim.optimizers
      • lmflow.optim.radam
      • lmflow.optim.sgd_schedule_free
      • lmflow.optim.sgdp
      • lmflow.optim.sophia
      • lmflow.optim.yogi
    • lmflow.pipeline
      • lmflow.pipeline.auto_pipeline
      • lmflow.pipeline.base_aligner
      • lmflow.pipeline.base_pipeline
      • lmflow.pipeline.base_tuner
      • lmflow.pipeline.dpo_aligner
      • lmflow.pipeline.dpov2_aligner
      • lmflow.pipeline.evaluator
      • lmflow.pipeline.finetuner
      • lmflow.pipeline.inferencer
      • lmflow.pipeline.iterative_dpo_aligner
      • lmflow.pipeline.raft_aligner
      • lmflow.pipeline.rm_inferencer
      • lmflow.pipeline.rm_tuner
      • lmflow.pipeline.utils
        • lmflow.pipeline.utils.dpov2_dataprocessor
        • lmflow.pipeline.utils.dpov2_trainer
        • lmflow.pipeline.utils.lisa_trainer_cache
        • lmflow.pipeline.utils.lisa_trainer_del
        • lmflow.pipeline.utils.lisa_trainer_fsdp
        • lmflow.pipeline.utils.memory_safe_dpov2_align
        • lmflow.pipeline.utils.memory_safe_vllm_inference
        • lmflow.pipeline.utils.peft_trainer
        • lmflow.pipeline.utils.raft_trainer
        • lmflow.pipeline.utils.rm_dataprocessor
        • lmflow.pipeline.utils.rm_trainer
      • lmflow.pipeline.vllm_inferencer
    • lmflow.tokenization
      • lmflow.tokenization.hf_decoder_model
      • lmflow.tokenization.hf_text_regression_model
    • lmflow.utils
      • lmflow.utils.common
      • lmflow.utils.constants
      • lmflow.utils.conversation_template
        • lmflow.utils.conversation_template.base
        • lmflow.utils.conversation_template.chatglm
        • lmflow.utils.conversation_template.chatml
        • lmflow.utils.conversation_template.deepseek
        • lmflow.utils.conversation_template.gemma
        • lmflow.utils.conversation_template.internlm
        • lmflow.utils.conversation_template.llama
        • lmflow.utils.conversation_template.phi
        • lmflow.utils.conversation_template.qwen
        • lmflow.utils.conversation_template.yi
        • lmflow.utils.conversation_template.zephyr
      • lmflow.utils.data_utils
      • lmflow.utils.flash_attention
        • lmflow.utils.flash_attention.bloom_flash_attention
        • lmflow.utils.flash_attention.gpt2_flash_attention
        • lmflow.utils.flash_attention.gpt_neo_flash_attention
        • lmflow.utils.flash_attention.llama_flash_attention
        • lmflow.utils.flash_attention.triton_flash_attention
      • lmflow.utils.llava_conversation_lib
      • lmflow.utils.model
      • lmflow.utils.multimodal
      • lmflow.utils.position_interpolation
        • lmflow.utils.position_interpolation.llama_rope_scaled_monkey_patch
      • lmflow.utils.versioning
    • lmflow.version
  • common
  • tensor
  • constants
  • model_params
  • API Reference
  • lmflow
  • lmflow.utils
  • lmflow.utils.flash_attention

lmflow.utils.flash_attention#

Submodules#

  • lmflow.utils.flash_attention.bloom_flash_attention
  • lmflow.utils.flash_attention.gpt2_flash_attention
  • lmflow.utils.flash_attention.gpt_neo_flash_attention
  • lmflow.utils.flash_attention.llama_flash_attention
  • lmflow.utils.flash_attention.triton_flash_attention

previous

lmflow.utils.data_utils

next

lmflow.utils.flash_attention.bloom_flash_attention
On this page
  • Submodules
Show Source

© Copyright LMFlow 2024.

Created using Sphinx 8.1.3.

Built with the PyData Sphinx Theme 0.16.0.