lmflow.models.hf_model_mixin

Attributes

logger
HF_AUTOMODEL_MAPPING
HF_AUTOMODEL_TYPE
LORA_TARGET_MODULES_MAPPING

Classes

HFModelMixin

Module Contents

lmflow.models.hf_model_mixin.logger

lmflow.models.hf_model_mixin.HF_AUTOMODEL_MAPPING

lmflow.models.hf_model_mixin.HF_AUTOMODEL_TYPE

lmflow.models.hf_model_mixin.LORA_TARGET_MODULES_MAPPING

class lmflow.models.hf_model_mixin.HFModelMixin(model_args: lmflow.args.ModelArguments, do_train: bool, device: str | None = 'gpu', hf_auto_model_additional_args: dict | None = None, *args, **kwargs)

Bases: lmflow.models.base_model.BaseModel

device = 'gpu'

model_args

hf_auto_model

do_train

tokenizer

torch_dtype

hf_model_config

quant_config = None

peft_config = None

_activated = False

__prepare_tokenizer(model_args: lmflow.args.ModelArguments) → transformers.PreTrainedTokenizer | transformers.PreTrainedTokenizerFast

__prepare_dtype(model_args: lmflow.args.ModelArguments) → torch.dtype

__prepare_model_config(model_args: lmflow.args.ModelArguments, hf_auto_model_additional_args: dict | None = None)

Prepare model configuration for hf auto register, Parameters ———- model_args : ModelArguments

LMFlow model arguments.

hf_auto_model_additional_argsOptional[dict], optional

Special configurations such as num_labels in AutoModelForSequenceClassification (commonly used in reward modeling) will not preset in __prepare_model_config, so it should be passed in hf_auto_model_additional_args.

Returns

configModelConfig

hf model config.

__prepare_quant_config(model_args: lmflow.args.ModelArguments)

__prepare_peft_config(model_args: lmflow.args.ModelArguments)

__model_module_inject(model_args: lmflow.args.ModelArguments) → None

Override some model modules with custom implementations.

Current implementations: - Position interpolation (model_args.do_rope_scaling):

replace llama embeddings with condense embeddings.

__prepare_model_for_training(model_args: lmflow.args.ModelArguments, hf_auto_model: HF_AUTOMODEL_TYPE)

__prepare_model_for_inference(model_args: lmflow.args.ModelArguments, hf_auto_model: HF_AUTOMODEL_TYPE)

__prepare_model_for_vllm_inference(model_args: lmflow.args.ModelArguments, gpu_memory_utilization: float, tensor_parallel_size: int, data_parallel_size: int = 1, max_model_len: int | None = None)

__prepare_model_for_sglang_inference(model_args: lmflow.args.ModelArguments, gpu_memory_utilization: float | None = None, tensor_parallel_size: int | None = None, enable_deterministic_inference: bool = False, attention_backend: str | None = None)

__fix_special_tokens()

activate_model_for_inference(inference_engine: Literal['huggingface', 'vllm', 'sglang'] = 'huggingface', gpu_memory_utilization: float | None = None, tensor_parallel_size: int | None = None, data_parallel_size: int = 1, max_model_len: int | None = None, enable_deterministic_inference: bool = False, attention_backend: str | None = None)

deactivate_model_for_inference(inference_engine: Literal['huggingface', 'vllm', 'sglang'] = 'huggingface')

Deactivate the model and release the resources.

NOTE: For vllm (>=0.8), the best-effort release below works for most single-GPU, inference-only use cases. It remains unreliable when tensor_parallel_size > 1, CUDA graphs are enabled, or the same process also holds an HF training model — in those cases use MemorySafeVLLMInferencer, which isolates inference in a subprocess. vllm still has no official in-process shutdown API (RFC vllm-project/vllm#24885); MemorySafeVLLMInferencer is kept for backward compatibility and will be migrated to vllm sleep mode in a follow-up.

get_max_length()

Return max acceptable input length in terms of tokens.

get_tokenizer()

Return the tokenizer of the model.

get_backend_model()

Return the backend model.