lmflow.models.hf_model_mixin

Attributes

logger
HF_AUTOMODEL_MAPPING
HF_AUTOMODEL_TYPE
LORA_TARGET_MODULES_MAPPING

Classes

HFModelMixin

Module Contents

lmflow.models.hf_model_mixin.logger

lmflow.models.hf_model_mixin.HF_AUTOMODEL_MAPPING

lmflow.models.hf_model_mixin.HF_AUTOMODEL_TYPE

lmflow.models.hf_model_mixin.LORA_TARGET_MODULES_MAPPING

class lmflow.models.hf_model_mixin.HFModelMixin(model_args: lmflow.args.ModelArguments, do_train: bool, device: str | None = 'gpu', hf_auto_model_additional_args: dict | None = None, *args, **kwargs)

Bases: lmflow.models.base_model.BaseModel

device = 'gpu'

model_args

hf_auto_model

do_train

tokenizer

torch_dtype

hf_model_config

quant_config = None

peft_config = None

activate_model_for_inference(inference_engine: Literal['huggingface', 'vllm', 'sglang'] = 'huggingface', gpu_memory_utilization: float | None = None, tensor_parallel_size: int | None = None, data_parallel_size: int = 1, max_model_len: int | None = None, enable_deterministic_inference: bool = False, attention_backend: str | None = None)

deactivate_model_for_inference(inference_engine: Literal['huggingface', 'vllm', 'sglang'] = 'huggingface')

Deactivate the model and release the resources.

NOTE: For vllm (>=0.8), the best-effort release below works for most single-GPU, inference-only use cases. It remains unreliable when tensor_parallel_size > 1, CUDA graphs are enabled, or the same process also holds an HF training model — in those cases use MemorySafeVLLMInferencer, which isolates inference in a subprocess. vllm still has no official in-process shutdown API (RFC vllm-project/vllm#24885); MemorySafeVLLMInferencer is kept for backward compatibility and will be migrated to vllm sleep mode in a follow-up.

get_max_length()

Return max acceptable input length in terms of tokens.

get_tokenizer()

Return the tokenizer of the model.

get_backend_model()

Return the backend model.