lmflow.models.hf_decoder_model ============================== .. py:module:: lmflow.models.hf_decoder_model .. autoapi-nested-parse:: This is a class called HFDecoderModel which is a wrapper around transformers model and tokenizer classes. It has several methods such as __init__, tokenize, and train that are used for training and fine-tuning the model. The __init__ method takes in several arguments such as model_args which are used to load the pretrained model and tokenizer, and initialize the training settings. The tokenize method is used to tokenize the input text and return the input IDs and attention masks that can be fed to the model for training or inference. Overall, this class provides a convenient interface for loading and fine-tuning transformer models and can be used for various NLP tasks such as language modeling, text classification, and question answering. Attributes ---------- .. autoapisummary:: lmflow.models.hf_decoder_model.logger Classes ------- .. autoapisummary:: lmflow.models.hf_decoder_model.HFDecoderModel Module Contents --------------- .. py:data:: logger .. py:class:: HFDecoderModel(model_args, do_train=True, device='gpu', **kwargs) Bases: :py:obj:`lmflow.models.decoder_model.DecoderModel`, :py:obj:`lmflow.models.hf_model_mixin.HFModelMixin`, :py:obj:`lmflow.models.interfaces.tunable.Tunable` Initializes a HFDecoderModel instance. :param model_args: Model arguments such as model name, path, revision, etc. :param do_train: Determines whether to prepare the model for training, including distribtued env, model placement, quantization, lora, etc. :type do_train: bool, default True :param args: Positional arguments. :type args: Optional. :param kwargs: Keyword arguments. :type kwargs: Optional. .. py:method:: tokenize(dataset: lmflow.datasets.dataset.Dataset, add_special_tokens=True, *args, **kwargs) -> lmflow.datasets.dataset.Dataset Tokenize the full dataset. :param dataset: :type dataset: lmflow.datasets.Dataset. :param args: Positional arguments. :type args: Optional. :param kwargs: Keyword arguments. :type kwargs: Optional. :returns: The tokenized dataset, without any leading or trailing special tokens (normally they are Begin-Of-Sentence or End-Of-Sentence tokens). :rtype: tokenized_datasets .. py:method:: encode(input: Union[str, list[str]], *args, **kwargs) -> Union[list[int], list[list[int]]] Encode one string or a batch of strings with the model tokenizer. :param input: Text input to encode. :type input: str or list[str] :param \*args: Positional tokenizer arguments. :param \*\*kwargs: Keyword tokenizer arguments. :returns: A batch encoding for list input, or token ids for string input. :rtype: transformers.BatchEncoding or list[int] .. rubric:: Examples A single string returns token IDs:: "Hello, world!" -> [101, 7592, 1010, 2088, 102] A list of strings returns a batch encoding with fields such as:: { "input_ids": tensor([[101, 7592, 1010, 2088, 102], ...]), "attention_mask": tensor([[1, 1, 1, 1, 1], [0, 0, 1, 1, 1]]) } .. py:method:: decode(input, **kwargs) -> Union[str, list[str]] Perform decoding process of the tokenizer. :param inputs: The token sequence. :type inputs: list or tensor. :param args: Positional arguments. :type args: Optional. :param kwargs: Keyword arguments. :type kwargs: Optional. :returns: The text decoded from the token inputs. if batch input,return the list of text [[101, 7592, 1010, 2088, 102],[101, 7592, 1010, 2088, 102]]-> ["Hello,world!","Hello,world!" if single input,return the text [101, 7592, 1010, 2088, 102]-> "Hello,world!" :rtype: outputs .. py:method:: inference(inputs: Union[str, list[str], torch.Tensor, lmflow.utils.protocol.DataProto], sampling_params: Optional[Union[dict, vllm.SamplingParams]] = None, return_logprob: bool = False, release_gpu: bool = False, inference_engine: Literal['huggingface', 'vllm', 'sglang'] = 'huggingface', gpu_memory_utilization: Optional[float] = None, tensor_parallel_size: Optional[int] = None, data_parallel_size: int = 1, max_model_len: Optional[int] = None, enable_deterministic_inference: bool = False, attention_backend: Optional[str] = None, **kwargs) -> Union[list, lmflow.utils.protocol.DataProto] Perform generation process of the model. :param inputs: The sequence used as a prompt for the generation or as model inputs to the model. When the inference engine is "vllm" or "sglang", this should be a DataProto. When the inference engine is "huggingface", this should be a tensor. :type inputs: Union[str, list[str], torch.Tensor, DataProto] :param sampling_params: The sampling parameters to use, by default None. :type sampling_params: Optional[Union[dict, "SamplingParams"]], optional :param return_logprob: Whether to return log probability during inference, by default False. :type return_logprob: bool, optional :param release_gpu: Whether to release the GPU resource after inference, by default False. :type release_gpu: bool, optional :param inference_engine: The inference engine to use, by default "huggingface". :type inference_engine: Literal["huggingface", "vllm", "sglang"], optional :param gpu_memory_utilization: The GPU memory utilization to use, by default None. :type gpu_memory_utilization: float, optional :param tensor_parallel_size: The tensor parallel size to use, by default None. :type tensor_parallel_size: int, optional :param data_parallel_size: The data parallel size for vllm inference, by default 1. :type data_parallel_size: int, optional :param max_model_len: Maximum model context length for vllm inference, by default None. :type max_model_len: int, optional :param enable_deterministic_inference: Whether to enable deterministic inference, by default False. :type enable_deterministic_inference: bool, optional :param attention_backend: The attention backend to use, by default None. :type attention_backend: Optional[str], optional :returns: The generated sequence output :rtype: outputs .. py:method:: prepare_inputs_for_inference(dataset: lmflow.datasets.dataset.Dataset, apply_chat_template: bool = True, inference_engine: Literal['huggingface', 'vllm', 'sglang'] = 'huggingface', sampling_params: Optional[dict] = None) -> Union[list[str], lmflow.utils.protocol.DataProto] .. py:method:: merge_lora_weights() .. py:method:: get_peft_without_qlora() .. py:method:: save(dir, save_full_model=False, *args, **kwargs) Perform generation process of the model. :param dir: The directory to save model and tokenizer :param save_full_model: Whether to save full model. :type save_full_model: Optional. :param kwargs: Keyword arguments. :type kwargs: Optional. :returns: The generated sequence output :rtype: outputs