lmflow.models.hf_decoder_model ============================== .. py:module:: lmflow.models.hf_decoder_model .. autoapi-nested-parse:: This is a class called HFDecoderModel which is a wrapper around transformers model and tokenizer classes. It has several methods such as __init__, tokenize, and train that are used for training and fine-tuning the model. The __init__ method takes in several arguments such as model_args, tune_strategy, and ds_config, which are used to load the pretrained model and tokenizer, and initialize the training settings. The tokenize method is used to tokenize the input text and return the input IDs and attention masks that can be fed to the model for training or inference. This class supports different tune_strategy options such as 'normal', 'none', 'lora', and 'adapter', which allow for different fine-tuning settings of the model. However, the 'lora' and 'adapter' strategies are not yet implemented. Overall, this class provides a convenient interface for loading and fine-tuning transformer models and can be used for various NLP tasks such as language modeling, text classification, and question answering. .. !! processed by numpydoc !! Attributes ---------- .. autoapisummary:: lmflow.models.hf_decoder_model.logger Classes ------- .. autoapisummary:: lmflow.models.hf_decoder_model.HFDecoderModel Module Contents --------------- .. py:data:: logger .. py:class:: HFDecoderModel(model_args, tune_strategy='normal', ds_config=None, device='gpu', use_accelerator=False, *args, **kwargs) Bases: :py:obj:`lmflow.models.decoder_model.DecoderModel`, :py:obj:`lmflow.models.hf_model_mixin.HFModelMixin`, :py:obj:`lmflow.models.interfaces.tunable.Tunable` Initializes a HFDecoderModel instance. :Parameters: **model_args** Model arguments such as model name, path, revision, etc. **tune_strategy** : str or none, default="normal". A string representing the dataset backend. Defaults to "huggingface". **ds_config** Deepspeed configuations. **args** : Optional. Positional arguments. **kwargs** : Optional. Keyword arguments. .. !! processed by numpydoc !! .. py:method:: tokenize(dataset, add_special_tokens=True, *args, **kwargs) -> lmflow.datasets.dataset.Dataset Tokenize the full dataset. :Parameters: **dataset** : lmflow.datasets.Dataset. .. **args** : Optional. Positional arguments. **kwargs** : Optional. Keyword arguments. :Returns: tokenized_datasets The tokenized dataset, without any leading or trailing special tokens (normally they are Begin-Of-Sentence or End-Of-Sentence tokens). .. !! processed by numpydoc !! .. py:method:: encode(input: Union[str, List[str]], *args, **kwargs) -> Union[List[int], List[List[int]]] Perform encoding process of the tokenizer. :Parameters: **inputs** : str or list. The text sequence. **args** : Optional. Positional arguments. **kwargs** : Optional. Keyword arguments. :Returns: outputs if string input,return the tokenized inputs. "Hello,world!"-> [101, 7592, 1010, 2088, 102] if batch input,return {input_ids,attention_mask,token_type_ids} ["Hello,world!","Hello!"]-> {'input_ids': tensor([[ 101, 7592, 1010, 2088, 102],...),'attention_mask': tensor([[1, 1, 1, 1, 1],[0,0,1,1,1]])} .. !! processed by numpydoc !! .. py:method:: decode(input, *args, **kwargs) -> Union[str, List[str]] Perform decoding process of the tokenizer. :Parameters: **inputs** : list or tensor. The token sequence. **args** : Optional. Positional arguments. **kwargs** : Optional. Keyword arguments. :Returns: outputs The text decoded from the token inputs. if batch input,return the list of text [[101, 7592, 1010, 2088, 102],[101, 7592, 1010, 2088, 102]]-> ["Hello,world!","Hello,world!" if single input,return the text [101, 7592, 1010, 2088, 102]-> "Hello,world!" .. !! processed by numpydoc !! .. py:method:: inference(inputs, release_gpu: bool = False, use_vllm: bool = False, **kwargs) Perform generation process of the model. :Parameters: **inputs** The sequence used as a prompt for the generation or as model inputs to the model. When using vllm inference, this should be a string or a list of strings. When using normal inference, this should be a tensor. **release_gpu** : bool, optional Whether to release the GPU resource after inference, by default False. **use_vllm** : bool, optional Whether to use VLLM for inference, by default False. **kwargs** : Optional. Keyword arguments. :Returns: outputs The generated sequence output .. !! processed by numpydoc !! .. py:method:: __inference(inputs, *args, **kwargs) Perform generation process of the model. :Parameters: **inputs** The **tokenized** sequence used as a prompt for the generation or as model inputs to the model. **args** : Optional. Positional arguments. **kwargs** : Optional. Keyword arguments. :Returns: outputs The generated sequence output .. !! processed by numpydoc !! .. py:method:: __vllm_inference(inputs: Union[str, List[str]], sampling_params: Optional[vllm.SamplingParams] = None, **kwargs) -> List[lmflow.utils.data_utils.VLLMInferenceResultWithInput] Perform VLLM inference process of the model. :Parameters: **inputs** : Union[str, List[str]] Prompt(s), string or a list of strings. **sampling_params** : Optional[SamplingParams], optional vllm SamplingParams object, by default None. :Returns: List[VLLMInferenceResultWithInput] Return a list of VLLMInferenceResultWithInput, where each element contains the input prompt and the corresponding output. When `sampling_params.detokenize = True`, the output would be a list of strings, contains sampling_params.n samples for the corresponding prompt. When `sampling_params.detokenize = False`, return a list of list of ints (token ids, no decoding after generation). .. !! processed by numpydoc !! .. py:method:: prepare_inputs_for_inference(dataset: lmflow.datasets.dataset.Dataset, apply_chat_template: bool = True, enable_distributed_inference: bool = False, use_vllm: bool = False, **kwargs) -> Union[List[str], ray.data.Dataset, Dict[str, torch.Tensor]] Prepare inputs for inference. :Parameters: **dataset** : lmflow.datasets.Dataset. The dataset used for inference. **args** : Optional. Positional arguments. **kwargs** : Optional. Keyword arguments. :Returns: outputs The prepared inputs for inference. .. !! processed by numpydoc !! .. py:method:: __prepare_inputs_for_vllm_inference(dataset: lmflow.datasets.dataset.Dataset, apply_chat_template: bool = True, enable_distributed_inference: bool = False) -> Union[List[str], ray.data.Dataset] .. py:method:: __prepare_inputs_for_inference(dataset: lmflow.datasets.dataset.Dataset, **kwargs) :abstractmethod: .. py:method:: merge_lora_weights() .. py:method:: get_peft_without_qlora() .. py:method:: save(dir, save_full_model=False, *args, **kwargs) Perform generation process of the model. :Parameters: **dir** The directory to save model and tokenizer **save_full_model** : Optional. Whether to save full model. **kwargs** : Optional. Keyword arguments. :Returns: outputs The generated sequence output .. !! processed by numpydoc !!