lmflow.models.hf_decoder_model#

This is a class called HFDecoderModel which is a wrapper around transformers model and tokenizer classes. It has several methods such as __init__, tokenize, and train that are used for training and fine-tuning the model. The __init__ method takes in several arguments such as model_args which are used to load the pretrained model and tokenizer, and initialize the training settings.

The tokenize method is used to tokenize the input text and return the input IDs and attention masks that can be fed to the model for training or inference.

Overall, this class provides a convenient interface for loading and fine-tuning transformer models and can be used for various NLP tasks such as language modeling, text classification, and question answering.

Attributes#

logger

Classes#

HFDecoderModel

Initializes a HFDecoderModel instance.

Module Contents#

lmflow.models.hf_decoder_model.logger[source]#
class lmflow.models.hf_decoder_model.HFDecoderModel(model_args, do_train=True, device='gpu', **kwargs)[source]#

Bases: lmflow.models.decoder_model.DecoderModel, lmflow.models.hf_model_mixin.HFModelMixin, lmflow.models.interfaces.tunable.Tunable

Initializes a HFDecoderModel instance.

Parameters:

  • model_args – Model arguments such as model name, path, revision, etc.

  • do_train (bool, default True) – Determines whether to prepare the model for training, including distribtued env, model placement, quantization, lora, etc.

  • args (Optional.) – Positional arguments.

  • kwargs (Optional.) – Keyword arguments.

tokenize(dataset: lmflow.datasets.dataset.Dataset, add_special_tokens=True, *args, **kwargs) → lmflow.datasets.dataset.Dataset[source]#

Tokenize the full dataset.

Parameters:

  • dataset (lmflow.datasets.Dataset.)

  • args (Optional.) – Positional arguments.

  • kwargs (Optional.) – Keyword arguments.

Returns:

The tokenized dataset, without any leading or trailing special tokens (normally they are Begin-Of-Sentence or End-Of-Sentence tokens).

Return type:

tokenized_datasets

encode(input: str | list[str], *args, **kwargs) → list[int] | list[list[int]][source]#

Encode one string or a batch of strings with the model tokenizer.

Parameters:

  • input (str or list[str]) – Text input to encode.

  • *args – Positional tokenizer arguments.

  • **kwargs – Keyword tokenizer arguments.

Returns:

A batch encoding for list input, or token ids for string input.

Return type:

transformers.BatchEncoding or list[int]

Examples

A single string returns token IDs:

"Hello, world!" -> [101, 7592, 1010, 2088, 102]

A list of strings returns a batch encoding with fields such as:

{
    "input_ids": tensor([[101, 7592, 1010, 2088, 102], ...]),
    "attention_mask": tensor([[1, 1, 1, 1, 1], [0, 0, 1, 1, 1]])
}
decode(input, **kwargs) → str | list[str][source]#

Perform decoding process of the tokenizer.

Parameters:

  • inputs (list or tensor.) – The token sequence.

  • args (Optional.) – Positional arguments.

  • kwargs (Optional.) – Keyword arguments.

Returns:

The text decoded from the token inputs. if batch input,return the list of text [[101, 7592, 1010, 2088, 102],[101, 7592, 1010, 2088, 102]]-> [“Hello,world!”,”Hello,world!” if single input,return the text [101, 7592, 1010, 2088, 102]-> “Hello,world!”

Return type:

outputs

inference(inputs: str | list[str] | torch.Tensor | lmflow.utils.protocol.DataProto, sampling_params: dict | vllm.SamplingParams | None = None, return_logprob: bool = False, release_gpu: bool = False, inference_engine: Literal['huggingface', 'vllm', 'sglang'] = 'huggingface', gpu_memory_utilization: float | None = None, tensor_parallel_size: int | None = None, data_parallel_size: int = 1, max_model_len: int | None = None, enable_deterministic_inference: bool = False, attention_backend: str | None = None, **kwargs) → list | lmflow.utils.protocol.DataProto[source]#

Perform generation process of the model.

Parameters:

  • inputs (Union[str, list[str], torch.Tensor, DataProto]) – The sequence used as a prompt for the generation or as model inputs to the model. When the inference engine is “vllm” or “sglang”, this should be a DataProto. When the inference engine is “huggingface”, this should be a tensor.

  • sampling_params (Optional[Union[dict, "SamplingParams"]], optional) – The sampling parameters to use, by default None.

  • return_logprob (bool, optional) – Whether to return log probability during inference, by default False.

  • release_gpu (bool, optional) – Whether to release the GPU resource after inference, by default False.

  • inference_engine (Literal["huggingface", "vllm", "sglang"], optional) – The inference engine to use, by default “huggingface”.

  • gpu_memory_utilization (float, optional) – The GPU memory utilization to use, by default None.

  • tensor_parallel_size (int, optional) – The tensor parallel size to use, by default None.

  • data_parallel_size (int, optional) – The data parallel size for vllm inference, by default 1.

  • max_model_len (int, optional) – Maximum model context length for vllm inference, by default None.

  • enable_deterministic_inference (bool, optional) – Whether to enable deterministic inference, by default False.

  • attention_backend (Optional[str], optional) – The attention backend to use, by default None.

Returns:

The generated sequence output

Return type:

outputs

prepare_inputs_for_inference(dataset: lmflow.datasets.dataset.Dataset, apply_chat_template: bool = True, inference_engine: Literal['huggingface', 'vllm', 'sglang'] = 'huggingface', sampling_params: dict | None = None) → list[str] | lmflow.utils.protocol.DataProto[source]#
merge_lora_weights()[source]#
get_peft_without_qlora()[source]#
save(dir, save_full_model=False, *args, **kwargs)[source]#

Perform generation process of the model.

Parameters:

  • dir – The directory to save model and tokenizer

  • save_full_model (Optional.) – Whether to save full model.

  • kwargs (Optional.) – Keyword arguments.

Returns:

The generated sequence output

Return type:

outputs