lmflow.models.hf_text_regression_model

Attributes

logger

Classes

HFTextRegressionModel

Initializes a HFTextRegressionModel instance.

Module Contents

lmflow.models.hf_text_regression_model.logger

class lmflow.models.hf_text_regression_model.HFTextRegressionModel(model_args: lmflow.args.ModelArguments, do_train: bool = False, device='gpu', **kwargs)

Bases: lmflow.models.text_regression_model.TextRegressionModel, lmflow.models.hf_model_mixin.HFModelMixin, lmflow.models.interfaces.tunable.Tunable

Initializes a HFTextRegressionModel instance.

Parameters:

model_args

Model arguments such as model name, path, revision, etc.

do_trainbool, default True

Determines whether to prepare the model for training, including distribtued env, model placement, quantization, lora, etc.

argsOptional.

Positional arguments.

kwargsOptional.

Keyword arguments.

tokenize(dataset: lmflow.datasets.dataset.Dataset, add_special_tokens=True, *args, **kwargs)

Tokenize the full dataset.

Parameters:

datasetlmflow.datasets.Dataset.

argsOptional.

Positional arguments.

kwargsOptional.

Keyword arguments.

Returns:

tokenized_datasets

The tokenized dataset, without any leading or trailing special tokens (normally they are Begin-Of-Sentence or End-Of-Sentence tokens).

inference(inputs, release_gpu: bool = False, use_vllm: bool = False, **kwargs) → list[float] | transformers.modeling_outputs.SequenceClassifierOutputWithPast

Perform generation process of the model.

Parameters:

inputs

The sequence used as a prompt for the generation or as model inputs to the model. When using vllm inference, this should be a string or a list of strings. When using normal inference, this should be a tensor.

release_gpubool, optional

Whether to release the GPU resource after inference, by default False.

use_vllmbool, optional

Whether to use VLLM for inference, by default False.

kwargsOptional.

Keyword arguments.

Returns:

outputs

The generated sequence output

__inference(inputs, **kwargs)

Perform generation process of the model.

Parameters:

inputs

The tokenized sequence used as a prompt for the generation or as model inputs to the model.

kwargsOptional.

Keyword arguments.

Returns:

outputs

The generated sequence output

abstractmethod __vllm_inference(inputs: str | list[str], sampling_params: vllm.SamplingParams | None = None, **kwargs) → list[list[str]] | list[list[list[int]]]

Perform VLLM inference process of the model.

Parameters:

inputsUnion[str, list[str]]

Prompt(s), string or a list of strings.

sampling_paramsOptional[SamplingParams], optional

vllm SamplingParams object, by default None.

Returns:

prepare_inputs_for_inference(dataset: lmflow.datasets.dataset.Dataset, enable_distributed_inference: bool = False, use_vllm: bool = False, **kwargs) → lmflow.datasets.dataset.Dataset | ray.data.Dataset

static postprocess_inference_outputs(dataset: lmflow.datasets.dataset.Dataset, scores: list[float] | list[list[float]])

static postprocess_distributed_inference_outputs(dataset: lmflow.datasets.dataset.Dataset, inference_result: list[lmflow.utils.data_utils.RewardModelInferenceResultWithInput])

save(dir, *args, **kwargs)

Perform generation process of the model.

Parameters:

dir

The directory to save model and tokenizer

kwargsOptional.

Keyword arguments.