lmflow.utils.data_utils

The program includes several functions: setting a random seed, loading data from a JSON file, batching data, and extracting answers from generated text.

Classes

VLLMInferenceResultWithInput	Structured vLLM inference result with its original input.
RewardModelInferenceResultWithInput	Structured reward-model inference result with its original input.

Functions

set_random_seed(seed)	Set the random seed for random, numpy, torch, torch.cuda.
load_data(file_name)	Load data with file name.
batchlize(examples, batch_size, random_shuffle)	Convert examples to a dataloader.
preview_file(file_path[, chars])	Returns the first and last specified number of characters from a file
get_dataset_type_fast(→ Union[str, None])	Get the type values from the first and last n lines of a large json dataset.
check_dataset_instances_key_fast(→ bool)	Check if the dataset instances key matches the instance_key.
answer_extraction(response[, answer_type])	Extract answers from generated text.
process_image_flag(text[, image_flag])

Module Contents

lmflow.utils.data_utils.set_random_seed(seed: int)

Set the random seed for random, numpy, torch, torch.cuda.

Parameters:

seed (int) – The default seed.

lmflow.utils.data_utils.load_data(file_name: str)

Load data with file name.

Parameters:

file_name (str) – The dataset file name.

Returns:

• inputs (list) – The input texts of the dataset.

• outputs (list) – The output texts file datasets.

• length (int) – The length of the dataset.

lmflow.utils.data_utils.batchlize(examples: list, batch_size: int, random_shuffle: bool)

Convert examples to a dataloader.

Parameters:

• examples (list) – Data list.

• batch_size (int) – Number of examples in each batch.

• random_shuffle (bool) – If true, the dataloader shuffle the training data.

Returns:

Dataloader with batch generator.

Return type:

list

lmflow.utils.data_utils.preview_file(file_path: str, chars: int = 100)

Returns the first and last specified number of characters from a file without loading the entire file into memory, working with any file type.

Parameters:

• file_path (str) – Path to the file to be previewed

• chars (int, optional) – Number of characters to show from start and end. Defaults to 100.

Returns:

(first_chars, last_chars) - The first and last characters from the file

Return type:

tuple

lmflow.utils.data_utils.get_dataset_type_fast(file_path: str, max_chars: int = 100) → str | None

Get the type values from the first and last n lines of a large json dataset.

lmflow.utils.data_utils.check_dataset_instances_key_fast(file_path: str, instances_key: str, max_lines: int = 100) → bool

Check if the dataset instances key matches the instance_key.

lmflow.utils.data_utils.answer_extraction(response, answer_type=None)

Extract answers from generated text.

Parameters:

• response (str) – plain string response.

• answer_type (str, optional) – Type of answer to extract.

Returns:

Decoded answer (such as A, B, C, D, E for multiple-choice QA).

Return type:

str

lmflow.utils.data_utils.process_image_flag(text, image_flag='<ImageHere>')

class lmflow.utils.data_utils.VLLMInferenceResultWithInput

Bases: TypedDict

Structured vLLM inference result with its original input.

input: str

output: list[str] | list[list[int]]

class lmflow.utils.data_utils.RewardModelInferenceResultWithInput

Bases: TypedDict

Structured reward-model inference result with its original input.

input: str

output: list[dict[str, str | float]]