lmflow.datasets.multi_modal_dataset#

This Python code defines a class Multi Modal Dataset.

Classes#

`CustomMultiModalDataset`	Dataset for Multi Modal data
`DataCollatorForSupervisedDataset`	Collate examples for supervised fine-tuning.

Functions#

`preprocess_multimodal_llava`(sources, data_args)
`tokenizer_image_token`(prompt, tokenizer[, ...])
`preprocess_llama_from_llava_plain`(sources, tokenizer)	Preprocess plain LLaVA samples without adding a prompt.
`preprocess_llama_from_llava_v1`(sources, tokenizer[, ...])	Preprocess LLaVA v1 samples and generate target labels.

Module Contents#

class lmflow.datasets.multi_modal_dataset.CustomMultiModalDataset(dataset_path: str, data_args: lmflow.args.DatasetArguments)[source]#

Bases: torch.utils.data.Dataset

Dataset for Multi Modal data

data_dict[source]#

data_args[source]#

image_folder[source]#

__len__()[source]#

register_tokenizer(tokenizer, image_processor=None)[source]#

__getitem__(i)[source]#

lmflow.datasets.multi_modal_dataset.preprocess_multimodal_llava(sources, data_args)[source]#

lmflow.datasets.multi_modal_dataset.tokenizer_image_token(prompt, tokenizer, image_token_index=IMAGE_TOKEN_INDEX, return_tensors=None)[source]#

lmflow.datasets.multi_modal_dataset.preprocess_llama_from_llava_plain(sources, tokenizer: transformers.PreTrainedTokenizer, has_image: bool = False)[source]#

Preprocess plain LLaVA samples without adding a prompt.

Parameters:

sources – Input samples containing text and image placeholders.
tokenizer (transformers.PreTrainedTokenizer) – Tokenizer used to encode text.
has_image (bool) – Whether the samples contain images.

Returns:

Model input_ids and labels.

Return type:

dict

lmflow.datasets.multi_modal_dataset.preprocess_llama_from_llava_v1(sources, tokenizer: transformers.PreTrainedTokenizer, has_image: bool = False)[source]#

Preprocess LLaVA v1 samples and generate target labels.

Parameters:

sources – Input samples containing text and image placeholders.
tokenizer (transformers.PreTrainedTokenizer) – Tokenizer used to encode text.
has_image (bool) – Whether the samples contain images.

Returns:

Model input_ids and labels.

Return type:

dict

class lmflow.datasets.multi_modal_dataset.DataCollatorForSupervisedDataset[source]#

Collate examples for supervised fine-tuning.

tokenizer: transformers.PreTrainedTokenizer[source]#

__call__(instances)[source]#