lmflow.datasets.multi_modal_dataset#

This Python code defines a class Multi Modal Dataset.

Classes#

`CustomMultiModalDataset`	Dataset for Multi Modal data
`DataCollatorForSupervisedDataset`	Collate examples for supervised fine-tuning.

Functions#

`preprocess_multimodal_llava`(sources, data_args)
`tokenizer_image_token`(prompt, tokenizer[, ...])
`preprocess_llama_from_llava_plain`(sources, tokenizer)	This function just add the image in the front of text.
`preprocess_llama_from_llava_v1`(sources, tokenizer[, ...])	This function add the prompt and then put the image after the prompt.

Module Contents#

class lmflow.datasets.multi_modal_dataset.CustomMultiModalDataset(dataset_path: str, data_args: lmflow.args.DatasetArguments)[source]#

Bases: torch.utils.data.Dataset

Dataset for Multi Modal data

data_dict[source]#

data_args[source]#

image_folder[source]#

__len__()[source]#

register_tokenizer(tokenizer, image_processor=None)[source]#

__getitem__(i)[source]#

lmflow.datasets.multi_modal_dataset.preprocess_multimodal_llava(sources, data_args)[source]#

lmflow.datasets.multi_modal_dataset.tokenizer_image_token(prompt, tokenizer, image_token_index=IMAGE_TOKEN_INDEX, return_tensors=None)[source]#

lmflow.datasets.multi_modal_dataset.preprocess_llama_from_llava_plain(sources, tokenizer: transformers.PreTrainedTokenizer, has_image: bool = False)[source]#

This function just add the image in the front of text. And don’t add any prompt. Args:

sources: The input data with text and image. tokenizer: The tokenizer to process text. has_image: Whether the input data has image.

Returns:: The input_ids and labels for the model.

lmflow.datasets.multi_modal_dataset.preprocess_llama_from_llava_v1(sources, tokenizer: transformers.PreTrainedTokenizer, has_image: bool = False)[source]#

This function add the prompt and then put the image after the prompt. So it needs additional code to generate the target label. Args:

sources: The input data with text and image. tokenizer: The tokenizer to process text. has_image: Whether the input data has image.

Returns:: The input_ids and labels for the model.

class lmflow.datasets.multi_modal_dataset.DataCollatorForSupervisedDataset[source]#

Bases: object

Collate examples for supervised fine-tuning.

tokenizer: transformers.PreTrainedTokenizer[source]#

__call__(instances)[source]#