lmflow.datasets.dataset#

This Python code defines a class Dataset with methods for initializing, loading, and manipulating datasets from different backends such as Hugging Face and JSON.

The Dataset class includes methods for loading datasets from a dictionary and a Hugging Face dataset, mapping datasets, and retrieving the backend dataset and arguments.

Attributes#

`logger`
`DATASET_TYPES`
`KEY_TYPE`
`KEY_INSTANCES`
`KEY_SCORE`

Classes#

Dataset

Initializes the Dataset object with the given parameters.

Module Contents#

lmflow.datasets.dataset.logger[source]#

lmflow.datasets.dataset.DATASET_TYPES = ['text_only', 'text2text', 'float_only', 'image_text', 'conversation', 'paired_conversation',...[source]#

lmflow.datasets.dataset.KEY_TYPE = 'type'[source]#

lmflow.datasets.dataset.KEY_INSTANCES = 'instances'[source]#

lmflow.datasets.dataset.KEY_SCORE = 'score'[source]#

class lmflow.datasets.dataset.Dataset(data_args: lmflow.args.DatasetArguments = None, backend: str = 'huggingface', *args, **kwargs)[source]#

Initializes the Dataset object with the given parameters.

Parameters:

data_args (DatasetArguments object.) – Contains the arguments required to load the dataset.
backend (str, default="huggingface") – A string representing the dataset backend. Defaults to “huggingface”.
args (Optional.) – Positional arguments.
kwargs (Optional.) – Keyword arguments.

data_args = None[source]#

backend = 'huggingface'[source]#

backend_dataset = None[source]#

type = None[source]#

dataset_path[source]#

__len__()[source]#

from_dict(dict_obj: dict, *args, **kwargs)[source]#

Populate this dataset from an LMFlow dataset dictionary.

The expected dictionary shape is:

{
    "type": TYPE,
    "instances": [
        {
            "key_1": VALUE_1_1,
            "key_2": VALUE_1_2,
            ...
        },
        {
            "key_1": VALUE_2_1,
            "key_2": VALUE_2_2,
            ...
        },
        ...
    ]
}

Parameters:

dict_obj (dict) – Dataset data containing type and instances keys.
*args – Positional arguments passed to the selected dataset backend.
**kwargs – Keyword arguments passed to the selected dataset backend.

Returns:

This dataset instance.

Return type:

Dataset

classmethod create_from_dict(dict_obj, *args, **kwargs)[source]#

Return type:: Returns a Dataset object given a dict.

to_dict()[source]#

Convert this dataset to the LMFlow dictionary format.

The returned dictionary has the following shape:

{
    "type": TYPE,
    "instances": [
        {
            "key_1": VALUE_1_1,
            "key_2": VALUE_1_2,
            ...
        },
        {
            "key_1": VALUE_2_1,
            "key_2": VALUE_2_2,
            ...
        },
        ...
    ]
}

Returns:: Dataset data containing type and instances keys.
Return type:: dict

to_list()[source]#: Returns a list of instances.

map(*args, **kwargs)[source]#

Parameters:

args (Optional.) – Positional arguments.
kwargs (Optional.) – Keyword arguments.

Returns:

self

Return type:

Dataset object.

get_backend() → str | None[source]#

Return type:: self.backend

get_backend_dataset()[source]#

Return type:: self.backend_dataset

get_fingerprint()[source]#

Return type:: Fingerprint of the backend_dataset which controls the cache

get_data_args()[source]#

Return type:: self.data_args

get_type() → str[source]#

Return type:: self.type

save(file_path: str, format: str = 'json')[source]#

Save the dataset to a json file.

Parameters:: file_path (str.) – The path to the file where the dataset will be saved.

sample(n: int, seed: int = 42)[source]#

Sample n instances from the dataset.

Parameters:: n (int.) – The number of instances to sample from the dataset.
Returns:: sample_dataset – A new dataset object containing the sampled instances.
Return type:: Dataset object.

train_test_split(test_size: float = 0.2, shuffle: bool = True, seed: int = 42)[source]#

Split the dataset into training and testing sets.

Parameters:

test_size (float, default=0.2.) – The proportion of the dataset that will be used for testing.

Returns:

train_dataset (Dataset object.) – A new dataset object containing the training instances.
test_dataset (Dataset object.) – A new dataset object containing the testing instances.

drop_instances(indices: list)[source]#

Drop instances from the dataset.

Parameters:: indices (list.) – A list of indices of the instances to drop from the dataset.

sanity_check(drop_invalid: bool = True)[source]#: Perform a sanity check on the dataset.

hf_dataset_sanity_check(drop_invalid: bool = True)[source]#: Perform a sanity check on the HuggingFace dataset.