lmflow.utils.data_utils
=======================

.. py:module:: lmflow.utils.data_utils

.. autoapi-nested-parse::

   The program includes several functions: setting a random seed,
   loading data from a JSON file, batching data, and extracting answers from generated text.



Classes
-------

.. autoapisummary::

   lmflow.utils.data_utils.VLLMInferenceResultWithInput
   lmflow.utils.data_utils.RewardModelInferenceResultWithInput


Functions
---------

.. autoapisummary::

   lmflow.utils.data_utils.set_random_seed
   lmflow.utils.data_utils.load_data
   lmflow.utils.data_utils.batchlize
   lmflow.utils.data_utils.preview_file
   lmflow.utils.data_utils.get_dataset_type_fast
   lmflow.utils.data_utils.check_dataset_instances_key_fast
   lmflow.utils.data_utils.answer_extraction
   lmflow.utils.data_utils.process_image_flag


Module Contents
---------------

.. py:function:: set_random_seed(seed: int)

   Set the random seed for `random`, `numpy`, `torch`, `torch.cuda`.

   :param seed: The default seed.
   :type seed: int


.. py:function:: load_data(file_name: str)

   Load data with file name.

   :param file_name: The dataset file name.
   :type file_name: str

   :returns: * **inputs** (*list*) -- The input texts of the dataset.
             * **outputs** (*list*) -- The output texts file datasets.
             * **length** (*int*) -- The length of the dataset.


.. py:function:: batchlize(examples: list, batch_size: int, random_shuffle: bool)

   Convert examples to a dataloader.

   :param examples: Data list.
   :type examples: list
   :param batch_size: Number of examples in each batch.
   :type batch_size: int
   :param random_shuffle: If true, the dataloader shuffle the training data.
   :type random_shuffle: bool

   :returns: Dataloader with batch generator.
   :rtype: list


.. py:function:: preview_file(file_path: str, chars: int = 100)

   Returns the first and last specified number of characters from a file
   without loading the entire file into memory, working with any file type.

   :param file_path: Path to the file to be previewed
   :type file_path: str
   :param chars: Number of characters to show from start and end. Defaults to 100.
   :type chars: int, optional

   :returns: (first_chars, last_chars) - The first and last characters from the file
   :rtype: tuple


.. py:function:: get_dataset_type_fast(file_path: str, max_chars: int = 100) -> Union[str, None]

   Get the type values from the first and last n lines of a large json dataset.


.. py:function:: check_dataset_instances_key_fast(file_path: str, instances_key: str, max_lines: int = 100) -> bool

   Check if the dataset instances key matches the instance_key.


.. py:function:: answer_extraction(response, answer_type=None)

   Extract answers from generated text.

   :param response: plain string response.
   :type response: str
   :param answer_type: Type of answer to extract.
   :type answer_type: str, optional

   :returns: Decoded answer (such as A, B, C, D, E for multiple-choice QA).
   :rtype: str


.. py:function:: process_image_flag(text, image_flag='<ImageHere>')

.. py:class:: VLLMInferenceResultWithInput

   Bases: :py:obj:`TypedDict`


   Structured vLLM inference result with its original input.


   .. py:attribute:: input
      :type:  str


   .. py:attribute:: output
      :type:  Union[list[str], list[list[int]]]


.. py:class:: RewardModelInferenceResultWithInput

   Bases: :py:obj:`TypedDict`


   Structured reward-model inference result with its original input.


   .. py:attribute:: input
      :type:  str


   .. py:attribute:: output
      :type:  list[dict[str, Union[str, float]]]