lmflow.utils.conversation_template.base

Attributes

logger
EMPTY_TEMPLATE
EMPTY_NO_SPECIAL_TOKENS_TEMPLATE

Classes

TemplateComponent	The minimal unit of a template, which can be a token, a string, or a list of tools.
Formatter	Helper class that provides a standard way to create an ABC using
EmptyFormatter	Helper class that provides a standard way to create an ABC using
StringFormatter	Helper class that provides a standard way to create an ABC using
ConversationTemplate
ConversationTemplateForTool

Module Contents

lmflow.utils.conversation_template.base.logger

class lmflow.utils.conversation_template.base.TemplateComponent

The minimal unit of a template, which can be a token, a string, or a list of tools.

Parameters:

typeLiteral[‘token’, ‘token_id’, ‘string’, ‘tools’]

• Type of the component.

• When the component is a token or a string, the content should be string.

The difference between the two is that token will be converted to token ids by the tokenizer.convert_tokens_to_ids() method, while string will be directly encoded by the tokenizer.encode() method. Specially, since the bos token and eos token are frequently used across different templates, we provide the convenience to use ‘bos_token’ and ‘eos_token’ to represent the actual bos and eos tokens when type of the TemplateComponent is token. For example:

`python TemplateComponent(type='token', content='bos_token') `

After encoding, the content will be replaced by the actual token id of the bos token. Please do remember that if you set the type to string, the tokenizer will try to encode the string ‘bos_token’ instead of providing the actual bos token.

• When the component is token_id, the content should be int or List[int], and

will be directly appended to the encoded token ids.

• Tools are not supported yet.

contentUnion[str, int, List[str], List[int]]

Content of the component.

type: Literal['token', 'token_id', 'string', 'tools']

content: str | int | List[str] | List[int]

mask: bool | None = True

__post_init__()

__repr__() → str

__str__() → str

class lmflow.utils.conversation_template.base.Formatter

Bases: abc.ABC

Helper class that provides a standard way to create an ABC using inheritance.

template: List[TemplateComponent] = []

abstract format(**kwargs) → List[TemplateComponent]

has_placeholder()

class lmflow.utils.conversation_template.base.EmptyFormatter

Bases: Formatter

Helper class that provides a standard way to create an ABC using inheritance.

__post_init__()

format(**kwargs) → list

Empty formatter for when no formatting is needed. This is useful when user has already applied formatting to the dataset.

Returns:

list

Original template.

class lmflow.utils.conversation_template.base.StringFormatter

Bases: Formatter

Helper class that provides a standard way to create an ABC using inheritance.

__post_init__()

format(**kwargs) → list

Format the string components with the provided keyword arguments. Mostly used for formatting system prompt, user and assistant messages.

Parameters:

**kwargsdict

Keyword arguments containing values to replace in the template components.

Returns:

list

Formatted template.

class lmflow.utils.conversation_template.base.ConversationTemplate

user_formatter: Formatter

assistant_formatter: Formatter

function_formatter: Formatter | None = None

observation_formatter: Formatter | None = None

system_formatter: Formatter | None = None

force_system: bool = False

tools_formatter: Formatter | None = None

separator: TemplateComponent | None = None

remove_last_sep: bool = False

special_starter: TemplateComponent | None = None

special_stopper: TemplateComponent | None = None

template_name: str | None = None

system_default: str | None = None

__post_init__()

encode_conversation(tokenizer: transformers.PreTrainedTokenizer, messages: List[Dict[str, str]], system: str | None = None, tools: List[str] | None = None, **kwargs) → Sequence[Tuple[List[int], List[int]]]

Messages here should be guaranteed to be in pairs, with the first message being the user message and the second message being the system message. Data example: ```json {

“conversation_id”: 2, “system”: “sysinfo1”, “tools”: [“tool_1_desc”], “messages”: [

{

“role”: “user”, “content”: “hi”

}, {

“role”: “assistant”, “content”: “Hello!”

}

]

}

_encode(tokenizer: transformers.PreTrainedTokenizer, messages: List[Dict[str, str]], system: str | None = None, tools: str | None = None, **kwargs) → Sequence[Tuple[List[int], List[int]]]

_encode_template(template: List[TemplateComponent], tokenizer: transformers.PreTrainedTokenizer, **kwargs) → List[int]

Encode template components into token ids.

Parameters:

templateList[TemplateComponent]

Formatted template components.

tokenizerPreTrainedTokenizer

Tokenizer to convert tokens into token ids.

Returns:

List[int]

Encoded token ids.

post_process_pairs(encoded_pairs, tokenizer)

remove_last_separator(encoded_pairs: Sequence[Tuple[List[int], List[int]]], tokenizer: transformers.PreTrainedTokenizer) → Sequence[Tuple[List[int], List[int]]]

add_special_starter(encoded_pairs: Sequence[Tuple[List[int], List[int]]], tokenizer: transformers.PreTrainedTokenizer) → Sequence[Tuple[List[int], List[int]]]

add_special_stopper(encoded_pairs: Sequence[Tuple[List[int], List[int]]], tokenizer: transformers.PreTrainedTokenizer) → Sequence[Tuple[List[int], List[int]]]

_ensure_id_list(obj: int | List[int]) → List[int]

Make sure the object is a list of integers. Useful for handling token ids.

class lmflow.utils.conversation_template.base.ConversationTemplateForTool

Bases: ConversationTemplate

encode_conversation(tokenizer: transformers.PreTrainedTokenizer, messages: List[Dict[str, str]], system: str | None = None, tools: List[str] | None = None, **kwargs) → Sequence[Tuple[List[int], List[int]]]

Messages here should be guaranteed to be in pairs, with the first message being the user message and the second message being the system message. Data example: ```json {

“conversation_id”: 2, “system”: “sysinfo1”, “tools”: [“tool_1_desc”], “messages”: [

{

“role”: “user”, “content”: “hi”

}, {

“role”: “assistant”, “content”: “Hello!”

}

]

}

_encode(tokenizer: transformers.PreTrainedTokenizer, messages: List[Dict[str, str]], system: str | None = None, tools: str | None = None, **kwargs) → Sequence[Tuple[List[int], List[int]]]

_encode_template(template: List[TemplateComponent], tokenizer: transformers.PreTrainedTokenizer, **kwargs) → List[int]

Encode template components into token ids.

Parameters:

templateList[TemplateComponent]

Formatted template components.

tokenizerPreTrainedTokenizer

Tokenizer to convert tokens into token ids.

Returns:

List[int]

Encoded token ids.

_handle_tools(tools: List[str] | None) → str

lmflow.utils.conversation_template.base.EMPTY_TEMPLATE

lmflow.utils.conversation_template.base.EMPTY_NO_SPECIAL_TOKENS_TEMPLATE