deberta

mindnlp.transformers.models.deberta.modeling_deberta

MindSpore DeBERTa model.

mindnlp.transformers.models.deberta.modeling_deberta.ContextPooler

Bases: Module

Represents a ContextPooler module used for pooling contextual embeddings in a neural network architecture.

This class inherits from nn.Module and provides methods for initializing the pooler, forwarding the pooled output based on hidden states, and retrieving the output dimension. The pooler consists of a dense layer and dropout mechanism for processing hidden states.

ATTRIBUTE	DESCRIPTION
dense	A dense layer for transforming input hidden states to pooler hidden size. TYPE: Linear
dropout	A dropout layer for stable dropout operations. TYPE: StableDropout
config	Configuration object containing pooler settings.

METHOD	DESCRIPTION
__init__	Initializes the ContextPooler with the given configuration.
forward	Constructs the pooled output by processing hidden states.
output_dim	Property that returns the output dimension based on the hidden size in the configuration.

Source code in mindnlp/transformers/models/deberta/modeling_deberta.py

class ContextPooler(nn.Module):

    """
    Represents a ContextPooler module used for pooling contextual embeddings in a neural network architecture.

    This class inherits from nn.Module and provides methods for initializing the pooler, forwarding the pooled output based on hidden states,
    and retrieving the output dimension. The pooler consists of a dense layer and dropout mechanism for processing hidden states.

    Attributes:
        dense (nn.Linear): A dense layer for transforming input hidden states to pooler hidden size.
        dropout (StableDropout): A dropout layer for stable dropout operations.
        config: Configuration object containing pooler settings.

    Methods:
        __init__(config): Initializes the ContextPooler with the given configuration.
        forward(hidden_states): Constructs the pooled output by processing hidden states.
        output_dim: Property that returns the output dimension based on the hidden size in the configuration.
    """
    def __init__(self, config):
        """
        Initializes a new instance of the ContextPooler class.

        Args:
            self: The instance of the ContextPooler class.
            config:
                An object of type 'config' that contains the configuration parameters for the ContextPooler.

                - Type: 'config'
                - Purpose: Specifies the configuration parameters for the ContextPooler.
                - Restrictions: None.

        Returns:
            None

        Raises:
            None
        """
        super().__init__()
        self.dense = nn.Linear(config.pooler_hidden_size, config.pooler_hidden_size)
        self.dropout = StableDropout(config.pooler_dropout)
        self.config = config

    def forward(self, hidden_states):
        """
        Args:
            self (ContextPooler): The instance of the ContextPooler class.
            hidden_states (tensor): A tensor containing hidden states.
                It is expected to have a specific shape and format for processing.

        Returns:
            pooled_output (tensor): The output tensor after the pooling operation.
                It represents the pooled context information.

        Raises:
            ValueError: If the hidden_states tensor does not meet the expected shape or format requirements.
            RuntimeError: If an error occurs during the pooling operation.

        """
        # We "pool" the model by simply taking the hidden state corresponding
        # to the first token.

        context_token = hidden_states[:, 0]
        context_token = self.dropout(context_token)
        pooled_output = self.dense(context_token)
        pooled_output = ACT2FN[self.config.pooler_hidden_act](pooled_output)
        return pooled_output

    @property
    def output_dim(self):
        """
        Method to retrieve the output dimension of the ContextPooler.

        Args:
            self (ContextPooler): An instance of the ContextPooler class.
                This parameter is required to access the configuration information.

        Returns:
            None: The method does not perform any computation but simply returns the output dimension.

        Raises:
            None.
        """
        return self.config.hidden_size

mindnlp.transformers.models.deberta.modeling_deberta.ContextPooler.output_dim property

Method to retrieve the output dimension of the ContextPooler.

PARAMETER	DESCRIPTION
self	An instance of the ContextPooler class. This parameter is required to access the configuration information. TYPE: ContextPooler

RETURNS	DESCRIPTION
None	The method does not perform any computation but simply returns the output dimension.

mindnlp.transformers.models.deberta.modeling_deberta.ContextPooler.__init__(config)

Initializes a new instance of the ContextPooler class.

PARAMETER	DESCRIPTION
self	The instance of the ContextPooler class.
config	An object of type 'config' that contains the configuration parameters for the ContextPooler. Type: 'config' Purpose: Specifies the configuration parameters for the ContextPooler. Restrictions: None.

RETURNS	DESCRIPTION
	None

Source code in mindnlp/transformers/models/deberta/modeling_deberta.py

def __init__(self, config):
    """
    Initializes a new instance of the ContextPooler class.

    Args:
        self: The instance of the ContextPooler class.
        config:
            An object of type 'config' that contains the configuration parameters for the ContextPooler.

            - Type: 'config'
            - Purpose: Specifies the configuration parameters for the ContextPooler.
            - Restrictions: None.

    Returns:
        None

    Raises:
        None
    """
    super().__init__()
    self.dense = nn.Linear(config.pooler_hidden_size, config.pooler_hidden_size)
    self.dropout = StableDropout(config.pooler_dropout)
    self.config = config

mindnlp.transformers.models.deberta.modeling_deberta.ContextPooler.forward(hidden_states)

PARAMETER	DESCRIPTION
self	The instance of the ContextPooler class. TYPE: ContextPooler
hidden_states	A tensor containing hidden states. It is expected to have a specific shape and format for processing. TYPE: tensor

RETURNS	DESCRIPTION
pooled_output	The output tensor after the pooling operation. It represents the pooled context information. TYPE: tensor

RAISES	DESCRIPTION
ValueError	If the hidden_states tensor does not meet the expected shape or format requirements.
RuntimeError	If an error occurs during the pooling operation.

Source code in mindnlp/transformers/models/deberta/modeling_deberta.py

def forward(self, hidden_states):
    """
    Args:
        self (ContextPooler): The instance of the ContextPooler class.
        hidden_states (tensor): A tensor containing hidden states.
            It is expected to have a specific shape and format for processing.

    Returns:
        pooled_output (tensor): The output tensor after the pooling operation.
            It represents the pooled context information.

    Raises:
        ValueError: If the hidden_states tensor does not meet the expected shape or format requirements.
        RuntimeError: If an error occurs during the pooling operation.

    """
    # We "pool" the model by simply taking the hidden state corresponding
    # to the first token.

    context_token = hidden_states[:, 0]
    context_token = self.dropout(context_token)
    pooled_output = self.dense(context_token)
    pooled_output = ACT2FN[self.config.pooler_hidden_act](pooled_output)
    return pooled_output

mindnlp.transformers.models.deberta.modeling_deberta.DebertaAttention

Bases: Module

This class represents the DebertaAttention module, which is a component of the DeBERTa model. It inherits from the nn.Module class.

DebertaAttention applies self-attention mechanism on the input hidden states, allowing the model to focus on different parts of the input sequence. It consists of a DisentangledSelfAttention layer and a DebertaSelfOutput layer.

PARAMETER	DESCRIPTION
config	A dictionary containing the configuration parameters for the DebertaAttention module. TYPE: dict

METHOD	DESCRIPTION
__init__	Initializes a new instance of DebertaAttention. Args: config (dict): A dictionary containing the configuration parameters for the DebertaAttention module.
forward	Applies the DebertaAttention mechanism on the input hidden states. Args: hidden_states (Tensor): The input hidden states of shape (batch_size, sequence_length, hidden_size). attention_mask (Tensor): The attention mask of shape (batch_size, sequence_length, sequence_length) where 1 indicates tokens to attend to and 0 indicates tokens to ignore. output_attentions (bool, optional): Whether to output the attention matrix. Defaults to False. query_states (Tensor, optional): The query states of shape (batch_size, sequence_length, hidden_size). If not provided, defaults to using the input hidden states. relative_pos (Tensor, optional): The relative positions of the tokens of shape (batch_size, sequence_length, sequence_length). rel_embeddings (Tensor, optional): The relative embeddings of shape (batch_size, sequence_length, hidden_size). Returns: Tensor or Tuple: The attention output tensor of shape (batch_size, sequence_length, hidden_size) or a tuple containing the attention output tensor and the attention matrix if output_attentions is True.

Source code in mindnlp/transformers/models/deberta/modeling_deberta.py

class DebertaAttention(nn.Module):

    """
    This class represents the DebertaAttention module, which is a component of the DeBERTa model.
    It inherits from the nn.Module class.

    DebertaAttention applies self-attention mechanism on the input hidden states, allowing the model to focus on
    different parts of the input sequence. It consists of a DisentangledSelfAttention layer and a
    DebertaSelfOutput layer.

    Args:
        config (dict): A dictionary containing the configuration parameters for the DebertaAttention module.

    Methods:
        __init__(self, config):
            Initializes a new instance of DebertaAttention.

            Args:

            - config (dict): A dictionary containing the configuration parameters for the DebertaAttention module.

        forward(self, hidden_states, attention_mask, output_attentions=False, query_states=None, relative_pos=None, rel_embeddings=None):
            Applies the DebertaAttention mechanism on the input hidden states.

            Args:

            - hidden_states (Tensor): The input hidden states of shape (batch_size, sequence_length, hidden_size).
            - attention_mask (Tensor): The attention mask of shape (batch_size, sequence_length, sequence_length)
            where 1 indicates tokens to attend to and 0 indicates tokens to ignore.
            - output_attentions (bool, optional): Whether to output the attention matrix.
            Defaults to False.
            - query_states (Tensor, optional): The query states of shape (batch_size, sequence_length, hidden_size).
            If not provided, defaults to using the input hidden states.
            - relative_pos (Tensor, optional): The relative positions of the tokens of shape
            (batch_size, sequence_length, sequence_length).
            - rel_embeddings (Tensor, optional): The relative embeddings of shape
            (batch_size, sequence_length, hidden_size).

            Returns:

            - Tensor or Tuple: The attention output tensor of shape (batch_size, sequence_length, hidden_size) or
            a tuple containing the attention output tensor and the attention matrix if output_attentions is True.
    """
    def __init__(self, config):
        """
        Initializes a new instance of the DebertaAttention class.

        Args:
            self (DebertaAttention): The current instance of the DebertaAttention class.
            config (object): The configuration object containing the settings for the attention module.
                It should provide the necessary parameters for initializing the DisentangledSelfAttention and
                DebertaSelfOutput instances.

        Returns:
            None.

        Raises:
            None.
        """
        super().__init__()
        self.self = DisentangledSelfAttention(config)
        self.output = DebertaSelfOutput(config)
        self.config = config

    def forward(
        self,
        hidden_states,
        attention_mask,
        output_attentions=False,
        query_states=None,
        relative_pos=None,
        rel_embeddings=None,
    ):
        """
        Constructs the DebertaAttention layer with the given parameters.

        Args:
            self: The DebertaAttention instance.
            hidden_states (torch.Tensor): The input hidden states with shape (batch_size, sequence_length, hidden_size).
            attention_mask (torch.Tensor): The attention mask with shape (batch_size, sequence_length).
            output_attentions (bool): Whether to output attention matrices.
            query_states (torch.Tensor): The query states with shape (batch_size, sequence_length, hidden_size).
                If not provided, defaults to hidden_states.
            relative_pos (torch.Tensor):
                The relative position encoding with shape (batch_size, sequence_length, sequence_length).
            rel_embeddings (torch.Tensor):
                The relative position embeddings with shape (num_relative_distances, hidden_size).

        Returns:
            None

        Raises:
            None
        """
        self_output = self.self(
            hidden_states,
            attention_mask,
            output_attentions,
            query_states=query_states,
            relative_pos=relative_pos,
            rel_embeddings=rel_embeddings,
        )
        if output_attentions:
            self_output, att_matrix = self_output
        if query_states is None:
            query_states = hidden_states
        attention_output = self.output(self_output, query_states)

        if output_attentions:
            return (attention_output, att_matrix)
        return attention_output

mindnlp.transformers.models.deberta.modeling_deberta.DebertaAttention.__init__(config)

Initializes a new instance of the DebertaAttention class.

PARAMETER	DESCRIPTION
self	The current instance of the DebertaAttention class. TYPE: DebertaAttention
config	The configuration object containing the settings for the attention module. It should provide the necessary parameters for initializing the DisentangledSelfAttention and DebertaSelfOutput instances. TYPE: object

RETURNS	DESCRIPTION
	None.

Source code in mindnlp/transformers/models/deberta/modeling_deberta.py

def __init__(self, config):
    """
    Initializes a new instance of the DebertaAttention class.

    Args:
        self (DebertaAttention): The current instance of the DebertaAttention class.
        config (object): The configuration object containing the settings for the attention module.
            It should provide the necessary parameters for initializing the DisentangledSelfAttention and
            DebertaSelfOutput instances.

    Returns:
        None.

    Raises:
        None.
    """
    super().__init__()
    self.self = DisentangledSelfAttention(config)
    self.output = DebertaSelfOutput(config)
    self.config = config

mindnlp.transformers.models.deberta.modeling_deberta.DebertaAttention.forward(hidden_states, attention_mask, output_attentions=False, query_states=None, relative_pos=None, rel_embeddings=None)

Constructs the DebertaAttention layer with the given parameters.

PARAMETER	DESCRIPTION
self	The DebertaAttention instance.
hidden_states	The input hidden states with shape (batch_size, sequence_length, hidden_size). TYPE: Tensor
attention_mask	The attention mask with shape (batch_size, sequence_length). TYPE: Tensor
output_attentions	Whether to output attention matrices. TYPE: bool DEFAULT: False
query_states	The query states with shape (batch_size, sequence_length, hidden_size). If not provided, defaults to hidden_states. TYPE: Tensor DEFAULT: None
relative_pos	The relative position encoding with shape (batch_size, sequence_length, sequence_length). TYPE: Tensor DEFAULT: None
rel_embeddings	The relative position embeddings with shape (num_relative_distances, hidden_size). TYPE: Tensor DEFAULT: None

RETURNS	DESCRIPTION
	None

Source code in mindnlp/transformers/models/deberta/modeling_deberta.py

def forward(
    self,
    hidden_states,
    attention_mask,
    output_attentions=False,
    query_states=None,
    relative_pos=None,
    rel_embeddings=None,
):
    """
    Constructs the DebertaAttention layer with the given parameters.

    Args:
        self: The DebertaAttention instance.
        hidden_states (torch.Tensor): The input hidden states with shape (batch_size, sequence_length, hidden_size).
        attention_mask (torch.Tensor): The attention mask with shape (batch_size, sequence_length).
        output_attentions (bool): Whether to output attention matrices.
        query_states (torch.Tensor): The query states with shape (batch_size, sequence_length, hidden_size).
            If not provided, defaults to hidden_states.
        relative_pos (torch.Tensor):
            The relative position encoding with shape (batch_size, sequence_length, sequence_length).
        rel_embeddings (torch.Tensor):
            The relative position embeddings with shape (num_relative_distances, hidden_size).

    Returns:
        None

    Raises:
        None
    """
    self_output = self.self(
        hidden_states,
        attention_mask,
        output_attentions,
        query_states=query_states,
        relative_pos=relative_pos,
        rel_embeddings=rel_embeddings,
    )
    if output_attentions:
        self_output, att_matrix = self_output
    if query_states is None:
        query_states = hidden_states
    attention_output = self.output(self_output, query_states)

    if output_attentions:
        return (attention_output, att_matrix)
    return attention_output

mindnlp.transformers.models.deberta.modeling_deberta.DebertaEmbeddings

Bases: Module

Construct the embeddings from word, position and token_type embeddings.

Source code in mindnlp/transformers/models/deberta/modeling_deberta.py

class DebertaEmbeddings(nn.Module):
    """Construct the embeddings from word, position and token_type embeddings."""
    def __init__(self, config):
        """
        Initializes the DebertaEmbeddings class.

        Args:
            self (object): Instance of the DebertaEmbeddings class.
            config (object):
                An object containing configuration parameters for the Deberta model.

                - Type: Custom class object.
                - Purpose: Specifies the model configuration including vocab size, hidden size, max position embeddings,
                type vocab size, etc.
                - Restrictions: Must be a valid configuration object.

        Returns:
            None.

        Raises:
            None.
        """
        super().__init__()
        pad_token_id = getattr(config, "pad_token_id", 0)
        self.embedding_size = getattr(config, "embedding_size", config.hidden_size)
        self.word_embeddings = nn.Embedding(config.vocab_size, self.embedding_size, padding_idx=pad_token_id)

        self.position_biased_input = getattr(config, "position_biased_input", True)
        if not self.position_biased_input:
            self.position_embeddings = None
        else:
            self.position_embeddings = nn.Embedding(config.max_position_embeddings, self.embedding_size)

        if config.type_vocab_size > 0:
            self.token_type_embeddings = nn.Embedding(config.type_vocab_size, self.embedding_size)

        if self.embedding_size != config.hidden_size:
            self.embed_proj = nn.Linear(self.embedding_size, config.hidden_size, bias=False)
        self.LayerNorm = DebertaLayerNorm(config.hidden_size, config.layer_norm_eps)
        self.dropout = StableDropout(config.hidden_dropout_prob)
        self.config = config

        # position_ids (1, len position emb) is contiguous in memory and exported when serialized
        self.position_ids = ops.arange(config.max_position_embeddings).expand((1, -1))

    def forward(self, input_ids=None, token_type_ids=None, position_ids=None, mask=None, inputs_embeds=None):
        """
        Constructs the embeddings for the Deberta model.

        Args:
            self (DebertaEmbeddings): An instance of the DebertaEmbeddings class.
            input_ids (Tensor, optional):
                A tensor of shape (batch_size, sequence_length) representing the input token IDs. Default is None.
            token_type_ids (Tensor, optional):
                A tensor of shape (batch_size, sequence_length) representing the token type IDs. Default is None.
            position_ids (Tensor, optional):
                A tensor of shape (batch_size, sequence_length) representing the position IDs. Default is None.
            mask (Tensor, optional):
                A tensor of shape (batch_size, sequence_length) representing the attention mask. Default is None.
            inputs_embeds (Tensor, optional):
                A tensor of shape (batch_size, sequence_length, embedding_size) representing the input embeddings.
                Default is None.

        Returns:
            Tensor:
                A tensor of shape (batch_size, sequence_length, embedding_size) representing the forwarded embeddings.

        Raises:
            None.
        """
        if input_ids is not None:
            input_shape = input_ids.shape
        else:
            input_shape = inputs_embeds.shape[:-1]

        seq_length = input_shape[1]

        if position_ids is None:
            position_ids = self.position_ids[:, :seq_length]

        if token_type_ids is None:
            token_type_ids = ops.zeros(input_shape, dtype=mindspore.int64)

        if inputs_embeds is None:
            inputs_embeds = self.word_embeddings(input_ids)

        if self.position_embeddings is not None:
            position_embeddings = self.position_embeddings(position_ids.long())
        else:
            position_embeddings = ops.zeros_like(inputs_embeds)

        embeddings = inputs_embeds
        if self.position_biased_input:
            embeddings += position_embeddings
        if self.config.type_vocab_size > 0:
            token_type_embeddings = self.token_type_embeddings(token_type_ids)
            embeddings += token_type_embeddings

        if self.embedding_size != self.config.hidden_size:
            embeddings = self.embed_proj(embeddings)

        embeddings = self.LayerNorm(embeddings)

        if mask is not None:
            if mask.ndim != embeddings.ndim:
                if mask.ndim == 4:
                    mask = mask.squeeze(1).squeeze(1)
                mask = mask.unsqueeze(2)
            mask = mask.to(embeddings.dtype)

            embeddings = embeddings * mask

        embeddings = self.dropout(embeddings)
        return embeddings

mindnlp.transformers.models.deberta.modeling_deberta.DebertaEmbeddings.__init__(config)

Initializes the DebertaEmbeddings class.

PARAMETER	DESCRIPTION
self	Instance of the DebertaEmbeddings class. TYPE: object
config	An object containing configuration parameters for the Deberta model. Type: Custom class object. Purpose: Specifies the model configuration including vocab size, hidden size, max position embeddings, type vocab size, etc. Restrictions: Must be a valid configuration object. TYPE: object

RETURNS	DESCRIPTION
	None.

Source code in mindnlp/transformers/models/deberta/modeling_deberta.py

def __init__(self, config):
    """
    Initializes the DebertaEmbeddings class.

    Args:
        self (object): Instance of the DebertaEmbeddings class.
        config (object):
            An object containing configuration parameters for the Deberta model.

            - Type: Custom class object.
            - Purpose: Specifies the model configuration including vocab size, hidden size, max position embeddings,
            type vocab size, etc.
            - Restrictions: Must be a valid configuration object.

    Returns:
        None.

    Raises:
        None.
    """
    super().__init__()
    pad_token_id = getattr(config, "pad_token_id", 0)
    self.embedding_size = getattr(config, "embedding_size", config.hidden_size)
    self.word_embeddings = nn.Embedding(config.vocab_size, self.embedding_size, padding_idx=pad_token_id)

    self.position_biased_input = getattr(config, "position_biased_input", True)
    if not self.position_biased_input:
        self.position_embeddings = None
    else:
        self.position_embeddings = nn.Embedding(config.max_position_embeddings, self.embedding_size)

    if config.type_vocab_size > 0:
        self.token_type_embeddings = nn.Embedding(config.type_vocab_size, self.embedding_size)

    if self.embedding_size != config.hidden_size:
        self.embed_proj = nn.Linear(self.embedding_size, config.hidden_size, bias=False)
    self.LayerNorm = DebertaLayerNorm(config.hidden_size, config.layer_norm_eps)
    self.dropout = StableDropout(config.hidden_dropout_prob)
    self.config = config

    # position_ids (1, len position emb) is contiguous in memory and exported when serialized
    self.position_ids = ops.arange(config.max_position_embeddings).expand((1, -1))

mindnlp.transformers.models.deberta.modeling_deberta.DebertaEmbeddings.forward(input_ids=None, token_type_ids=None, position_ids=None, mask=None, inputs_embeds=None)

Constructs the embeddings for the Deberta model.

PARAMETER	DESCRIPTION
self	An instance of the DebertaEmbeddings class. TYPE: DebertaEmbeddings
input_ids	A tensor of shape (batch_size, sequence_length) representing the input token IDs. Default is None. TYPE: Tensor DEFAULT: None
token_type_ids	A tensor of shape (batch_size, sequence_length) representing the token type IDs. Default is None. TYPE: Tensor DEFAULT: None
position_ids	A tensor of shape (batch_size, sequence_length) representing the position IDs. Default is None. TYPE: Tensor DEFAULT: None
mask	A tensor of shape (batch_size, sequence_length) representing the attention mask. Default is None. TYPE: Tensor DEFAULT: None
inputs_embeds	A tensor of shape (batch_size, sequence_length, embedding_size) representing the input embeddings. Default is None. TYPE: Tensor DEFAULT: None

RETURNS	DESCRIPTION
Tensor	A tensor of shape (batch_size, sequence_length, embedding_size) representing the forwarded embeddings.

Source code in mindnlp/transformers/models/deberta/modeling_deberta.py

def forward(self, input_ids=None, token_type_ids=None, position_ids=None, mask=None, inputs_embeds=None):
    """
    Constructs the embeddings for the Deberta model.

    Args:
        self (DebertaEmbeddings): An instance of the DebertaEmbeddings class.
        input_ids (Tensor, optional):
            A tensor of shape (batch_size, sequence_length) representing the input token IDs. Default is None.
        token_type_ids (Tensor, optional):
            A tensor of shape (batch_size, sequence_length) representing the token type IDs. Default is None.
        position_ids (Tensor, optional):
            A tensor of shape (batch_size, sequence_length) representing the position IDs. Default is None.
        mask (Tensor, optional):
            A tensor of shape (batch_size, sequence_length) representing the attention mask. Default is None.
        inputs_embeds (Tensor, optional):
            A tensor of shape (batch_size, sequence_length, embedding_size) representing the input embeddings.
            Default is None.

    Returns:
        Tensor:
            A tensor of shape (batch_size, sequence_length, embedding_size) representing the forwarded embeddings.

    Raises:
        None.
    """
    if input_ids is not None:
        input_shape = input_ids.shape
    else:
        input_shape = inputs_embeds.shape[:-1]

    seq_length = input_shape[1]

    if position_ids is None:
        position_ids = self.position_ids[:, :seq_length]

    if token_type_ids is None:
        token_type_ids = ops.zeros(input_shape, dtype=mindspore.int64)

    if inputs_embeds is None:
        inputs_embeds = self.word_embeddings(input_ids)

    if self.position_embeddings is not None:
        position_embeddings = self.position_embeddings(position_ids.long())
    else:
        position_embeddings = ops.zeros_like(inputs_embeds)

    embeddings = inputs_embeds
    if self.position_biased_input:
        embeddings += position_embeddings
    if self.config.type_vocab_size > 0:
        token_type_embeddings = self.token_type_embeddings(token_type_ids)
        embeddings += token_type_embeddings

    if self.embedding_size != self.config.hidden_size:
        embeddings = self.embed_proj(embeddings)

    embeddings = self.LayerNorm(embeddings)

    if mask is not None:
        if mask.ndim != embeddings.ndim:
            if mask.ndim == 4:
                mask = mask.squeeze(1).squeeze(1)
            mask = mask.unsqueeze(2)
        mask = mask.to(embeddings.dtype)

        embeddings = embeddings * mask

    embeddings = self.dropout(embeddings)
    return embeddings

mindnlp.transformers.models.deberta.modeling_deberta.DebertaEncoder

Bases: Module

Modified BertEncoder with relative position bias support

Source code in mindnlp/transformers/models/deberta/modeling_deberta.py

class DebertaEncoder(nn.Module):
    """Modified BertEncoder with relative position bias support"""
    def __init__(self, config):
        """
        Initialize the DebertaEncoder class with the provided configuration.

        Args:
            self (DebertaEncoder): The instance of the DebertaEncoder class.
            config (object):
                An object containing configuration settings for the DebertaEncoder.

                - The configuration should include the following attributes:

                    - num_hidden_layers (int): Number of hidden layers.
                    - relative_attention (bool): Flag indicating whether relative attention is used.
                    - max_relative_positions (int): Maximum number of relative positions.
                        If not provided or less than 1, defaults to config.max_position_embeddings.
                    - hidden_size (int): Size of the hidden layer.
                    - max_position_embeddings (int): Maximum number of position embeddings.

        Returns:
            None.

        Raises:
            None.
        """
        super().__init__()
        self.layer = nn.ModuleList([DebertaLayer(config) for _ in range(config.num_hidden_layers)])
        self.relative_attention = getattr(config, "relative_attention", False)
        if self.relative_attention:
            self.max_relative_positions = getattr(config, "max_relative_positions", -1)
            if self.max_relative_positions < 1:
                self.max_relative_positions = config.max_position_embeddings
            self.rel_embeddings = nn.Embedding(self.max_relative_positions * 2, config.hidden_size)
        self.gradient_checkpointing = False

    def get_rel_embedding(self):
        """
        Retrieve the relative embeddings from the DebertaEncoder.

        Args:
            self (DebertaEncoder): The instance of the DebertaEncoder class.

        Returns:
            None: Returns the relative embeddings if self.relative_attention is True, otherwise returns None.

        Raises:
            None.
        """
        rel_embeddings = self.rel_embeddings.weight if self.relative_attention else None
        return rel_embeddings

    def get_attention_mask(self, attention_mask):
        """
        This method calculates the attention mask for the DebertaEncoder.

        Args:
            self (object): The instance of the DebertaEncoder class.
            attention_mask (tensor): The attention mask tensor.
                It can be of dimension 2 or 3. For a 2-dimensional tensor, it is expected to be of shape
                (batch_size, sequence_length) representing the attention mask for each token in the input sequence.
                For a 3-dimensional tensor, it is expected to be of shape (batch_size, num_heads, sequence_length)
                representing the attention mask for each head in the multi-head attention mechanism.

        Returns:
            None: This method does not return any value. The attention_mask parameter is modified in place.

        Raises:
            ValueError: If the attention_mask tensor is not of dimension 2 or 3, a ValueError is raised.
            RuntimeError: If there is a runtime error during the calculation, a RuntimeError may be raised.
        """
        if attention_mask.ndim <= 2:
            extended_attention_mask = attention_mask.unsqueeze(1).unsqueeze(2)
            attention_mask = extended_attention_mask * extended_attention_mask.squeeze(-2).unsqueeze(-1)
        elif attention_mask.ndim == 3:
            attention_mask = attention_mask.unsqueeze(1)

        return attention_mask

    def get_rel_pos(self, hidden_states, query_states=None, relative_pos=None):
        """
        Method:
            get_rel_pos

        Description:
            This method calculates and returns the relative position tensor used for relative attention in the
            DebertaEncoder class.

        Args:
            self (DebertaEncoder): The instance of the DebertaEncoder class.
            hidden_states (Tensor): The input tensor representing the hidden states.
            query_states (Tensor, optional): The input tensor representing the query states. Default is None.
            relative_pos (Tensor, optional): The input tensor representing the relative positions. Default is None.

        Returns:
            None

        Raises:
            None

        Note:
            The 'query_states' and 'relative_pos' parameters are optional.
            If 'relative_attention' is True and 'relative_pos' is not provided,
            this method will automatically build the relative position tensor using 'query_states' or
            'hidden_states' shape.

        Example:
            ```python
            >>> # Create an instance of DebertaEncoder class
            >>> encoder = DebertaEncoder()
            ...
            >>> # Call the get_rel_pos method
            >>> encoder.get_rel_pos(hidden_states, query_states, relative_pos)
            ```
        """
        if self.relative_attention and relative_pos is None:
            q = query_states.shape[-2] if query_states is not None else hidden_states.shape[-2]
            relative_pos = build_relative_position(q, hidden_states.shape[-2])
        return relative_pos

    def forward(
        self,
        hidden_states,
        attention_mask,
        output_hidden_states=True,
        output_attentions=False,
        query_states=None,
        relative_pos=None,
        return_dict=True,
    ):
        """
        This method forwards the DebertaEncoder by processing the input hidden states and attention mask.

        Args:
            self (object): The instance of the DebertaEncoder class.
            hidden_states (Sequence or object): The input hidden states for the encoder.
                It can be a Sequence of hidden states or a single hidden state object.
            attention_mask (Tensor): The attention mask to be applied to the input hidden states.
            output_hidden_states (bool, optional): Indicates whether to return all hidden states. Defaults to True.
            output_attentions (bool, optional): Indicates whether to return attentions. Defaults to False.
            query_states (object, optional): The query states for the encoder. Defaults to None.
            relative_pos (object, optional): The relative position information. Defaults to None.
            return_dict (bool, optional): Indicates whether to return the output as a BaseModelOutput instance.
                Defaults to True.

        Returns:
            None.

        Raises:
            ValueError: If the input parameters are invalid or incompatible.
            RuntimeError: If there is a runtime error during the execution of the method.
            TypeError: If the input types are incorrect or incompatible.
        """
        attention_mask = self.get_attention_mask(attention_mask)
        relative_pos = self.get_rel_pos(hidden_states, query_states, relative_pos)

        all_hidden_states = () if output_hidden_states else None
        all_attentions = () if output_attentions else None

        if isinstance(hidden_states, Sequence):
            next_kv = hidden_states[0]
        else:
            next_kv = hidden_states
        rel_embeddings = self.get_rel_embedding()
        for i, layer_module in enumerate(self.layer):
            if output_hidden_states:
                all_hidden_states = all_hidden_states + (hidden_states,)

            if self.gradient_checkpointing and self.training:
                hidden_states = self._gradient_checkpointing_func(
                    layer_module.__call__,
                    next_kv,
                    attention_mask,
                    query_states,
                    relative_pos,
                    rel_embeddings,
                    output_attentions,
                )
            else:
                hidden_states = layer_module(
                    next_kv,
                    attention_mask,
                    query_states=query_states,
                    relative_pos=relative_pos,
                    rel_embeddings=rel_embeddings,
                    output_attentions=output_attentions,
                )

            if output_attentions:
                hidden_states, att_m = hidden_states

            if query_states is not None:
                query_states = hidden_states
                if isinstance(hidden_states, Sequence):
                    next_kv = hidden_states[i + 1] if i + 1 < len(self.layer) else None
            else:
                next_kv = hidden_states

            if output_attentions:
                all_attentions = all_attentions + (att_m,)

        if output_hidden_states:
            all_hidden_states = all_hidden_states + (hidden_states,)

        if not return_dict:
            return tuple(v for v in [hidden_states, all_hidden_states, all_attentions] if v is not None)
        return BaseModelOutput(
            last_hidden_state=hidden_states, hidden_states=all_hidden_states, attentions=all_attentions
        )

mindnlp.transformers.models.deberta.modeling_deberta.DebertaEncoder.__init__(config)

Initialize the DebertaEncoder class with the provided configuration.

PARAMETER	DESCRIPTION
self	The instance of the DebertaEncoder class. TYPE: DebertaEncoder
config	An object containing configuration settings for the DebertaEncoder. The configuration should include the following attributes: num_hidden_layers (int): Number of hidden layers. relative_attention (bool): Flag indicating whether relative attention is used. max_relative_positions (int): Maximum number of relative positions. If not provided or less than 1, defaults to config.max_position_embeddings. hidden_size (int): Size of the hidden layer. max_position_embeddings (int): Maximum number of position embeddings. TYPE: object

RETURNS	DESCRIPTION
	None.

Source code in mindnlp/transformers/models/deberta/modeling_deberta.py

def __init__(self, config):
    """
    Initialize the DebertaEncoder class with the provided configuration.

    Args:
        self (DebertaEncoder): The instance of the DebertaEncoder class.
        config (object):
            An object containing configuration settings for the DebertaEncoder.

            - The configuration should include the following attributes:

                - num_hidden_layers (int): Number of hidden layers.
                - relative_attention (bool): Flag indicating whether relative attention is used.
                - max_relative_positions (int): Maximum number of relative positions.
                    If not provided or less than 1, defaults to config.max_position_embeddings.
                - hidden_size (int): Size of the hidden layer.
                - max_position_embeddings (int): Maximum number of position embeddings.

    Returns:
        None.

    Raises:
        None.
    """
    super().__init__()
    self.layer = nn.ModuleList([DebertaLayer(config) for _ in range(config.num_hidden_layers)])
    self.relative_attention = getattr(config, "relative_attention", False)
    if self.relative_attention:
        self.max_relative_positions = getattr(config, "max_relative_positions", -1)
        if self.max_relative_positions < 1:
            self.max_relative_positions = config.max_position_embeddings
        self.rel_embeddings = nn.Embedding(self.max_relative_positions * 2, config.hidden_size)
    self.gradient_checkpointing = False

mindnlp.transformers.models.deberta.modeling_deberta.DebertaEncoder.forward(hidden_states, attention_mask, output_hidden_states=True, output_attentions=False, query_states=None, relative_pos=None, return_dict=True)

This method forwards the DebertaEncoder by processing the input hidden states and attention mask.

PARAMETER	DESCRIPTION
self	The instance of the DebertaEncoder class. TYPE: object
hidden_states	The input hidden states for the encoder. It can be a Sequence of hidden states or a single hidden state object. TYPE: Sequence or object
attention_mask	The attention mask to be applied to the input hidden states. TYPE: Tensor
output_hidden_states	Indicates whether to return all hidden states. Defaults to True. TYPE: bool DEFAULT: True
output_attentions	Indicates whether to return attentions. Defaults to False. TYPE: bool DEFAULT: False
query_states	The query states for the encoder. Defaults to None. TYPE: object DEFAULT: None
relative_pos	The relative position information. Defaults to None. TYPE: object DEFAULT: None
return_dict	Indicates whether to return the output as a BaseModelOutput instance. Defaults to True. TYPE: bool DEFAULT: True

RETURNS	DESCRIPTION
	None.

RAISES	DESCRIPTION
ValueError	If the input parameters are invalid or incompatible.
RuntimeError	If there is a runtime error during the execution of the method.
TypeError	If the input types are incorrect or incompatible.

Source code in mindnlp/transformers/models/deberta/modeling_deberta.py

def forward(
    self,
    hidden_states,
    attention_mask,
    output_hidden_states=True,
    output_attentions=False,
    query_states=None,
    relative_pos=None,
    return_dict=True,
):
    """
    This method forwards the DebertaEncoder by processing the input hidden states and attention mask.

    Args:
        self (object): The instance of the DebertaEncoder class.
        hidden_states (Sequence or object): The input hidden states for the encoder.
            It can be a Sequence of hidden states or a single hidden state object.
        attention_mask (Tensor): The attention mask to be applied to the input hidden states.
        output_hidden_states (bool, optional): Indicates whether to return all hidden states. Defaults to True.
        output_attentions (bool, optional): Indicates whether to return attentions. Defaults to False.
        query_states (object, optional): The query states for the encoder. Defaults to None.
        relative_pos (object, optional): The relative position information. Defaults to None.
        return_dict (bool, optional): Indicates whether to return the output as a BaseModelOutput instance.
            Defaults to True.

    Returns:
        None.

    Raises:
        ValueError: If the input parameters are invalid or incompatible.
        RuntimeError: If there is a runtime error during the execution of the method.
        TypeError: If the input types are incorrect or incompatible.
    """
    attention_mask = self.get_attention_mask(attention_mask)
    relative_pos = self.get_rel_pos(hidden_states, query_states, relative_pos)

    all_hidden_states = () if output_hidden_states else None
    all_attentions = () if output_attentions else None

    if isinstance(hidden_states, Sequence):
        next_kv = hidden_states[0]
    else:
        next_kv = hidden_states
    rel_embeddings = self.get_rel_embedding()
    for i, layer_module in enumerate(self.layer):
        if output_hidden_states:
            all_hidden_states = all_hidden_states + (hidden_states,)

        if self.gradient_checkpointing and self.training:
            hidden_states = self._gradient_checkpointing_func(
                layer_module.__call__,
                next_kv,
                attention_mask,
                query_states,
                relative_pos,
                rel_embeddings,
                output_attentions,
            )
        else:
            hidden_states = layer_module(
                next_kv,
                attention_mask,
                query_states=query_states,
                relative_pos=relative_pos,
                rel_embeddings=rel_embeddings,
                output_attentions=output_attentions,
            )

        if output_attentions:
            hidden_states, att_m = hidden_states

        if query_states is not None:
            query_states = hidden_states
            if isinstance(hidden_states, Sequence):
                next_kv = hidden_states[i + 1] if i + 1 < len(self.layer) else None
        else:
            next_kv = hidden_states

        if output_attentions:
            all_attentions = all_attentions + (att_m,)

    if output_hidden_states:
        all_hidden_states = all_hidden_states + (hidden_states,)

    if not return_dict:
        return tuple(v for v in [hidden_states, all_hidden_states, all_attentions] if v is not None)
    return BaseModelOutput(
        last_hidden_state=hidden_states, hidden_states=all_hidden_states, attentions=all_attentions
    )

mindnlp.transformers.models.deberta.modeling_deberta.DebertaEncoder.get_attention_mask(attention_mask)

This method calculates the attention mask for the DebertaEncoder.

PARAMETER	DESCRIPTION
self	The instance of the DebertaEncoder class. TYPE: object
attention_mask	The attention mask tensor. It can be of dimension 2 or 3. For a 2-dimensional tensor, it is expected to be of shape (batch_size, sequence_length) representing the attention mask for each token in the input sequence. For a 3-dimensional tensor, it is expected to be of shape (batch_size, num_heads, sequence_length) representing the attention mask for each head in the multi-head attention mechanism. TYPE: tensor

RETURNS	DESCRIPTION
None	This method does not return any value. The attention_mask parameter is modified in place.

RAISES	DESCRIPTION
ValueError	If the attention_mask tensor is not of dimension 2 or 3, a ValueError is raised.
RuntimeError	If there is a runtime error during the calculation, a RuntimeError may be raised.

Source code in mindnlp/transformers/models/deberta/modeling_deberta.py

def get_attention_mask(self, attention_mask):
    """
    This method calculates the attention mask for the DebertaEncoder.

    Args:
        self (object): The instance of the DebertaEncoder class.
        attention_mask (tensor): The attention mask tensor.
            It can be of dimension 2 or 3. For a 2-dimensional tensor, it is expected to be of shape
            (batch_size, sequence_length) representing the attention mask for each token in the input sequence.
            For a 3-dimensional tensor, it is expected to be of shape (batch_size, num_heads, sequence_length)
            representing the attention mask for each head in the multi-head attention mechanism.

    Returns:
        None: This method does not return any value. The attention_mask parameter is modified in place.

    Raises:
        ValueError: If the attention_mask tensor is not of dimension 2 or 3, a ValueError is raised.
        RuntimeError: If there is a runtime error during the calculation, a RuntimeError may be raised.
    """
    if attention_mask.ndim <= 2:
        extended_attention_mask = attention_mask.unsqueeze(1).unsqueeze(2)
        attention_mask = extended_attention_mask * extended_attention_mask.squeeze(-2).unsqueeze(-1)
    elif attention_mask.ndim == 3:
        attention_mask = attention_mask.unsqueeze(1)

    return attention_mask

mindnlp.transformers.models.deberta.modeling_deberta.DebertaEncoder.get_rel_embedding()

Retrieve the relative embeddings from the DebertaEncoder.

PARAMETER	DESCRIPTION
self	The instance of the DebertaEncoder class. TYPE: DebertaEncoder

RETURNS	DESCRIPTION
None	Returns the relative embeddings if self.relative_attention is True, otherwise returns None.

Source code in mindnlp/transformers/models/deberta/modeling_deberta.py

def get_rel_embedding(self):
    """
    Retrieve the relative embeddings from the DebertaEncoder.

    Args:
        self (DebertaEncoder): The instance of the DebertaEncoder class.

    Returns:
        None: Returns the relative embeddings if self.relative_attention is True, otherwise returns None.

    Raises:
        None.
    """
    rel_embeddings = self.rel_embeddings.weight if self.relative_attention else None
    return rel_embeddings

mindnlp.transformers.models.deberta.modeling_deberta.DebertaEncoder.get_rel_pos(hidden_states, query_states=None, relative_pos=None)

Method

get_rel_pos

Description

This method calculates and returns the relative position tensor used for relative attention in the DebertaEncoder class.

PARAMETER	DESCRIPTION
self	The instance of the DebertaEncoder class. TYPE: DebertaEncoder
hidden_states	The input tensor representing the hidden states. TYPE: Tensor
query_states	The input tensor representing the query states. Default is None. TYPE: Tensor DEFAULT: None
relative_pos	The input tensor representing the relative positions. Default is None. TYPE: Tensor DEFAULT: None

RETURNS	DESCRIPTION
	None

Note

The 'query_states' and 'relative_pos' parameters are optional. If 'relative_attention' is True and 'relative_pos' is not provided, this method will automatically build the relative position tensor using 'query_states' or 'hidden_states' shape.

Example

>>> # Create an instance of DebertaEncoder class
>>> encoder = DebertaEncoder()
...
>>> # Call the get_rel_pos method
>>> encoder.get_rel_pos(hidden_states, query_states, relative_pos)

Source code in mindnlp/transformers/models/deberta/modeling_deberta.py

def get_rel_pos(self, hidden_states, query_states=None, relative_pos=None):
    """
    Method:
        get_rel_pos

    Description:
        This method calculates and returns the relative position tensor used for relative attention in the
        DebertaEncoder class.

    Args:
        self (DebertaEncoder): The instance of the DebertaEncoder class.
        hidden_states (Tensor): The input tensor representing the hidden states.
        query_states (Tensor, optional): The input tensor representing the query states. Default is None.
        relative_pos (Tensor, optional): The input tensor representing the relative positions. Default is None.

    Returns:
        None

    Raises:
        None

    Note:
        The 'query_states' and 'relative_pos' parameters are optional.
        If 'relative_attention' is True and 'relative_pos' is not provided,
        this method will automatically build the relative position tensor using 'query_states' or
        'hidden_states' shape.

    Example:
        ```python
        >>> # Create an instance of DebertaEncoder class
        >>> encoder = DebertaEncoder()
        ...
        >>> # Call the get_rel_pos method
        >>> encoder.get_rel_pos(hidden_states, query_states, relative_pos)
        ```
    """
    if self.relative_attention and relative_pos is None:
        q = query_states.shape[-2] if query_states is not None else hidden_states.shape[-2]
        relative_pos = build_relative_position(q, hidden_states.shape[-2])
    return relative_pos

mindnlp.transformers.models.deberta.modeling_deberta.DebertaForMaskedLM

Bases: DebertaPreTrainedModel

DebertaForMaskedLM is a class that represents a DeBERTa model for masked language modeling. This class is designed to be used for generating predictions and computing loss in a masked language modeling task. It inherits from DebertaPreTrainedModel, providing additional functionality specific to masked language modeling tasks.

ATTRIBUTE	DESCRIPTION
deberta	A DebertaModel instance used for processing input sequences.
cls	A DebertaOnlyMLMHead instance responsible for generating prediction scores for masked tokens.

METHOD	DESCRIPTION
get_output_embeddings	Retrieves the decoder embeddings used for output predictions.
set_output_embeddings	Sets new decoder embeddings for output predictions.
forward	Constructs the DeBERTa model for masked language modeling, including processing input data, generating predictions, and computing the masked language modeling loss.

The 'forward' method takes various input parameters such as input_ids, attention_mask, labels, etc., and returns a MaskedLMOutput object containing the loss, prediction scores, hidden states, and attentions. It also allows for customization of return types based on the 'return_dict' parameter.

Note

Ensure proper input data formatting as described in the docstring of the 'forward' method for accurate predictions and loss computation.

Source code in mindnlp/transformers/models/deberta/modeling_deberta.py

class DebertaForMaskedLM(DebertaPreTrainedModel):

    """
    DebertaForMaskedLM is a class that represents a DeBERTa model for masked language modeling.
    This class is designed to be used for generating predictions and computing loss in a masked language modeling task.
    It inherits from DebertaPreTrainedModel, providing additional functionality specific to masked language modeling tasks.

    Attributes:
        deberta: A DebertaModel instance used for processing input sequences.
        cls: A DebertaOnlyMLMHead instance responsible for generating prediction scores for masked tokens.

    Methods:
        get_output_embeddings: Retrieves the decoder embeddings used for output predictions.
        set_output_embeddings: Sets new decoder embeddings for output predictions.
        forward: Constructs the DeBERTa model for masked language modeling, including processing input data,
            generating predictions, and computing the masked language modeling loss.

    The 'forward' method takes various input parameters such as input_ids, attention_mask, labels, etc., and returns
    a MaskedLMOutput object containing the loss, prediction scores, hidden states, and attentions.
    It also allows for customization of return types based on the 'return_dict' parameter.

    Note:
        Ensure proper input data formatting as described in the docstring of the 'forward' method for accurate
        predictions and loss computation.
    """
    _tied_weights_keys = ["cls.predictions.decoder.weight", "cls.predictions.decoder.bias"]

    def __init__(self, config):
        """
        Initialize the DebertaForMaskedLM class.

        Args:
            self (DebertaForMaskedLM): The instance of the DebertaForMaskedLM class.
            config (object): The configuration object containing parameters for the Deberta model.

        Returns:
            None.

        Raises:
            TypeError: If the config parameter is not provided or is of an incorrect type.
            ValueError: If the config object is missing required attributes.
        """
        super().__init__(config)

        self.deberta = DebertaModel(config)
        self.cls = DebertaOnlyMLMHead(config)

        # Initialize weights and apply final processing
        self.post_init()

    def get_output_embeddings(self):
        """
        Retrieve the output embeddings from the DebertaForMaskedLM model.

        Args:
            self (DebertaForMaskedLM): The instance of the DebertaForMaskedLM class.

        Returns:
            decoder: This method returns the output embeddings obtained from the predictions decoder of the model.

        Raises:
            None.
        """
        return self.cls.predictions.decoder

    def set_output_embeddings(self, new_embeddings):
        """
        Sets the output embeddings for the DebertaForMaskedLM model.

        Args:
            self (DebertaForMaskedLM): The instance of the DebertaForMaskedLM class.
            new_embeddings (Tensor): The new embeddings to be set as the output embeddings.
                It should be of shape (vocab_size, hidden_size).

        Returns:
            None.

        Raises:
            None.
        """
        self.cls.predictions.decoder = new_embeddings

    def forward(
        self,
        input_ids: Optional[mindspore.Tensor] = None,
        attention_mask: Optional[mindspore.Tensor] = None,
        token_type_ids: Optional[mindspore.Tensor] = None,
        position_ids: Optional[mindspore.Tensor] = None,
        inputs_embeds: Optional[mindspore.Tensor] = None,
        labels: Optional[mindspore.Tensor] = None,
        output_attentions: Optional[bool] = None,
        output_hidden_states: Optional[bool] = None,
        return_dict: Optional[bool] = None,
    ) -> Union[Tuple, MaskedLMOutput]:
        r"""
        Args:
            labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*):
                Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ...,
                config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the
                loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`
        """
        return_dict = return_dict if return_dict is not None else self.config.use_return_dict

        outputs = self.deberta(
            input_ids,
            attention_mask=attention_mask,
            token_type_ids=token_type_ids,
            position_ids=position_ids,
            inputs_embeds=inputs_embeds,
            output_attentions=output_attentions,
            output_hidden_states=output_hidden_states,
            return_dict=return_dict,
        )

        sequence_output = outputs[0]
        prediction_scores = self.cls(sequence_output)

        masked_lm_loss = None
        if labels is not None:
            masked_lm_loss = ops.cross_entropy(prediction_scores.view(-1, self.config.vocab_size), labels.view(-1))

        if not return_dict:
            output = (prediction_scores,) + outputs[1:]
            return ((masked_lm_loss,) + output) if masked_lm_loss is not None else output

        return MaskedLMOutput(
            loss=masked_lm_loss,
            logits=prediction_scores,
            hidden_states=outputs.hidden_states,
            attentions=outputs.attentions,
        )

mindnlp.transformers.models.deberta.modeling_deberta.DebertaForMaskedLM.__init__(config)

Initialize the DebertaForMaskedLM class.

PARAMETER	DESCRIPTION
self	The instance of the DebertaForMaskedLM class. TYPE: DebertaForMaskedLM
config	The configuration object containing parameters for the Deberta model. TYPE: object

RETURNS	DESCRIPTION
	None.

RAISES	DESCRIPTION
TypeError	If the config parameter is not provided or is of an incorrect type.
ValueError	If the config object is missing required attributes.

Source code in mindnlp/transformers/models/deberta/modeling_deberta.py

def __init__(self, config):
    """
    Initialize the DebertaForMaskedLM class.

    Args:
        self (DebertaForMaskedLM): The instance of the DebertaForMaskedLM class.
        config (object): The configuration object containing parameters for the Deberta model.

    Returns:
        None.

    Raises:
        TypeError: If the config parameter is not provided or is of an incorrect type.
        ValueError: If the config object is missing required attributes.
    """
    super().__init__(config)

    self.deberta = DebertaModel(config)
    self.cls = DebertaOnlyMLMHead(config)

    # Initialize weights and apply final processing
    self.post_init()

mindnlp.transformers.models.deberta.modeling_deberta.DebertaForMaskedLM.forward(input_ids=None, attention_mask=None, token_type_ids=None, position_ids=None, inputs_embeds=None, labels=None, output_attentions=None, output_hidden_states=None, return_dict=None)

PARAMETER	DESCRIPTION
labels	Labels for computing the masked language modeling loss. Indices should be in [-100, 0, ..., config.vocab_size] (see input_ids docstring) Tokens with indices set to -100 are ignored (masked), the loss is only computed for the tokens with labels in [0, ..., config.vocab_size] TYPE: `torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional* DEFAULT: None

Source code in mindnlp/transformers/models/deberta/modeling_deberta.py

def forward(
    self,
    input_ids: Optional[mindspore.Tensor] = None,
    attention_mask: Optional[mindspore.Tensor] = None,
    token_type_ids: Optional[mindspore.Tensor] = None,
    position_ids: Optional[mindspore.Tensor] = None,
    inputs_embeds: Optional[mindspore.Tensor] = None,
    labels: Optional[mindspore.Tensor] = None,
    output_attentions: Optional[bool] = None,
    output_hidden_states: Optional[bool] = None,
    return_dict: Optional[bool] = None,
) -> Union[Tuple, MaskedLMOutput]:
    r"""
    Args:
        labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*):
            Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ...,
            config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the
            loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`
    """
    return_dict = return_dict if return_dict is not None else self.config.use_return_dict

    outputs = self.deberta(
        input_ids,
        attention_mask=attention_mask,
        token_type_ids=token_type_ids,
        position_ids=position_ids,
        inputs_embeds=inputs_embeds,
        output_attentions=output_attentions,
        output_hidden_states=output_hidden_states,
        return_dict=return_dict,
    )

    sequence_output = outputs[0]
    prediction_scores = self.cls(sequence_output)

    masked_lm_loss = None
    if labels is not None:
        masked_lm_loss = ops.cross_entropy(prediction_scores.view(-1, self.config.vocab_size), labels.view(-1))

    if not return_dict:
        output = (prediction_scores,) + outputs[1:]
        return ((masked_lm_loss,) + output) if masked_lm_loss is not None else output

    return MaskedLMOutput(
        loss=masked_lm_loss,
        logits=prediction_scores,
        hidden_states=outputs.hidden_states,
        attentions=outputs.attentions,
    )

mindnlp.transformers.models.deberta.modeling_deberta.DebertaForMaskedLM.get_output_embeddings()

Retrieve the output embeddings from the DebertaForMaskedLM model.

PARAMETER	DESCRIPTION
self	The instance of the DebertaForMaskedLM class. TYPE: DebertaForMaskedLM

RETURNS	DESCRIPTION
decoder	This method returns the output embeddings obtained from the predictions decoder of the model.

Source code in mindnlp/transformers/models/deberta/modeling_deberta.py

def get_output_embeddings(self):
    """
    Retrieve the output embeddings from the DebertaForMaskedLM model.

    Args:
        self (DebertaForMaskedLM): The instance of the DebertaForMaskedLM class.

    Returns:
        decoder: This method returns the output embeddings obtained from the predictions decoder of the model.

    Raises:
        None.
    """
    return self.cls.predictions.decoder

mindnlp.transformers.models.deberta.modeling_deberta.DebertaForMaskedLM.set_output_embeddings(new_embeddings)

Sets the output embeddings for the DebertaForMaskedLM model.

PARAMETER	DESCRIPTION
self	The instance of the DebertaForMaskedLM class. TYPE: DebertaForMaskedLM
new_embeddings	The new embeddings to be set as the output embeddings. It should be of shape (vocab_size, hidden_size). TYPE: Tensor

RETURNS	DESCRIPTION
	None.

Source code in mindnlp/transformers/models/deberta/modeling_deberta.py

def set_output_embeddings(self, new_embeddings):
    """
    Sets the output embeddings for the DebertaForMaskedLM model.

    Args:
        self (DebertaForMaskedLM): The instance of the DebertaForMaskedLM class.
        new_embeddings (Tensor): The new embeddings to be set as the output embeddings.
            It should be of shape (vocab_size, hidden_size).

    Returns:
        None.

    Raises:
        None.
    """
    self.cls.predictions.decoder = new_embeddings

mindnlp.transformers.models.deberta.modeling_deberta.DebertaForQuestionAnswering

Bases: DebertaPreTrainedModel

This class represents a Deberta model for question answering tasks. It inherits functionality from the DebertaPreTrainedModel class. The DebertaForQuestionAnswering class includes methods for initializing the model with configuration, and for forwarding the model by processing input data and producing question answering model outputs. The forward method takes various input tensors such as input_ids, attention_mask, token_type_ids, position_ids, and inputs_embeds, and returns QuestionAnsweringModelOutput. It also supports optional parameters for controlling the output format and behavior. The class provides detailed documentation for the forward method, including explanations of the input and output parameters and their respective shapes and types. Additionally, the class handles the computation of total loss for question answering tasks based on start and end positions, and returns the final model outputs as a QuestionAnsweringModelOutput object.

Source code in mindnlp/transformers/models/deberta/modeling_deberta.py

class DebertaForQuestionAnswering(DebertaPreTrainedModel):

    """
    This class represents a Deberta model for question answering tasks. It inherits functionality from the
    DebertaPreTrainedModel class.
    The DebertaForQuestionAnswering class includes methods for initializing the model with configuration,
    and for forwarding the model by processing input data and producing question answering model outputs.
    The forward method takes various input tensors such as input_ids, attention_mask, token_type_ids, position_ids,
    and inputs_embeds, and returns QuestionAnsweringModelOutput.
    It also supports optional parameters for controlling the output format and behavior.
    The class provides detailed documentation for the forward method, including explanations of the input and output
    parameters and their respective shapes and types.
    Additionally, the class handles the computation of total loss for question answering tasks based on start
    and end positions, and returns the final model outputs as a QuestionAnsweringModelOutput object.
    """
    def __init__(self, config):
        """
        Initializes a new instance of the DebertaForQuestionAnswering class.

        Args:
            self: The instance of the class.
            config: An instance of the configuration class containing the model configuration.

        Returns:
            None.

        Raises:
            None.
        """
        super().__init__(config)
        self.num_labels = config.num_labels

        self.deberta = DebertaModel(config)
        self.qa_outputs = nn.Linear(config.hidden_size, config.num_labels)

        # Initialize weights and apply final processing
        self.post_init()

    def forward(
        self,
        input_ids: Optional[mindspore.Tensor] = None,
        attention_mask: Optional[mindspore.Tensor] = None,
        token_type_ids: Optional[mindspore.Tensor] = None,
        position_ids: Optional[mindspore.Tensor] = None,
        inputs_embeds: Optional[mindspore.Tensor] = None,
        start_positions: Optional[mindspore.Tensor] = None,
        end_positions: Optional[mindspore.Tensor] = None,
        output_attentions: Optional[bool] = None,
        output_hidden_states: Optional[bool] = None,
        return_dict: Optional[bool] = None,
    ) -> Union[Tuple, QuestionAnsweringModelOutput]:
        r"""
        Args:
            start_positions (`torch.LongTensor` of shape `(batch_size,)`, *optional*):
                Labels for position (index) of the start of the labelled span for computing the token classification loss.
                Positions are clamped to the length of the sequence (`sequence_length`). Position outside of the sequence
                are not taken into account for computing the loss.
            end_positions (`torch.LongTensor` of shape `(batch_size,)`, *optional*):
                Labels for position (index) of the end of the labelled span for computing the token classification loss.
                Positions are clamped to the length of the sequence (`sequence_length`). Position outside of the sequence
                are not taken into account for computing the loss.
        """
        return_dict = return_dict if return_dict is not None else self.config.use_return_dict

        outputs = self.deberta(
            input_ids,
            attention_mask=attention_mask,
            token_type_ids=token_type_ids,
            position_ids=position_ids,
            inputs_embeds=inputs_embeds,
            output_attentions=output_attentions,
            output_hidden_states=output_hidden_states,
            return_dict=return_dict,
        )

        sequence_output = outputs[0]

        logits = self.qa_outputs(sequence_output)
        start_logits, end_logits = logits.split(1, axis=-1)
        start_logits = start_logits.squeeze(-1)
        end_logits = end_logits.squeeze(-1)

        total_loss = None
        if start_positions is not None and end_positions is not None:
            # If we are on multi-GPU, split add a dimension
            if len(start_positions.shape) > 1:
                start_positions = start_positions.squeeze(-1)
            if len(end_positions.shape) > 1:
                end_positions = end_positions.squeeze(-1)
            # sometimes the start/end positions are outside our model inputs, we ignore these terms
            ignored_index = start_logits.shape[1]
            start_positions = start_positions.clamp(0, ignored_index)
            end_positions = end_positions.clamp(0, ignored_index)

            start_loss = ops.cross_entropy(start_logits, start_positions, ignore_index=ignored_index)
            end_loss = ops.cross_entropy(end_logits, end_positions, ignore_index=ignored_index)
            total_loss = (start_loss + end_loss) / 2

        if not return_dict:
            output = (start_logits, end_logits) + outputs[1:]
            return ((total_loss,) + output) if total_loss is not None else output

        return QuestionAnsweringModelOutput(
            loss=total_loss,
            start_logits=start_logits,
            end_logits=end_logits,
            hidden_states=outputs.hidden_states,
            attentions=outputs.attentions,
        )

mindnlp.transformers.models.deberta.modeling_deberta.DebertaForQuestionAnswering.__init__(config)

Initializes a new instance of the DebertaForQuestionAnswering class.

PARAMETER	DESCRIPTION
self	The instance of the class.
config	An instance of the configuration class containing the model configuration.

RETURNS	DESCRIPTION
	None.

Source code in mindnlp/transformers/models/deberta/modeling_deberta.py

def __init__(self, config):
    """
    Initializes a new instance of the DebertaForQuestionAnswering class.

    Args:
        self: The instance of the class.
        config: An instance of the configuration class containing the model configuration.

    Returns:
        None.

    Raises:
        None.
    """
    super().__init__(config)
    self.num_labels = config.num_labels

    self.deberta = DebertaModel(config)
    self.qa_outputs = nn.Linear(config.hidden_size, config.num_labels)

    # Initialize weights and apply final processing
    self.post_init()

mindnlp.transformers.models.deberta.modeling_deberta.DebertaForQuestionAnswering.forward(input_ids=None, attention_mask=None, token_type_ids=None, position_ids=None, inputs_embeds=None, start_positions=None, end_positions=None, output_attentions=None, output_hidden_states=None, return_dict=None)

PARAMETER	DESCRIPTION
start_positions	Labels for position (index) of the start of the labelled span for computing the token classification loss. Positions are clamped to the length of the sequence (sequence_length). Position outside of the sequence are not taken into account for computing the loss. TYPE: `torch.LongTensor` of shape `(batch_size,)`, *optional* DEFAULT: None
end_positions	Labels for position (index) of the end of the labelled span for computing the token classification loss. Positions are clamped to the length of the sequence (sequence_length). Position outside of the sequence are not taken into account for computing the loss. TYPE: `torch.LongTensor` of shape `(batch_size,)`, *optional* DEFAULT: None

Source code in mindnlp/transformers/models/deberta/modeling_deberta.py

def forward(
    self,
    input_ids: Optional[mindspore.Tensor] = None,
    attention_mask: Optional[mindspore.Tensor] = None,
    token_type_ids: Optional[mindspore.Tensor] = None,
    position_ids: Optional[mindspore.Tensor] = None,
    inputs_embeds: Optional[mindspore.Tensor] = None,
    start_positions: Optional[mindspore.Tensor] = None,
    end_positions: Optional[mindspore.Tensor] = None,
    output_attentions: Optional[bool] = None,
    output_hidden_states: Optional[bool] = None,
    return_dict: Optional[bool] = None,
) -> Union[Tuple, QuestionAnsweringModelOutput]:
    r"""
    Args:
        start_positions (`torch.LongTensor` of shape `(batch_size,)`, *optional*):
            Labels for position (index) of the start of the labelled span for computing the token classification loss.
            Positions are clamped to the length of the sequence (`sequence_length`). Position outside of the sequence
            are not taken into account for computing the loss.
        end_positions (`torch.LongTensor` of shape `(batch_size,)`, *optional*):
            Labels for position (index) of the end of the labelled span for computing the token classification loss.
            Positions are clamped to the length of the sequence (`sequence_length`). Position outside of the sequence
            are not taken into account for computing the loss.
    """
    return_dict = return_dict if return_dict is not None else self.config.use_return_dict

    outputs = self.deberta(
        input_ids,
        attention_mask=attention_mask,
        token_type_ids=token_type_ids,
        position_ids=position_ids,
        inputs_embeds=inputs_embeds,
        output_attentions=output_attentions,
        output_hidden_states=output_hidden_states,
        return_dict=return_dict,
    )

    sequence_output = outputs[0]

    logits = self.qa_outputs(sequence_output)
    start_logits, end_logits = logits.split(1, axis=-1)
    start_logits = start_logits.squeeze(-1)
    end_logits = end_logits.squeeze(-1)

    total_loss = None
    if start_positions is not None and end_positions is not None:
        # If we are on multi-GPU, split add a dimension
        if len(start_positions.shape) > 1:
            start_positions = start_positions.squeeze(-1)
        if len(end_positions.shape) > 1:
            end_positions = end_positions.squeeze(-1)
        # sometimes the start/end positions are outside our model inputs, we ignore these terms
        ignored_index = start_logits.shape[1]
        start_positions = start_positions.clamp(0, ignored_index)
        end_positions = end_positions.clamp(0, ignored_index)

        start_loss = ops.cross_entropy(start_logits, start_positions, ignore_index=ignored_index)
        end_loss = ops.cross_entropy(end_logits, end_positions, ignore_index=ignored_index)
        total_loss = (start_loss + end_loss) / 2

    if not return_dict:
        output = (start_logits, end_logits) + outputs[1:]
        return ((total_loss,) + output) if total_loss is not None else output

    return QuestionAnsweringModelOutput(
        loss=total_loss,
        start_logits=start_logits,
        end_logits=end_logits,
        hidden_states=outputs.hidden_states,
        attentions=outputs.attentions,
    )

mindnlp.transformers.models.deberta.modeling_deberta.DebertaForSequenceClassification

Bases: DebertaPreTrainedModel

DebertaForSequenceClassification is a class that represents a DeBERTa model for sequence classification tasks. It inherits from DebertaPreTrainedModel and provides functionalities for sequenceclassification using the DeBERTa model architecture.

The class includes methods for initializing the model, getting and setting input embeddings, and forwarding the model for sequence classification tasks. The 'forward' method takes input tensors such as input_ids, attention_mask, token_type_ids, position_ids, inputs_embeds, and labels to perform sequence classification. It utilizes the DeBERTa model, a context pooler, and a classifier to generate logits for the input sequences and compute the loss based on the specified problem type.

The 'forward' method also handles different problem types such as regression, single-label classification, and multi-label classification by adjusting the loss computation accordingly. The class provides flexibility in handling various types of sequence classification tasks and supports configurable return options.

For more detailed information on the methods and parameters of DebertaForSequenceClassification, refer to the class implementation and the DeBERTa documentation.

Source code in mindnlp/transformers/models/deberta/modeling_deberta.py

class DebertaForSequenceClassification(DebertaPreTrainedModel):

    """
    DebertaForSequenceClassification is a class that represents a DeBERTa model for sequence classification tasks.
    It inherits from DebertaPreTrainedModel and provides functionalities for sequenceclassification using the
    DeBERTa model architecture.

    The class includes methods for initializing the model, getting and setting input embeddings, and forwarding
    the model for sequence classification tasks. The 'forward' method takes input tensors such as  input_ids,
    attention_mask, token_type_ids, position_ids, inputs_embeds, and labels to perform sequence classification.
    It utilizes the DeBERTa model, a context pooler, and a classifier to generate logits for the input sequences and
    compute the loss based on the specified problem type.

    The 'forward' method also handles different problem types such as regression, single-label classification,
    and multi-label classification by adjusting the loss computation accordingly.
    The class provides flexibility in handling various types of sequence classification tasks and supports configurable
    return options.

    For more detailed information on the methods and parameters of DebertaForSequenceClassification,
    refer to the class implementation and the DeBERTa documentation.
    """
    def __init__(self, config):
        """
        Initializes the DebertaForSequenceClassification class.

        Args:
            self (DebertaForSequenceClassification): The instance of the DebertaForSequenceClassification class.
            config (object): The configuration object containing various settings for the model.

        Returns:
            None.

        Raises:
            AttributeError: If the 'num_labels' attribute is missing in the configuration object.
            TypeError: If the 'num_labels' attribute in the configuration object is not an integer.
            ValueError: If the 'cls_dropout' attribute is not a valid dropout value.
        """
        super().__init__(config)

        num_labels = getattr(config, "num_labels", 2)
        self.num_labels = num_labels

        self.deberta = DebertaModel(config)
        self.pooler = ContextPooler(config)
        output_dim = self.pooler.output_dim

        self.classifier = nn.Linear(output_dim, num_labels)
        drop_out = getattr(config, "cls_dropout", None)
        drop_out = self.config.hidden_dropout_prob if drop_out is None else drop_out
        self.dropout = StableDropout(drop_out)

        # Initialize weights and apply final processing
        self.post_init()

    def get_input_embeddings(self):
        """
        Method to retrieve the input embeddings from the Deberta model for sequence classification.

        Args:
            self (DebertaForSequenceClassification): The instance of the DebertaForSequenceClassification class.
                This parameter is used to access the Deberta model's input embeddings.

        Returns:
            None:
                This method returns None as it simply delegates the call to the Deberta model to retrieve the
                input embeddings.

        Raises:
            None
        """
        return self.deberta.get_input_embeddings()

    def set_input_embeddings(self, new_embeddings):
        """
        Sets the input embeddings for the Deberta model in the DebertaForSequenceClassification class.

        Args:
            self (DebertaForSequenceClassification): The instance of the DebertaForSequenceClassification class.
            new_embeddings (torch.nn.Embedding): The new input embeddings to be set for the Deberta model.

        Returns:
            None.

        Raises:
            None.
        """
        self.deberta.set_input_embeddings(new_embeddings)

    def forward(
        self,
        input_ids: Optional[mindspore.Tensor] = None,
        attention_mask: Optional[mindspore.Tensor] = None,
        token_type_ids: Optional[mindspore.Tensor] = None,
        position_ids: Optional[mindspore.Tensor] = None,
        inputs_embeds: Optional[mindspore.Tensor] = None,
        labels: Optional[mindspore.Tensor] = None,
        output_attentions: Optional[bool] = None,
        output_hidden_states: Optional[bool] = None,
        return_dict: Optional[bool] = None,
    ) -> Union[Tuple, SequenceClassifierOutput]:
        r"""
        Args:
            labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*):
                Labels for computing the sequence classification/regression loss. Indices should be in `[0, ...,
                config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If
                `config.num_labels > 1` a classification loss is computed (Cross-Entropy).
        """
        return_dict = return_dict if return_dict is not None else self.config.use_return_dict

        outputs = self.deberta(
            input_ids,
            token_type_ids=token_type_ids,
            attention_mask=attention_mask,
            position_ids=position_ids,
            inputs_embeds=inputs_embeds,
            output_attentions=output_attentions,
            output_hidden_states=output_hidden_states,
            return_dict=return_dict,
        )

        encoder_layer = outputs[0]
        pooled_output = self.pooler(encoder_layer)
        pooled_output = self.dropout(pooled_output)
        logits = self.classifier(pooled_output)

        loss = None
        if labels is not None:
            if self.config.problem_type is None:
                if self.num_labels == 1:
                    # regression task
                    logits = logits.view(-1).to(labels.dtype)
                    loss = ops.mse_loss(logits, labels.view(-1))
                elif labels.ndim == 1 or labels.shape[-1] == 1:
                    label_index = (labels >= 0).nonzero()
                    labels = labels.long()
                    if label_index.shape[0] > 0:
                        labeled_logits = ops.gather_elements(
                            logits, 0, label_index.expand(label_index.shape[0], logits.shape[1])
                        )
                        labels = ops.gather_elements(labels, 0, label_index.view(-1))
                        loss = ops.cross_entropy(labeled_logits.view(-1, self.num_labels).float(), labels.view(-1))
                    else:
                        loss = mindspore.tensor(0).to(logits)
                else:
                    log_softmax = nn.LogSoftmax(-1)
                    loss = -((log_softmax(logits) * labels).sum(-1)).mean()
            elif self.config.problem_type == "regression":
                if self.num_labels == 1:
                    loss = ops.mse_loss(logits.squeeze(), labels.squeeze())
                else:
                    loss = ops.mse_loss(logits, labels)
            elif self.config.problem_type == "single_label_classification":
                loss = ops.cross_entropy(logits.view(-1, self.num_labels), labels.view(-1))
            elif self.config.problem_type == "multi_label_classification":
                loss = ops.binary_cross_entropy(logits, labels)
        if not return_dict:
            output = (logits,) + outputs[1:]
            return ((loss,) + output) if loss is not None else output

        return SequenceClassifierOutput(
            loss=loss, logits=logits, hidden_states=outputs.hidden_states, attentions=outputs.attentions
        )

mindnlp.transformers.models.deberta.modeling_deberta.DebertaForSequenceClassification.__init__(config)

Initializes the DebertaForSequenceClassification class.

PARAMETER	DESCRIPTION
self	The instance of the DebertaForSequenceClassification class. TYPE: DebertaForSequenceClassification
config	The configuration object containing various settings for the model. TYPE: object

RETURNS	DESCRIPTION
	None.

RAISES	DESCRIPTION
AttributeError	If the 'num_labels' attribute is missing in the configuration object.
TypeError	If the 'num_labels' attribute in the configuration object is not an integer.
ValueError	If the 'cls_dropout' attribute is not a valid dropout value.

Source code in mindnlp/transformers/models/deberta/modeling_deberta.py

def __init__(self, config):
    """
    Initializes the DebertaForSequenceClassification class.

    Args:
        self (DebertaForSequenceClassification): The instance of the DebertaForSequenceClassification class.
        config (object): The configuration object containing various settings for the model.

    Returns:
        None.

    Raises:
        AttributeError: If the 'num_labels' attribute is missing in the configuration object.
        TypeError: If the 'num_labels' attribute in the configuration object is not an integer.
        ValueError: If the 'cls_dropout' attribute is not a valid dropout value.
    """
    super().__init__(config)

    num_labels = getattr(config, "num_labels", 2)
    self.num_labels = num_labels

    self.deberta = DebertaModel(config)
    self.pooler = ContextPooler(config)
    output_dim = self.pooler.output_dim

    self.classifier = nn.Linear(output_dim, num_labels)
    drop_out = getattr(config, "cls_dropout", None)
    drop_out = self.config.hidden_dropout_prob if drop_out is None else drop_out
    self.dropout = StableDropout(drop_out)

    # Initialize weights and apply final processing
    self.post_init()

mindnlp.transformers.models.deberta.modeling_deberta.DebertaForSequenceClassification.forward(input_ids=None, attention_mask=None, token_type_ids=None, position_ids=None, inputs_embeds=None, labels=None, output_attentions=None, output_hidden_states=None, return_dict=None)

PARAMETER	DESCRIPTION
labels	Labels for computing the sequence classification/regression loss. Indices should be in [0, ..., config.num_labels - 1]. If config.num_labels == 1 a regression loss is computed (Mean-Square loss), If config.num_labels > 1 a classification loss is computed (Cross-Entropy). TYPE: `torch.LongTensor` of shape `(batch_size,)`, *optional* DEFAULT: None

Source code in mindnlp/transformers/models/deberta/modeling_deberta.py

def forward(
    self,
    input_ids: Optional[mindspore.Tensor] = None,
    attention_mask: Optional[mindspore.Tensor] = None,
    token_type_ids: Optional[mindspore.Tensor] = None,
    position_ids: Optional[mindspore.Tensor] = None,
    inputs_embeds: Optional[mindspore.Tensor] = None,
    labels: Optional[mindspore.Tensor] = None,
    output_attentions: Optional[bool] = None,
    output_hidden_states: Optional[bool] = None,
    return_dict: Optional[bool] = None,
) -> Union[Tuple, SequenceClassifierOutput]:
    r"""
    Args:
        labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*):
            Labels for computing the sequence classification/regression loss. Indices should be in `[0, ...,
            config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If
            `config.num_labels > 1` a classification loss is computed (Cross-Entropy).
    """
    return_dict = return_dict if return_dict is not None else self.config.use_return_dict

    outputs = self.deberta(
        input_ids,
        token_type_ids=token_type_ids,
        attention_mask=attention_mask,
        position_ids=position_ids,
        inputs_embeds=inputs_embeds,
        output_attentions=output_attentions,
        output_hidden_states=output_hidden_states,
        return_dict=return_dict,
    )

    encoder_layer = outputs[0]
    pooled_output = self.pooler(encoder_layer)
    pooled_output = self.dropout(pooled_output)
    logits = self.classifier(pooled_output)

    loss = None
    if labels is not None:
        if self.config.problem_type is None:
            if self.num_labels == 1:
                # regression task
                logits = logits.view(-1).to(labels.dtype)
                loss = ops.mse_loss(logits, labels.view(-1))
            elif labels.ndim == 1 or labels.shape[-1] == 1:
                label_index = (labels >= 0).nonzero()
                labels = labels.long()
                if label_index.shape[0] > 0:
                    labeled_logits = ops.gather_elements(
                        logits, 0, label_index.expand(label_index.shape[0], logits.shape[1])
                    )
                    labels = ops.gather_elements(labels, 0, label_index.view(-1))
                    loss = ops.cross_entropy(labeled_logits.view(-1, self.num_labels).float(), labels.view(-1))
                else:
                    loss = mindspore.tensor(0).to(logits)
            else:
                log_softmax = nn.LogSoftmax(-1)
                loss = -((log_softmax(logits) * labels).sum(-1)).mean()
        elif self.config.problem_type == "regression":
            if self.num_labels == 1:
                loss = ops.mse_loss(logits.squeeze(), labels.squeeze())
            else:
                loss = ops.mse_loss(logits, labels)
        elif self.config.problem_type == "single_label_classification":
            loss = ops.cross_entropy(logits.view(-1, self.num_labels), labels.view(-1))
        elif self.config.problem_type == "multi_label_classification":
            loss = ops.binary_cross_entropy(logits, labels)
    if not return_dict:
        output = (logits,) + outputs[1:]
        return ((loss,) + output) if loss is not None else output

    return SequenceClassifierOutput(
        loss=loss, logits=logits, hidden_states=outputs.hidden_states, attentions=outputs.attentions
    )

mindnlp.transformers.models.deberta.modeling_deberta.DebertaForSequenceClassification.get_input_embeddings()

Method to retrieve the input embeddings from the Deberta model for sequence classification.

PARAMETER	DESCRIPTION
self	The instance of the DebertaForSequenceClassification class. This parameter is used to access the Deberta model's input embeddings. TYPE: DebertaForSequenceClassification

RETURNS	DESCRIPTION
None	This method returns None as it simply delegates the call to the Deberta model to retrieve the input embeddings.

Source code in mindnlp/transformers/models/deberta/modeling_deberta.py

def get_input_embeddings(self):
    """
    Method to retrieve the input embeddings from the Deberta model for sequence classification.

    Args:
        self (DebertaForSequenceClassification): The instance of the DebertaForSequenceClassification class.
            This parameter is used to access the Deberta model's input embeddings.

    Returns:
        None:
            This method returns None as it simply delegates the call to the Deberta model to retrieve the
            input embeddings.

    Raises:
        None
    """
    return self.deberta.get_input_embeddings()

mindnlp.transformers.models.deberta.modeling_deberta.DebertaForSequenceClassification.set_input_embeddings(new_embeddings)

Sets the input embeddings for the Deberta model in the DebertaForSequenceClassification class.

PARAMETER	DESCRIPTION
self	The instance of the DebertaForSequenceClassification class. TYPE: DebertaForSequenceClassification
new_embeddings	The new input embeddings to be set for the Deberta model. TYPE: Embedding

RETURNS	DESCRIPTION
	None.

Source code in mindnlp/transformers/models/deberta/modeling_deberta.py

def set_input_embeddings(self, new_embeddings):
    """
    Sets the input embeddings for the Deberta model in the DebertaForSequenceClassification class.

    Args:
        self (DebertaForSequenceClassification): The instance of the DebertaForSequenceClassification class.
        new_embeddings (torch.nn.Embedding): The new input embeddings to be set for the Deberta model.

    Returns:
        None.

    Raises:
        None.
    """
    self.deberta.set_input_embeddings(new_embeddings)

mindnlp.transformers.models.deberta.modeling_deberta.DebertaForTokenClassification

Bases: DebertaPreTrainedModel

This class represents a token classification model based on the DeBERTa architecture. It is designed to perform token-level classification tasks such as named entity recognition or part-of-speech tagging.

The DebertaForTokenClassification class extends the DebertaPreTrainedModel class and inherits its functionality and attributes.

ATTRIBUTE	DESCRIPTION
`num_labels`	The number of labels for token classification.
`deberta`	The DeBERTa model used for feature extraction.
`dropout`	A dropout layer for regularization.
`classifier`	A fully connected layer for classification.

METHOD	DESCRIPTION
`__init__	Initializes the DebertaForTokenClassification instance.
`forward	Performs the forward pass of the model and returns the output. Args: input_ids: An optional tensor representing the input token IDs. attention_mask: An optional tensor representing the attention mask. token_type_ids: An optional tensor representing the token type IDs. position_ids: An optional tensor representing the position IDs. inputs_embeds: An optional tensor representing the input embeddings. labels: An optional tensor representing the labels for computing the token classification loss. output_attentions: An optional boolean indicating whether to output attentions. output_hidden_states: An optional boolean indicating whether to output hidden states. return_dict: An optional boolean indicating whether to return the output as a dictionary. Returns: If return_dict is False, returns a tuple containing the loss, logits, and other outputs. If return_dict is True, returns a TokenClassifierOutput object containing the loss, logits, hidden states, and attentions.

Source code in mindnlp/transformers/models/deberta/modeling_deberta.py

class DebertaForTokenClassification(DebertaPreTrainedModel):

    """
    This class represents a token classification model based on the DeBERTa architecture.
    It is designed to perform token-level classification tasks such as named entity recognition or part-of-speech tagging.

    The `DebertaForTokenClassification` class extends the `DebertaPreTrainedModel` class and inherits its functionality
    and attributes.

    Attributes:
        `num_labels`: The number of labels for token classification.
        `deberta`: The DeBERTa model used for feature extraction.
        `dropout`: A dropout layer for regularization.
        `classifier`: A fully connected layer for classification.

    Methods:
        `__init__(self, config)`: Initializes the `DebertaForTokenClassification` instance.
        `forward(self, input_ids, attention_mask, token_type_ids, position_ids, inputs_embeds, labels, output_attentions, output_hidden_states, return_dict)`:
            Performs the forward pass of the model and returns the output.

            Args:

            - `input_ids`: An optional tensor representing the input token IDs.
            - `attention_mask`: An optional tensor representing the attention mask.
            - `token_type_ids`: An optional tensor representing the token type IDs.
            - `position_ids`: An optional tensor representing the position IDs.
            - `inputs_embeds`: An optional tensor representing the input embeddings.
            - `labels`: An optional tensor representing the labels for computing the token classification loss.
            - `output_attentions`: An optional boolean indicating whether to output attentions.
            - `output_hidden_states`: An optional boolean indicating whether to output hidden states.
            - `return_dict`: An optional boolean indicating whether to return the output as a dictionary.

            Returns:

            - If `return_dict` is False,
            returns a tuple containing the loss, logits, and other outputs.
            - If `return_dict` is True,
            returns a `TokenClassifierOutput` object containing the loss, logits, hidden states, and attentions.
    """
    def __init__(self, config):
        """
        __init__

        Initializes an instance of the DebertaForTokenClassification class.
        Args:
            self: DebertaForTokenClassification
                The instance of the DebertaForTokenClassification class.
            config: DebertaConfig
                The configuration object containing the model configuration settings.
                It is used to set up the model architecture and hyperparameters.
                Required and must be an instance of DebertaConfig.

        Returns:
            None.

        Raises:
            None.
        """
        super().__init__(config)
        self.num_labels = config.num_labels

        self.deberta = DebertaModel(config)
        self.dropout = nn.Dropout(p=config.hidden_dropout_prob)
        self.classifier = nn.Linear(config.hidden_size, config.num_labels)

        # Initialize weights and apply final processing
        self.post_init()

    def forward(
        self,
        input_ids: Optional[mindspore.Tensor] = None,
        attention_mask: Optional[mindspore.Tensor] = None,
        token_type_ids: Optional[mindspore.Tensor] = None,
        position_ids: Optional[mindspore.Tensor] = None,
        inputs_embeds: Optional[mindspore.Tensor] = None,
        labels: Optional[mindspore.Tensor] = None,
        output_attentions: Optional[bool] = None,
        output_hidden_states: Optional[bool] = None,
        return_dict: Optional[bool] = None,
    ) -> Union[Tuple, TokenClassifierOutput]:
        r"""
        Args:
            labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*):
                Labels for computing the token classification loss. Indices should be in `[0, ..., config.num_labels - 1]`.
        """
        return_dict = return_dict if return_dict is not None else self.config.use_return_dict

        outputs = self.deberta(
            input_ids,
            attention_mask=attention_mask,
            token_type_ids=token_type_ids,
            position_ids=position_ids,
            inputs_embeds=inputs_embeds,
            output_attentions=output_attentions,
            output_hidden_states=output_hidden_states,
            return_dict=return_dict,
        )

        sequence_output = outputs[0]

        sequence_output = self.dropout(sequence_output)
        logits = self.classifier(sequence_output)

        loss = None
        if labels is not None:
            loss = ops.cross_entropy(logits.view(-1, self.num_labels), labels.view(-1))

        if not return_dict:
            output = (logits,) + outputs[1:]
            return ((loss,) + output) if loss is not None else output

        return TokenClassifierOutput(
            loss=loss, logits=logits, hidden_states=outputs.hidden_states, attentions=outputs.attentions
        )

mindnlp.transformers.models.deberta.modeling_deberta.DebertaForTokenClassification.__init__(config)

init

Initializes an instance of the DebertaForTokenClassification class. Args: self: DebertaForTokenClassification The instance of the DebertaForTokenClassification class. config: DebertaConfig The configuration object containing the model configuration settings. It is used to set up the model architecture and hyperparameters. Required and must be an instance of DebertaConfig.

RETURNS	DESCRIPTION
	None.

Source code in mindnlp/transformers/models/deberta/modeling_deberta.py

def __init__(self, config):
    """
    __init__

    Initializes an instance of the DebertaForTokenClassification class.
    Args:
        self: DebertaForTokenClassification
            The instance of the DebertaForTokenClassification class.
        config: DebertaConfig
            The configuration object containing the model configuration settings.
            It is used to set up the model architecture and hyperparameters.
            Required and must be an instance of DebertaConfig.

    Returns:
        None.

    Raises:
        None.
    """
    super().__init__(config)
    self.num_labels = config.num_labels

    self.deberta = DebertaModel(config)
    self.dropout = nn.Dropout(p=config.hidden_dropout_prob)
    self.classifier = nn.Linear(config.hidden_size, config.num_labels)

    # Initialize weights and apply final processing
    self.post_init()

mindnlp.transformers.models.deberta.modeling_deberta.DebertaForTokenClassification.forward(input_ids=None, attention_mask=None, token_type_ids=None, position_ids=None, inputs_embeds=None, labels=None, output_attentions=None, output_hidden_states=None, return_dict=None)

PARAMETER	DESCRIPTION
labels	Labels for computing the token classification loss. Indices should be in [0, ..., config.num_labels - 1]. TYPE: `torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional* DEFAULT: None

Source code in mindnlp/transformers/models/deberta/modeling_deberta.py

def forward(
    self,
    input_ids: Optional[mindspore.Tensor] = None,
    attention_mask: Optional[mindspore.Tensor] = None,
    token_type_ids: Optional[mindspore.Tensor] = None,
    position_ids: Optional[mindspore.Tensor] = None,
    inputs_embeds: Optional[mindspore.Tensor] = None,
    labels: Optional[mindspore.Tensor] = None,
    output_attentions: Optional[bool] = None,
    output_hidden_states: Optional[bool] = None,
    return_dict: Optional[bool] = None,
) -> Union[Tuple, TokenClassifierOutput]:
    r"""
    Args:
        labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*):
            Labels for computing the token classification loss. Indices should be in `[0, ..., config.num_labels - 1]`.
    """
    return_dict = return_dict if return_dict is not None else self.config.use_return_dict

    outputs = self.deberta(
        input_ids,
        attention_mask=attention_mask,
        token_type_ids=token_type_ids,
        position_ids=position_ids,
        inputs_embeds=inputs_embeds,
        output_attentions=output_attentions,
        output_hidden_states=output_hidden_states,
        return_dict=return_dict,
    )

    sequence_output = outputs[0]

    sequence_output = self.dropout(sequence_output)
    logits = self.classifier(sequence_output)

    loss = None
    if labels is not None:
        loss = ops.cross_entropy(logits.view(-1, self.num_labels), labels.view(-1))

    if not return_dict:
        output = (logits,) + outputs[1:]
        return ((loss,) + output) if loss is not None else output

    return TokenClassifierOutput(
        loss=loss, logits=logits, hidden_states=outputs.hidden_states, attentions=outputs.attentions
    )

mindnlp.transformers.models.deberta.modeling_deberta.DebertaIntermediate

Bases: Module

DebertaIntermediate represents an intermediate layer in the DeBERTa neural network architecture for natural language processing tasks. This class inherits from nn.Module and contains methods for initializing the layer and performing computations on hidden states. The layer consists of a dense transformation followed by an activation function specified in the configuration.

ATTRIBUTE	DESCRIPTION
dense	A dense layer with hidden size and intermediate size specified in the configuration. TYPE: Linear
intermediate_act_fn	The activation function applied to the hidden states. TYPE: function

METHOD	DESCRIPTION
__init__	Initializes the DebertaIntermediate layer with the provided configuration.
forward	Applies the dense transformation and activation function to the input hidden states.

Source code in mindnlp/transformers/models/deberta/modeling_deberta.py

class DebertaIntermediate(nn.Module):

    """
    DebertaIntermediate represents an intermediate layer in the DeBERTa neural network architecture for natural
    language processing tasks.
    This class inherits from nn.Module and contains methods for initializing the layer and performing computations
    on hidden states.
    The layer consists of a dense transformation followed by an activation function specified in the configuration.

    Attributes:
        dense (nn.Linear): A dense layer with hidden size and intermediate size specified in the configuration.
        intermediate_act_fn (function): The activation function applied to the hidden states.

    Methods:
        __init__: Initializes the DebertaIntermediate layer with the provided configuration.
        forward: Applies the dense transformation and activation function to the input hidden states.

    """
    def __init__(self, config):
        """
        Initializes a new instance of the DebertaIntermediate class.

        Args:
            self: The object itself.
            config (object):
                An object containing the configuration parameters for the DebertaIntermediate class.
                It should have the following properties:

                - hidden_size (int): The size of the hidden layer in the intermediate module.
                - intermediate_size (int): The size of the intermediate layer.
                - hidden_act (str or object): The activation function for the hidden layer.

                    - If it is a string, it should be one of the supported activation functions.
                    - If it is an object, it should be a callable that takes a single argument.

        Returns:
            None.

        Raises:
            None.
        """
        super().__init__()
        self.dense = nn.Linear(config.hidden_size, config.intermediate_size)
        if isinstance(config.hidden_act, str):
            self.intermediate_act_fn = ACT2FN[config.hidden_act]
        else:
            self.intermediate_act_fn = config.hidden_act

    def forward(self, hidden_states: mindspore.Tensor) -> mindspore.Tensor:
        """
        Constructs the intermediate layer of the Deberta model.
        This method takes in the hidden states tensor and applies a series of transformations to it in order to
        forward the intermediate layer of the Deberta model. The hidden states tensor is first passed through a dense
        layer, followed by an activation function specified by 'intermediate_act_fn'.
        The resulting tensor represents the intermediate hidden states and is returned as the output of this method.

        Args:
            self (DebertaIntermediate): The instance of the DebertaIntermediate class.
            hidden_states (mindspore.Tensor): The input hidden states tensor.

        Returns:
            mindspore.Tensor: The tensor representing the output hidden states.

        Raises:
            None:

        Note:
            The 'intermediate_act_fn' attribute should be set prior to calling this method to specify the desired
            activation function.

        Example:
            ```python
            >>> intermediate_layer = DebertaIntermediate()
            >>> hidden_states = mindspore.Tensor([0.1, 0.2, 0.3])
            >>> output = intermediate_layer.forward(hidden_states)
            ```
        """
        hidden_states = self.dense(hidden_states)
        hidden_states = self.intermediate_act_fn(hidden_states)
        return hidden_states

mindnlp.transformers.models.deberta.modeling_deberta.DebertaIntermediate.__init__(config)

Initializes a new instance of the DebertaIntermediate class.

PARAMETER	DESCRIPTION
self	The object itself.
config	An object containing the configuration parameters for the DebertaIntermediate class. It should have the following properties: hidden_size (int): The size of the hidden layer in the intermediate module. intermediate_size (int): The size of the intermediate layer. hidden_act (str or object): The activation function for the hidden layer. If it is a string, it should be one of the supported activation functions. If it is an object, it should be a callable that takes a single argument. TYPE: object

RETURNS	DESCRIPTION
	None.

Source code in mindnlp/transformers/models/deberta/modeling_deberta.py

def __init__(self, config):
    """
    Initializes a new instance of the DebertaIntermediate class.

    Args:
        self: The object itself.
        config (object):
            An object containing the configuration parameters for the DebertaIntermediate class.
            It should have the following properties:

            - hidden_size (int): The size of the hidden layer in the intermediate module.
            - intermediate_size (int): The size of the intermediate layer.
            - hidden_act (str or object): The activation function for the hidden layer.

                - If it is a string, it should be one of the supported activation functions.
                - If it is an object, it should be a callable that takes a single argument.

    Returns:
        None.

    Raises:
        None.
    """
    super().__init__()
    self.dense = nn.Linear(config.hidden_size, config.intermediate_size)
    if isinstance(config.hidden_act, str):
        self.intermediate_act_fn = ACT2FN[config.hidden_act]
    else:
        self.intermediate_act_fn = config.hidden_act

mindnlp.transformers.models.deberta.modeling_deberta.DebertaIntermediate.forward(hidden_states)

Constructs the intermediate layer of the Deberta model. This method takes in the hidden states tensor and applies a series of transformations to it in order to forward the intermediate layer of the Deberta model. The hidden states tensor is first passed through a dense layer, followed by an activation function specified by 'intermediate_act_fn'. The resulting tensor represents the intermediate hidden states and is returned as the output of this method.

PARAMETER	DESCRIPTION
self	The instance of the DebertaIntermediate class. TYPE: DebertaIntermediate
hidden_states	The input hidden states tensor. TYPE: Tensor

RETURNS	DESCRIPTION
Tensor	mindspore.Tensor: The tensor representing the output hidden states.

RAISES	DESCRIPTION
None

Note

The 'intermediate_act_fn' attribute should be set prior to calling this method to specify the desired activation function.

Example

>>> intermediate_layer = DebertaIntermediate()
>>> hidden_states = mindspore.Tensor([0.1, 0.2, 0.3])
>>> output = intermediate_layer.forward(hidden_states)

Source code in mindnlp/transformers/models/deberta/modeling_deberta.py

def forward(self, hidden_states: mindspore.Tensor) -> mindspore.Tensor:
    """
    Constructs the intermediate layer of the Deberta model.
    This method takes in the hidden states tensor and applies a series of transformations to it in order to
    forward the intermediate layer of the Deberta model. The hidden states tensor is first passed through a dense
    layer, followed by an activation function specified by 'intermediate_act_fn'.
    The resulting tensor represents the intermediate hidden states and is returned as the output of this method.

    Args:
        self (DebertaIntermediate): The instance of the DebertaIntermediate class.
        hidden_states (mindspore.Tensor): The input hidden states tensor.

    Returns:
        mindspore.Tensor: The tensor representing the output hidden states.

    Raises:
        None:

    Note:
        The 'intermediate_act_fn' attribute should be set prior to calling this method to specify the desired
        activation function.

    Example:
        ```python
        >>> intermediate_layer = DebertaIntermediate()
        >>> hidden_states = mindspore.Tensor([0.1, 0.2, 0.3])
        >>> output = intermediate_layer.forward(hidden_states)
        ```
    """
    hidden_states = self.dense(hidden_states)
    hidden_states = self.intermediate_act_fn(hidden_states)
    return hidden_states

mindnlp.transformers.models.deberta.modeling_deberta.DebertaLMPredictionHead

Bases: Module

DebertaLMPredictionHead represents the prediction head for language model tasks in a DeBERTa model. This class inherits from nn.Module.

ATTRIBUTE	DESCRIPTION
transform	An instance of DebertaPredictionHeadTransform for transforming hidden states. TYPE: DebertaPredictionHeadTransform
embedding_size	The size of the embedding layer, defaults to the hidden size if not specified in config. TYPE: int
decoder	A fully connected layer for decoding hidden states to predict the next token. TYPE: Linear
bias	The bias parameter for the decoder layer. TYPE: Parameter

METHOD	DESCRIPTION
__init__	Initializes the DebertaLMPredictionHead with the provided configuration.
forward	Constructs the prediction head by applying transformations and decoding the hidden states.

Source code in mindnlp/transformers/models/deberta/modeling_deberta.py

class DebertaLMPredictionHead(nn.Module):

    """
    DebertaLMPredictionHead represents the prediction head for language model tasks in a DeBERTa model.
    This class inherits from nn.Module.

    Attributes:
        transform (DebertaPredictionHeadTransform):
            An instance of DebertaPredictionHeadTransform for transforming hidden states.
        embedding_size (int): The size of the embedding layer, defaults to the hidden size if not specified in config.
        decoder (nn.Linear): A fully connected layer for decoding hidden states to predict the next token.
        bias (Parameter): The bias parameter for the decoder layer.

    Methods:
        __init__: Initializes the DebertaLMPredictionHead with the provided configuration.
        forward: Constructs the prediction head by applying transformations and decoding the hidden states.

    """
    def __init__(self, config):
        """
        Initializes an instance of the DebertaLMPredictionHead class.

        Args:
            self: The current object instance.
            config (obj):
                An object containing configuration parameters for the DebertaLMPredictionHead.

                - Type: object
                - Purpose: Specifies the configuration settings for the DebertaLMPredictionHead.
                - Restrictions: None

        Returns:
            None

        Raises:
            None
        """
        super().__init__()
        self.transform = DebertaPredictionHeadTransform(config)

        self.embedding_size = getattr(config, "embedding_size", config.hidden_size)
        # The output weights are the same as the input embeddings, but there is
        # an output-only bias for each token.
        self.decoder = nn.Linear(self.embedding_size, config.vocab_size, bias=False)

        self.bias = Parameter(ops.zeros(config.vocab_size))

        # Need a link between the two variables so that the bias is correctly resized with `resize_token_embeddings`
        self.decoder.bias = self.bias

    def forward(self, hidden_states):
        """
        This method forwards the prediction head for DebertaLM model.

        Args:
            self (DebertaLMPredictionHead): An instance of the DebertaLMPredictionHead class.
            hidden_states (tensor): The hidden states to be processed for prediction.

        Returns:
            None: The processed hidden states after passing through the transformation and decoder layers.

        Raises:
            None.
        """
        hidden_states = self.transform(hidden_states)
        hidden_states = self.decoder(hidden_states)
        return hidden_states

mindnlp.transformers.models.deberta.modeling_deberta.DebertaLMPredictionHead.__init__(config)

Initializes an instance of the DebertaLMPredictionHead class.

PARAMETER	DESCRIPTION
self	The current object instance.
config	An object containing configuration parameters for the DebertaLMPredictionHead. Type: object Purpose: Specifies the configuration settings for the DebertaLMPredictionHead. Restrictions: None TYPE: obj

RETURNS	DESCRIPTION
	None

Source code in mindnlp/transformers/models/deberta/modeling_deberta.py

def __init__(self, config):
    """
    Initializes an instance of the DebertaLMPredictionHead class.

    Args:
        self: The current object instance.
        config (obj):
            An object containing configuration parameters for the DebertaLMPredictionHead.

            - Type: object
            - Purpose: Specifies the configuration settings for the DebertaLMPredictionHead.
            - Restrictions: None

    Returns:
        None

    Raises:
        None
    """
    super().__init__()
    self.transform = DebertaPredictionHeadTransform(config)

    self.embedding_size = getattr(config, "embedding_size", config.hidden_size)
    # The output weights are the same as the input embeddings, but there is
    # an output-only bias for each token.
    self.decoder = nn.Linear(self.embedding_size, config.vocab_size, bias=False)

    self.bias = Parameter(ops.zeros(config.vocab_size))

    # Need a link between the two variables so that the bias is correctly resized with `resize_token_embeddings`
    self.decoder.bias = self.bias

mindnlp.transformers.models.deberta.modeling_deberta.DebertaLMPredictionHead.forward(hidden_states)

This method forwards the prediction head for DebertaLM model.

PARAMETER	DESCRIPTION
self	An instance of the DebertaLMPredictionHead class. TYPE: DebertaLMPredictionHead
hidden_states	The hidden states to be processed for prediction. TYPE: tensor

RETURNS	DESCRIPTION
None	The processed hidden states after passing through the transformation and decoder layers.

Source code in mindnlp/transformers/models/deberta/modeling_deberta.py

def forward(self, hidden_states):
    """
    This method forwards the prediction head for DebertaLM model.

    Args:
        self (DebertaLMPredictionHead): An instance of the DebertaLMPredictionHead class.
        hidden_states (tensor): The hidden states to be processed for prediction.

    Returns:
        None: The processed hidden states after passing through the transformation and decoder layers.

    Raises:
        None.
    """
    hidden_states = self.transform(hidden_states)
    hidden_states = self.decoder(hidden_states)
    return hidden_states

mindnlp.transformers.models.deberta.modeling_deberta.DebertaLayer

Bases: Module

Represents a single layer in the DeBERTa model, containing modules for attention, intermediate processing, and output computation.

This class inherits from nn.Module and is responsible for processing input hidden states through attention mechanisms, intermediate processing, and final output computation. It provides a 'forward' method to perform these operations and return the final layer output.

ATTRIBUTE	DESCRIPTION
attention	Module for performing attention mechanism computation. TYPE: DebertaAttention
intermediate	Module for intermediate processing of attention output. TYPE: DebertaIntermediate
output	Module for computing final output based on intermediate processed data. TYPE: DebertaOutput

METHOD

DESCRIPTION

forward

Process the input hidden states through attention, intermediate, and output modules to compute the final layer output. Args: hidden_states (Tensor): Input hidden states to be processed. attention_mask (Tensor): Mask for attention calculation. query_states (Tensor, optional): Query states for attention mechanism. Default is None. relative_pos (Tensor, optional): Relative position information for attention computation. Default is None. rel_embeddings (Tensor, optional): Relative embeddings for attention computation. Default is None. output_attentions (bool, optional): Flag indicating whether to output attention matrices. Default is False. Returns: layer_output (Tensor): Final computed output of the layer. att_matrix (Tensor, optional): Attention matrix if 'output_attentions' is True. Otherwise, None.

Note

If 'output_attentions' is set to True, the 'forward' method will return both the final layer output and the attention matrix.

Source code in mindnlp/transformers/models/deberta/modeling_deberta.py

class DebertaLayer(nn.Module):

    """
    Represents a single layer in the DeBERTa model, containing modules for attention, intermediate processing,
    and output computation.

    This class inherits from nn.Module and is responsible for processing input hidden states through attention mechanisms,
    intermediate processing, and final output computation.
    It provides a 'forward' method to perform these operations and return the final layer output.

    Attributes:
        attention (DebertaAttention): Module for performing attention mechanism computation.
        intermediate (DebertaIntermediate): Module for intermediate processing of attention output.
        output (DebertaOutput): Module for computing final output based on intermediate processed data.

    Methods:
        forward:
            Process the input hidden states through attention, intermediate,
            and output modules to compute the final layer output.

            Args:

            - hidden_states (Tensor): Input hidden states to be processed.
            - attention_mask (Tensor): Mask for attention calculation.
            - query_states (Tensor, optional): Query states for attention mechanism. Default is None.
            - relative_pos (Tensor, optional): Relative position information for attention computation. Default is None.
            - rel_embeddings (Tensor, optional): Relative embeddings for attention computation. Default is None.
            - output_attentions (bool, optional): Flag indicating whether to output attention matrices. Default is False.

            Returns:

            - layer_output (Tensor): Final computed output of the layer.
            - att_matrix (Tensor, optional): Attention matrix if 'output_attentions' is True. Otherwise, None.

    Note:
        If 'output_attentions' is set to True, the 'forward' method will return both the final layer output and the
        attention matrix.
    """
    def __init__(self, config):
        """
        Initialize a DebertaLayer instance.

        Args:
            self (object): The instance of the DebertaLayer class.
            config (object): An object containing configuration settings for the DebertaLayer.
                It is used to customize the behavior of the layer during initialization.

        Returns:
            None.

        Raises:
            None.
        """
        super().__init__()
        self.attention = DebertaAttention(config)
        self.intermediate = DebertaIntermediate(config)
        self.output = DebertaOutput(config)

    def forward(
        self,
        hidden_states,
        attention_mask,
        query_states=None,
        relative_pos=None,
        rel_embeddings=None,
        output_attentions=False,
    ):
        """
        Constructs the DebertaLayer by performing attention, intermediate, and output operations.

        Args:
            self (object): The class instance.
            hidden_states (torch.Tensor): The input hidden states tensor.
            attention_mask (torch.Tensor): The attention mask tensor to mask out padded tokens.
            query_states (torch.Tensor, optional): The tensor representing query states for attention computation.
                Defaults to None.
            relative_pos (torch.Tensor, optional): The tensor representing relative positions for attention computation.
                Defaults to None.
            rel_embeddings (torch.Tensor, optional): The tensor containing relative embeddings for attention computation.
                Defaults to None.
            output_attentions (bool): Flag indicating whether to output attention matrices. Defaults to False.

        Returns:
            None.

        Raises:
            ValueError: If the dimensions of the input tensors are incompatible.
            TypeError: If the input parameters are not of the expected types.
        """
        attention_output = self.attention(
            hidden_states,
            attention_mask,
            output_attentions=output_attentions,
            query_states=query_states,
            relative_pos=relative_pos,
            rel_embeddings=rel_embeddings,
        )
        if output_attentions:
            attention_output, att_matrix = attention_output
        intermediate_output = self.intermediate(attention_output)
        layer_output = self.output(intermediate_output, attention_output)
        if output_attentions:
            return (layer_output, att_matrix)
        return layer_output

mindnlp.transformers.models.deberta.modeling_deberta.DebertaLayer.__init__(config)

Initialize a DebertaLayer instance.

PARAMETER	DESCRIPTION
self	The instance of the DebertaLayer class. TYPE: object
config	An object containing configuration settings for the DebertaLayer. It is used to customize the behavior of the layer during initialization. TYPE: object

RETURNS	DESCRIPTION
	None.

Source code in mindnlp/transformers/models/deberta/modeling_deberta.py

def __init__(self, config):
    """
    Initialize a DebertaLayer instance.

    Args:
        self (object): The instance of the DebertaLayer class.
        config (object): An object containing configuration settings for the DebertaLayer.
            It is used to customize the behavior of the layer during initialization.

    Returns:
        None.

    Raises:
        None.
    """
    super().__init__()
    self.attention = DebertaAttention(config)
    self.intermediate = DebertaIntermediate(config)
    self.output = DebertaOutput(config)

mindnlp.transformers.models.deberta.modeling_deberta.DebertaLayer.forward(hidden_states, attention_mask, query_states=None, relative_pos=None, rel_embeddings=None, output_attentions=False)

Constructs the DebertaLayer by performing attention, intermediate, and output operations.

PARAMETER	DESCRIPTION
self	The class instance. TYPE: object
hidden_states	The input hidden states tensor. TYPE: Tensor
attention_mask	The attention mask tensor to mask out padded tokens. TYPE: Tensor
query_states	The tensor representing query states for attention computation. Defaults to None. TYPE: Tensor DEFAULT: None
relative_pos	The tensor representing relative positions for attention computation. Defaults to None. TYPE: Tensor DEFAULT: None
rel_embeddings	The tensor containing relative embeddings for attention computation. Defaults to None. TYPE: Tensor DEFAULT: None
output_attentions	Flag indicating whether to output attention matrices. Defaults to False. TYPE: bool DEFAULT: False

RETURNS	DESCRIPTION
	None.

RAISES	DESCRIPTION
ValueError	If the dimensions of the input tensors are incompatible.
TypeError	If the input parameters are not of the expected types.

Source code in mindnlp/transformers/models/deberta/modeling_deberta.py

def forward(
    self,
    hidden_states,
    attention_mask,
    query_states=None,
    relative_pos=None,
    rel_embeddings=None,
    output_attentions=False,
):
    """
    Constructs the DebertaLayer by performing attention, intermediate, and output operations.

    Args:
        self (object): The class instance.
        hidden_states (torch.Tensor): The input hidden states tensor.
        attention_mask (torch.Tensor): The attention mask tensor to mask out padded tokens.
        query_states (torch.Tensor, optional): The tensor representing query states for attention computation.
            Defaults to None.
        relative_pos (torch.Tensor, optional): The tensor representing relative positions for attention computation.
            Defaults to None.
        rel_embeddings (torch.Tensor, optional): The tensor containing relative embeddings for attention computation.
            Defaults to None.
        output_attentions (bool): Flag indicating whether to output attention matrices. Defaults to False.

    Returns:
        None.

    Raises:
        ValueError: If the dimensions of the input tensors are incompatible.
        TypeError: If the input parameters are not of the expected types.
    """
    attention_output = self.attention(
        hidden_states,
        attention_mask,
        output_attentions=output_attentions,
        query_states=query_states,
        relative_pos=relative_pos,
        rel_embeddings=rel_embeddings,
    )
    if output_attentions:
        attention_output, att_matrix = attention_output
    intermediate_output = self.intermediate(attention_output)
    layer_output = self.output(intermediate_output, attention_output)
    if output_attentions:
        return (layer_output, att_matrix)
    return layer_output

mindnlp.transformers.models.deberta.modeling_deberta.DebertaLayerNorm

Bases: Module

LayerNorm module in the TF style (epsilon inside the square root).

Source code in mindnlp/transformers/models/deberta/modeling_deberta.py

class DebertaLayerNorm(nn.Module):
    """LayerNorm module in the TF style (epsilon inside the square root)."""
    def __init__(self, size, eps=1e-12):
        """
        Initializes an instance of the DebertaLayerNorm class.

        Args:
            self: The instance of the class.
            size (int): The size of the layer normalization parameters.
                It determines the shape of the weight and bias tensors.
            eps (float, optional): The epsilon value used for numerical stability.
                It prevents division by zero. Default is 1e-12.

        Returns:
            None

        Raises:
            None
        """
        super().__init__()
        self.weight = Parameter(ops.ones(size))
        self.bias = Parameter(ops.zeros(size))
        self.variance_epsilon = eps

    def forward(self, hidden_states):
        """
        This method forwards layer normalization for hidden states in a Deberta model.

        Args:
            self (DebertaLayerNorm): The instance of the DebertaLayerNorm class.
            hidden_states (torch.Tensor): The input hidden states tensor to be normalized.
                Should be a tensor of dtype float32.

        Returns:
            None: The method performs layer normalization on the hidden_states tensor in place.

        Raises:
            ValueError: If the input hidden_states tensor is not of dtype float32.
            RuntimeError: If any runtime error occurs during the normalization process.
        """
        input_type = hidden_states.dtype
        hidden_states = hidden_states.float()
        mean = hidden_states.mean(-1, keep_dims=True)
        variance = (hidden_states - mean).pow(2).mean(-1, keep_dims=True)
        hidden_states = (hidden_states - mean) / ops.sqrt(variance + self.variance_epsilon)
        hidden_states = hidden_states.to(input_type)
        y = self.weight * hidden_states + self.bias
        return y

mindnlp.transformers.models.deberta.modeling_deberta.DebertaLayerNorm.__init__(size, eps=1e-12)

Initializes an instance of the DebertaLayerNorm class.

PARAMETER	DESCRIPTION
self	The instance of the class.
size	The size of the layer normalization parameters. It determines the shape of the weight and bias tensors. TYPE: int
eps	The epsilon value used for numerical stability. It prevents division by zero. Default is 1e-12. TYPE: float DEFAULT: 1e-12

RETURNS	DESCRIPTION
	None

Source code in mindnlp/transformers/models/deberta/modeling_deberta.py

def __init__(self, size, eps=1e-12):
    """
    Initializes an instance of the DebertaLayerNorm class.

    Args:
        self: The instance of the class.
        size (int): The size of the layer normalization parameters.
            It determines the shape of the weight and bias tensors.
        eps (float, optional): The epsilon value used for numerical stability.
            It prevents division by zero. Default is 1e-12.

    Returns:
        None

    Raises:
        None
    """
    super().__init__()
    self.weight = Parameter(ops.ones(size))
    self.bias = Parameter(ops.zeros(size))
    self.variance_epsilon = eps

mindnlp.transformers.models.deberta.modeling_deberta.DebertaLayerNorm.forward(hidden_states)

This method forwards layer normalization for hidden states in a Deberta model.

PARAMETER	DESCRIPTION
self	The instance of the DebertaLayerNorm class. TYPE: DebertaLayerNorm
hidden_states	The input hidden states tensor to be normalized. Should be a tensor of dtype float32. TYPE: Tensor

RETURNS	DESCRIPTION
None	The method performs layer normalization on the hidden_states tensor in place.

RAISES	DESCRIPTION
ValueError	If the input hidden_states tensor is not of dtype float32.
RuntimeError	If any runtime error occurs during the normalization process.

Source code in mindnlp/transformers/models/deberta/modeling_deberta.py

def forward(self, hidden_states):
    """
    This method forwards layer normalization for hidden states in a Deberta model.

    Args:
        self (DebertaLayerNorm): The instance of the DebertaLayerNorm class.
        hidden_states (torch.Tensor): The input hidden states tensor to be normalized.
            Should be a tensor of dtype float32.

    Returns:
        None: The method performs layer normalization on the hidden_states tensor in place.

    Raises:
        ValueError: If the input hidden_states tensor is not of dtype float32.
        RuntimeError: If any runtime error occurs during the normalization process.
    """
    input_type = hidden_states.dtype
    hidden_states = hidden_states.float()
    mean = hidden_states.mean(-1, keep_dims=True)
    variance = (hidden_states - mean).pow(2).mean(-1, keep_dims=True)
    hidden_states = (hidden_states - mean) / ops.sqrt(variance + self.variance_epsilon)
    hidden_states = hidden_states.to(input_type)
    y = self.weight * hidden_states + self.bias
    return y

mindnlp.transformers.models.deberta.modeling_deberta.DebertaModel

Bases: DebertaPreTrainedModel

DebertaModel class represents a DeBERTa model for natural language processing tasks. This class inherits functionalities from DebertaPreTrainedModel and implements methods for initializing the model, getting and setting input embeddings, and forwarding the model output.

ATTRIBUTE	DESCRIPTION
embeddings	The embeddings module of the DeBERTa model. TYPE: DebertaEmbeddings
encoder	The encoder module of the DeBERTa model. TYPE: DebertaEncoder
z_steps	Number of Z steps used in the model. TYPE: int
config	Configuration object for the model.

METHOD	DESCRIPTION
__init__	Initializes the DebertaModel with the provided configuration.
get_input_embeddings	Retrieves the word embeddings from the input embeddings.
set_input_embeddings	Sets new word embeddings for the input embeddings.
_prune_heads	Prunes heads of the model based on the provided dictionary.
forward	Constructs the model output based on the input parameters.

RAISES	DESCRIPTION
NotImplementedError	If the prune function is called as it is not implemented in the DeBERTa model.
ValueError	If both input_ids and inputs_embeds are specified simultaneously, or if neither input_ids nor inputs_embeds are provided.

RETURNS	DESCRIPTION
	Tuple or BaseModelOutput: Depending on the configuration settings, returns either a tuple or a BaseModelOutput object containing the model output.

Note

This class is designed for use in natural language processing tasks and leverages the DeBERTa architecture for efficient modeling.

Source code in mindnlp/transformers/models/deberta/modeling_deberta.py

class DebertaModel(DebertaPreTrainedModel):

    """
    DebertaModel class represents a DeBERTa model for natural language processing tasks.
    This class inherits functionalities from DebertaPreTrainedModel and implements methods for initializing the model,
    getting and setting input embeddings, and forwarding the model output.

    Attributes:
        embeddings (DebertaEmbeddings): The embeddings module of the DeBERTa model.
        encoder (DebertaEncoder): The encoder module of the DeBERTa model.
        z_steps (int): Number of Z steps used in the model.
        config: Configuration object for the model.

    Methods:
        __init__: Initializes the DebertaModel with the provided configuration.
        get_input_embeddings: Retrieves the word embeddings from the input embeddings.
        set_input_embeddings: Sets new word embeddings for the input embeddings.
        _prune_heads: Prunes heads of the model based on the provided dictionary.
        forward: Constructs the model output based on the input parameters.

    Raises:
        NotImplementedError: If the prune function is called as it is not implemented in the DeBERTa model.
        ValueError: If both input_ids and inputs_embeds are specified simultaneously,
            or if neither input_ids nor inputs_embeds are provided.

    Returns:
        Tuple or BaseModelOutput:
            Depending on the configuration settings, returns either a tuple or a BaseModelOutput object
            containing the model output.

    Note:
        This class is designed for use in natural language processing tasks and leverages the DeBERTa architecture
        for efficient modeling.

    """
    def __init__(self, config):
        """
        Initializes a new instance of the DebertaModel class.

        Args:
            self: The instance of the class.
            config (object): The configuration object containing the model configuration parameters.

        Returns:
            None.

        Raises:
            None.
        """
        super().__init__(config)

        self.embeddings = DebertaEmbeddings(config)
        self.encoder = DebertaEncoder(config)
        self.z_steps = 0
        self.config = config
        # Initialize weights and apply final processing
        self.post_init()

    def get_input_embeddings(self):
        """
        Retrieve the input embeddings from the DebertaModel.

        Args:
            self (DebertaModel): An instance of the DebertaModel class.

        Returns:
            None.

        Raises:
            None.
        """
        return self.embeddings.word_embeddings

    def set_input_embeddings(self, new_embeddings):
        """
        Method to set the input embeddings for a DebertaModel instance.

        Args:
            self (DebertaModel): The instance of the DebertaModel class.
            new_embeddings (object): New input embeddings to be set for the model.
                It should be of the appropriate type compatible with the model's word_embeddings attribute.

        Returns:
            None.

        Raises:
            TypeError: If the new_embeddings parameter is not of the expected type.
            ValueError: If the new_embeddings parameter is invalid or incompatible with the model.
        """
        self.embeddings.word_embeddings = new_embeddings

    def _prune_heads(self, heads_to_prune):
        """
        Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base
        class PreTrainedModel
        """
        raise NotImplementedError("The prune function is not implemented in DeBERTa model.")

    def forward(
        self,
        input_ids: Optional[mindspore.Tensor] = None,
        attention_mask: Optional[mindspore.Tensor] = None,
        token_type_ids: Optional[mindspore.Tensor] = None,
        position_ids: Optional[mindspore.Tensor] = None,
        inputs_embeds: Optional[mindspore.Tensor] = None,
        output_attentions: Optional[bool] = None,
        output_hidden_states: Optional[bool] = None,
        return_dict: Optional[bool] = None,
    ) -> Union[Tuple, BaseModelOutput]:
        """
        This method forwards a DebertaModel based on the provided input parameters.

        Args:
            self (object): The instance of the DebertaModel class.
            input_ids (Optional[mindspore.Tensor]): The input tensor containing token indices. Default is None.
            attention_mask (Optional[mindspore.Tensor]):
                The attention mask tensor to specify which tokens should be attended to. Default is None.
            token_type_ids (Optional[mindspore.Tensor]): The tensor specifying the type of each token. Default is None.
            position_ids (Optional[mindspore.Tensor]): The tensor containing position indices of tokens. Default is None.
            inputs_embeds (Optional[mindspore.Tensor]):
                The tensor containing precomputed embeddings for input tokens. Default is None.
            output_attentions (Optional[bool]): Flag to indicate whether to output attentions. Default is None.
            output_hidden_states (Optional[bool]): Flag to indicate whether to output hidden states. Default is None.
            return_dict (Optional[bool]): Flag to indicate whether to return output as a dictionary. Default is None.

        Returns:
            Union[Tuple, BaseModelOutput]: The output value, which can either be a tuple or a BaseModelOutput object,
                containing the forwarded DebertaModel.

        Raises:
            ValueError: Raised if both input_ids and inputs_embeds are specified simultaneously.
            ValueError: Raised if neither input_ids nor inputs_embeds are specified.
        """
        output_attentions = output_attentions if output_attentions is not None else self.config.output_attentions
        output_hidden_states = (
            output_hidden_states if output_hidden_states is not None else self.config.output_hidden_states
        )
        return_dict = return_dict if return_dict is not None else self.config.use_return_dict

        if input_ids is not None and inputs_embeds is not None:
            raise ValueError("You cannot specify both input_ids and inputs_embeds at the same time")
        if input_ids is not None:
            self.warn_if_padding_and_no_attention_mask(input_ids, attention_mask)
            input_shape = input_ids.shape
        elif inputs_embeds is not None:
            input_shape = inputs_embeds.shape[:-1]
        else:
            raise ValueError("You have to specify either input_ids or inputs_embeds")

        if attention_mask is None:
            attention_mask = ops.ones(input_shape)
        if token_type_ids is None:
            token_type_ids = ops.zeros(input_shape, dtype=mindspore.int64)

        embedding_output = self.embeddings(
            input_ids=input_ids,
            token_type_ids=token_type_ids,
            position_ids=position_ids,
            mask=attention_mask,
            inputs_embeds=inputs_embeds,
        )

        encoder_outputs = self.encoder(
            embedding_output,
            attention_mask,
            output_hidden_states=True,
            output_attentions=output_attentions,
            return_dict=return_dict,
        )
        encoded_layers = encoder_outputs[1]

        if self.z_steps > 1:
            hidden_states = encoded_layers[-2]
            layers = [self.encoder.layer[-1] for _ in range(self.z_steps)]
            query_states = encoded_layers[-1]
            rel_embeddings = self.encoder.get_rel_embedding()
            attention_mask = self.encoder.get_attention_mask(attention_mask)
            rel_pos = self.encoder.get_rel_pos(embedding_output)
            for layer in layers[1:]:
                query_states = layer(
                    hidden_states,
                    attention_mask,
                    output_attentions=False,
                    query_states=query_states,
                    relative_pos=rel_pos,
                    rel_embeddings=rel_embeddings,
                )
                encoded_layers.append(query_states)

        sequence_output = encoded_layers[-1]

        if not return_dict:
            return (sequence_output,) + encoder_outputs[(1 if output_hidden_states else 2) :]

        return BaseModelOutput(
            last_hidden_state=sequence_output,
            hidden_states=encoder_outputs.hidden_states if output_hidden_states else None,
            attentions=encoder_outputs.attentions,
        )

mindnlp.transformers.models.deberta.modeling_deberta.DebertaModel.__init__(config)

Initializes a new instance of the DebertaModel class.

PARAMETER	DESCRIPTION
self	The instance of the class.
config	The configuration object containing the model configuration parameters. TYPE: object

RETURNS	DESCRIPTION
	None.

Source code in mindnlp/transformers/models/deberta/modeling_deberta.py

def __init__(self, config):
    """
    Initializes a new instance of the DebertaModel class.

    Args:
        self: The instance of the class.
        config (object): The configuration object containing the model configuration parameters.

    Returns:
        None.

    Raises:
        None.
    """
    super().__init__(config)

    self.embeddings = DebertaEmbeddings(config)
    self.encoder = DebertaEncoder(config)
    self.z_steps = 0
    self.config = config
    # Initialize weights and apply final processing
    self.post_init()

mindnlp.transformers.models.deberta.modeling_deberta.DebertaModel.forward(input_ids=None, attention_mask=None, token_type_ids=None, position_ids=None, inputs_embeds=None, output_attentions=None, output_hidden_states=None, return_dict=None)

This method forwards a DebertaModel based on the provided input parameters.

PARAMETER	DESCRIPTION
self	The instance of the DebertaModel class. TYPE: object
input_ids	The input tensor containing token indices. Default is None. TYPE: Optional[Tensor] DEFAULT: None
attention_mask	The attention mask tensor to specify which tokens should be attended to. Default is None. TYPE: Optional[Tensor] DEFAULT: None
token_type_ids	The tensor specifying the type of each token. Default is None. TYPE: Optional[Tensor] DEFAULT: None
position_ids	The tensor containing position indices of tokens. Default is None. TYPE: Optional[Tensor] DEFAULT: None
inputs_embeds	The tensor containing precomputed embeddings for input tokens. Default is None. TYPE: Optional[Tensor] DEFAULT: None
output_attentions	Flag to indicate whether to output attentions. Default is None. TYPE: Optional[bool] DEFAULT: None
output_hidden_states	Flag to indicate whether to output hidden states. Default is None. TYPE: Optional[bool] DEFAULT: None
return_dict	Flag to indicate whether to return output as a dictionary. Default is None. TYPE: Optional[bool] DEFAULT: None

RETURNS	DESCRIPTION
Union[Tuple, BaseModelOutput]	Union[Tuple, BaseModelOutput]: The output value, which can either be a tuple or a BaseModelOutput object, containing the forwarded DebertaModel.

RAISES	DESCRIPTION
ValueError	Raised if both input_ids and inputs_embeds are specified simultaneously.
ValueError	Raised if neither input_ids nor inputs_embeds are specified.

Source code in mindnlp/transformers/models/deberta/modeling_deberta.py

def forward(
    self,
    input_ids: Optional[mindspore.Tensor] = None,
    attention_mask: Optional[mindspore.Tensor] = None,
    token_type_ids: Optional[mindspore.Tensor] = None,
    position_ids: Optional[mindspore.Tensor] = None,
    inputs_embeds: Optional[mindspore.Tensor] = None,
    output_attentions: Optional[bool] = None,
    output_hidden_states: Optional[bool] = None,
    return_dict: Optional[bool] = None,
) -> Union[Tuple, BaseModelOutput]:
    """
    This method forwards a DebertaModel based on the provided input parameters.

    Args:
        self (object): The instance of the DebertaModel class.
        input_ids (Optional[mindspore.Tensor]): The input tensor containing token indices. Default is None.
        attention_mask (Optional[mindspore.Tensor]):
            The attention mask tensor to specify which tokens should be attended to. Default is None.
        token_type_ids (Optional[mindspore.Tensor]): The tensor specifying the type of each token. Default is None.
        position_ids (Optional[mindspore.Tensor]): The tensor containing position indices of tokens. Default is None.
        inputs_embeds (Optional[mindspore.Tensor]):
            The tensor containing precomputed embeddings for input tokens. Default is None.
        output_attentions (Optional[bool]): Flag to indicate whether to output attentions. Default is None.
        output_hidden_states (Optional[bool]): Flag to indicate whether to output hidden states. Default is None.
        return_dict (Optional[bool]): Flag to indicate whether to return output as a dictionary. Default is None.

    Returns:
        Union[Tuple, BaseModelOutput]: The output value, which can either be a tuple or a BaseModelOutput object,
            containing the forwarded DebertaModel.

    Raises:
        ValueError: Raised if both input_ids and inputs_embeds are specified simultaneously.
        ValueError: Raised if neither input_ids nor inputs_embeds are specified.
    """
    output_attentions = output_attentions if output_attentions is not None else self.config.output_attentions
    output_hidden_states = (
        output_hidden_states if output_hidden_states is not None else self.config.output_hidden_states
    )
    return_dict = return_dict if return_dict is not None else self.config.use_return_dict

    if input_ids is not None and inputs_embeds is not None:
        raise ValueError("You cannot specify both input_ids and inputs_embeds at the same time")
    if input_ids is not None:
        self.warn_if_padding_and_no_attention_mask(input_ids, attention_mask)
        input_shape = input_ids.shape
    elif inputs_embeds is not None:
        input_shape = inputs_embeds.shape[:-1]
    else:
        raise ValueError("You have to specify either input_ids or inputs_embeds")

    if attention_mask is None:
        attention_mask = ops.ones(input_shape)
    if token_type_ids is None:
        token_type_ids = ops.zeros(input_shape, dtype=mindspore.int64)

    embedding_output = self.embeddings(
        input_ids=input_ids,
        token_type_ids=token_type_ids,
        position_ids=position_ids,
        mask=attention_mask,
        inputs_embeds=inputs_embeds,
    )

    encoder_outputs = self.encoder(
        embedding_output,
        attention_mask,
        output_hidden_states=True,
        output_attentions=output_attentions,
        return_dict=return_dict,
    )
    encoded_layers = encoder_outputs[1]

    if self.z_steps > 1:
        hidden_states = encoded_layers[-2]
        layers = [self.encoder.layer[-1] for _ in range(self.z_steps)]
        query_states = encoded_layers[-1]
        rel_embeddings = self.encoder.get_rel_embedding()
        attention_mask = self.encoder.get_attention_mask(attention_mask)
        rel_pos = self.encoder.get_rel_pos(embedding_output)
        for layer in layers[1:]:
            query_states = layer(
                hidden_states,
                attention_mask,
                output_attentions=False,
                query_states=query_states,
                relative_pos=rel_pos,
                rel_embeddings=rel_embeddings,
            )
            encoded_layers.append(query_states)

    sequence_output = encoded_layers[-1]

    if not return_dict:
        return (sequence_output,) + encoder_outputs[(1 if output_hidden_states else 2) :]

    return BaseModelOutput(
        last_hidden_state=sequence_output,
        hidden_states=encoder_outputs.hidden_states if output_hidden_states else None,
        attentions=encoder_outputs.attentions,
    )

mindnlp.transformers.models.deberta.modeling_deberta.DebertaModel.get_input_embeddings()

Retrieve the input embeddings from the DebertaModel.

PARAMETER	DESCRIPTION
self	An instance of the DebertaModel class. TYPE: DebertaModel

RETURNS	DESCRIPTION
	None.

Source code in mindnlp/transformers/models/deberta/modeling_deberta.py

def get_input_embeddings(self):
    """
    Retrieve the input embeddings from the DebertaModel.

    Args:
        self (DebertaModel): An instance of the DebertaModel class.

    Returns:
        None.

    Raises:
        None.
    """
    return self.embeddings.word_embeddings

mindnlp.transformers.models.deberta.modeling_deberta.DebertaModel.set_input_embeddings(new_embeddings)

Method to set the input embeddings for a DebertaModel instance.

PARAMETER	DESCRIPTION
self	The instance of the DebertaModel class. TYPE: DebertaModel
new_embeddings	New input embeddings to be set for the model. It should be of the appropriate type compatible with the model's word_embeddings attribute. TYPE: object

RETURNS	DESCRIPTION
	None.

RAISES	DESCRIPTION
TypeError	If the new_embeddings parameter is not of the expected type.
ValueError	If the new_embeddings parameter is invalid or incompatible with the model.

Source code in mindnlp/transformers/models/deberta/modeling_deberta.py

def set_input_embeddings(self, new_embeddings):
    """
    Method to set the input embeddings for a DebertaModel instance.

    Args:
        self (DebertaModel): The instance of the DebertaModel class.
        new_embeddings (object): New input embeddings to be set for the model.
            It should be of the appropriate type compatible with the model's word_embeddings attribute.

    Returns:
        None.

    Raises:
        TypeError: If the new_embeddings parameter is not of the expected type.
        ValueError: If the new_embeddings parameter is invalid or incompatible with the model.
    """
    self.embeddings.word_embeddings = new_embeddings

mindnlp.transformers.models.deberta.modeling_deberta.DebertaOnlyMLMHead

Bases: Module

This class represents a Deberta Masked Language Model (MLM) head for generating prediction scores from sequence output. It inherits from nn.Module and contains methods for initializing the MLM head and forwarding prediction scores.

ATTRIBUTE	DESCRIPTION
predictions	A DebertaLMPredictionHead object for generating prediction scores.

METHOD	DESCRIPTION
__init__	Initializes the DebertaOnlyMLMHead with the given configuration.
forward	Constructs prediction scores from the provided sequence output.

Source code in mindnlp/transformers/models/deberta/modeling_deberta.py

class DebertaOnlyMLMHead(nn.Module):

    """
    This class represents a Deberta Masked Language Model (MLM) head for generating prediction scores from sequence output.
    It inherits from nn.Module and contains methods for initializing the MLM head and forwarding prediction scores.

    Attributes:
        predictions: A DebertaLMPredictionHead object for generating prediction scores.

    Methods:
        __init__: Initializes the DebertaOnlyMLMHead with the given configuration.
        forward: Constructs prediction scores from the provided sequence output.
    """
    def __init__(self, config):
        """
        Initializes an instance of the DebertaOnlyMLMHead class.

        Args:
            self: The instance of the class.
            config: A configuration object containing the necessary settings for the DebertaOnlyMLMHead.

        Returns:
            None

        Raises:
            None
        """
        super().__init__()
        self.predictions = DebertaLMPredictionHead(config)

    def forward(self, sequence_output):
        """
        Class:
            DebertaOnlyMLMHead

        Method:
            forward

        Description:
            This method forwards prediction scores based on the given sequence output.

        Args:
            self: (object) The instance of the DebertaOnlyMLMHead class.
            sequence_output: (object) The sequence output from the model for which prediction scores need to be generated.

        Returns:
            None.

        Raises:
            None.
        """
        prediction_scores = self.predictions(sequence_output)
        return prediction_scores

mindnlp.transformers.models.deberta.modeling_deberta.DebertaOnlyMLMHead.__init__(config)

Initializes an instance of the DebertaOnlyMLMHead class.

PARAMETER	DESCRIPTION
self	The instance of the class.
config	A configuration object containing the necessary settings for the DebertaOnlyMLMHead.

RETURNS	DESCRIPTION
	None

Source code in mindnlp/transformers/models/deberta/modeling_deberta.py

def __init__(self, config):
    """
    Initializes an instance of the DebertaOnlyMLMHead class.

    Args:
        self: The instance of the class.
        config: A configuration object containing the necessary settings for the DebertaOnlyMLMHead.

    Returns:
        None

    Raises:
        None
    """
    super().__init__()
    self.predictions = DebertaLMPredictionHead(config)

mindnlp.transformers.models.deberta.modeling_deberta.DebertaOnlyMLMHead.forward(sequence_output)

Class

DebertaOnlyMLMHead

Method

forward

Description

This method forwards prediction scores based on the given sequence output.

PARAMETER	DESCRIPTION
self	(object) The instance of the DebertaOnlyMLMHead class.
sequence_output	(object) The sequence output from the model for which prediction scores need to be generated.

RETURNS	DESCRIPTION
	None.

Source code in mindnlp/transformers/models/deberta/modeling_deberta.py

def forward(self, sequence_output):
    """
    Class:
        DebertaOnlyMLMHead

    Method:
        forward

    Description:
        This method forwards prediction scores based on the given sequence output.

    Args:
        self: (object) The instance of the DebertaOnlyMLMHead class.
        sequence_output: (object) The sequence output from the model for which prediction scores need to be generated.

    Returns:
        None.

    Raises:
        None.
    """
    prediction_scores = self.predictions(sequence_output)
    return prediction_scores

mindnlp.transformers.models.deberta.modeling_deberta.DebertaOutput

Bases: Module

This class represents the output layer of the Deberta model. It inherits from the nn.Module class and is responsible for applying the final transformations to the hidden states.

ATTRIBUTE	DESCRIPTION
dense	A dense layer that transforms the hidden states to an intermediate size. TYPE: Linear
LayerNorm	A layer normalization module that normalizes the hidden states. TYPE: DebertaLayerNorm
dropout	A dropout layer that applies dropout to the hidden states. TYPE: StableDropout
config	The configuration object for the Deberta model.

METHOD	DESCRIPTION
__init__	Initializes the DebertaOutput instance. Args: config: The configuration object for the Deberta model.
forward	Applies the final transformations to the hidden states. Args: hidden_states: The input hidden states. input_tensor: The original input tensor. Returns: The transformed hidden states after applying the intermediate dense layer, dropout, and layer normalization.

Source code in mindnlp/transformers/models/deberta/modeling_deberta.py

class DebertaOutput(nn.Module):

    """
    This class represents the output layer of the Deberta model.
    It inherits from the nn.Module class and is responsible for applying the final transformations to the hidden states.

    Attributes:
        dense (nn.Linear): A dense layer that transforms the hidden states to an intermediate size.
        LayerNorm (DebertaLayerNorm): A layer normalization module that normalizes the hidden states.
        dropout (StableDropout): A dropout layer that applies dropout to the hidden states.
        config: The configuration object for the Deberta model.

    Methods:
        __init__(self, config):
            Initializes the DebertaOutput instance.

            Args:

            - config: The configuration object for the Deberta model.

        forward(self, hidden_states, input_tensor):
            Applies the final transformations to the hidden states.

            Args:

            - hidden_states: The input hidden states.
            - input_tensor: The original input tensor.

            Returns:

            - The transformed hidden states after applying the intermediate dense layer, dropout, and layer normalization.
    """
    def __init__(self, config):
        """
        Initializes a new instance of the DebertaOutput class.

        Args:
            self: The instance of the DebertaOutput class.
            config:
                An instance of the configuration class containing the parameters for the DebertaOutput layer.

                - Type: object
                - Purpose: Specifies the configuration settings for the DebertaOutput layer.
                - Restrictions: Must be a valid instance of the configuration class.

        Returns:
            None.

        Raises:
            None.
        """
        super().__init__()
        self.dense = nn.Linear(config.intermediate_size, config.hidden_size)
        self.LayerNorm = DebertaLayerNorm(config.hidden_size, config.layer_norm_eps)
        self.dropout = StableDropout(config.hidden_dropout_prob)
        self.config = config

    def forward(self, hidden_states, input_tensor):
        """
        Constructs the output of the Deberta model by performing a series of operations.

        Args:
            self (DebertaOutput): The instance of the DebertaOutput class.
            hidden_states (Tensor): The input hidden states.
                This tensor represents the intermediate outputs of the model.
            input_tensor (Tensor): The input tensor to be added to the hidden states.

        Returns:
            None

        Raises:
            None
        """
        hidden_states = self.dense(hidden_states)
        hidden_states = self.dropout(hidden_states)
        hidden_states = self.LayerNorm(hidden_states + input_tensor)
        return hidden_states

mindnlp.transformers.models.deberta.modeling_deberta.DebertaOutput.__init__(config)

Initializes a new instance of the DebertaOutput class.

PARAMETER	DESCRIPTION
self	The instance of the DebertaOutput class.
config	An instance of the configuration class containing the parameters for the DebertaOutput layer. Type: object Purpose: Specifies the configuration settings for the DebertaOutput layer. Restrictions: Must be a valid instance of the configuration class.

RETURNS	DESCRIPTION
	None.

Source code in mindnlp/transformers/models/deberta/modeling_deberta.py

def __init__(self, config):
    """
    Initializes a new instance of the DebertaOutput class.

    Args:
        self: The instance of the DebertaOutput class.
        config:
            An instance of the configuration class containing the parameters for the DebertaOutput layer.

            - Type: object
            - Purpose: Specifies the configuration settings for the DebertaOutput layer.
            - Restrictions: Must be a valid instance of the configuration class.

    Returns:
        None.

    Raises:
        None.
    """
    super().__init__()
    self.dense = nn.Linear(config.intermediate_size, config.hidden_size)
    self.LayerNorm = DebertaLayerNorm(config.hidden_size, config.layer_norm_eps)
    self.dropout = StableDropout(config.hidden_dropout_prob)
    self.config = config

mindnlp.transformers.models.deberta.modeling_deberta.DebertaOutput.forward(hidden_states, input_tensor)

Constructs the output of the Deberta model by performing a series of operations.

PARAMETER	DESCRIPTION
self	The instance of the DebertaOutput class. TYPE: DebertaOutput
hidden_states	The input hidden states. This tensor represents the intermediate outputs of the model. TYPE: Tensor
input_tensor	The input tensor to be added to the hidden states. TYPE: Tensor

RETURNS	DESCRIPTION
	None

Source code in mindnlp/transformers/models/deberta/modeling_deberta.py

def forward(self, hidden_states, input_tensor):
    """
    Constructs the output of the Deberta model by performing a series of operations.

    Args:
        self (DebertaOutput): The instance of the DebertaOutput class.
        hidden_states (Tensor): The input hidden states.
            This tensor represents the intermediate outputs of the model.
        input_tensor (Tensor): The input tensor to be added to the hidden states.

    Returns:
        None

    Raises:
        None
    """
    hidden_states = self.dense(hidden_states)
    hidden_states = self.dropout(hidden_states)
    hidden_states = self.LayerNorm(hidden_states + input_tensor)
    return hidden_states

mindnlp.transformers.models.deberta.modeling_deberta.DebertaPreTrainedModel

Bases: PreTrainedModel

An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models.

Source code in mindnlp/transformers/models/deberta/modeling_deberta.py

class DebertaPreTrainedModel(PreTrainedModel):
    """
    An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained
    models.
    """
    config_class = DebertaConfig
    base_model_prefix = "deberta"
    _keys_to_ignore_on_load_unexpected = ["position_embeddings"]
    supports_gradient_checkpointing = True

    def _init_weights(self, cell):
        """Initialize the weights"""
        if isinstance(cell, nn.Linear):
            # Slightly different from the TF version which uses truncated_normal for initialization
            # cf https://github.com/pytorch/pytorch/pull/5617
            cell.weight.set_data(initializer(Normal(self.config.initializer_range),
                                                    cell.weight.shape, cell.weight.dtype))
            if cell.bias:
                cell.bias.set_data(initializer('zeros', cell.bias.shape, cell.bias.dtype))
        elif isinstance(cell, nn.Embedding):
            weight = np.random.normal(0.0, self.config.initializer_range, cell.weight.shape)
            if cell.padding_idx:
                weight[cell.padding_idx] = 0

            cell.weight.set_data(Tensor(weight, cell.weight.dtype))

mindnlp.transformers.models.deberta.modeling_deberta.DebertaPredictionHeadTransform

Bases: Module

Represents a prediction head transformation module for the DeBERTa model.

This class defines a prediction head transformation module for the DeBERTa model, which includes operations such as dense layer, activation function transformation, and layer normalization.

ATTRIBUTE	DESCRIPTION
embedding_size	The size of the embedding used in the transformation. TYPE: int
dense	The dense layer used for transformation. TYPE: Linear
transform_act_fn	The activation function used for transformation. TYPE: function
LayerNorm	The layer normalization module applied to the hidden states. TYPE: LayerNorm

METHOD	DESCRIPTION
__init__	Initializes the DebertaPredictionHeadTransform instance with the given configuration.
forward	Constructs the prediction head transformation on the input hidden states.

Note

This class inherits from nn.Module and is designed specifically for the DeBERTa model.

Source code in mindnlp/transformers/models/deberta/modeling_deberta.py

class DebertaPredictionHeadTransform(nn.Module):

    """
    Represents a prediction head transformation module for the DeBERTa model.

    This class defines a prediction head transformation module for the DeBERTa model,
    which includes operations such as dense layer, activation function transformation, and layer normalization.

    Attributes:
        embedding_size (int): The size of the embedding used in the transformation.
        dense (nn.Linear): The dense layer used for transformation.
        transform_act_fn (function): The activation function used for transformation.
        LayerNorm (nn.LayerNorm): The layer normalization module applied to the hidden states.

    Methods:
        __init__: Initializes the DebertaPredictionHeadTransform instance with the given configuration.
        forward: Constructs the prediction head transformation on the input hidden states.

    Note:
        This class inherits from nn.Module and is designed specifically for the DeBERTa model.

    """
    def __init__(self, config):
        """
        Initializes the DebertaPredictionHeadTransform class.

        Args:
            self (DebertaPredictionHeadTransform): The instance of the DebertaPredictionHeadTransform class.
            config (object): The configuration object containing parameters for the prediction head.
                It should include the following attributes:

                - embedding_size (int, optional): The size of the embedding. Defaults to config.hidden_size.
                - hidden_size (int): The size of the hidden layer.
                - hidden_act (str or object): The activation function for the hidden layer.
                If a string, it should be a key in the ACT2FN dictionary.
                - layer_norm_eps (float): The epsilon value for LayerNorm.

        Returns:
            None.

        Raises:
            TypeError: If the config parameter is not of the expected type.
            KeyError: If the config.hidden_act is a string that does not match any key in the ACT2FN dictionary.
            ValueError: If the config does not contain the required attributes.
        """
        super().__init__()
        self.embedding_size = getattr(config, "embedding_size", config.hidden_size)

        self.dense = nn.Linear(config.hidden_size, self.embedding_size)
        if isinstance(config.hidden_act, str):
            self.transform_act_fn = ACT2FN[config.hidden_act]
        else:
            self.transform_act_fn = config.hidden_act
        self.LayerNorm = nn.LayerNorm(self.embedding_size, eps=config.layer_norm_eps)

    def forward(self, hidden_states):
        """
        This method 'forward' is defined within the class 'DebertaPredictionHeadTransform' and is responsible for
        processing the hidden states.

        Args:
            self: An instance of the 'DebertaPredictionHeadTransform' class.
            hidden_states:
                A tensor representing the hidden states to be processed.
                It is of type 'Tensor' and is expected to contain the information to be transformed.

        Returns:
            hidden_states: A tensor containing the transformed hidden states after processing.
                It is of type 'Tensor' and represents the result of the transformation operation.

        Raises:
            This method does not explicitly raise any exceptions.
        """
        hidden_states = self.dense(hidden_states)
        hidden_states = self.transform_act_fn(hidden_states)
        hidden_states = self.LayerNorm(hidden_states)
        return hidden_states

mindnlp.transformers.models.deberta.modeling_deberta.DebertaPredictionHeadTransform.__init__(config)

Initializes the DebertaPredictionHeadTransform class.

PARAMETER	DESCRIPTION
self	The instance of the DebertaPredictionHeadTransform class. TYPE: DebertaPredictionHeadTransform
config	The configuration object containing parameters for the prediction head. It should include the following attributes: embedding_size (int, optional): The size of the embedding. Defaults to config.hidden_size. hidden_size (int): The size of the hidden layer. hidden_act (str or object): The activation function for the hidden layer. If a string, it should be a key in the ACT2FN dictionary. layer_norm_eps (float): The epsilon value for LayerNorm. TYPE: object

RETURNS	DESCRIPTION
	None.

RAISES	DESCRIPTION
TypeError	If the config parameter is not of the expected type.
KeyError	If the config.hidden_act is a string that does not match any key in the ACT2FN dictionary.
ValueError	If the config does not contain the required attributes.

Source code in mindnlp/transformers/models/deberta/modeling_deberta.py

def __init__(self, config):
    """
    Initializes the DebertaPredictionHeadTransform class.

    Args:
        self (DebertaPredictionHeadTransform): The instance of the DebertaPredictionHeadTransform class.
        config (object): The configuration object containing parameters for the prediction head.
            It should include the following attributes:

            - embedding_size (int, optional): The size of the embedding. Defaults to config.hidden_size.
            - hidden_size (int): The size of the hidden layer.
            - hidden_act (str or object): The activation function for the hidden layer.
            If a string, it should be a key in the ACT2FN dictionary.
            - layer_norm_eps (float): The epsilon value for LayerNorm.

    Returns:
        None.

    Raises:
        TypeError: If the config parameter is not of the expected type.
        KeyError: If the config.hidden_act is a string that does not match any key in the ACT2FN dictionary.
        ValueError: If the config does not contain the required attributes.
    """
    super().__init__()
    self.embedding_size = getattr(config, "embedding_size", config.hidden_size)

    self.dense = nn.Linear(config.hidden_size, self.embedding_size)
    if isinstance(config.hidden_act, str):
        self.transform_act_fn = ACT2FN[config.hidden_act]
    else:
        self.transform_act_fn = config.hidden_act
    self.LayerNorm = nn.LayerNorm(self.embedding_size, eps=config.layer_norm_eps)

mindnlp.transformers.models.deberta.modeling_deberta.DebertaPredictionHeadTransform.forward(hidden_states)

This method 'forward' is defined within the class 'DebertaPredictionHeadTransform' and is responsible for processing the hidden states.

PARAMETER	DESCRIPTION
self	An instance of the 'DebertaPredictionHeadTransform' class.
hidden_states	A tensor representing the hidden states to be processed. It is of type 'Tensor' and is expected to contain the information to be transformed.

RETURNS	DESCRIPTION
hidden_states	A tensor containing the transformed hidden states after processing. It is of type 'Tensor' and represents the result of the transformation operation.

Source code in mindnlp/transformers/models/deberta/modeling_deberta.py

def forward(self, hidden_states):
    """
    This method 'forward' is defined within the class 'DebertaPredictionHeadTransform' and is responsible for
    processing the hidden states.

    Args:
        self: An instance of the 'DebertaPredictionHeadTransform' class.
        hidden_states:
            A tensor representing the hidden states to be processed.
            It is of type 'Tensor' and is expected to contain the information to be transformed.

    Returns:
        hidden_states: A tensor containing the transformed hidden states after processing.
            It is of type 'Tensor' and represents the result of the transformation operation.

    Raises:
        This method does not explicitly raise any exceptions.
    """
    hidden_states = self.dense(hidden_states)
    hidden_states = self.transform_act_fn(hidden_states)
    hidden_states = self.LayerNorm(hidden_states)
    return hidden_states

mindnlp.transformers.models.deberta.modeling_deberta.DebertaSelfOutput

Bases: Module

Represents the output layer for the DeBERTa model, responsible for transforming hidden states and applying normalization and dropout.

This class inherits from nn.Module and contains methods to initialize the output layer components, including dense transformation, layer normalization, and dropout. The 'forward' method takes hidden states and input tensor, applies transformations, and returns the final hidden states after normalization and dropout.

ATTRIBUTE	DESCRIPTION
dense	A fully connected layer for transforming hidden states. TYPE: Linear
LayerNorm	Layer normalization applied to the hidden states. TYPE: DebertaLayerNorm
dropout	Dropout regularization to prevent overfitting. TYPE: StableDropout

METHOD	DESCRIPTION
__init__	Initializes the output layer components with the given configuration.
forward	Applies transformations to hidden states and input tensor to produce final hidden states.

Source code in mindnlp/transformers/models/deberta/modeling_deberta.py

class DebertaSelfOutput(nn.Module):

    """
    Represents the output layer for the DeBERTa model, responsible for transforming hidden states and applying normalization and dropout.

    This class inherits from nn.Module and contains methods to initialize the output layer components,
    including dense transformation, layer normalization, and dropout.
    The 'forward' method takes hidden states and input tensor, applies transformations,
    and returns the final hidden states after normalization and dropout.

    Attributes:
        dense (nn.Linear): A fully connected layer for transforming hidden states.
        LayerNorm (DebertaLayerNorm): Layer normalization applied to the hidden states.
        dropout (StableDropout): Dropout regularization to prevent overfitting.

    Methods:
        __init__(self, config): Initializes the output layer components with the given configuration.
        forward(self, hidden_states, input_tensor):
            Applies transformations to hidden states and input tensor to produce final hidden states.

    """
    def __init__(self, config):
        """
        Initializes an instance of the DebertaSelfOutput class.

        Args:
            self (DebertaSelfOutput): The current instance of the class.
            config: The configuration object containing the settings for the Deberta model.

        Returns:
            None

        Raises:
            None
        """
        super().__init__()
        self.dense = nn.Linear(config.hidden_size, config.hidden_size)
        self.LayerNorm = DebertaLayerNorm(config.hidden_size, config.layer_norm_eps)
        self.dropout = StableDropout(config.hidden_dropout_prob)

    def forward(self, hidden_states, input_tensor):
        """
        Method 'forward' in the class 'DebertaSelfOutput'.

        This method forwards the hidden states by applying a series of operations on the input hidden states and the input tensor.

        Args:
            self:
                Instance of the DebertaSelfOutput class.

                - Type: DebertaSelfOutput
                - Purpose: Represents the current instance of the class.

            hidden_states:
                Hidden states that need to be processed.

                - Type: tensor
                - Purpose: Represents the input hidden states that will undergo transformation.

            input_tensor:
                Input tensor to be added to the processed hidden states.

                - Type: tensor
                - Purpose: Represents the input tensor to be added to the processed hidden states.

        Returns:
            None.

        Raises:
            None
        """
        hidden_states = self.dense(hidden_states)
        hidden_states = self.dropout(hidden_states)
        hidden_states = self.LayerNorm(hidden_states + input_tensor)
        return hidden_states

mindnlp.transformers.models.deberta.modeling_deberta.DebertaSelfOutput.__init__(config)

Initializes an instance of the DebertaSelfOutput class.

PARAMETER	DESCRIPTION
self	The current instance of the class. TYPE: DebertaSelfOutput
config	The configuration object containing the settings for the Deberta model.

RETURNS	DESCRIPTION
	None

Source code in mindnlp/transformers/models/deberta/modeling_deberta.py

def __init__(self, config):
    """
    Initializes an instance of the DebertaSelfOutput class.

    Args:
        self (DebertaSelfOutput): The current instance of the class.
        config: The configuration object containing the settings for the Deberta model.

    Returns:
        None

    Raises:
        None
    """
    super().__init__()
    self.dense = nn.Linear(config.hidden_size, config.hidden_size)
    self.LayerNorm = DebertaLayerNorm(config.hidden_size, config.layer_norm_eps)
    self.dropout = StableDropout(config.hidden_dropout_prob)

mindnlp.transformers.models.deberta.modeling_deberta.DebertaSelfOutput.forward(hidden_states, input_tensor)

Method 'forward' in the class 'DebertaSelfOutput'.

This method forwards the hidden states by applying a series of operations on the input hidden states and the input tensor.

PARAMETER	DESCRIPTION
self	Instance of the DebertaSelfOutput class. Type: DebertaSelfOutput Purpose: Represents the current instance of the class.
hidden_states	Hidden states that need to be processed. Type: tensor Purpose: Represents the input hidden states that will undergo transformation.
input_tensor	Input tensor to be added to the processed hidden states. Type: tensor Purpose: Represents the input tensor to be added to the processed hidden states.

RETURNS	DESCRIPTION
	None.

Source code in mindnlp/transformers/models/deberta/modeling_deberta.py

def forward(self, hidden_states, input_tensor):
    """
    Method 'forward' in the class 'DebertaSelfOutput'.

    This method forwards the hidden states by applying a series of operations on the input hidden states and the input tensor.

    Args:
        self:
            Instance of the DebertaSelfOutput class.

            - Type: DebertaSelfOutput
            - Purpose: Represents the current instance of the class.

        hidden_states:
            Hidden states that need to be processed.

            - Type: tensor
            - Purpose: Represents the input hidden states that will undergo transformation.

        input_tensor:
            Input tensor to be added to the processed hidden states.

            - Type: tensor
            - Purpose: Represents the input tensor to be added to the processed hidden states.

    Returns:
        None.

    Raises:
        None
    """
    hidden_states = self.dense(hidden_states)
    hidden_states = self.dropout(hidden_states)
    hidden_states = self.LayerNorm(hidden_states + input_tensor)
    return hidden_states

mindnlp.transformers.models.deberta.modeling_deberta.DisentangledSelfAttention

Bases: Module

Disentangled self-attention module

PARAMETER	DESCRIPTION
config	A model config class instance with the configuration to build a new model. The schema is similar to BertConfig, for more details, please refer [DebertaConfig] TYPE: `str`

Source code in mindnlp/transformers/models/deberta/modeling_deberta.py

class DisentangledSelfAttention(nn.Module):
    """
    Disentangled self-attention module

    Parameters:
        config (`str`):
            A model config class instance with the configuration to build a new model. The schema is similar to
            *BertConfig*, for more details, please refer [`DebertaConfig`]

    """
    def __init__(self, config):
        """
        Initializes a DisentangledSelfAttention object with the given configuration.

        Args:
            self (DisentangledSelfAttention): The object itself.
            config: A configuration object that contains various parameters for the self-attention mechanism.

        Returns:
            None

        Raises:
            ValueError: If the hidden size is not a multiple of the number of attention heads.

        Note:
            The hidden size should be a multiple of the number of attention heads in order to ensure proper
            functioning of the self-attention mechanism.
        """
        super().__init__()
        if config.hidden_size % config.num_attention_heads != 0:
            raise ValueError(
                f"The hidden size ({config.hidden_size}) is not a multiple of the number of attention "
                f"heads ({config.num_attention_heads})"
            )
        self.num_attention_heads = config.num_attention_heads
        self.attention_head_size = int(config.hidden_size / config.num_attention_heads)
        self.all_head_size = self.num_attention_heads * self.attention_head_size
        self.in_proj = nn.Linear(config.hidden_size, self.all_head_size * 3, bias=False)
        self.q_bias = Parameter(ops.zeros((self.all_head_size), dtype=mindspore.float32))
        self.v_bias = Parameter(ops.zeros((self.all_head_size), dtype=mindspore.float32))
        self.pos_att_type = config.pos_att_type if config.pos_att_type is not None else []

        self.relative_attention = getattr(config, "relative_attention", False)
        self.talking_head = getattr(config, "talking_head", False)

        if self.talking_head:
            self.head_logits_proj = nn.Linear(config.num_attention_heads, config.num_attention_heads, bias=False)
            self.head_weights_proj = nn.Linear(config.num_attention_heads, config.num_attention_heads, bias=False)

        if self.relative_attention:
            self.max_relative_positions = getattr(config, "max_relative_positions", -1)
            if self.max_relative_positions < 1:
                self.max_relative_positions = config.max_position_embeddings
            self.pos_dropout = StableDropout(config.hidden_dropout_prob)

            if "c2p" in self.pos_att_type:
                self.pos_proj = nn.Linear(config.hidden_size, self.all_head_size, bias=False)
            if "p2c" in self.pos_att_type:
                self.pos_q_proj = nn.Linear(config.hidden_size, self.all_head_size)

        self.dropout = StableDropout(config.attention_probs_dropout_prob)
        self.softmax = XSoftmax(-1)

    def swapaxes_for_scores(self, x):
        """
        Performs a swap axis operation on the input tensor for scores in the DisentangledSelfAttention class.

        Args:
            self (DisentangledSelfAttention): An instance of the DisentangledSelfAttention class.
            x (torch.Tensor): The input tensor to be operated on.
                It should have a shape of (batch_size, seq_length, hidden_size).

        Returns:
            torch.Tensor: The transformed tensor after swapping the axes.
                The shape of the returned tensor is (batch_size, num_attention_heads, seq_length, -1).

        Raises:
            None.

        Note:
            - The method assumes that the input tensor has a rank of at least 3.
            - The parameter 'self.num_attention_heads' is expected to be a positive integer representing the number
            of attention heads.
            - The last dimension in the returned tensor is determined by the shape of the input tensor.

        Example:
            ```python
            >>> attention = DisentangledSelfAttention()
            >>> input_tensor = torch.randn(32, 10, 512)
            >>> output_tensor = attention.swapaxes_for_scores(input_tensor)
            ```
        """
        new_x_shape = x.shape[:-1] + (self.num_attention_heads, -1)
        x = x.view(new_x_shape)
        return x.permute(0, 2, 1, 3)

    def forward(
        self,
        hidden_states,
        attention_mask,
        output_attentions=False,
        query_states=None,
        relative_pos=None,
        rel_embeddings=None,
    ):
        """
        Call the module

        Args:
            hidden_states (`torch.FloatTensor`):
                Input states to the module usually the output from previous layer, it will be the Q,K and V in
                *Attention(Q,K,V)*

            attention_mask (`torch.BoolTensor`):
                An attention mask matrix of shape [*B*, *N*, *N*] where *B* is the batch size, *N* is the maximum
                sequence length in which element [i,j] = *1* means the *i* th token in the input can attend to the *j*
                th token.

            output_attentions (`bool`, optional):
                Whether return the attention matrix.

            query_states (`torch.FloatTensor`, optional):
                The *Q* state in *Attention(Q,K,V)*.

            relative_pos (`torch.LongTensor`):
                The relative position encoding between the tokens in the sequence. It's of shape [*B*, *N*, *N*] with
                values ranging in [*-max_relative_positions*, *max_relative_positions*].

            rel_embeddings (`torch.FloatTensor`):
                The embedding of relative distances. It's a tensor of shape [\\(2 \\times
                \\text{max_relative_positions}\\), *hidden_size*].


        """
        if query_states is None:
            qp = self.in_proj(hidden_states)  # .split(self.all_head_size, dim=-1)
            query_layer, key_layer, value_layer = self.swapaxes_for_scores(qp).chunk(3, axis=-1)
        else:

            def linear(w, b, x):
                if b is not None:
                    return ops.matmul(x, w.t()) + b.t()
                return ops.matmul(x, w.t())  # + b.t()

            ws = self.in_proj.weight.chunk(self.num_attention_heads * 3, dim=0)
            qkvw = [ops.cat([ws[i * 3 + k] for i in range(self.num_attention_heads)], axis=0) for k in range(3)]
            qkvb = [None] * 3

            q = linear(qkvw[0], qkvb[0], query_states.to(dtype=qkvw[0].dtype))
            k, v = [linear(qkvw[i], qkvb[i], hidden_states.to(dtype=qkvw[i].dtype)) for i in range(1, 3)]
            query_layer, key_layer, value_layer = [self.swapaxes_for_scores(x) for x in [q, k, v]]

        query_layer = query_layer + self.swapaxes_for_scores(self.q_bias[None, None, :])
        value_layer = value_layer + self.swapaxes_for_scores(self.v_bias[None, None, :])

        rel_att = None
        # Take the dot product between "query" and "key" to get the raw attention scores.
        scale_factor = 1 + len(self.pos_att_type)
        scale = ops.sqrt(mindspore.tensor(query_layer.shape[-1], dtype=mindspore.float32) * scale_factor)
        query_layer = query_layer / scale.to(dtype=query_layer.dtype)
        attention_scores = ops.matmul(query_layer, key_layer.swapaxes(-1, -2))
        if self.relative_attention:
            rel_embeddings = self.pos_dropout(rel_embeddings)
            rel_att = self.disentangled_att_bias(query_layer, key_layer, relative_pos, rel_embeddings, scale_factor)

        if rel_att is not None:
            attention_scores = attention_scores + rel_att

        # bxhxlxd
        if self.talking_head:
            attention_scores = self.head_logits_proj(attention_scores.permute(0, 2, 3, 1)).permute(0, 3, 1, 2)

        attention_probs = self.softmax(attention_scores, attention_mask)
        attention_probs = self.dropout(attention_probs)
        if self.talking_head:
            attention_probs = self.head_weights_proj(attention_probs.permute(0, 2, 3, 1)).permute(0, 3, 1, 2)

        context_layer = ops.matmul(attention_probs, value_layer)
        context_layer = context_layer.permute(0, 2, 1, 3)
        new_context_layer_shape = context_layer.shape[:-2] + (-1,)
        context_layer = context_layer.view(new_context_layer_shape)
        if output_attentions:
            return (context_layer, attention_probs)
        return context_layer

    def disentangled_att_bias(self, query_layer, key_layer, relative_pos, rel_embeddings, scale_factor):
        """
        Perform disentangled attention bias calculation in the DisentangledSelfAttention class.

        Args:
            self (DisentangledSelfAttention): An instance of the DisentangledSelfAttention class.
            query_layer (Tensor): Input tensor representing the query layer of shape [batch_size, seq_length, hidden_size].
            key_layer (Tensor): Input tensor representing the key layer of shape [batch_size, seq_length, hidden_size].
            relative_pos (Tensor or None): Optional input tensor representing the relative positions of shape
                [batch_size, seq_length, seq_length] or [seq_length, seq_length].
                If None, relative positions are calculated using the build_relative_position function.
            rel_embeddings (Tensor): Input tensor representing the relative position embeddings of shape
                [2 * max_relative_positions, hidden_size].
            scale_factor (float): Scaling factor for the calculation.

        Returns:
            score (Tensor): Output tensor representing the disentangled attention bias score of shape
                [batch_size, seq_length, seq_length].

        Raises:
            ValueError: If the dimension of relative_pos is not 2 or 3 or 4.

        Note:
            - The method calculates the disentangled attention bias score using the query and key layers,
            relative positions, and relative position embeddings.
            - The attention bias score is calculated based on the 'c2p' and 'p2c' types of positional attention
            specified in the pos_att_type attribute of the DisentangledSelfAttention instance.
            - The score is returned as a Tensor.
        """
        if relative_pos is None:
            q = query_layer.shape[-2]
            relative_pos = build_relative_position(q, key_layer.shape[-2])
        if relative_pos.ndim == 2:
            relative_pos = relative_pos.unsqueeze(0).unsqueeze(0)
        elif relative_pos.ndim == 3:
            relative_pos = relative_pos.unsqueeze(1)
        # bxhxqxk
        elif relative_pos.ndim != 4:
            raise ValueError(f"Relative position ids must be of dim 2 or 3 or 4. {relative_pos.ndim}")

        att_span = min(max(query_layer.shape[-2], key_layer.shape[-2]), self.max_relative_positions)
        relative_pos = relative_pos.long()
        rel_embeddings = rel_embeddings[
            self.max_relative_positions - att_span : self.max_relative_positions + att_span, :
        ].unsqueeze(0)

        score = 0

        # content->position
        if "c2p" in self.pos_att_type:
            pos_key_layer = self.pos_proj(rel_embeddings)
            pos_key_layer = self.swapaxes_for_scores(pos_key_layer)
            c2p_att = ops.matmul(query_layer, pos_key_layer.swapaxes(-1, -2))
            c2p_pos = ops.clamp(relative_pos + att_span, 0, att_span * 2 - 1)
            c2p_att = ops.gather_elements(c2p_att, dim=-1, index=c2p_dynamic_expand(c2p_pos, query_layer, relative_pos))
            score += c2p_att

        # position->content
        if "p2c" in self.pos_att_type:
            pos_query_layer = self.pos_q_proj(rel_embeddings)
            pos_query_layer = self.swapaxes_for_scores(pos_query_layer)
            pos_query_layer /= ops.sqrt(mindspore.tensor(pos_query_layer.shape[-1], dtype=mindspore.float32) * scale_factor)
            if query_layer.shape[-2] != key_layer.shape[-2]:
                r_pos = build_relative_position(key_layer.shape[-2], key_layer.shape[-2])
            else:
                r_pos = relative_pos
            p2c_pos = ops.clamp(-r_pos + att_span, 0, att_span * 2 - 1)
            p2c_att = ops.matmul(key_layer, pos_query_layer.swapaxes(-1, -2).to(dtype=key_layer.dtype))
            p2c_att = ops.gather_elements(
                p2c_att, dim=-1, index=p2c_dynamic_expand(p2c_pos, query_layer, key_layer)
            ).swapaxes(-1, -2)

            if query_layer.shape[-2] != key_layer.shape[-2]:
                pos_index = relative_pos[:, :, :, 0].unsqueeze(-1)
                p2c_att = ops.gather_elements(p2c_att, dim=-2, index=pos_dynamic_expand(pos_index, p2c_att, key_layer))
            score += p2c_att

        return score

mindnlp.transformers.models.deberta.modeling_deberta.DisentangledSelfAttention.__init__(config)

Initializes a DisentangledSelfAttention object with the given configuration.

PARAMETER	DESCRIPTION
self	The object itself. TYPE: DisentangledSelfAttention
config	A configuration object that contains various parameters for the self-attention mechanism.

RETURNS	DESCRIPTION
	None

RAISES	DESCRIPTION
ValueError	If the hidden size is not a multiple of the number of attention heads.

Note

The hidden size should be a multiple of the number of attention heads in order to ensure proper functioning of the self-attention mechanism.

Source code in mindnlp/transformers/models/deberta/modeling_deberta.py

def __init__(self, config):
    """
    Initializes a DisentangledSelfAttention object with the given configuration.

    Args:
        self (DisentangledSelfAttention): The object itself.
        config: A configuration object that contains various parameters for the self-attention mechanism.

    Returns:
        None

    Raises:
        ValueError: If the hidden size is not a multiple of the number of attention heads.

    Note:
        The hidden size should be a multiple of the number of attention heads in order to ensure proper
        functioning of the self-attention mechanism.
    """
    super().__init__()
    if config.hidden_size % config.num_attention_heads != 0:
        raise ValueError(
            f"The hidden size ({config.hidden_size}) is not a multiple of the number of attention "
            f"heads ({config.num_attention_heads})"
        )
    self.num_attention_heads = config.num_attention_heads
    self.attention_head_size = int(config.hidden_size / config.num_attention_heads)
    self.all_head_size = self.num_attention_heads * self.attention_head_size
    self.in_proj = nn.Linear(config.hidden_size, self.all_head_size * 3, bias=False)
    self.q_bias = Parameter(ops.zeros((self.all_head_size), dtype=mindspore.float32))
    self.v_bias = Parameter(ops.zeros((self.all_head_size), dtype=mindspore.float32))
    self.pos_att_type = config.pos_att_type if config.pos_att_type is not None else []

    self.relative_attention = getattr(config, "relative_attention", False)
    self.talking_head = getattr(config, "talking_head", False)

    if self.talking_head:
        self.head_logits_proj = nn.Linear(config.num_attention_heads, config.num_attention_heads, bias=False)
        self.head_weights_proj = nn.Linear(config.num_attention_heads, config.num_attention_heads, bias=False)

    if self.relative_attention:
        self.max_relative_positions = getattr(config, "max_relative_positions", -1)
        if self.max_relative_positions < 1:
            self.max_relative_positions = config.max_position_embeddings
        self.pos_dropout = StableDropout(config.hidden_dropout_prob)

        if "c2p" in self.pos_att_type:
            self.pos_proj = nn.Linear(config.hidden_size, self.all_head_size, bias=False)
        if "p2c" in self.pos_att_type:
            self.pos_q_proj = nn.Linear(config.hidden_size, self.all_head_size)

    self.dropout = StableDropout(config.attention_probs_dropout_prob)
    self.softmax = XSoftmax(-1)

mindnlp.transformers.models.deberta.modeling_deberta.DisentangledSelfAttention.disentangled_att_bias(query_layer, key_layer, relative_pos, rel_embeddings, scale_factor)

Perform disentangled attention bias calculation in the DisentangledSelfAttention class.

PARAMETER	DESCRIPTION
self	An instance of the DisentangledSelfAttention class. TYPE: DisentangledSelfAttention
query_layer	Input tensor representing the query layer of shape [batch_size, seq_length, hidden_size]. TYPE: Tensor
key_layer	Input tensor representing the key layer of shape [batch_size, seq_length, hidden_size]. TYPE: Tensor
relative_pos	Optional input tensor representing the relative positions of shape [batch_size, seq_length, seq_length] or [seq_length, seq_length]. If None, relative positions are calculated using the build_relative_position function. TYPE: Tensor or None
rel_embeddings	Input tensor representing the relative position embeddings of shape [2 * max_relative_positions, hidden_size]. TYPE: Tensor
scale_factor	Scaling factor for the calculation. TYPE: float

RETURNS	DESCRIPTION
score	Output tensor representing the disentangled attention bias score of shape [batch_size, seq_length, seq_length]. TYPE: Tensor

RAISES	DESCRIPTION
ValueError	If the dimension of relative_pos is not 2 or 3 or 4.

Note

• The method calculates the disentangled attention bias score using the query and key layers, relative positions, and relative position embeddings.
• The attention bias score is calculated based on the 'c2p' and 'p2c' types of positional attention specified in the pos_att_type attribute of the DisentangledSelfAttention instance.
• The score is returned as a Tensor.

Source code in mindnlp/transformers/models/deberta/modeling_deberta.py

def disentangled_att_bias(self, query_layer, key_layer, relative_pos, rel_embeddings, scale_factor):
    """
    Perform disentangled attention bias calculation in the DisentangledSelfAttention class.

    Args:
        self (DisentangledSelfAttention): An instance of the DisentangledSelfAttention class.
        query_layer (Tensor): Input tensor representing the query layer of shape [batch_size, seq_length, hidden_size].
        key_layer (Tensor): Input tensor representing the key layer of shape [batch_size, seq_length, hidden_size].
        relative_pos (Tensor or None): Optional input tensor representing the relative positions of shape
            [batch_size, seq_length, seq_length] or [seq_length, seq_length].
            If None, relative positions are calculated using the build_relative_position function.
        rel_embeddings (Tensor): Input tensor representing the relative position embeddings of shape
            [2 * max_relative_positions, hidden_size].
        scale_factor (float): Scaling factor for the calculation.

    Returns:
        score (Tensor): Output tensor representing the disentangled attention bias score of shape
            [batch_size, seq_length, seq_length].

    Raises:
        ValueError: If the dimension of relative_pos is not 2 or 3 or 4.

    Note:
        - The method calculates the disentangled attention bias score using the query and key layers,
        relative positions, and relative position embeddings.
        - The attention bias score is calculated based on the 'c2p' and 'p2c' types of positional attention
        specified in the pos_att_type attribute of the DisentangledSelfAttention instance.
        - The score is returned as a Tensor.
    """
    if relative_pos is None:
        q = query_layer.shape[-2]
        relative_pos = build_relative_position(q, key_layer.shape[-2])
    if relative_pos.ndim == 2:
        relative_pos = relative_pos.unsqueeze(0).unsqueeze(0)
    elif relative_pos.ndim == 3:
        relative_pos = relative_pos.unsqueeze(1)
    # bxhxqxk
    elif relative_pos.ndim != 4:
        raise ValueError(f"Relative position ids must be of dim 2 or 3 or 4. {relative_pos.ndim}")

    att_span = min(max(query_layer.shape[-2], key_layer.shape[-2]), self.max_relative_positions)
    relative_pos = relative_pos.long()
    rel_embeddings = rel_embeddings[
        self.max_relative_positions - att_span : self.max_relative_positions + att_span, :
    ].unsqueeze(0)

    score = 0

    # content->position
    if "c2p" in self.pos_att_type:
        pos_key_layer = self.pos_proj(rel_embeddings)
        pos_key_layer = self.swapaxes_for_scores(pos_key_layer)
        c2p_att = ops.matmul(query_layer, pos_key_layer.swapaxes(-1, -2))
        c2p_pos = ops.clamp(relative_pos + att_span, 0, att_span * 2 - 1)
        c2p_att = ops.gather_elements(c2p_att, dim=-1, index=c2p_dynamic_expand(c2p_pos, query_layer, relative_pos))
        score += c2p_att

    # position->content
    if "p2c" in self.pos_att_type:
        pos_query_layer = self.pos_q_proj(rel_embeddings)
        pos_query_layer = self.swapaxes_for_scores(pos_query_layer)
        pos_query_layer /= ops.sqrt(mindspore.tensor(pos_query_layer.shape[-1], dtype=mindspore.float32) * scale_factor)
        if query_layer.shape[-2] != key_layer.shape[-2]:
            r_pos = build_relative_position(key_layer.shape[-2], key_layer.shape[-2])
        else:
            r_pos = relative_pos
        p2c_pos = ops.clamp(-r_pos + att_span, 0, att_span * 2 - 1)
        p2c_att = ops.matmul(key_layer, pos_query_layer.swapaxes(-1, -2).to(dtype=key_layer.dtype))
        p2c_att = ops.gather_elements(
            p2c_att, dim=-1, index=p2c_dynamic_expand(p2c_pos, query_layer, key_layer)
        ).swapaxes(-1, -2)

        if query_layer.shape[-2] != key_layer.shape[-2]:
            pos_index = relative_pos[:, :, :, 0].unsqueeze(-1)
            p2c_att = ops.gather_elements(p2c_att, dim=-2, index=pos_dynamic_expand(pos_index, p2c_att, key_layer))
        score += p2c_att

    return score

mindnlp.transformers.models.deberta.modeling_deberta.DisentangledSelfAttention.forward(hidden_states, attention_mask, output_attentions=False, query_states=None, relative_pos=None, rel_embeddings=None)

Call the module

PARAMETER	DESCRIPTION
hidden_states	Input states to the module usually the output from previous layer, it will be the Q,K and V in Attention(Q,K,V) TYPE: `torch.FloatTensor`
attention_mask	An attention mask matrix of shape [B, N, N] where B is the batch size, N is the maximum sequence length in which element [i,j] = 1 means the i th token in the input can attend to the j th token. TYPE: `torch.BoolTensor`
output_attentions	Whether return the attention matrix. TYPE: `bool` DEFAULT: False
query_states	The Q state in Attention(Q,K,V). TYPE: `torch.FloatTensor` DEFAULT: None
relative_pos	The relative position encoding between the tokens in the sequence. It's of shape [B, N, N] with values ranging in [-max_relative_positions, max_relative_positions]. TYPE: `torch.LongTensor` DEFAULT: None
rel_embeddings	The embedding of relative distances. It's a tensor of shape [\(2 \times \text{max_relative_positions}\), hidden_size]. TYPE: `torch.FloatTensor` DEFAULT: None

Source code in mindnlp/transformers/models/deberta/modeling_deberta.py

def forward(
    self,
    hidden_states,
    attention_mask,
    output_attentions=False,
    query_states=None,
    relative_pos=None,
    rel_embeddings=None,
):
    """
    Call the module

    Args:
        hidden_states (`torch.FloatTensor`):
            Input states to the module usually the output from previous layer, it will be the Q,K and V in
            *Attention(Q,K,V)*

        attention_mask (`torch.BoolTensor`):
            An attention mask matrix of shape [*B*, *N*, *N*] where *B* is the batch size, *N* is the maximum
            sequence length in which element [i,j] = *1* means the *i* th token in the input can attend to the *j*
            th token.

        output_attentions (`bool`, optional):
            Whether return the attention matrix.

        query_states (`torch.FloatTensor`, optional):
            The *Q* state in *Attention(Q,K,V)*.

        relative_pos (`torch.LongTensor`):
            The relative position encoding between the tokens in the sequence. It's of shape [*B*, *N*, *N*] with
            values ranging in [*-max_relative_positions*, *max_relative_positions*].

        rel_embeddings (`torch.FloatTensor`):
            The embedding of relative distances. It's a tensor of shape [\\(2 \\times
            \\text{max_relative_positions}\\), *hidden_size*].


    """
    if query_states is None:
        qp = self.in_proj(hidden_states)  # .split(self.all_head_size, dim=-1)
        query_layer, key_layer, value_layer = self.swapaxes_for_scores(qp).chunk(3, axis=-1)
    else:

        def linear(w, b, x):
            if b is not None:
                return ops.matmul(x, w.t()) + b.t()
            return ops.matmul(x, w.t())  # + b.t()

        ws = self.in_proj.weight.chunk(self.num_attention_heads * 3, dim=0)
        qkvw = [ops.cat([ws[i * 3 + k] for i in range(self.num_attention_heads)], axis=0) for k in range(3)]
        qkvb = [None] * 3

        q = linear(qkvw[0], qkvb[0], query_states.to(dtype=qkvw[0].dtype))
        k, v = [linear(qkvw[i], qkvb[i], hidden_states.to(dtype=qkvw[i].dtype)) for i in range(1, 3)]
        query_layer, key_layer, value_layer = [self.swapaxes_for_scores(x) for x in [q, k, v]]

    query_layer = query_layer + self.swapaxes_for_scores(self.q_bias[None, None, :])
    value_layer = value_layer + self.swapaxes_for_scores(self.v_bias[None, None, :])

    rel_att = None
    # Take the dot product between "query" and "key" to get the raw attention scores.
    scale_factor = 1 + len(self.pos_att_type)
    scale = ops.sqrt(mindspore.tensor(query_layer.shape[-1], dtype=mindspore.float32) * scale_factor)
    query_layer = query_layer / scale.to(dtype=query_layer.dtype)
    attention_scores = ops.matmul(query_layer, key_layer.swapaxes(-1, -2))
    if self.relative_attention:
        rel_embeddings = self.pos_dropout(rel_embeddings)
        rel_att = self.disentangled_att_bias(query_layer, key_layer, relative_pos, rel_embeddings, scale_factor)

    if rel_att is not None:
        attention_scores = attention_scores + rel_att

    # bxhxlxd
    if self.talking_head:
        attention_scores = self.head_logits_proj(attention_scores.permute(0, 2, 3, 1)).permute(0, 3, 1, 2)

    attention_probs = self.softmax(attention_scores, attention_mask)
    attention_probs = self.dropout(attention_probs)
    if self.talking_head:
        attention_probs = self.head_weights_proj(attention_probs.permute(0, 2, 3, 1)).permute(0, 3, 1, 2)

    context_layer = ops.matmul(attention_probs, value_layer)
    context_layer = context_layer.permute(0, 2, 1, 3)
    new_context_layer_shape = context_layer.shape[:-2] + (-1,)
    context_layer = context_layer.view(new_context_layer_shape)
    if output_attentions:
        return (context_layer, attention_probs)
    return context_layer

mindnlp.transformers.models.deberta.modeling_deberta.DisentangledSelfAttention.swapaxes_for_scores(x)

Performs a swap axis operation on the input tensor for scores in the DisentangledSelfAttention class.

PARAMETER	DESCRIPTION
self	An instance of the DisentangledSelfAttention class. TYPE: DisentangledSelfAttention
x	The input tensor to be operated on. It should have a shape of (batch_size, seq_length, hidden_size). TYPE: Tensor

RETURNS	DESCRIPTION
	torch.Tensor: The transformed tensor after swapping the axes. The shape of the returned tensor is (batch_size, num_attention_heads, seq_length, -1).

Note

• The method assumes that the input tensor has a rank of at least 3.
• The parameter 'self.num_attention_heads' is expected to be a positive integer representing the number of attention heads.
• The last dimension in the returned tensor is determined by the shape of the input tensor.

Example

>>> attention = DisentangledSelfAttention()
>>> input_tensor = torch.randn(32, 10, 512)
>>> output_tensor = attention.swapaxes_for_scores(input_tensor)

Source code in mindnlp/transformers/models/deberta/modeling_deberta.py

def swapaxes_for_scores(self, x):
    """
    Performs a swap axis operation on the input tensor for scores in the DisentangledSelfAttention class.

    Args:
        self (DisentangledSelfAttention): An instance of the DisentangledSelfAttention class.
        x (torch.Tensor): The input tensor to be operated on.
            It should have a shape of (batch_size, seq_length, hidden_size).

    Returns:
        torch.Tensor: The transformed tensor after swapping the axes.
            The shape of the returned tensor is (batch_size, num_attention_heads, seq_length, -1).

    Raises:
        None.

    Note:
        - The method assumes that the input tensor has a rank of at least 3.
        - The parameter 'self.num_attention_heads' is expected to be a positive integer representing the number
        of attention heads.
        - The last dimension in the returned tensor is determined by the shape of the input tensor.

    Example:
        ```python
        >>> attention = DisentangledSelfAttention()
        >>> input_tensor = torch.randn(32, 10, 512)
        >>> output_tensor = attention.swapaxes_for_scores(input_tensor)
        ```
    """
    new_x_shape = x.shape[:-1] + (self.num_attention_heads, -1)
    x = x.view(new_x_shape)
    return x.permute(0, 2, 1, 3)

mindnlp.transformers.models.deberta.modeling_deberta.DropoutContext

Represents a context for managing dropout operations within a neural network.

This class defines a context for managing dropout operations, including setting the dropout rate, mask, scaling factor, and reusing masks across iterations. It is designed to be used within a neural network framework to control dropout behavior during training.

ATTRIBUTE	DESCRIPTION
dropout	The dropout rate to be applied. TYPE: float
mask	The mask array used for applying dropout. TYPE: ndarray or None
scale	The scaling factor applied to the output. TYPE: float
reuse_mask	Flag indicating whether to reuse the mask across iterations. TYPE: bool

Source code in mindnlp/transformers/models/deberta/modeling_deberta.py

class DropoutContext:

    """
    Represents a context for managing dropout operations within a neural network.

    This class defines a context for managing dropout operations,
    including setting the dropout rate, mask, scaling factor, and reusing masks across iterations.
    It is designed to be used within a neural network framework to control dropout behavior during training.

    Attributes:
        dropout (float): The dropout rate to be applied.
        mask (ndarray or None): The mask array used for applying dropout.
        scale (float): The scaling factor applied to the output.
        reuse_mask (bool): Flag indicating whether to reuse the mask across iterations.

    """
    def __init__(self):
        """
        Initialize a DropoutContext object.

        Args:
            self: The instance of the DropoutContext class.

        Returns:
            None.

        Raises:
            None.
        """
        self.dropout = 0
        self.mask = None
        self.scale = 1
        self.reuse_mask = True

mindnlp.transformers.models.deberta.modeling_deberta.DropoutContext.__init__()

Initialize a DropoutContext object.

PARAMETER	DESCRIPTION
self	The instance of the DropoutContext class.

RETURNS	DESCRIPTION
	None.

Source code in mindnlp/transformers/models/deberta/modeling_deberta.py

def __init__(self):
    """
    Initialize a DropoutContext object.

    Args:
        self: The instance of the DropoutContext class.

    Returns:
        None.

    Raises:
        None.
    """
    self.dropout = 0
    self.mask = None
    self.scale = 1
    self.reuse_mask = True

mindnlp.transformers.models.deberta.modeling_deberta.StableDropout

Bases: Module

Optimized dropout module for stabilizing the training

PARAMETER	DESCRIPTION
drop_prob	the dropout probabilities TYPE: float

Source code in mindnlp/transformers/models/deberta/modeling_deberta.py

class StableDropout(nn.Module):
    """
    Optimized dropout module for stabilizing the training

    Args:
        drop_prob (float): the dropout probabilities
    """
    def __init__(self, drop_prob):
        """Initialize the StableDropout object.

        This method is called when a new instance of the StableDropout class is created.
        It initializes the object with the given drop probability and sets the count and context_stack attributes to
        their initial values.

        Args:
            self (StableDropout): The instance of the StableDropout class.
            drop_prob (float): The probability of dropping a value during dropout. Must be between 0 and 1 (inclusive).

        Returns:
            None.

        Raises:
            None.
        """
        super().__init__()
        self.drop_prob = drop_prob
        self.count = 0
        self.context_stack = None

    def forward(self, x):
        """
        Call the module

        Args:
            x (`mindspore.tensor`): The input tensor to apply dropout
        """
        if self.training and self.drop_prob > 0:
            return XDropout(self.get_context())(x)
        return x

    def clear_context(self):
        """
        Clears the context of the StableDropout class.

        Args:
            self (StableDropout): An instance of the StableDropout class.

        Returns:
            None.

        Raises:
            None.
        """
        self.count = 0
        self.context_stack = None

    def init_context(self, reuse_mask=True, scale=1):
        """
        Initializes the context stack for the StableDropout class.

        Args:
            self: The instance of the StableDropout class.
            reuse_mask (bool, optional): Indicates whether the dropout mask should be reused or not. Defaults to True.
            scale (int, optional): The scaling factor applied to the dropout mask. Defaults to 1.

        Returns:
            None.

        Raises:
            None.
        """
        if self.context_stack is None:
            self.context_stack = []
        self.count = 0
        for c in self.context_stack:
            c.reuse_mask = reuse_mask
            c.scale = scale

    def get_context(self):
        """
        Args:
            self (StableDropout): The instance of the StableDropout class invoking the method.
                This parameter is required for accessing the instance attributes and methods.

        Returns:
            None.

        Raises:
            None.
        """
        if self.context_stack is not None:
            if self.count >= len(self.context_stack):
                self.context_stack.append(DropoutContext())
            ctx = self.context_stack[self.count]
            ctx.dropout = self.drop_prob
            self.count += 1
            return ctx
        return self.drop_prob

mindnlp.transformers.models.deberta.modeling_deberta.StableDropout.__init__(drop_prob)

Initialize the StableDropout object.

This method is called when a new instance of the StableDropout class is created. It initializes the object with the given drop probability and sets the count and context_stack attributes to their initial values.

PARAMETER	DESCRIPTION
self	The instance of the StableDropout class. TYPE: StableDropout
drop_prob	The probability of dropping a value during dropout. Must be between 0 and 1 (inclusive). TYPE: float

RETURNS	DESCRIPTION
	None.

Source code in mindnlp/transformers/models/deberta/modeling_deberta.py

def __init__(self, drop_prob):
    """Initialize the StableDropout object.

    This method is called when a new instance of the StableDropout class is created.
    It initializes the object with the given drop probability and sets the count and context_stack attributes to
    their initial values.

    Args:
        self (StableDropout): The instance of the StableDropout class.
        drop_prob (float): The probability of dropping a value during dropout. Must be between 0 and 1 (inclusive).

    Returns:
        None.

    Raises:
        None.
    """
    super().__init__()
    self.drop_prob = drop_prob
    self.count = 0
    self.context_stack = None

mindnlp.transformers.models.deberta.modeling_deberta.StableDropout.clear_context()

Clears the context of the StableDropout class.

PARAMETER	DESCRIPTION
self	An instance of the StableDropout class. TYPE: StableDropout

RETURNS	DESCRIPTION
	None.

Source code in mindnlp/transformers/models/deberta/modeling_deberta.py

def clear_context(self):
    """
    Clears the context of the StableDropout class.

    Args:
        self (StableDropout): An instance of the StableDropout class.

    Returns:
        None.

    Raises:
        None.
    """
    self.count = 0
    self.context_stack = None

mindnlp.transformers.models.deberta.modeling_deberta.StableDropout.forward(x)

Call the module

PARAMETER	DESCRIPTION
x	The input tensor to apply dropout TYPE: `mindspore.tensor`

Source code in mindnlp/transformers/models/deberta/modeling_deberta.py

def forward(self, x):
    """
    Call the module

    Args:
        x (`mindspore.tensor`): The input tensor to apply dropout
    """
    if self.training and self.drop_prob > 0:
        return XDropout(self.get_context())(x)
    return x

mindnlp.transformers.models.deberta.modeling_deberta.StableDropout.get_context()

PARAMETER	DESCRIPTION
self	The instance of the StableDropout class invoking the method. This parameter is required for accessing the instance attributes and methods. TYPE: StableDropout

RETURNS	DESCRIPTION
	None.

Source code in mindnlp/transformers/models/deberta/modeling_deberta.py

def get_context(self):
    """
    Args:
        self (StableDropout): The instance of the StableDropout class invoking the method.
            This parameter is required for accessing the instance attributes and methods.

    Returns:
        None.

    Raises:
        None.
    """
    if self.context_stack is not None:
        if self.count >= len(self.context_stack):
            self.context_stack.append(DropoutContext())
        ctx = self.context_stack[self.count]
        ctx.dropout = self.drop_prob
        self.count += 1
        return ctx
    return self.drop_prob

mindnlp.transformers.models.deberta.modeling_deberta.StableDropout.init_context(reuse_mask=True, scale=1)

Initializes the context stack for the StableDropout class.

PARAMETER	DESCRIPTION
self	The instance of the StableDropout class.
reuse_mask	Indicates whether the dropout mask should be reused or not. Defaults to True. TYPE: bool DEFAULT: True
scale	The scaling factor applied to the dropout mask. Defaults to 1. TYPE: int DEFAULT: 1

RETURNS	DESCRIPTION
	None.

Source code in mindnlp/transformers/models/deberta/modeling_deberta.py

def init_context(self, reuse_mask=True, scale=1):
    """
    Initializes the context stack for the StableDropout class.

    Args:
        self: The instance of the StableDropout class.
        reuse_mask (bool, optional): Indicates whether the dropout mask should be reused or not. Defaults to True.
        scale (int, optional): The scaling factor applied to the dropout mask. Defaults to 1.

    Returns:
        None.

    Raises:
        None.
    """
    if self.context_stack is None:
        self.context_stack = []
    self.count = 0
    for c in self.context_stack:
        c.reuse_mask = reuse_mask
        c.scale = scale

mindnlp.transformers.models.deberta.modeling_deberta.XDropout

Bases: Module

Optimized dropout function to save computation and memory by using mask operation instead of multiplication.

Source code in mindnlp/transformers/models/deberta/modeling_deberta.py

class XDropout(nn.Module):
    """Optimized dropout function to save computation and memory by using mask operation instead of multiplication."""
    def __init__(self, local_ctx):
        """
        Initialize a new instance of the XDropout class.

        Args:
            self (object): The instance of the XDropout class.
            local_ctx (object): The local context for the XDropout instance.

        Returns:
            None.

        Raises:
            None.
        """
        super().__init__()
        self.local_ctx = local_ctx
        self.scale = 0
        self.mask = None

    def forward(self, inputs):
        """
        Constructs a masked and scaled version of the input tensor using the XDropout method.

        Args:
            self (XDropout): An instance of the XDropout class.
            inputs (torch.Tensor): The input tensor to be masked and scaled.

        Returns:
            None.

        Raises:
            None.
        """
        mask, dropout = get_mask(inputs, self.local_ctx)
        self.scale = 1.0 / (1 - dropout)
        self.mask = mask
        if dropout > 0:
            return inputs.masked_fill(mask, 0) * self.scale
        return inputs

mindnlp.transformers.models.deberta.modeling_deberta.XDropout.__init__(local_ctx)

Initialize a new instance of the XDropout class.

PARAMETER	DESCRIPTION
self	The instance of the XDropout class. TYPE: object
local_ctx	The local context for the XDropout instance. TYPE: object

RETURNS	DESCRIPTION
	None.

Source code in mindnlp/transformers/models/deberta/modeling_deberta.py

def __init__(self, local_ctx):
    """
    Initialize a new instance of the XDropout class.

    Args:
        self (object): The instance of the XDropout class.
        local_ctx (object): The local context for the XDropout instance.

    Returns:
        None.

    Raises:
        None.
    """
    super().__init__()
    self.local_ctx = local_ctx
    self.scale = 0
    self.mask = None

mindnlp.transformers.models.deberta.modeling_deberta.XDropout.forward(inputs)

Constructs a masked and scaled version of the input tensor using the XDropout method.

PARAMETER	DESCRIPTION
self	An instance of the XDropout class. TYPE: XDropout
inputs	The input tensor to be masked and scaled. TYPE: Tensor

RETURNS	DESCRIPTION
	None.

Source code in mindnlp/transformers/models/deberta/modeling_deberta.py

def forward(self, inputs):
    """
    Constructs a masked and scaled version of the input tensor using the XDropout method.

    Args:
        self (XDropout): An instance of the XDropout class.
        inputs (torch.Tensor): The input tensor to be masked and scaled.

    Returns:
        None.

    Raises:
        None.
    """
    mask, dropout = get_mask(inputs, self.local_ctx)
    self.scale = 1.0 / (1 - dropout)
    self.mask = mask
    if dropout > 0:
        return inputs.masked_fill(mask, 0) * self.scale
    return inputs

mindnlp.transformers.models.deberta.modeling_deberta.XSoftmax

Bases: Module

Masked Softmax which is optimized for saving memory

PARAMETER	DESCRIPTION
input	The input tensor that will apply softmax. TYPE: `mindspore.tensor`
mask	The mask matrix where 0 indicate that element will be ignored in the softmax calculation. TYPE: `torch.IntTensor`
dim	The dimension that will apply softmax TYPE: int DEFAULT: -1

Example

>>> import torch
>>> from transformers.models.deberta.modeling_deberta import XSoftmax
...
>>> # Make a tensor
>>> x = torch.randn([4, 20, 100])
...
>>> # Create a mask
>>> mask = (x > 0).int()
...
>>> # Specify the dimension to apply softmax
>>> dim = -1
...
>>> y = XSoftmax.apply(x, mask, dim)

Source code in mindnlp/transformers/models/deberta/modeling_deberta.py

class XSoftmax(nn.Module):
    """
    Masked Softmax which is optimized for saving memory

    Args:
        input (`mindspore.tensor`): The input tensor that will apply softmax.
        mask (`torch.IntTensor`):
            The mask matrix where 0 indicate that element will be ignored in the softmax calculation.
        dim (int): The dimension that will apply softmax

    Example:
        ```python
        >>> import torch
        >>> from transformers.models.deberta.modeling_deberta import XSoftmax
        ...
        >>> # Make a tensor
        >>> x = torch.randn([4, 20, 100])
        ...
        >>> # Create a mask
        >>> mask = (x > 0).int()
        ...
        >>> # Specify the dimension to apply softmax
        >>> dim = -1
        ...
        >>> y = XSoftmax.apply(x, mask, dim)
        ```
    """
    def __init__(self, dim=-1):
        """
        Initializes an instance of the XSoftmax class.

        Args:
            self: The instance of the XSoftmax class.
            dim (int): The dimension along which the softmax operation is performed. Default is -1.
                The value of dim must be a non-negative integer or -1. If -1, the operation is performed
                along the last dimension of the input tensor.

        Returns:
            None.

        Raises:
            None.

        """
        super().__init__()
        self.dim = dim

    def forward(self, input, mask):
        """
        Constructs a softmax operation with masking for a given input tensor.

        Args:
            self (XSoftmax): An instance of the XSoftmax class.
            input (Tensor): The input tensor on which the softmax operation is performed.
            mask (Tensor): A tensor representing the mask used for masking certain elements in the input tensor.

        Returns:
            None: The method modifies the input tensor in-place and does not return any value.

        Raises:
            TypeError: If the input tensor or the mask tensor is not of the expected type.
            ValueError: If the dimensions of the input tensor and the mask tensor do not match.
            RuntimeError: If an error occurs during the softmax operation or masking process.
        """
        rmask = ~(mask.to(mindspore.bool_))

        output = input.masked_fill(rmask, mindspore.tensor(finfo(input.dtype, 'min')))
        output = ops.softmax(output, self.dim)
        output = output.masked_fill(rmask, 0)
        return output

    def brop(self, input, mask, output, grad_output):
        """
        This method, 'brop', is a member of the 'XSoftmax' class and performs a specific operation on the given input,
        mask, output, and grad_output parameters.

        Args:
            self: An instance of the 'XSoftmax' class.
            input: The input parameter of type <input_type>. It represents the input value used in the operation.
            mask: The mask parameter of type <mask_type>.
                It represents a mask used in the operation.
                <Additional details about the purpose and restrictions of the mask parameter.>
            output: The output parameter of type <output_type>. It represents the output value of the operation.
            grad_output: The grad_output parameter of type <grad_output_type>.
                It represents the gradient of the output value.

        Returns:
            dx: A value of type <dx_type>.
                It represents the final result of the operation.
                <Additional details about the purpose and format of the dx value.>
            None.

        Raises:
            <Exception1>: <Description of when and why this exception may be raised.>
            <Exception2>: <Description of when and why this exception may be raised.>
            <Additional exceptions that may be raised during the execution of the method.>
        """
        dx = ops.mul(output, ops.sub(grad_output, ops.sum(ops.mul(output, grad_output), self.dim, keepdim=True)))
        return dx, None

mindnlp.transformers.models.deberta.modeling_deberta.XSoftmax.__init__(dim=-1)

Initializes an instance of the XSoftmax class.

PARAMETER	DESCRIPTION
self	The instance of the XSoftmax class.
dim	The dimension along which the softmax operation is performed. Default is -1. The value of dim must be a non-negative integer or -1. If -1, the operation is performed along the last dimension of the input tensor. TYPE: int DEFAULT: -1

RETURNS	DESCRIPTION
	None.

Source code in mindnlp/transformers/models/deberta/modeling_deberta.py

def __init__(self, dim=-1):
    """
    Initializes an instance of the XSoftmax class.

    Args:
        self: The instance of the XSoftmax class.
        dim (int): The dimension along which the softmax operation is performed. Default is -1.
            The value of dim must be a non-negative integer or -1. If -1, the operation is performed
            along the last dimension of the input tensor.

    Returns:
        None.

    Raises:
        None.

    """
    super().__init__()
    self.dim = dim

mindnlp.transformers.models.deberta.modeling_deberta.XSoftmax.brop(input, mask, output, grad_output)

This method, 'brop', is a member of the 'XSoftmax' class and performs a specific operation on the given input, mask, output, and grad_output parameters.

PARAMETER	DESCRIPTION
self	An instance of the 'XSoftmax' class.
input	The input parameter of type . It represents the input value used in the operation.
mask	The mask parameter of type . It represents a mask used in the operation.
output	The output parameter of type . It represents the output value of the operation.
grad_output	The grad_output parameter of type . It represents the gradient of the output value.

RETURNS	DESCRIPTION
dx	A value of type . It represents the final result of the operation.
	None.

RAISES	DESCRIPTION
<Exception1>
<Exception2>

Source code in mindnlp/transformers/models/deberta/modeling_deberta.py

def brop(self, input, mask, output, grad_output):
    """
    This method, 'brop', is a member of the 'XSoftmax' class and performs a specific operation on the given input,
    mask, output, and grad_output parameters.

    Args:
        self: An instance of the 'XSoftmax' class.
        input: The input parameter of type <input_type>. It represents the input value used in the operation.
        mask: The mask parameter of type <mask_type>.
            It represents a mask used in the operation.
            <Additional details about the purpose and restrictions of the mask parameter.>
        output: The output parameter of type <output_type>. It represents the output value of the operation.
        grad_output: The grad_output parameter of type <grad_output_type>.
            It represents the gradient of the output value.

    Returns:
        dx: A value of type <dx_type>.
            It represents the final result of the operation.
            <Additional details about the purpose and format of the dx value.>
        None.

    Raises:
        <Exception1>: <Description of when and why this exception may be raised.>
        <Exception2>: <Description of when and why this exception may be raised.>
        <Additional exceptions that may be raised during the execution of the method.>
    """
    dx = ops.mul(output, ops.sub(grad_output, ops.sum(ops.mul(output, grad_output), self.dim, keepdim=True)))
    return dx, None

mindnlp.transformers.models.deberta.modeling_deberta.XSoftmax.forward(input, mask)

Constructs a softmax operation with masking for a given input tensor.

PARAMETER	DESCRIPTION
self	An instance of the XSoftmax class. TYPE: XSoftmax
input	The input tensor on which the softmax operation is performed. TYPE: Tensor
mask	A tensor representing the mask used for masking certain elements in the input tensor. TYPE: Tensor

RETURNS	DESCRIPTION
None	The method modifies the input tensor in-place and does not return any value.

RAISES	DESCRIPTION
TypeError	If the input tensor or the mask tensor is not of the expected type.
ValueError	If the dimensions of the input tensor and the mask tensor do not match.
RuntimeError	If an error occurs during the softmax operation or masking process.

Source code in mindnlp/transformers/models/deberta/modeling_deberta.py

def forward(self, input, mask):
    """
    Constructs a softmax operation with masking for a given input tensor.

    Args:
        self (XSoftmax): An instance of the XSoftmax class.
        input (Tensor): The input tensor on which the softmax operation is performed.
        mask (Tensor): A tensor representing the mask used for masking certain elements in the input tensor.

    Returns:
        None: The method modifies the input tensor in-place and does not return any value.

    Raises:
        TypeError: If the input tensor or the mask tensor is not of the expected type.
        ValueError: If the dimensions of the input tensor and the mask tensor do not match.
        RuntimeError: If an error occurs during the softmax operation or masking process.
    """
    rmask = ~(mask.to(mindspore.bool_))

    output = input.masked_fill(rmask, mindspore.tensor(finfo(input.dtype, 'min')))
    output = ops.softmax(output, self.dim)
    output = output.masked_fill(rmask, 0)
    return output

mindnlp.transformers.models.deberta.modeling_deberta.build_relative_position(query_size, key_size)

Build relative position according to the query and key

We assume the absolute position of query \(P_q\) is range from (0, query_size) and the absolute position of key \(P_k\) is range from (0, key_size), The relative positions from query to key is \(R_{q \rightarrow k} = P_q - P_k\)

PARAMETER	DESCRIPTION
query_size	the length of query TYPE: int
key_size	the length of key TYPE: int

Return

torch.LongTensor: A tensor with shape [1, query_size, key_size]

Source code in mindnlp/transformers/models/deberta/modeling_deberta.py

def build_relative_position(query_size, key_size):
    """
    Build relative position according to the query and key

    We assume the absolute position of query \\(P_q\\) is range from (0, query_size) and the absolute position of key
    \\(P_k\\) is range from (0, key_size), The relative positions from query to key is \\(R_{q \\rightarrow k} = P_q -
    P_k\\)

    Args:
        query_size (int): the length of query
        key_size (int): the length of key

    Return:
        `torch.LongTensor`: A tensor with shape [1, query_size, key_size]

    """
    q_ids = ops.arange(query_size, dtype=mindspore.int64)
    k_ids = ops.arange(key_size, dtype=mindspore.int64)
    rel_pos_ids = q_ids[:, None] - k_ids.view(1, -1).repeat(query_size, 1)
    rel_pos_ids = rel_pos_ids[:query_size, :]
    rel_pos_ids = rel_pos_ids.unsqueeze(0)
    return rel_pos_ids

mindnlp.transformers.models.deberta.modeling_deberta.c2p_dynamic_expand(c2p_pos, query_layer, relative_pos)

Converts the input Cartesian coordinates to polar coordinates by dynamically expanding the Cartesian coordinates based on the shape of the query layer and relative positions.

PARAMETER	DESCRIPTION
c2p_pos	The input Cartesian coordinates. Expected shape is [batch_size, height, width, num_features]. TYPE: Tensor
query_layer	The query layer. Used to determine the shape of the expanded Cartesian coordinates. Expected shape is [batch_size, query_height, query_width, query_features]. TYPE: Tensor
relative_pos	The relative positions. Used to determine the shape of the expanded Cartesian coordinates. Expected shape is [relative_features]. TYPE: Tensor

RETURNS	DESCRIPTION
	None.

Source code in mindnlp/transformers/models/deberta/modeling_deberta.py

def c2p_dynamic_expand(c2p_pos, query_layer, relative_pos):
    """
    Converts the input Cartesian coordinates to polar coordinates by dynamically expanding the Cartesian coordinates
    based on the shape of the query layer and relative positions.

    Args:
        c2p_pos (Tensor): The input Cartesian coordinates. Expected shape is [batch_size, height, width, num_features].
        query_layer (Tensor): The query layer. Used to determine the shape of the expanded Cartesian coordinates.
            Expected shape is [batch_size, query_height, query_width, query_features].
        relative_pos (Tensor): The relative positions. Used to determine the shape of the expanded Cartesian coordinates.
            Expected shape is [relative_features].

    Returns:
        None.

    Raises:
        None.
    """
    return c2p_pos.expand([query_layer.shape[0], query_layer.shape[1], query_layer.shape[2], relative_pos.shape[-1]])

mindnlp.transformers.models.deberta.modeling_deberta.get_mask(input, local_context)

PARAMETER	DESCRIPTION
input	The input tensor for which the dropout mask is generated. TYPE: Tensor
local_context	The local context containing information about dropout parameters. If a DropoutContext object is provided, the dropout mask will be generated based on its parameters. If a float value is provided, it will be used as the dropout rate. TYPE: DropoutContext or float

RETURNS	DESCRIPTION
None	The function returns the generated dropout mask, or None if no mask is generated.

RAISES	DESCRIPTION
ValueError	If the local_context is not of type DropoutContext.

Source code in mindnlp/transformers/models/deberta/modeling_deberta.py

def get_mask(input, local_context):
    """
    Args:
        input (Tensor): The input tensor for which the dropout mask is generated.
        local_context (DropoutContext or float):
            The local context containing information about dropout parameters.

            - If a DropoutContext object is provided, the dropout mask will be generated based on its parameters.
            - If a float value is provided, it will be used as the dropout rate.

    Returns:
        None: The function returns the generated dropout mask, or None if no mask is generated.

    Raises:
        ValueError: If the local_context is not of type DropoutContext.
    """
    if not isinstance(local_context, DropoutContext):
        dropout = local_context
        mask = None
    else:
        dropout = local_context.dropout
        dropout *= local_context.scale
        mask = local_context.mask if local_context.reuse_mask else None

    if dropout > 0 and mask is None:
        mask = (1 - ops.zeros_like(input).bernoulli(1 - dropout)).to(mindspore.bool_)

    if isinstance(local_context, DropoutContext):
        if local_context.mask is None:
            local_context.mask = mask

    return mask, dropout

mindnlp.transformers.models.deberta.modeling_deberta.p2c_dynamic_expand(c2p_pos, query_layer, key_layer)

Transforms the given c2p_pos tensor into a dynamic expanded tensor.

PARAMETER	DESCRIPTION
c2p_pos	The tensor representing the c2p position. TYPE: Tensor
query_layer	The tensor representing the query layer. TYPE: Tensor
key_layer	The tensor representing the key layer. TYPE: Tensor

RETURNS	DESCRIPTION
	torch.Tensor: The dynamic expanded tensor obtained by expanding the c2p_pos tensor. The shape of the returned tensor is [query_layer.shape[0], query_layer.shape[1], key_layer.shape[-2], key_layer.shape[-2]].

Source code in mindnlp/transformers/models/deberta/modeling_deberta.py

def p2c_dynamic_expand(c2p_pos, query_layer, key_layer):
    """
    Transforms the given c2p_pos tensor into a dynamic expanded tensor.

    Args:
        c2p_pos (torch.Tensor): The tensor representing the c2p position.
        query_layer (torch.Tensor): The tensor representing the query layer.
        key_layer (torch.Tensor): The tensor representing the key layer.

    Returns:
        torch.Tensor: The dynamic expanded tensor obtained by expanding the c2p_pos tensor.
            The shape of the returned tensor is [query_layer.shape[0], query_layer.shape[1], key_layer.shape[-2],
            key_layer.shape[-2]].

    Raises:
        None.
    """
    return c2p_pos.expand([query_layer.shape[0], query_layer.shape[1], key_layer.shape[-2], key_layer.shape[-2]])

mindnlp.transformers.models.deberta.modeling_deberta.pos_dynamic_expand(pos_index, p2c_att, key_layer)

PARAMETER	DESCRIPTION
pos_index	A tensor representing positional indices. TYPE: Tensor
p2c_att	A tensor containing attention weights. TYPE: Tensor
key_layer	A tensor representing key layer values. TYPE: Tensor

RETURNS	DESCRIPTION
	None.

Source code in mindnlp/transformers/models/deberta/modeling_deberta.py

def pos_dynamic_expand(pos_index, p2c_att, key_layer):
    """
    Args:
        pos_index (torch.Tensor): A tensor representing positional indices.
        p2c_att (torch.Tensor): A tensor containing attention weights.
        key_layer (torch.Tensor): A tensor representing key layer values.

    Returns:
        None.

    Raises:
        None
    """
    return pos_index.expand(p2c_att.shape[:2] + (pos_index.shape[-2], key_layer.shape[-2]))

mindnlp.transformers.models.deberta.configuration_deberta

DeBERTa model configuration

mindnlp.transformers.models.deberta.configuration_deberta.DebertaConfig

Bases: PretrainedConfig

This is the configuration class to store the configuration of a [DebertaModel] or a [TFDebertaModel]. It is used to instantiate a DeBERTa model according to the specified arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of the DeBERTa microsoft/deberta-base architecture.

Configuration objects inherit from [PretrainedConfig] and can be used to control the model outputs. Read the documentation from [PretrainedConfig] for more information.

PARAMETER	DESCRIPTION
vocab_size	Vocabulary size of the DeBERTa model. Defines the number of different tokens that can be represented by the inputs_ids passed when calling [DebertaModel] or [TFDebertaModel]. TYPE: `int`, *optional*, defaults to 30522 DEFAULT: 50265
hidden_size	Dimensionality of the encoder layers and the pooler layer. TYPE: `int`, *optional*, defaults to 768 DEFAULT: 768
num_hidden_layers	Number of hidden layers in the Transformer encoder. TYPE: `int`, *optional*, defaults to 12 DEFAULT: 12
num_attention_heads	Number of attention heads for each attention layer in the Transformer encoder. TYPE: `int`, *optional*, defaults to 12 DEFAULT: 12
intermediate_size	Dimensionality of the "intermediate" (often named feed-forward) layer in the Transformer encoder. TYPE: `int`, *optional*, defaults to 3072 DEFAULT: 3072
hidden_act	The non-linear activation function (function or string) in the encoder and pooler. If string, "gelu", "relu", "silu", "gelu", "tanh", "gelu_fast", "mish", "linear", "sigmoid" and "gelu_new" are supported. TYPE: `str` or `Callable`, *optional*, defaults to `"gelu"` DEFAULT: 'gelu'
hidden_dropout_prob	The dropout probability for all fully connected layers in the embeddings, encoder, and pooler. TYPE: `float`, *optional*, defaults to 0.1 DEFAULT: 0.1
attention_probs_dropout_prob	The dropout ratio for the attention probabilities. TYPE: `float`, *optional*, defaults to 0.1 DEFAULT: 0.1
max_position_embeddings	The maximum sequence length that this model might ever be used with. Typically set this to something large just in case (e.g., 512 or 1024 or 2048). TYPE: `int`, *optional*, defaults to 512 DEFAULT: 512
type_vocab_size	The vocabulary size of the token_type_ids passed when calling [DebertaModel] or [TFDebertaModel]. TYPE: `int`, *optional*, defaults to 2 DEFAULT: 0
initializer_range	The standard deviation of the truncated_normal_initializer for initializing all weight matrices. TYPE: `float`, *optional*, defaults to 0.02 DEFAULT: 0.02
layer_norm_eps	The epsilon used by the layer normalization layers. TYPE: `float`, *optional*, defaults to 1e-12 DEFAULT: 1e-07
relative_attention	Whether use relative position encoding. TYPE: `bool`, *optional*, defaults to `False` DEFAULT: False
max_relative_positions	The range of relative positions [-max_position_embeddings, max_position_embeddings]. Use the same value as max_position_embeddings. TYPE: `int`, *optional*, defaults to 1 DEFAULT: -1
pad_token_id	The value used to pad input_ids. TYPE: `int`, *optional*, defaults to 0 DEFAULT: 0
position_biased_input	Whether add absolute position embedding to content embedding. TYPE: `bool`, *optional*, defaults to `True` DEFAULT: True
pos_att_type	The type of relative position attention, it can be a combination of ["p2c", "c2p"], e.g. ["p2c"], ["p2c", "c2p"]. TYPE: `List[str]`, *optional* DEFAULT: None
layer_norm_eps	The epsilon used by the layer normalization layers. TYPE: `float`, optional, defaults to 1e-12 DEFAULT: 1e-07

Example

>>> from transformers import DebertaConfig, DebertaModel
...
>>> # Initializing a DeBERTa microsoft/deberta-base style configuration
>>> configuration = DebertaConfig()
...
>>> # Initializing a model (with random weights) from the microsoft/deberta-base style configuration
>>> model = DebertaModel(configuration)
...
>>> # Accessing the model configuration
>>> configuration = model.config

Source code in mindnlp/transformers/models/deberta/configuration_deberta.py

class DebertaConfig(PretrainedConfig):
    r"""
    This is the configuration class to store the configuration of a [`DebertaModel`] or a [`TFDebertaModel`]. It is
    used to instantiate a DeBERTa model according to the specified arguments, defining the model architecture.
    Instantiating a configuration with the defaults will yield a similar configuration to that of the DeBERTa
    [microsoft/deberta-base](https://hf-mirror.com/microsoft/deberta-base) architecture.

    Configuration objects inherit from [`PretrainedConfig`] and can be used to control the model outputs. Read the
    documentation from [`PretrainedConfig`] for more information.

    Arguments:
        vocab_size (`int`, *optional*, defaults to 30522):
            Vocabulary size of the DeBERTa model. Defines the number of different tokens that can be represented by the
            `inputs_ids` passed when calling [`DebertaModel`] or [`TFDebertaModel`].
        hidden_size (`int`, *optional*, defaults to 768):
            Dimensionality of the encoder layers and the pooler layer.
        num_hidden_layers (`int`, *optional*, defaults to 12):
            Number of hidden layers in the Transformer encoder.
        num_attention_heads (`int`, *optional*, defaults to 12):
            Number of attention heads for each attention layer in the Transformer encoder.
        intermediate_size (`int`, *optional*, defaults to 3072):
            Dimensionality of the "intermediate" (often named feed-forward) layer in the Transformer encoder.
        hidden_act (`str` or `Callable`, *optional*, defaults to `"gelu"`):
            The non-linear activation function (function or string) in the encoder and pooler. If string, `"gelu"`,
            `"relu"`, `"silu"`, `"gelu"`, `"tanh"`, `"gelu_fast"`, `"mish"`, `"linear"`, `"sigmoid"` and `"gelu_new"`
            are supported.
        hidden_dropout_prob (`float`, *optional*, defaults to 0.1):
            The dropout probability for all fully connected layers in the embeddings, encoder, and pooler.
        attention_probs_dropout_prob (`float`, *optional*, defaults to 0.1):
            The dropout ratio for the attention probabilities.
        max_position_embeddings (`int`, *optional*, defaults to 512):
            The maximum sequence length that this model might ever be used with. Typically set this to something large
            just in case (e.g., 512 or 1024 or 2048).
        type_vocab_size (`int`, *optional*, defaults to 2):
            The vocabulary size of the `token_type_ids` passed when calling [`DebertaModel`] or [`TFDebertaModel`].
        initializer_range (`float`, *optional*, defaults to 0.02):
            The standard deviation of the truncated_normal_initializer for initializing all weight matrices.
        layer_norm_eps (`float`, *optional*, defaults to 1e-12):
            The epsilon used by the layer normalization layers.
        relative_attention (`bool`, *optional*, defaults to `False`):
            Whether use relative position encoding.
        max_relative_positions (`int`, *optional*, defaults to 1):
            The range of relative positions `[-max_position_embeddings, max_position_embeddings]`. Use the same value
            as `max_position_embeddings`.
        pad_token_id (`int`, *optional*, defaults to 0):
            The value used to pad input_ids.
        position_biased_input (`bool`, *optional*, defaults to `True`):
            Whether add absolute position embedding to content embedding.
        pos_att_type (`List[str]`, *optional*):
            The type of relative position attention, it can be a combination of `["p2c", "c2p"]`, e.g. `["p2c"]`,
            `["p2c", "c2p"]`.
        layer_norm_eps (`float`, optional, defaults to 1e-12):
            The epsilon used by the layer normalization layers.

    Example:
        ```python
        >>> from transformers import DebertaConfig, DebertaModel
        ...
        >>> # Initializing a DeBERTa microsoft/deberta-base style configuration
        >>> configuration = DebertaConfig()
        ...
        >>> # Initializing a model (with random weights) from the microsoft/deberta-base style configuration
        >>> model = DebertaModel(configuration)
        ...
        >>> # Accessing the model configuration
        >>> configuration = model.config
        ```
    """
    model_type = "deberta"

    def __init__(
        self,
        vocab_size=50265,
        hidden_size=768,
        num_hidden_layers=12,
        num_attention_heads=12,
        intermediate_size=3072,
        hidden_act="gelu",
        hidden_dropout_prob=0.1,
        attention_probs_dropout_prob=0.1,
        max_position_embeddings=512,
        type_vocab_size=0,
        initializer_range=0.02,
        layer_norm_eps=1e-7,
        relative_attention=False,
        max_relative_positions=-1,
        pad_token_id=0,
        position_biased_input=True,
        pos_att_type=None,
        pooler_dropout=0,
        pooler_hidden_act="gelu",
        **kwargs,
    ):
        """
        Initialize a DebertaConfig object.

        Args:
            self: The object instance.
            vocab_size (int, optional): The size of the vocabulary. Default is 50265.
            hidden_size (int, optional): The size of the hidden layers. Default is 768.
            num_hidden_layers (int, optional): The number of hidden layers. Default is 12.
            num_attention_heads (int, optional): The number of attention heads. Default is 12.
            intermediate_size (int, optional): The size of the intermediate layers. Default is 3072.
            hidden_act (str, optional): The activation function for hidden layers. Default is 'gelu'.
            hidden_dropout_prob (float, optional): The dropout probability for hidden layers. Default is 0.1.
            attention_probs_dropout_prob (float, optional): The dropout probability for attention probabilities. Default is 0.1.
            max_position_embeddings (int, optional): The maximum position embeddings. Default is 512.
            type_vocab_size (int, optional): The size of the type vocabulary. Default is 0.
            initializer_range (float, optional): The range for parameter initialization. Default is 0.02.
            layer_norm_eps (float): The epsilon value for layer normalization. Default is 1e-07.
            relative_attention (bool, optional): Whether to use relative attention. Default is False.
            max_relative_positions (int, optional): The maximum relative positions for relative attention. Default is -1.
            pad_token_id (int, optional): The token ID for padding. Default is 0.
            position_biased_input (bool, optional): Whether to use position-biased input. Default is True.
            pos_att_type (str or list of str, optional): The type of positional attention. Default is None.
            pooler_dropout (float, optional): The dropout probability for the pooler layer. Default is 0.
            pooler_hidden_act (str, optional): The activation function for the pooler layer. Default is 'gelu'.

        Returns:
            None.

        Raises:
            None.
        """
        super().__init__(**kwargs)

        self.hidden_size = hidden_size
        self.num_hidden_layers = num_hidden_layers
        self.num_attention_heads = num_attention_heads
        self.intermediate_size = intermediate_size
        self.hidden_act = hidden_act
        self.hidden_dropout_prob = hidden_dropout_prob
        self.attention_probs_dropout_prob = attention_probs_dropout_prob
        self.max_position_embeddings = max_position_embeddings
        self.type_vocab_size = type_vocab_size
        self.initializer_range = initializer_range
        self.relative_attention = relative_attention
        self.max_relative_positions = max_relative_positions
        self.pad_token_id = pad_token_id
        self.position_biased_input = position_biased_input

        # Backwards compatibility
        if isinstance(pos_att_type, str):
            pos_att_type = [x.strip() for x in pos_att_type.lower().split("|")]

        self.pos_att_type = pos_att_type
        self.vocab_size = vocab_size
        self.layer_norm_eps = layer_norm_eps

        self.pooler_hidden_size = kwargs.get("pooler_hidden_size", hidden_size)
        self.pooler_dropout = pooler_dropout
        self.pooler_hidden_act = pooler_hidden_act

mindnlp.transformers.models.deberta.configuration_deberta.DebertaConfig.__init__(vocab_size=50265, hidden_size=768, num_hidden_layers=12, num_attention_heads=12, intermediate_size=3072, hidden_act='gelu', hidden_dropout_prob=0.1, attention_probs_dropout_prob=0.1, max_position_embeddings=512, type_vocab_size=0, initializer_range=0.02, layer_norm_eps=1e-07, relative_attention=False, max_relative_positions=-1, pad_token_id=0, position_biased_input=True, pos_att_type=None, pooler_dropout=0, pooler_hidden_act='gelu', **kwargs)

Initialize a DebertaConfig object.

PARAMETER	DESCRIPTION
self	The object instance.
vocab_size	The size of the vocabulary. Default is 50265. TYPE: int DEFAULT: 50265
hidden_size	The size of the hidden layers. Default is 768. TYPE: int DEFAULT: 768
num_hidden_layers	The number of hidden layers. Default is 12. TYPE: int DEFAULT: 12
num_attention_heads	The number of attention heads. Default is 12. TYPE: int DEFAULT: 12
intermediate_size	The size of the intermediate layers. Default is 3072. TYPE: int DEFAULT: 3072
hidden_act	The activation function for hidden layers. Default is 'gelu'. TYPE: str DEFAULT: 'gelu'
hidden_dropout_prob	The dropout probability for hidden layers. Default is 0.1. TYPE: float DEFAULT: 0.1
attention_probs_dropout_prob	The dropout probability for attention probabilities. Default is 0.1. TYPE: float DEFAULT: 0.1
max_position_embeddings	The maximum position embeddings. Default is 512. TYPE: int DEFAULT: 512
type_vocab_size	The size of the type vocabulary. Default is 0. TYPE: int DEFAULT: 0
initializer_range	The range for parameter initialization. Default is 0.02. TYPE: float DEFAULT: 0.02
layer_norm_eps	The epsilon value for layer normalization. Default is 1e-07. TYPE: float DEFAULT: 1e-07
relative_attention	Whether to use relative attention. Default is False. TYPE: bool DEFAULT: False
max_relative_positions	The maximum relative positions for relative attention. Default is -1. TYPE: int DEFAULT: -1
pad_token_id	The token ID for padding. Default is 0. TYPE: int DEFAULT: 0
position_biased_input	Whether to use position-biased input. Default is True. TYPE: bool DEFAULT: True
pos_att_type	The type of positional attention. Default is None. TYPE: str or list of str DEFAULT: None
pooler_dropout	The dropout probability for the pooler layer. Default is 0. TYPE: float DEFAULT: 0
pooler_hidden_act	The activation function for the pooler layer. Default is 'gelu'. TYPE: str DEFAULT: 'gelu'

RETURNS	DESCRIPTION
	None.

Source code in mindnlp/transformers/models/deberta/configuration_deberta.py

def __init__(
    self,
    vocab_size=50265,
    hidden_size=768,
    num_hidden_layers=12,
    num_attention_heads=12,
    intermediate_size=3072,
    hidden_act="gelu",
    hidden_dropout_prob=0.1,
    attention_probs_dropout_prob=0.1,
    max_position_embeddings=512,
    type_vocab_size=0,
    initializer_range=0.02,
    layer_norm_eps=1e-7,
    relative_attention=False,
    max_relative_positions=-1,
    pad_token_id=0,
    position_biased_input=True,
    pos_att_type=None,
    pooler_dropout=0,
    pooler_hidden_act="gelu",
    **kwargs,
):
    """
    Initialize a DebertaConfig object.

    Args:
        self: The object instance.
        vocab_size (int, optional): The size of the vocabulary. Default is 50265.
        hidden_size (int, optional): The size of the hidden layers. Default is 768.
        num_hidden_layers (int, optional): The number of hidden layers. Default is 12.
        num_attention_heads (int, optional): The number of attention heads. Default is 12.
        intermediate_size (int, optional): The size of the intermediate layers. Default is 3072.
        hidden_act (str, optional): The activation function for hidden layers. Default is 'gelu'.
        hidden_dropout_prob (float, optional): The dropout probability for hidden layers. Default is 0.1.
        attention_probs_dropout_prob (float, optional): The dropout probability for attention probabilities. Default is 0.1.
        max_position_embeddings (int, optional): The maximum position embeddings. Default is 512.
        type_vocab_size (int, optional): The size of the type vocabulary. Default is 0.
        initializer_range (float, optional): The range for parameter initialization. Default is 0.02.
        layer_norm_eps (float): The epsilon value for layer normalization. Default is 1e-07.
        relative_attention (bool, optional): Whether to use relative attention. Default is False.
        max_relative_positions (int, optional): The maximum relative positions for relative attention. Default is -1.
        pad_token_id (int, optional): The token ID for padding. Default is 0.
        position_biased_input (bool, optional): Whether to use position-biased input. Default is True.
        pos_att_type (str or list of str, optional): The type of positional attention. Default is None.
        pooler_dropout (float, optional): The dropout probability for the pooler layer. Default is 0.
        pooler_hidden_act (str, optional): The activation function for the pooler layer. Default is 'gelu'.

    Returns:
        None.

    Raises:
        None.
    """
    super().__init__(**kwargs)

    self.hidden_size = hidden_size
    self.num_hidden_layers = num_hidden_layers
    self.num_attention_heads = num_attention_heads
    self.intermediate_size = intermediate_size
    self.hidden_act = hidden_act
    self.hidden_dropout_prob = hidden_dropout_prob
    self.attention_probs_dropout_prob = attention_probs_dropout_prob
    self.max_position_embeddings = max_position_embeddings
    self.type_vocab_size = type_vocab_size
    self.initializer_range = initializer_range
    self.relative_attention = relative_attention
    self.max_relative_positions = max_relative_positions
    self.pad_token_id = pad_token_id
    self.position_biased_input = position_biased_input

    # Backwards compatibility
    if isinstance(pos_att_type, str):
        pos_att_type = [x.strip() for x in pos_att_type.lower().split("|")]

    self.pos_att_type = pos_att_type
    self.vocab_size = vocab_size
    self.layer_norm_eps = layer_norm_eps

    self.pooler_hidden_size = kwargs.get("pooler_hidden_size", hidden_size)
    self.pooler_dropout = pooler_dropout
    self.pooler_hidden_act = pooler_hidden_act

mindnlp.transformers.models.deberta.tokenization_deberta

Tokenization class for model DeBERTa.

mindnlp.transformers.models.deberta.tokenization_deberta.DebertaTokenizer

Bases: PreTrainedTokenizer

Construct a DeBERTa tokenizer. Based on byte-level Byte-Pair-Encoding.

This tokenizer has been trained to treat spaces like parts of the tokens (a bit like sentencepiece) so a word will be encoded differently whether it is at the beginning of the sentence (without space) or not:

Example

>>> from transformers import DebertaTokenizer
...
>>> tokenizer = DebertaTokenizer.from_pretrained("microsoft/deberta-base")
>>> tokenizer("Hello world")["input_ids"]
[1, 31414, 232, 2]
>>> tokenizer(" Hello world")["input_ids"]
[1, 20920, 232, 2]

You can get around that behavior by passing add_prefix_space=True when instantiating this tokenizer or when you call it on some text, but since the model was not pretrained this way, it might yield a decrease in performance.

When used with is_split_into_words=True, this tokenizer will add a space before each word (even the first one).

This tokenizer inherits from [PreTrainedTokenizer] which contains most of the main methods. Users should refer to this superclass for more information regarding those methods.

PARAMETER	DESCRIPTION
vocab_file	Path to the vocabulary file. TYPE: `str`
merges_file	Path to the merges file. TYPE: `str`
errors	Paradigm to follow when decoding bytes to UTF-8. See bytes.decode for more information. TYPE: `str`, *optional*, defaults to `"replace"` DEFAULT: 'replace'
bos_token	The beginning of sequence token. TYPE: `str`, *optional*, defaults to `"[CLS]"` DEFAULT: '[CLS]'
eos_token	The end of sequence token. TYPE: `str`, *optional*, defaults to `"[SEP]"` DEFAULT: '[SEP]'
sep_token	The separator token, which is used when building a sequence from multiple sequences, e.g. two sequences for sequence classification or for a text and a question for question answering. It is also used as the last token of a sequence built with special tokens. TYPE: `str`, *optional*, defaults to `"[SEP]"` DEFAULT: '[SEP]'
cls_token	The classifier token which is used when doing sequence classification (classification of the whole sequence instead of per-token classification). It is the first token of the sequence when built with special tokens. TYPE: `str`, *optional*, defaults to `"[CLS]"` DEFAULT: '[CLS]'
unk_token	The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this token instead. TYPE: `str`, *optional*, defaults to `"[UNK]"` DEFAULT: '[UNK]'
pad_token	The token used for padding, for example when batching sequences of different lengths. TYPE: `str`, *optional*, defaults to `"[PAD]"` DEFAULT: '[PAD]'
mask_token	The token used for masking values. This is the token used when training this model with masked language modeling. This is the token which the model will try to predict. TYPE: `str`, *optional*, defaults to `"[MASK]"` DEFAULT: '[MASK]'
add_prefix_space	Whether or not to add an initial space to the input. This allows to treat the leading word just as any other word. (Deberta tokenizer detect beginning of words by the preceding space). TYPE: `bool`, *optional*, defaults to `False` DEFAULT: False
add_bos_token	Whether or not to add an initial <|endoftext|> to the input. This allows to treat the leading word just as any other word. TYPE: `bool`, *optional*, defaults to `False` DEFAULT: False

Source code in mindnlp/transformers/models/deberta/tokenization_deberta.py

class DebertaTokenizer(PreTrainedTokenizer):
    """
    Construct a DeBERTa tokenizer. Based on byte-level Byte-Pair-Encoding.

    This tokenizer has been trained to treat spaces like parts of the tokens (a bit like sentencepiece) so a word will
    be encoded differently whether it is at the beginning of the sentence (without space) or not:

    Example:
        ```python
        >>> from transformers import DebertaTokenizer
        ...
        >>> tokenizer = DebertaTokenizer.from_pretrained("microsoft/deberta-base")
        >>> tokenizer("Hello world")["input_ids"]
        [1, 31414, 232, 2]
        >>> tokenizer(" Hello world")["input_ids"]
        [1, 20920, 232, 2]
        ```

    You can get around that behavior by passing `add_prefix_space=True` when instantiating this tokenizer or when you
    call it on some text, but since the model was not pretrained this way, it might yield a decrease in performance.

    <Tip>

    When used with `is_split_into_words=True`, this tokenizer will add a space before each word (even the first one).

    </Tip>

    This tokenizer inherits from [`PreTrainedTokenizer`] which contains most of the main methods. Users should refer to
    this superclass for more information regarding those methods.

    Args:
        vocab_file (`str`):
            Path to the vocabulary file.
        merges_file (`str`):
            Path to the merges file.
        errors (`str`, *optional*, defaults to `"replace"`):
            Paradigm to follow when decoding bytes to UTF-8. See
            [bytes.decode](https://docs.python.org/3/library/stdtypes.html#bytes.decode) for more information.
        bos_token (`str`, *optional*, defaults to `"[CLS]"`):
            The beginning of sequence token.
        eos_token (`str`, *optional*, defaults to `"[SEP]"`):
            The end of sequence token.
        sep_token (`str`, *optional*, defaults to `"[SEP]"`):
            The separator token, which is used when building a sequence from multiple sequences, e.g. two sequences for
            sequence classification or for a text and a question for question answering. It is also used as the last
            token of a sequence built with special tokens.
        cls_token (`str`, *optional*, defaults to `"[CLS]"`):
            The classifier token which is used when doing sequence classification (classification of the whole sequence
            instead of per-token classification). It is the first token of the sequence when built with special tokens.
        unk_token (`str`, *optional*, defaults to `"[UNK]"`):
            The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this
            token instead.
        pad_token (`str`, *optional*, defaults to `"[PAD]"`):
            The token used for padding, for example when batching sequences of different lengths.
        mask_token (`str`, *optional*, defaults to `"[MASK]"`):
            The token used for masking values. This is the token used when training this model with masked language
            modeling. This is the token which the model will try to predict.
        add_prefix_space (`bool`, *optional*, defaults to `False`):
            Whether or not to add an initial space to the input. This allows to treat the leading word just as any
            other word. (Deberta tokenizer detect beginning of words by the preceding space).
        add_bos_token (`bool`, *optional*, defaults to `False`):
            Whether or not to add an initial <|endoftext|> to the input. This allows to treat the leading word just as
            any other word.
    """
    vocab_files_names = VOCAB_FILES_NAMES
    pretrained_vocab_files_map = PRETRAINED_VOCAB_FILES_MAP
    max_model_input_sizes = PRETRAINED_POSITIONAL_EMBEDDINGS_SIZES
    model_input_names = ["input_ids", "attention_mask", "token_type_ids"]

    def __init__(
        self,
        vocab_file,
        merges_file,
        errors="replace",
        bos_token="[CLS]",
        eos_token="[SEP]",
        sep_token="[SEP]",
        cls_token="[CLS]",
        unk_token="[UNK]",
        pad_token="[PAD]",
        mask_token="[MASK]",
        add_prefix_space=False,
        add_bos_token=False,
        **kwargs,
    ):
        """
        Initialize a DebertaTokenizer object.

        Args:
            self: The instance of the class.
            vocab_file (str): The path to the vocabulary file.
            merges_file (str): The path to the merges file.
            errors (str, optional): The error handling strategy. Default is 'replace'.
            bos_token (str, optional): Beginning of sentence token. Default is '[CLS]'.
            eos_token (str, optional): End of sentence token. Default is '[SEP]'.
            sep_token (str, optional): Separator token. Default is '[SEP]'.
            cls_token (str, optional): Classification token. Default is '[CLS]'.
            unk_token (str, optional): Token for unknown words. Default is '[UNK]'.
            pad_token (str, optional): Token for padding. Default is '[PAD]'.
            mask_token (str, optional): Token for masking. Default is '[MASK]'.
            add_prefix_space (bool, optional): Whether to add prefix space. Default is False.
            add_bos_token (bool, optional): Whether to add beginning of sentence token. Default is False.

        Returns:
            None.

        Raises:
            IOError: If there is an issue with opening the vocab_file or merges_file.
            Exception: Any other unexpected error that may occur during initialization.
        """
        bos_token = AddedToken(bos_token, special=True) if isinstance(bos_token, str) else bos_token
        eos_token = AddedToken(eos_token, special=True) if isinstance(eos_token, str) else eos_token
        sep_token = AddedToken(sep_token, special=True) if isinstance(sep_token, str) else sep_token
        cls_token = AddedToken(cls_token, special=True) if isinstance(cls_token, str) else cls_token
        unk_token = AddedToken(unk_token, special=True) if isinstance(unk_token, str) else unk_token
        pad_token = AddedToken(pad_token, special=True) if isinstance(pad_token, str) else pad_token

        # Mask token behave like a normal word, i.e. include the space before it
        mask_token = AddedToken(mask_token, lstrip=True, rstrip=False) if isinstance(mask_token, str) else mask_token
        self.add_bos_token = add_bos_token

        with open(vocab_file, encoding="utf-8") as vocab_handle:
            self.encoder = json.load(vocab_handle)
        self.decoder = {v: k for k, v in self.encoder.items()}
        self.errors = errors  # how to handle errors in decoding
        self.byte_encoder = bytes_to_unicode()
        self.byte_decoder = {v: k for k, v in self.byte_encoder.items()}
        with open(merges_file, encoding="utf-8") as merges_handle:
            bpe_merges = merges_handle.read().split("\n")[1:-1]
        bpe_merges = [tuple(merge.split()) for merge in bpe_merges]
        self.bpe_ranks = dict(zip(bpe_merges, range(len(bpe_merges))))
        self.cache = {}
        self.add_prefix_space = add_prefix_space

        # Should have added re.IGNORECASE so BPE merges can happen for capitalized versions of contractions
        self.pat = re.compile(r"""'s|'t|'re|'ve|'m|'ll|'d| ?\p{L}+| ?\p{N}+| ?[^\s\p{L}\p{N}]+|\s+(?!\S)|\s+""")

        super().__init__(
            errors=errors,
            bos_token=bos_token,
            eos_token=eos_token,
            unk_token=unk_token,
            sep_token=sep_token,
            cls_token=cls_token,
            pad_token=pad_token,
            mask_token=mask_token,
            add_prefix_space=add_prefix_space,
            add_bos_token=add_bos_token,
            **kwargs,
        )

    @property
    # Copied from transformers.models.gpt2.tokenization_gpt2.GPT2Tokenizer.vocab_size
    def vocab_size(self):
        """
        Returns the size of the vocabulary used by the DebertaTokenizer.

        Args:
            self (DebertaTokenizer): The instance of the DebertaTokenizer class.

        Returns:
            int: The number of unique tokens in the vocabulary.

        Raises:
            None.
        """
        return len(self.encoder)

    # Copied from transformers.models.gpt2.tokenization_gpt2.GPT2Tokenizer.get_vocab
    def get_vocab(self):
        """
        Returns the vocabulary of the DebertaTokenizer.

        Args:
            self (DebertaTokenizer): An instance of the DebertaTokenizer class.

        Returns:
            dict: The vocabulary of the tokenizer,
                which is a dictionary containing the encoder mappings of the tokenizer's tokens and any added tokens.

        Raises:
            None: This method does not raise any exceptions.
        """
        return dict(self.encoder, **self.added_tokens_encoder)

    # Copied from transformers.models.gpt2.tokenization_gpt2.GPT2Tokenizer.bpe
    def bpe(self, token):
        """
        Performs Byte Pair Encoding (BPE) on the given token.

        Args:
            self (DebertaTokenizer): An instance of the DebertaTokenizer class.
            token (str): The token to be encoded using BPE.

        Returns:
            str: The encoded token after applying BPE.

        Raises:
            None.

        This method applies BPE to the given token by iteratively replacing the most frequent pairs of characters in
        the token with a single character.
        If the token is already present in the cache, the cached value is returned.
        Otherwise, the token is converted to a tuple of characters. Pairs of characters in the tuple are obtained
        using the 'get_pairs' function. If no pairs are found, the original token is returned.

        The method then enters a loop where it selects the most frequent pair from the pairs obtained.
        If the selected pair is not present in the 'bpe_ranks' dictionary, the loop is terminated.
        Otherwise, the first and second characters of the pair are extracted.

        A new word list, 'new_word', is created to store the modified characters of the token.
        The method iterates over the characters of the token and checks if the current character matches the first
        character of the selected pair.
        If it does, and the next character is the second character of the pair, the pair is replaced with a
        single character by appending it to 'new_word' and incrementing the index by 2.
        Otherwise, the current character is appended to 'new_word' and the index is incremented by 1.

        The modified 'new_word' is converted back to a tuple and assigned to 'word'.
        If the length of 'word' becomes 1, indicating that the BPE process is complete, the loop is terminated.
        Otherwise, new pairs are obtained from 'word' and the process is repeated until 'word' is of length 1.

        Finally, 'word' is converted to a string by joining the characters with spaces.
        The encoded token is stored in the cache for future use and returned.

        Note:
            - This method assumes the presence of the 'get_pairs' function and the 'bpe_ranks' dictionary.

        """
        if token in self.cache:
            return self.cache[token]
        word = tuple(token)
        pairs = get_pairs(word)

        if not pairs:
            return token

        while True:
            bigram = min(pairs, key=lambda pair: self.bpe_ranks.get(pair, float("inf")))
            if bigram not in self.bpe_ranks:
                break
            first, second = bigram
            new_word = []
            i = 0
            while i < len(word):
                try:
                    j = word.index(first, i)
                except ValueError:
                    new_word.extend(word[i:])
                    break
                else:
                    new_word.extend(word[i:j])
                    i = j

                if word[i] == first and i < len(word) - 1 and word[i + 1] == second:
                    new_word.append(first + second)
                    i += 2
                else:
                    new_word.append(word[i])
                    i += 1
            new_word = tuple(new_word)
            word = new_word
            if len(word) == 1:
                break
            pairs = get_pairs(word)
        word = " ".join(word)
        self.cache[token] = word
        return word

    def build_inputs_with_special_tokens(
        self, token_ids_0: List[int], token_ids_1: Optional[List[int]] = None
    ) -> List[int]:
        """
        Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and
        adding special tokens. A DeBERTa sequence has the following format:

        - single sequence: [CLS] X [SEP]
        - pair of sequences: [CLS] A [SEP] B [SEP]

        Args:
            token_ids_0 (`List[int]`):
                List of IDs to which the special tokens will be added.
            token_ids_1 (`List[int]`, *optional*):
                Optional second list of IDs for sequence pairs.

        Returns:
            `List[int]`: List of [input IDs](../glossary#input-ids) with the appropriate special tokens.
        """
        if token_ids_1 is None:
            return [self.cls_token_id] + token_ids_0 + [self.sep_token_id]
        cls = [self.cls_token_id]
        sep = [self.sep_token_id]
        return cls + token_ids_0 + sep + token_ids_1 + sep

    def get_special_tokens_mask(
        self, token_ids_0: List[int], token_ids_1: Optional[List[int]] = None, already_has_special_tokens: bool = False
    ) -> List[int]:
        """
        Retrieves sequence ids from a token list that has no special tokens added. This method is called when adding
        special tokens using the tokenizer `prepare_for_model` or `encode_plus` methods.

        Args:
            token_ids_0 (`List[int]`):
                List of IDs.
            token_ids_1 (`List[int]`, *optional*):
                Optional second list of IDs for sequence pairs.
            already_has_special_tokens (`bool`, *optional*, defaults to `False`):
                Whether or not the token list is already formatted with special tokens for the model.

        Returns:
            `List[int]`: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token.
        """
        if already_has_special_tokens:
            return super().get_special_tokens_mask(
                token_ids_0=token_ids_0, token_ids_1=token_ids_1, already_has_special_tokens=True
            )

        if token_ids_1 is None:
            return [1] + ([0] * len(token_ids_0)) + [1]
        return [1] + ([0] * len(token_ids_0)) + [1] + ([0] * len(token_ids_1)) + [1]

    def create_token_type_ids_from_sequences(
        self, token_ids_0: List[int], token_ids_1: Optional[List[int]] = None
    ) -> List[int]:
        """
        Create a mask from the two sequences passed to be used in a sequence-pair classification task. A DeBERTa
        sequence pair mask has the following format:

        ```
        0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1
        | first sequence    | second sequence |
        ```

        If `token_ids_1` is `None`, this method only returns the first portion of the mask (0s).

        Args:
            token_ids_0 (`List[int]`):
                List of IDs.
            token_ids_1 (`List[int]`, *optional*):
                Optional second list of IDs for sequence pairs.

        Returns:
            `List[int]`: List of [token type IDs](../glossary#token-type-ids) according to the given sequence(s).
        """
        sep = [self.sep_token_id]
        cls = [self.cls_token_id]

        if token_ids_1 is None:
            return len(cls + token_ids_0 + sep) * [0]
        return len(cls + token_ids_0 + sep) * [0] + len(token_ids_1 + sep) * [1]

    # Copied from transformers.models.gpt2.tokenization_gpt2.GPT2Tokenizer._tokenize
    def _tokenize(self, text):
        """Tokenize a string."""
        bpe_tokens = []
        for token in re.findall(self.pat, text):
            token = "".join(
                self.byte_encoder[b] for b in token.encode("utf-8")
            )  # Maps all our bytes to unicode strings, avoiding control tokens of the BPE (spaces in our case)
            bpe_tokens.extend(bpe_token for bpe_token in self.bpe(token).split(" "))
        return bpe_tokens

    # Copied from transformers.models.gpt2.tokenization_gpt2.GPT2Tokenizer._convert_token_to_id
    def _convert_token_to_id(self, token):
        """Converts a token (str) in an id using the vocab."""
        return self.encoder.get(token, self.encoder.get(self.unk_token))

    # Copied from transformers.models.gpt2.tokenization_gpt2.GPT2Tokenizer._convert_id_to_token
    def _convert_id_to_token(self, index):
        """Converts an index (integer) in a token (str) using the vocab."""
        return self.decoder.get(index)

    # Copied from transformers.models.gpt2.tokenization_gpt2.GPT2Tokenizer.convert_tokens_to_string
    def convert_tokens_to_string(self, tokens):
        """Converts a sequence of tokens (string) in a single string."""
        text = "".join(tokens)
        text = bytearray([self.byte_decoder[c] for c in text]).decode("utf-8", errors=self.errors)
        return text

    # Copied from transformers.models.gpt2.tokenization_gpt2.GPT2Tokenizer.save_vocabulary
    def save_vocabulary(self, save_directory: str, filename_prefix: Optional[str] = None) -> Tuple[str]:
        """
        Save the vocabulary to files in the specified directory with an optional filename prefix.

        Args:
            self (DebertaTokenizer): The instance of the DebertaTokenizer class.
            save_directory (str): The directory path where the vocabulary files will be saved.
            filename_prefix (Optional[str]): An optional prefix to be added to the filenames. Defaults to None.

        Returns:
            Tuple[str]: A tuple containing the paths to the saved vocabulary file and merge file.

        Raises:
            FileNotFoundError: If the specified save_directory does not exist.
            IOError: If there is an issue encountered while writing to the vocabulary or merge files.
            RuntimeError: If the BPE merge indices are not consecutive,
                indicating a potential corruption in the tokenizer.
        """
        if not os.path.isdir(save_directory):
            logger.error(f"Vocabulary path ({save_directory}) should be a directory")
            return
        vocab_file = os.path.join(
            save_directory, (filename_prefix + "-" if filename_prefix else "") + VOCAB_FILES_NAMES["vocab_file"]
        )
        merge_file = os.path.join(
            save_directory, (filename_prefix + "-" if filename_prefix else "") + VOCAB_FILES_NAMES["merges_file"]
        )

        with open(vocab_file, "w", encoding="utf-8") as f:
            f.write(json.dumps(self.encoder, indent=2, sort_keys=True, ensure_ascii=False) + "\n")

        index = 0
        with open(merge_file, "w", encoding="utf-8") as writer:
            writer.write("#version: 0.2\n")
            for bpe_tokens, token_index in sorted(self.bpe_ranks.items(), key=lambda kv: kv[1]):
                if index != token_index:
                    logger.warning(
                        f"Saving vocabulary to {merge_file}: BPE merge indices are not consecutive."
                        " Please check that the tokenizer is not corrupted!"
                    )
                    index = token_index
                writer.write(" ".join(bpe_tokens) + "\n")
                index += 1

        return vocab_file, merge_file

    def prepare_for_tokenization(self, text, is_split_into_words=False, **kwargs):
        """
        This method prepares the input text for tokenization by potentially adding a prefix space based on
        the provided parameters.

        Args:
            self: The instance of the DebertaTokenizer class.
            text (str): The input text to be tokenized.
            is_split_into_words (bool): A flag indicating whether the text is already split into words. Default is False.

        Returns:
            None: This method modifies the input text in place and does not return any value.

        Raises:
            None.
        """
        add_prefix_space = kwargs.pop("add_prefix_space", self.add_prefix_space)
        if (is_split_into_words or add_prefix_space) and (len(text) > 0 and not text[0].isspace()):
            text = " " + text
        return (text, kwargs)

mindnlp.transformers.models.deberta.tokenization_deberta.DebertaTokenizer.vocab_size property

Returns the size of the vocabulary used by the DebertaTokenizer.

PARAMETER	DESCRIPTION
self	The instance of the DebertaTokenizer class. TYPE: DebertaTokenizer

RETURNS	DESCRIPTION
int	The number of unique tokens in the vocabulary.

mindnlp.transformers.models.deberta.tokenization_deberta.DebertaTokenizer.__init__(vocab_file, merges_file, errors='replace', bos_token='[CLS]', eos_token='[SEP]', sep_token='[SEP]', cls_token='[CLS]', unk_token='[UNK]', pad_token='[PAD]', mask_token='[MASK]', add_prefix_space=False, add_bos_token=False, **kwargs)

Initialize a DebertaTokenizer object.

PARAMETER	DESCRIPTION
self	The instance of the class.
vocab_file	The path to the vocabulary file. TYPE: str
merges_file	The path to the merges file. TYPE: str
errors	The error handling strategy. Default is 'replace'. TYPE: str DEFAULT: 'replace'
bos_token	Beginning of sentence token. Default is '[CLS]'. TYPE: str DEFAULT: '[CLS]'
eos_token	End of sentence token. Default is '[SEP]'. TYPE: str DEFAULT: '[SEP]'
sep_token	Separator token. Default is '[SEP]'. TYPE: str DEFAULT: '[SEP]'
cls_token	Classification token. Default is '[CLS]'. TYPE: str DEFAULT: '[CLS]'
unk_token	Token for unknown words. Default is '[UNK]'. TYPE: str DEFAULT: '[UNK]'
pad_token	Token for padding. Default is '[PAD]'. TYPE: str DEFAULT: '[PAD]'
mask_token	Token for masking. Default is '[MASK]'. TYPE: str DEFAULT: '[MASK]'
add_prefix_space	Whether to add prefix space. Default is False. TYPE: bool DEFAULT: False
add_bos_token	Whether to add beginning of sentence token. Default is False. TYPE: bool DEFAULT: False

RETURNS	DESCRIPTION
	None.

RAISES	DESCRIPTION
IOError	If there is an issue with opening the vocab_file or merges_file.
Exception	Any other unexpected error that may occur during initialization.

Source code in mindnlp/transformers/models/deberta/tokenization_deberta.py

def __init__(
    self,
    vocab_file,
    merges_file,
    errors="replace",
    bos_token="[CLS]",
    eos_token="[SEP]",
    sep_token="[SEP]",
    cls_token="[CLS]",
    unk_token="[UNK]",
    pad_token="[PAD]",
    mask_token="[MASK]",
    add_prefix_space=False,
    add_bos_token=False,
    **kwargs,
):
    """
    Initialize a DebertaTokenizer object.

    Args:
        self: The instance of the class.
        vocab_file (str): The path to the vocabulary file.
        merges_file (str): The path to the merges file.
        errors (str, optional): The error handling strategy. Default is 'replace'.
        bos_token (str, optional): Beginning of sentence token. Default is '[CLS]'.
        eos_token (str, optional): End of sentence token. Default is '[SEP]'.
        sep_token (str, optional): Separator token. Default is '[SEP]'.
        cls_token (str, optional): Classification token. Default is '[CLS]'.
        unk_token (str, optional): Token for unknown words. Default is '[UNK]'.
        pad_token (str, optional): Token for padding. Default is '[PAD]'.
        mask_token (str, optional): Token for masking. Default is '[MASK]'.
        add_prefix_space (bool, optional): Whether to add prefix space. Default is False.
        add_bos_token (bool, optional): Whether to add beginning of sentence token. Default is False.

    Returns:
        None.

    Raises:
        IOError: If there is an issue with opening the vocab_file or merges_file.
        Exception: Any other unexpected error that may occur during initialization.
    """
    bos_token = AddedToken(bos_token, special=True) if isinstance(bos_token, str) else bos_token
    eos_token = AddedToken(eos_token, special=True) if isinstance(eos_token, str) else eos_token
    sep_token = AddedToken(sep_token, special=True) if isinstance(sep_token, str) else sep_token
    cls_token = AddedToken(cls_token, special=True) if isinstance(cls_token, str) else cls_token
    unk_token = AddedToken(unk_token, special=True) if isinstance(unk_token, str) else unk_token
    pad_token = AddedToken(pad_token, special=True) if isinstance(pad_token, str) else pad_token

    # Mask token behave like a normal word, i.e. include the space before it
    mask_token = AddedToken(mask_token, lstrip=True, rstrip=False) if isinstance(mask_token, str) else mask_token
    self.add_bos_token = add_bos_token

    with open(vocab_file, encoding="utf-8") as vocab_handle:
        self.encoder = json.load(vocab_handle)
    self.decoder = {v: k for k, v in self.encoder.items()}
    self.errors = errors  # how to handle errors in decoding
    self.byte_encoder = bytes_to_unicode()
    self.byte_decoder = {v: k for k, v in self.byte_encoder.items()}
    with open(merges_file, encoding="utf-8") as merges_handle:
        bpe_merges = merges_handle.read().split("\n")[1:-1]
    bpe_merges = [tuple(merge.split()) for merge in bpe_merges]
    self.bpe_ranks = dict(zip(bpe_merges, range(len(bpe_merges))))
    self.cache = {}
    self.add_prefix_space = add_prefix_space

    # Should have added re.IGNORECASE so BPE merges can happen for capitalized versions of contractions
    self.pat = re.compile(r"""'s|'t|'re|'ve|'m|'ll|'d| ?\p{L}+| ?\p{N}+| ?[^\s\p{L}\p{N}]+|\s+(?!\S)|\s+""")

    super().__init__(
        errors=errors,
        bos_token=bos_token,
        eos_token=eos_token,
        unk_token=unk_token,
        sep_token=sep_token,
        cls_token=cls_token,
        pad_token=pad_token,
        mask_token=mask_token,
        add_prefix_space=add_prefix_space,
        add_bos_token=add_bos_token,
        **kwargs,
    )

mindnlp.transformers.models.deberta.tokenization_deberta.DebertaTokenizer.bpe(token)

Performs Byte Pair Encoding (BPE) on the given token.

PARAMETER	DESCRIPTION
self	An instance of the DebertaTokenizer class. TYPE: DebertaTokenizer
token	The token to be encoded using BPE. TYPE: str

RETURNS	DESCRIPTION
str	The encoded token after applying BPE.

This method applies BPE to the given token by iteratively replacing the most frequent pairs of characters in the token with a single character. If the token is already present in the cache, the cached value is returned. Otherwise, the token is converted to a tuple of characters. Pairs of characters in the tuple are obtained using the 'get_pairs' function. If no pairs are found, the original token is returned.

The method then enters a loop where it selects the most frequent pair from the pairs obtained. If the selected pair is not present in the 'bpe_ranks' dictionary, the loop is terminated. Otherwise, the first and second characters of the pair are extracted.

A new word list, 'new_word', is created to store the modified characters of the token. The method iterates over the characters of the token and checks if the current character matches the first character of the selected pair. If it does, and the next character is the second character of the pair, the pair is replaced with a single character by appending it to 'new_word' and incrementing the index by 2. Otherwise, the current character is appended to 'new_word' and the index is incremented by 1.

The modified 'new_word' is converted back to a tuple and assigned to 'word'. If the length of 'word' becomes 1, indicating that the BPE process is complete, the loop is terminated. Otherwise, new pairs are obtained from 'word' and the process is repeated until 'word' is of length 1.

Finally, 'word' is converted to a string by joining the characters with spaces. The encoded token is stored in the cache for future use and returned.

Note

• This method assumes the presence of the 'get_pairs' function and the 'bpe_ranks' dictionary.

Source code in mindnlp/transformers/models/deberta/tokenization_deberta.py

def bpe(self, token):
    """
    Performs Byte Pair Encoding (BPE) on the given token.

    Args:
        self (DebertaTokenizer): An instance of the DebertaTokenizer class.
        token (str): The token to be encoded using BPE.

    Returns:
        str: The encoded token after applying BPE.

    Raises:
        None.

    This method applies BPE to the given token by iteratively replacing the most frequent pairs of characters in
    the token with a single character.
    If the token is already present in the cache, the cached value is returned.
    Otherwise, the token is converted to a tuple of characters. Pairs of characters in the tuple are obtained
    using the 'get_pairs' function. If no pairs are found, the original token is returned.

    The method then enters a loop where it selects the most frequent pair from the pairs obtained.
    If the selected pair is not present in the 'bpe_ranks' dictionary, the loop is terminated.
    Otherwise, the first and second characters of the pair are extracted.

    A new word list, 'new_word', is created to store the modified characters of the token.
    The method iterates over the characters of the token and checks if the current character matches the first
    character of the selected pair.
    If it does, and the next character is the second character of the pair, the pair is replaced with a
    single character by appending it to 'new_word' and incrementing the index by 2.
    Otherwise, the current character is appended to 'new_word' and the index is incremented by 1.

    The modified 'new_word' is converted back to a tuple and assigned to 'word'.
    If the length of 'word' becomes 1, indicating that the BPE process is complete, the loop is terminated.
    Otherwise, new pairs are obtained from 'word' and the process is repeated until 'word' is of length 1.

    Finally, 'word' is converted to a string by joining the characters with spaces.
    The encoded token is stored in the cache for future use and returned.

    Note:
        - This method assumes the presence of the 'get_pairs' function and the 'bpe_ranks' dictionary.

    """
    if token in self.cache:
        return self.cache[token]
    word = tuple(token)
    pairs = get_pairs(word)

    if not pairs:
        return token

    while True:
        bigram = min(pairs, key=lambda pair: self.bpe_ranks.get(pair, float("inf")))
        if bigram not in self.bpe_ranks:
            break
        first, second = bigram
        new_word = []
        i = 0
        while i < len(word):
            try:
                j = word.index(first, i)
            except ValueError:
                new_word.extend(word[i:])
                break
            else:
                new_word.extend(word[i:j])
                i = j

            if word[i] == first and i < len(word) - 1 and word[i + 1] == second:
                new_word.append(first + second)
                i += 2
            else:
                new_word.append(word[i])
                i += 1
        new_word = tuple(new_word)
        word = new_word
        if len(word) == 1:
            break
        pairs = get_pairs(word)
    word = " ".join(word)
    self.cache[token] = word
    return word

mindnlp.transformers.models.deberta.tokenization_deberta.DebertaTokenizer.build_inputs_with_special_tokens(token_ids_0, token_ids_1=None)

Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and adding special tokens. A DeBERTa sequence has the following format:

• single sequence: [CLS] X [SEP]
• pair of sequences: [CLS] A [SEP] B [SEP]

PARAMETER	DESCRIPTION
token_ids_0	List of IDs to which the special tokens will be added. TYPE: `List[int]`
token_ids_1	Optional second list of IDs for sequence pairs. TYPE: `List[int]`, *optional* DEFAULT: None

RETURNS	DESCRIPTION
List[int]	List[int]: List of input IDs with the appropriate special tokens.

Source code in mindnlp/transformers/models/deberta/tokenization_deberta.py

def build_inputs_with_special_tokens(
    self, token_ids_0: List[int], token_ids_1: Optional[List[int]] = None
) -> List[int]:
    """
    Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and
    adding special tokens. A DeBERTa sequence has the following format:

    - single sequence: [CLS] X [SEP]
    - pair of sequences: [CLS] A [SEP] B [SEP]

    Args:
        token_ids_0 (`List[int]`):
            List of IDs to which the special tokens will be added.
        token_ids_1 (`List[int]`, *optional*):
            Optional second list of IDs for sequence pairs.

    Returns:
        `List[int]`: List of [input IDs](../glossary#input-ids) with the appropriate special tokens.
    """
    if token_ids_1 is None:
        return [self.cls_token_id] + token_ids_0 + [self.sep_token_id]
    cls = [self.cls_token_id]
    sep = [self.sep_token_id]
    return cls + token_ids_0 + sep + token_ids_1 + sep

mindnlp.transformers.models.deberta.tokenization_deberta.DebertaTokenizer.convert_tokens_to_string(tokens)

Converts a sequence of tokens (string) in a single string.

Source code in mindnlp/transformers/models/deberta/tokenization_deberta.py

def convert_tokens_to_string(self, tokens):
    """Converts a sequence of tokens (string) in a single string."""
    text = "".join(tokens)
    text = bytearray([self.byte_decoder[c] for c in text]).decode("utf-8", errors=self.errors)
    return text

mindnlp.transformers.models.deberta.tokenization_deberta.DebertaTokenizer.create_token_type_ids_from_sequences(token_ids_0, token_ids_1=None)

Create a mask from the two sequences passed to be used in a sequence-pair classification task. A DeBERTa sequence pair mask has the following format:

0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1
| first sequence    | second sequence |

If token_ids_1 is None, this method only returns the first portion of the mask (0s).

PARAMETER	DESCRIPTION
token_ids_0	List of IDs. TYPE: `List[int]`
token_ids_1	Optional second list of IDs for sequence pairs. TYPE: `List[int]`, *optional* DEFAULT: None

RETURNS	DESCRIPTION
List[int]	List[int]: List of token type IDs according to the given sequence(s).

Source code in mindnlp/transformers/models/deberta/tokenization_deberta.py

def create_token_type_ids_from_sequences(
    self, token_ids_0: List[int], token_ids_1: Optional[List[int]] = None
) -> List[int]:
    """
    Create a mask from the two sequences passed to be used in a sequence-pair classification task. A DeBERTa
    sequence pair mask has the following format:

    ```
    0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1
    | first sequence    | second sequence |
    ```

    If `token_ids_1` is `None`, this method only returns the first portion of the mask (0s).

    Args:
        token_ids_0 (`List[int]`):
            List of IDs.
        token_ids_1 (`List[int]`, *optional*):
            Optional second list of IDs for sequence pairs.

    Returns:
        `List[int]`: List of [token type IDs](../glossary#token-type-ids) according to the given sequence(s).
    """
    sep = [self.sep_token_id]
    cls = [self.cls_token_id]

    if token_ids_1 is None:
        return len(cls + token_ids_0 + sep) * [0]
    return len(cls + token_ids_0 + sep) * [0] + len(token_ids_1 + sep) * [1]

mindnlp.transformers.models.deberta.tokenization_deberta.DebertaTokenizer.get_special_tokens_mask(token_ids_0, token_ids_1=None, already_has_special_tokens=False)

Retrieves sequence ids from a token list that has no special tokens added. This method is called when adding special tokens using the tokenizer prepare_for_model or encode_plus methods.

PARAMETER	DESCRIPTION
token_ids_0	List of IDs. TYPE: `List[int]`
token_ids_1	Optional second list of IDs for sequence pairs. TYPE: `List[int]`, *optional* DEFAULT: None
already_has_special_tokens	Whether or not the token list is already formatted with special tokens for the model. TYPE: `bool`, *optional*, defaults to `False` DEFAULT: False

RETURNS	DESCRIPTION
List[int]	List[int]: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token.

Source code in mindnlp/transformers/models/deberta/tokenization_deberta.py

def get_special_tokens_mask(
    self, token_ids_0: List[int], token_ids_1: Optional[List[int]] = None, already_has_special_tokens: bool = False
) -> List[int]:
    """
    Retrieves sequence ids from a token list that has no special tokens added. This method is called when adding
    special tokens using the tokenizer `prepare_for_model` or `encode_plus` methods.

    Args:
        token_ids_0 (`List[int]`):
            List of IDs.
        token_ids_1 (`List[int]`, *optional*):
            Optional second list of IDs for sequence pairs.
        already_has_special_tokens (`bool`, *optional*, defaults to `False`):
            Whether or not the token list is already formatted with special tokens for the model.

    Returns:
        `List[int]`: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token.
    """
    if already_has_special_tokens:
        return super().get_special_tokens_mask(
            token_ids_0=token_ids_0, token_ids_1=token_ids_1, already_has_special_tokens=True
        )

    if token_ids_1 is None:
        return [1] + ([0] * len(token_ids_0)) + [1]
    return [1] + ([0] * len(token_ids_0)) + [1] + ([0] * len(token_ids_1)) + [1]

mindnlp.transformers.models.deberta.tokenization_deberta.DebertaTokenizer.get_vocab()

Returns the vocabulary of the DebertaTokenizer.

PARAMETER	DESCRIPTION
self	An instance of the DebertaTokenizer class. TYPE: DebertaTokenizer

RETURNS	DESCRIPTION
dict	The vocabulary of the tokenizer, which is a dictionary containing the encoder mappings of the tokenizer's tokens and any added tokens.

RAISES	DESCRIPTION
None	This method does not raise any exceptions.

Source code in mindnlp/transformers/models/deberta/tokenization_deberta.py

def get_vocab(self):
    """
    Returns the vocabulary of the DebertaTokenizer.

    Args:
        self (DebertaTokenizer): An instance of the DebertaTokenizer class.

    Returns:
        dict: The vocabulary of the tokenizer,
            which is a dictionary containing the encoder mappings of the tokenizer's tokens and any added tokens.

    Raises:
        None: This method does not raise any exceptions.
    """
    return dict(self.encoder, **self.added_tokens_encoder)

mindnlp.transformers.models.deberta.tokenization_deberta.DebertaTokenizer.prepare_for_tokenization(text, is_split_into_words=False, **kwargs)

This method prepares the input text for tokenization by potentially adding a prefix space based on the provided parameters.

PARAMETER	DESCRIPTION
self	The instance of the DebertaTokenizer class.
text	The input text to be tokenized. TYPE: str
is_split_into_words	A flag indicating whether the text is already split into words. Default is False. TYPE: bool DEFAULT: False

RETURNS	DESCRIPTION
None	This method modifies the input text in place and does not return any value.

Source code in mindnlp/transformers/models/deberta/tokenization_deberta.py

def prepare_for_tokenization(self, text, is_split_into_words=False, **kwargs):
    """
    This method prepares the input text for tokenization by potentially adding a prefix space based on
    the provided parameters.

    Args:
        self: The instance of the DebertaTokenizer class.
        text (str): The input text to be tokenized.
        is_split_into_words (bool): A flag indicating whether the text is already split into words. Default is False.

    Returns:
        None: This method modifies the input text in place and does not return any value.

    Raises:
        None.
    """
    add_prefix_space = kwargs.pop("add_prefix_space", self.add_prefix_space)
    if (is_split_into_words or add_prefix_space) and (len(text) > 0 and not text[0].isspace()):
        text = " " + text
    return (text, kwargs)

mindnlp.transformers.models.deberta.tokenization_deberta.DebertaTokenizer.save_vocabulary(save_directory, filename_prefix=None)

Save the vocabulary to files in the specified directory with an optional filename prefix.

PARAMETER	DESCRIPTION
self	The instance of the DebertaTokenizer class. TYPE: DebertaTokenizer
save_directory	The directory path where the vocabulary files will be saved. TYPE: str
filename_prefix	An optional prefix to be added to the filenames. Defaults to None. TYPE: Optional[str] DEFAULT: None

RETURNS	DESCRIPTION
Tuple[str]	Tuple[str]: A tuple containing the paths to the saved vocabulary file and merge file.

RAISES	DESCRIPTION
FileNotFoundError	If the specified save_directory does not exist.
IOError	If there is an issue encountered while writing to the vocabulary or merge files.
RuntimeError	If the BPE merge indices are not consecutive, indicating a potential corruption in the tokenizer.

Source code in mindnlp/transformers/models/deberta/tokenization_deberta.py

def save_vocabulary(self, save_directory: str, filename_prefix: Optional[str] = None) -> Tuple[str]:
    """
    Save the vocabulary to files in the specified directory with an optional filename prefix.

    Args:
        self (DebertaTokenizer): The instance of the DebertaTokenizer class.
        save_directory (str): The directory path where the vocabulary files will be saved.
        filename_prefix (Optional[str]): An optional prefix to be added to the filenames. Defaults to None.

    Returns:
        Tuple[str]: A tuple containing the paths to the saved vocabulary file and merge file.

    Raises:
        FileNotFoundError: If the specified save_directory does not exist.
        IOError: If there is an issue encountered while writing to the vocabulary or merge files.
        RuntimeError: If the BPE merge indices are not consecutive,
            indicating a potential corruption in the tokenizer.
    """
    if not os.path.isdir(save_directory):
        logger.error(f"Vocabulary path ({save_directory}) should be a directory")
        return
    vocab_file = os.path.join(
        save_directory, (filename_prefix + "-" if filename_prefix else "") + VOCAB_FILES_NAMES["vocab_file"]
    )
    merge_file = os.path.join(
        save_directory, (filename_prefix + "-" if filename_prefix else "") + VOCAB_FILES_NAMES["merges_file"]
    )

    with open(vocab_file, "w", encoding="utf-8") as f:
        f.write(json.dumps(self.encoder, indent=2, sort_keys=True, ensure_ascii=False) + "\n")

    index = 0
    with open(merge_file, "w", encoding="utf-8") as writer:
        writer.write("#version: 0.2\n")
        for bpe_tokens, token_index in sorted(self.bpe_ranks.items(), key=lambda kv: kv[1]):
            if index != token_index:
                logger.warning(
                    f"Saving vocabulary to {merge_file}: BPE merge indices are not consecutive."
                    " Please check that the tokenizer is not corrupted!"
                )
                index = token_index
            writer.write(" ".join(bpe_tokens) + "\n")
            index += 1

    return vocab_file, merge_file

mindnlp.transformers.models.deberta.tokenization_deberta.bytes_to_unicode()

Returns list of utf-8 byte and a mapping to unicode strings. We specifically avoids mapping to whitespace/control characters the bpe code barfs on.

The reversible bpe codes work on unicode strings. This means you need a large # of unicode characters in your vocab if you want to avoid UNKs. When you're at something like a 10B token dataset you end up needing around 5K for decent coverage. This is a significant percentage of your normal, say, 32K bpe vocab. To avoid that, we want lookup tables between utf-8 bytes and unicode strings.

Source code in mindnlp/transformers/models/deberta/tokenization_deberta.py

def bytes_to_unicode():
    """
    Returns list of utf-8 byte and a mapping to unicode strings. We specifically avoids mapping to whitespace/control
    characters the bpe code barfs on.

    The reversible bpe codes work on unicode strings. This means you need a large # of unicode characters in your vocab
    if you want to avoid UNKs. When you're at something like a 10B token dataset you end up needing around 5K for
    decent coverage. This is a significant percentage of your normal, say, 32K bpe vocab. To avoid that, we want lookup
    tables between utf-8 bytes and unicode strings.
    """
    bs = (
        list(range(ord("!"), ord("~") + 1)) + list(range(ord("¡"), ord("¬") + 1)) + list(range(ord("®"), ord("ÿ") + 1))
    )
    cs = bs[:]
    n = 0
    for b in range(2**8):
        if b not in bs:
            bs.append(b)
            cs.append(2**8 + n)
            n += 1
    cs = [chr(n) for n in cs]
    return dict(zip(bs, cs))

mindnlp.transformers.models.deberta.tokenization_deberta.get_pairs(word)

Return set of symbol pairs in a word.

Word is represented as tuple of symbols (symbols being variable-length strings).

Source code in mindnlp/transformers/models/deberta/tokenization_deberta.py

def get_pairs(word):
    """
    Return set of symbol pairs in a word.

    Word is represented as tuple of symbols (symbols being variable-length strings).
    """
    pairs = set()
    prev_char = word[0]
    for char in word[1:]:
        pairs.add((prev_char, char))
        prev_char = char
    return pairs

mindnlp.transformers.models.deberta.tokenization_deberta_fast

Fast Tokenization class for model DeBERTa.

mindnlp.transformers.models.deberta.tokenization_deberta_fast.DebertaTokenizerFast

Bases: PreTrainedTokenizerFast

Construct a "fast" DeBERTa tokenizer (backed by HuggingFace's tokenizers library). Based on byte-level Byte-Pair-Encoding.

This tokenizer has been trained to treat spaces like parts of the tokens (a bit like sentencepiece) so a word will be encoded differently whether it is at the beginning of the sentence (without space) or not:

Example

>>> from transformers import DebertaTokenizerFast
...
>>> tokenizer = DebertaTokenizerFast.from_pretrained("microsoft/deberta-base")
>>> tokenizer("Hello world")["input_ids"]
[1, 31414, 232, 2]
>>> tokenizer(" Hello world")["input_ids"]
[1, 20920, 232, 2]

You can get around that behavior by passing add_prefix_space=True when instantiating this tokenizer, but since the model was not pretrained this way, it might yield a decrease in performance.

When used with is_split_into_words=True, this tokenizer needs to be instantiated with add_prefix_space=True.

This tokenizer inherits from [PreTrainedTokenizerFast] which contains most of the main methods. Users should refer to this superclass for more information regarding those methods.

PARAMETER	DESCRIPTION
vocab_file	Path to the vocabulary file. TYPE: `str`, *optional* DEFAULT: None
merges_file	Path to the merges file. TYPE: `str`, *optional* DEFAULT: None
tokenizer_file	The path to a tokenizer file to use instead of the vocab file. TYPE: `str`, *optional* DEFAULT: None
errors	Paradigm to follow when decoding bytes to UTF-8. See bytes.decode for more information. TYPE: `str`, *optional*, defaults to `"replace"` DEFAULT: 'replace'
bos_token	The beginning of sequence token. TYPE: `str`, *optional*, defaults to `"[CLS]"` DEFAULT: '[CLS]'
eos_token	The end of sequence token. TYPE: `str`, *optional*, defaults to `"[SEP]"` DEFAULT: '[SEP]'
sep_token	The separator token, which is used when building a sequence from multiple sequences, e.g. two sequences for sequence classification or for a text and a question for question answering. It is also used as the last token of a sequence built with special tokens. TYPE: `str`, *optional*, defaults to `"[SEP]"` DEFAULT: '[SEP]'
cls_token	The classifier token which is used when doing sequence classification (classification of the whole sequence instead of per-token classification). It is the first token of the sequence when built with special tokens. TYPE: `str`, *optional*, defaults to `"[CLS]"` DEFAULT: '[CLS]'
unk_token	The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this token instead. TYPE: `str`, *optional*, defaults to `"[UNK]"` DEFAULT: '[UNK]'
pad_token	The token used for padding, for example when batching sequences of different lengths. TYPE: `str`, *optional*, defaults to `"[PAD]"` DEFAULT: '[PAD]'
mask_token	The token used for masking values. This is the token used when training this model with masked language modeling. This is the token which the model will try to predict. TYPE: `str`, *optional*, defaults to `"[MASK]"` DEFAULT: '[MASK]'
add_prefix_space	Whether or not to add an initial space to the input. This allows to treat the leading word just as any other word. (Deberta tokenizer detect beginning of words by the preceding space). TYPE: `bool`, *optional*, defaults to `False` DEFAULT: False

Source code in mindnlp/transformers/models/deberta/tokenization_deberta_fast.py

class DebertaTokenizerFast(PreTrainedTokenizerFast):
    """
    Construct a "fast" DeBERTa tokenizer (backed by HuggingFace's *tokenizers* library). Based on byte-level
    Byte-Pair-Encoding.

    This tokenizer has been trained to treat spaces like parts of the tokens (a bit like sentencepiece) so a word will
    be encoded differently whether it is at the beginning of the sentence (without space) or not:

    Example:
        ```python
        >>> from transformers import DebertaTokenizerFast
        ...
        >>> tokenizer = DebertaTokenizerFast.from_pretrained("microsoft/deberta-base")
        >>> tokenizer("Hello world")["input_ids"]
        [1, 31414, 232, 2]
        >>> tokenizer(" Hello world")["input_ids"]
        [1, 20920, 232, 2]
        ```

    You can get around that behavior by passing `add_prefix_space=True` when instantiating this tokenizer, but since
    the model was not pretrained this way, it might yield a decrease in performance.

    <Tip>

    When used with `is_split_into_words=True`, this tokenizer needs to be instantiated with `add_prefix_space=True`.

    </Tip>

    This tokenizer inherits from [`PreTrainedTokenizerFast`] which contains most of the main methods. Users should
    refer to this superclass for more information regarding those methods.

    Args:
        vocab_file (`str`, *optional*):
            Path to the vocabulary file.
        merges_file (`str`, *optional*):
            Path to the merges file.
        tokenizer_file (`str`, *optional*):
            The path to a tokenizer file to use instead of the vocab file.
        errors (`str`, *optional*, defaults to `"replace"`):
            Paradigm to follow when decoding bytes to UTF-8. See
            [bytes.decode](https://docs.python.org/3/library/stdtypes.html#bytes.decode) for more information.
        bos_token (`str`, *optional*, defaults to `"[CLS]"`):
            The beginning of sequence token.
        eos_token (`str`, *optional*, defaults to `"[SEP]"`):
            The end of sequence token.
        sep_token (`str`, *optional*, defaults to `"[SEP]"`):
            The separator token, which is used when building a sequence from multiple sequences, e.g. two sequences for
            sequence classification or for a text and a question for question answering. It is also used as the last
            token of a sequence built with special tokens.
        cls_token (`str`, *optional*, defaults to `"[CLS]"`):
            The classifier token which is used when doing sequence classification (classification of the whole sequence
            instead of per-token classification). It is the first token of the sequence when built with special tokens.
        unk_token (`str`, *optional*, defaults to `"[UNK]"`):
            The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this
            token instead.
        pad_token (`str`, *optional*, defaults to `"[PAD]"`):
            The token used for padding, for example when batching sequences of different lengths.
        mask_token (`str`, *optional*, defaults to `"[MASK]"`):
            The token used for masking values. This is the token used when training this model with masked language
            modeling. This is the token which the model will try to predict.
        add_prefix_space (`bool`, *optional*, defaults to `False`):
            Whether or not to add an initial space to the input. This allows to treat the leading word just as any
            other word. (Deberta tokenizer detect beginning of words by the preceding space).
    """
    vocab_files_names = VOCAB_FILES_NAMES
    pretrained_vocab_files_map = PRETRAINED_VOCAB_FILES_MAP
    max_model_input_sizes = PRETRAINED_POSITIONAL_EMBEDDINGS_SIZES
    model_input_names = ["input_ids", "attention_mask", "token_type_ids"]
    slow_tokenizer_class = DebertaTokenizer

    def __init__(
        self,
        vocab_file=None,
        merges_file=None,
        tokenizer_file=None,
        errors="replace",
        bos_token="[CLS]",
        eos_token="[SEP]",
        sep_token="[SEP]",
        cls_token="[CLS]",
        unk_token="[UNK]",
        pad_token="[PAD]",
        mask_token="[MASK]",
        add_prefix_space=False,
        **kwargs,
    ):
        """
        Initialize a DebertaTokenizerFast object.

        Args:
            self (DebertaTokenizerFast): An instance of the DebertaTokenizerFast class.
            vocab_file (str, optional): The path to the vocabulary file. Defaults to None.
            merges_file (str, optional): The path to the merges file. Defaults to None.
            tokenizer_file (str, optional): The path to the tokenizer file. Defaults to None.
            errors (str, optional): Specifies how to handle encoding and decoding errors. Defaults to 'replace'.
            bos_token (str, optional): The beginning of sentence token. Defaults to '[CLS]'.
            eos_token (str, optional): The end of sentence token. Defaults to '[SEP]'.
            sep_token (str, optional): The separator token. Defaults to '[SEP]'.
            cls_token (str, optional): The classification token. Defaults to '[CLS]'.
            unk_token (str, optional): The unknown token. Defaults to '[UNK]'.
            pad_token (str, optional): The padding token. Defaults to '[PAD]'.
            mask_token (str, optional): The mask token. Defaults to '[MASK]'.
            add_prefix_space (bool, optional): Whether to add a space before each token. Defaults to False.

        Returns:
            None.

        Raises:
            None.
        """
        super().__init__(
            vocab_file,
            merges_file,
            tokenizer_file=tokenizer_file,
            errors=errors,
            bos_token=bos_token,
            eos_token=eos_token,
            unk_token=unk_token,
            sep_token=sep_token,
            cls_token=cls_token,
            pad_token=pad_token,
            mask_token=mask_token,
            add_prefix_space=add_prefix_space,
            **kwargs,
        )
        self.add_bos_token = kwargs.pop("add_bos_token", False)

        pre_tok_state = json.loads(self.backend_tokenizer.pre_tokenizer.__getstate__())
        if pre_tok_state.get("add_prefix_space", add_prefix_space) != add_prefix_space:
            pre_tok_class = getattr(pre_tokenizers, pre_tok_state.pop("type"))
            pre_tok_state["add_prefix_space"] = add_prefix_space
            self.backend_tokenizer.pre_tokenizer = pre_tok_class(**pre_tok_state)

        self.add_prefix_space = add_prefix_space

    @property
    def mask_token(self) -> str:
        """
        Returns:
            `str`: Mask token, to use when training a model with masked-language modeling.
                Log an error if used while not having been set.

        Deberta tokenizer has a special mask token to be used in the fill-mask pipeline. The mask token will greedily
        comprise the space before the *[MASK]*.
        """
        if self._mask_token is None:
            if self.verbose:
                logger.error("Using mask_token, but it is not set yet.")
            return None
        return str(self._mask_token)

    @mask_token.setter
    def mask_token(self, value):
        """
        Overriding the default behavior of the mask token to have it eat the space before it.
        """
        # Mask token behave like a normal word, i.e. include the space before it
        # So we set lstrip to True
        value = AddedToken(value, lstrip=True, rstrip=False) if isinstance(value, str) else value
        self._mask_token = value

    def build_inputs_with_special_tokens(
        self, token_ids_0: List[int], token_ids_1: Optional[List[int]] = None
    ) -> List[int]:
        """
        Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and
        adding special tokens. A DeBERTa sequence has the following format:

        - single sequence: [CLS] X [SEP]
        - pair of sequences: [CLS] A [SEP] B [SEP]

        Args:
            token_ids_0 (`List[int]`):
                List of IDs to which the special tokens will be added.
            token_ids_1 (`List[int]`, *optional*):
                Optional second list of IDs for sequence pairs.

        Returns:
            `List[int]`: List of [input IDs](../glossary#input-ids) with the appropriate special tokens.
        """
        if token_ids_1 is None:
            return [self.cls_token_id] + token_ids_0 + [self.sep_token_id]
        cls = [self.cls_token_id]
        sep = [self.sep_token_id]
        return cls + token_ids_0 + sep + token_ids_1 + sep

    def create_token_type_ids_from_sequences(
        self, token_ids_0: List[int], token_ids_1: Optional[List[int]] = None
    ) -> List[int]:
        """
        Create a mask from the two sequences passed to be used in a sequence-pair classification task. A DeBERTa
        sequence pair mask has the following format:
        ```
        0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1
        | first sequence    | second sequence |
        ```

        If `token_ids_1` is `None`, this method only returns the first portion of the mask (0s).

        Args:
            token_ids_0 (`List[int]`):
                List of IDs.
            token_ids_1 (`List[int]`, *optional*):
                Optional second list of IDs for sequence pairs.

        Returns:
            `List[int]`: List of [token type IDs](../glossary#token-type-ids) according to the given sequence(s).
        """
        sep = [self.sep_token_id]
        cls = [self.cls_token_id]

        if token_ids_1 is None:
            return len(cls + token_ids_0 + sep) * [0]
        return len(cls + token_ids_0 + sep) * [0] + len(token_ids_1 + sep) * [1]

    # Copied from transformers.models.gpt2.tokenization_gpt2_fast.GPT2TokenizerFast._batch_encode_plus
    def _batch_encode_plus(self, *args, **kwargs) -> BatchEncoding:
        """
        Encodes a batch of inputs into their tokenized form using the DebertaTokenizerFast.

        Args:
            self: An instance of the DebertaTokenizerFast class.

        Returns:
            A BatchEncoding object that represents the tokenized inputs.

        Raises:
            AssertionError: If the 'is_split_into_words' parameter is set to True
                but the DebertaTokenizerFast instance is not instantiated with 'add_prefix_space=True'.
        """
        is_split_into_words = kwargs.get("is_split_into_words", False)
        assert self.add_prefix_space or not is_split_into_words, (
            f"You need to instantiate {self.__class__.__name__} with add_prefix_space=True "
            "to use it with pretokenized inputs."
        )

        return super()._batch_encode_plus(*args, **kwargs)

    # Copied from transformers.models.gpt2.tokenization_gpt2_fast.GPT2TokenizerFast._encode_plus
    def _encode_plus(self, *args, **kwargs) -> BatchEncoding:
        """
        Encodes the input into a batch of model inputs and returns a BatchEncoding object.

        Args:
            self (DebertaTokenizerFast): An instance of the DebertaTokenizerFast class.

        Returns:
            BatchEncoding: A BatchEncoding object containing the encoded inputs.

        Raises:
            AssertionError: If `is_split_into_words` is True and `add_prefix_space` is False,
                an AssertionError is raised with a message indicating that the DebertaTokenizerFast class
                needs to be instantiated with `add_prefix_space=True` to use it with pretokenized inputs.
        """
        is_split_into_words = kwargs.get("is_split_into_words", False)

        assert self.add_prefix_space or not is_split_into_words, (
            f"You need to instantiate {self.__class__.__name__} with add_prefix_space=True "
            "to use it with pretokenized inputs."
        )

        return super()._encode_plus(*args, **kwargs)

    # Copied from transformers.models.gpt2.tokenization_gpt2_fast.GPT2TokenizerFast.save_vocabulary
    def save_vocabulary(self, save_directory: str, filename_prefix: Optional[str] = None) -> Tuple[str]:
        """
        Save the vocabulary files of the tokenizer model to a specified directory.

        Args:
            self: Instance of the DebertaTokenizerFast class.
            save_directory (str): The directory path where the vocabulary files will be saved.
            filename_prefix (Optional[str]): An optional prefix to be added to the saved filenames. Default is None.

        Returns:
            Tuple[str]: A tuple containing the paths of the saved vocabulary files.

        Raises:
            None.
        """
        files = self._tokenizer.model.save(save_directory, name=filename_prefix)
        return tuple(files)

mindnlp.transformers.models.deberta.tokenization_deberta_fast.DebertaTokenizerFast.mask_token: str property writable

RETURNS	DESCRIPTION
str	str: Mask token, to use when training a model with masked-language modeling. Log an error if used while not having been set.

Deberta tokenizer has a special mask token to be used in the fill-mask pipeline. The mask token will greedily comprise the space before the [MASK].

mindnlp.transformers.models.deberta.tokenization_deberta_fast.DebertaTokenizerFast.__init__(vocab_file=None, merges_file=None, tokenizer_file=None, errors='replace', bos_token='[CLS]', eos_token='[SEP]', sep_token='[SEP]', cls_token='[CLS]', unk_token='[UNK]', pad_token='[PAD]', mask_token='[MASK]', add_prefix_space=False, **kwargs)

Initialize a DebertaTokenizerFast object.

PARAMETER	DESCRIPTION
self	An instance of the DebertaTokenizerFast class. TYPE: DebertaTokenizerFast
vocab_file	The path to the vocabulary file. Defaults to None. TYPE: str DEFAULT: None
merges_file	The path to the merges file. Defaults to None. TYPE: str DEFAULT: None
tokenizer_file	The path to the tokenizer file. Defaults to None. TYPE: str DEFAULT: None
errors	Specifies how to handle encoding and decoding errors. Defaults to 'replace'. TYPE: str DEFAULT: 'replace'
bos_token	The beginning of sentence token. Defaults to '[CLS]'. TYPE: str DEFAULT: '[CLS]'
eos_token	The end of sentence token. Defaults to '[SEP]'. TYPE: str DEFAULT: '[SEP]'
sep_token	The separator token. Defaults to '[SEP]'. TYPE: str DEFAULT: '[SEP]'
cls_token	The classification token. Defaults to '[CLS]'. TYPE: str DEFAULT: '[CLS]'
unk_token	The unknown token. Defaults to '[UNK]'. TYPE: str DEFAULT: '[UNK]'
pad_token	The padding token. Defaults to '[PAD]'. TYPE: str DEFAULT: '[PAD]'
mask_token	The mask token. Defaults to '[MASK]'. TYPE: str DEFAULT: '[MASK]'
add_prefix_space	Whether to add a space before each token. Defaults to False. TYPE: bool DEFAULT: False

RETURNS	DESCRIPTION
	None.

Source code in mindnlp/transformers/models/deberta/tokenization_deberta_fast.py

def __init__(
    self,
    vocab_file=None,
    merges_file=None,
    tokenizer_file=None,
    errors="replace",
    bos_token="[CLS]",
    eos_token="[SEP]",
    sep_token="[SEP]",
    cls_token="[CLS]",
    unk_token="[UNK]",
    pad_token="[PAD]",
    mask_token="[MASK]",
    add_prefix_space=False,
    **kwargs,
):
    """
    Initialize a DebertaTokenizerFast object.

    Args:
        self (DebertaTokenizerFast): An instance of the DebertaTokenizerFast class.
        vocab_file (str, optional): The path to the vocabulary file. Defaults to None.
        merges_file (str, optional): The path to the merges file. Defaults to None.
        tokenizer_file (str, optional): The path to the tokenizer file. Defaults to None.
        errors (str, optional): Specifies how to handle encoding and decoding errors. Defaults to 'replace'.
        bos_token (str, optional): The beginning of sentence token. Defaults to '[CLS]'.
        eos_token (str, optional): The end of sentence token. Defaults to '[SEP]'.
        sep_token (str, optional): The separator token. Defaults to '[SEP]'.
        cls_token (str, optional): The classification token. Defaults to '[CLS]'.
        unk_token (str, optional): The unknown token. Defaults to '[UNK]'.
        pad_token (str, optional): The padding token. Defaults to '[PAD]'.
        mask_token (str, optional): The mask token. Defaults to '[MASK]'.
        add_prefix_space (bool, optional): Whether to add a space before each token. Defaults to False.

    Returns:
        None.

    Raises:
        None.
    """
    super().__init__(
        vocab_file,
        merges_file,
        tokenizer_file=tokenizer_file,
        errors=errors,
        bos_token=bos_token,
        eos_token=eos_token,
        unk_token=unk_token,
        sep_token=sep_token,
        cls_token=cls_token,
        pad_token=pad_token,
        mask_token=mask_token,
        add_prefix_space=add_prefix_space,
        **kwargs,
    )
    self.add_bos_token = kwargs.pop("add_bos_token", False)

    pre_tok_state = json.loads(self.backend_tokenizer.pre_tokenizer.__getstate__())
    if pre_tok_state.get("add_prefix_space", add_prefix_space) != add_prefix_space:
        pre_tok_class = getattr(pre_tokenizers, pre_tok_state.pop("type"))
        pre_tok_state["add_prefix_space"] = add_prefix_space
        self.backend_tokenizer.pre_tokenizer = pre_tok_class(**pre_tok_state)

    self.add_prefix_space = add_prefix_space

mindnlp.transformers.models.deberta.tokenization_deberta_fast.DebertaTokenizerFast.build_inputs_with_special_tokens(token_ids_0, token_ids_1=None)

Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and adding special tokens. A DeBERTa sequence has the following format:

• single sequence: [CLS] X [SEP]
• pair of sequences: [CLS] A [SEP] B [SEP]

PARAMETER	DESCRIPTION
token_ids_0	List of IDs to which the special tokens will be added. TYPE: `List[int]`
token_ids_1	Optional second list of IDs for sequence pairs. TYPE: `List[int]`, *optional* DEFAULT: None

RETURNS	DESCRIPTION
List[int]	List[int]: List of input IDs with the appropriate special tokens.

Source code in mindnlp/transformers/models/deberta/tokenization_deberta_fast.py

def build_inputs_with_special_tokens(
    self, token_ids_0: List[int], token_ids_1: Optional[List[int]] = None
) -> List[int]:
    """
    Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and
    adding special tokens. A DeBERTa sequence has the following format:

    - single sequence: [CLS] X [SEP]
    - pair of sequences: [CLS] A [SEP] B [SEP]

    Args:
        token_ids_0 (`List[int]`):
            List of IDs to which the special tokens will be added.
        token_ids_1 (`List[int]`, *optional*):
            Optional second list of IDs for sequence pairs.

    Returns:
        `List[int]`: List of [input IDs](../glossary#input-ids) with the appropriate special tokens.
    """
    if token_ids_1 is None:
        return [self.cls_token_id] + token_ids_0 + [self.sep_token_id]
    cls = [self.cls_token_id]
    sep = [self.sep_token_id]
    return cls + token_ids_0 + sep + token_ids_1 + sep

mindnlp.transformers.models.deberta.tokenization_deberta_fast.DebertaTokenizerFast.create_token_type_ids_from_sequences(token_ids_0, token_ids_1=None)

Create a mask from the two sequences passed to be used in a sequence-pair classification task. A DeBERTa sequence pair mask has the following format:

0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1
| first sequence    | second sequence |

If token_ids_1 is None, this method only returns the first portion of the mask (0s).

PARAMETER	DESCRIPTION
token_ids_0	List of IDs. TYPE: `List[int]`
token_ids_1	Optional second list of IDs for sequence pairs. TYPE: `List[int]`, *optional* DEFAULT: None

RETURNS	DESCRIPTION
List[int]	List[int]: List of token type IDs according to the given sequence(s).

Source code in mindnlp/transformers/models/deberta/tokenization_deberta_fast.py

def create_token_type_ids_from_sequences(
    self, token_ids_0: List[int], token_ids_1: Optional[List[int]] = None
) -> List[int]:
    """
    Create a mask from the two sequences passed to be used in a sequence-pair classification task. A DeBERTa
    sequence pair mask has the following format:
    ```
    0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1
    | first sequence    | second sequence |
    ```

    If `token_ids_1` is `None`, this method only returns the first portion of the mask (0s).

    Args:
        token_ids_0 (`List[int]`):
            List of IDs.
        token_ids_1 (`List[int]`, *optional*):
            Optional second list of IDs for sequence pairs.

    Returns:
        `List[int]`: List of [token type IDs](../glossary#token-type-ids) according to the given sequence(s).
    """
    sep = [self.sep_token_id]
    cls = [self.cls_token_id]

    if token_ids_1 is None:
        return len(cls + token_ids_0 + sep) * [0]
    return len(cls + token_ids_0 + sep) * [0] + len(token_ids_1 + sep) * [1]

mindnlp.transformers.models.deberta.tokenization_deberta_fast.DebertaTokenizerFast.save_vocabulary(save_directory, filename_prefix=None)

Save the vocabulary files of the tokenizer model to a specified directory.

PARAMETER	DESCRIPTION
self	Instance of the DebertaTokenizerFast class.
save_directory	The directory path where the vocabulary files will be saved. TYPE: str
filename_prefix	An optional prefix to be added to the saved filenames. Default is None. TYPE: Optional[str] DEFAULT: None

RETURNS	DESCRIPTION
Tuple[str]	Tuple[str]: A tuple containing the paths of the saved vocabulary files.

Source code in mindnlp/transformers/models/deberta/tokenization_deberta_fast.py

def save_vocabulary(self, save_directory: str, filename_prefix: Optional[str] = None) -> Tuple[str]:
    """
    Save the vocabulary files of the tokenizer model to a specified directory.

    Args:
        self: Instance of the DebertaTokenizerFast class.
        save_directory (str): The directory path where the vocabulary files will be saved.
        filename_prefix (Optional[str]): An optional prefix to be added to the saved filenames. Default is None.

    Returns:
        Tuple[str]: A tuple containing the paths of the saved vocabulary files.

    Raises:
        None.
    """
    files = self._tokenizer.model.save(save_directory, name=filename_prefix)
    return tuple(files)