IA3

mindnlp.peft.tuners.ia3.config

IA3 Config

mindnlp.peft.tuners.ia3.config.IA3Config dataclass

Bases: PeftConfig

This is the configuration class to store the configuration of a [IA3Model].

PARAMETER	DESCRIPTION
target_modules	The names of the modules to apply the adapter to. If this is specified, only the modules with the specified names will be replaced. When passing a string, a regex match will be performed. When passing a list of strings, either an exact match will be performed or it is checked if the name of the module ends with any of the passed strings. If this is specified as 'all-linear', then all linear/Conv1D modules are chosen, excluding the output layer. If this is not specified, modules will be chosen according to the model architecture. If the architecture is not known, an error will be raised -- in this case, you should specify the target modules manually. TYPE: `Optional[Union[List[str], str]]` DEFAULT: None
feedforward_modules	The names of the modules to be treated as feedforward modules, as in the original paper. These modules will have (IA)³ vectors multiplied to the input, instead of the output. feedforward_modules must be a name or a subset of names present in target_modules. TYPE: `Optional[Union[List[str], str]]` DEFAULT: None
fan_in_fan_out	Set this to True if the layer to replace stores weight like (fan_in, fan_out). For example, gpt-2 uses Conv1D which stores weights like (fan_in, fan_out) and hence this should be set to True. TYPE: `bool` DEFAULT: False
modules_to_save	List of modules apart from (IA)³ layers to be set as trainable and saved in the final checkpoint. TYPE: `Optional[List[str]]` DEFAULT: None
init_ia3_weights	Whether to initialize the vectors in the (IA)³ layers, defaults to True. Setting this to False is discouraged. TYPE: `bool` DEFAULT: True

Source code in mindnlp/peft/tuners/ia3/config.py

@dataclass
class IA3Config(PeftConfig):
    """
    This is the configuration class to store the configuration of a [`IA3Model`].

    Args:
        target_modules (`Optional[Union[List[str], str]]`):
            The names of the modules to apply the adapter to. If this is specified, only the modules with the specified
            names will be replaced. When passing a string, a regex match will be performed. When passing a list of
            strings, either an exact match will be performed or it is checked if the name of the module ends with any
            of the passed strings. If this is specified as 'all-linear', then all linear/Conv1D modules are chosen,
            excluding the output layer. If this is not specified, modules will be chosen according to the model
            architecture. If the architecture is not known, an error will be raised -- in this case, you should specify
            the target modules manually.
        feedforward_modules (`Optional[Union[List[str], str]]`):
            The names of the modules to be treated as feedforward modules, as in the original paper. These modules will
            have (IA)³ vectors multiplied to the input, instead of the output. `feedforward_modules` must be a name or
            a subset of names present in `target_modules`.
        fan_in_fan_out (`bool`):
            Set this to True if the layer to replace stores weight like (fan_in, fan_out). For example, gpt-2 uses
            `Conv1D` which stores weights like (fan_in, fan_out) and hence this should be set to `True`.
        modules_to_save (`Optional[List[str]]`):
            List of modules apart from (IA)³ layers to be set as trainable and saved in the final checkpoint.
        init_ia3_weights (`bool`):
            Whether to initialize the vectors in the (IA)³ layers, defaults to `True`. Setting this to `False` is
            discouraged.
    """

    target_modules: Optional[Union[List[str], str]] = field(
        default=None,
        metadata={
            "help": (
                "List of module names or regex expression of the module names to replace with (IA)³."
                "For example, ['q', 'v'] or '.*decoder.*(SelfAttention|EncDecAttention).*(q|v)$'."
                "This can also be a wildcard 'all-linear' which matches all linear/Conv1D layers except the output layer."
                "If not specified, modules will be chosen according to the model architecture, If the architecture is "
                "not known, an error will be raised -- in this case, you should specify the target modules manually."
            ),
        },
    )
    feedforward_modules: Optional[Union[List[str], str]] = field(
        default=None,
        metadata={
            "help": "List of module names or a regex expression of module names which are feedforward"
            "For example, ['output.dense']"
        },
    )
    fan_in_fan_out: bool = field(
        default=False,
        metadata={"help": "Set this to True if the layer to replace stores weight like (fan_in, fan_out)"},
    )
    modules_to_save: Optional[List[str]] = field(
        default=None,
        metadata={
            "help": "List of modules apart from (IA)^3 layers to be set as trainable and saved in the final checkpoint. "
            "For example, in Sequence Classification or Token Classification tasks, "
            "the final layer `classifier/score` are randomly initialized and as such need to be trainable and saved."
        },
    )
    init_ia3_weights: bool = field(
        default=True,
        metadata={"help": "Whether to initialize the vectors in the (IA)^3 layers."},
    )

    def __post_init__(self):
        self.peft_type = PeftType.IA3
        self.target_modules = (
            set(self.target_modules) if isinstance(self.target_modules, list) else self.target_modules
        )
        self.feedforward_modules = (
            set(self.feedforward_modules) if isinstance(self.feedforward_modules, list) else self.feedforward_modules
        )

        # check if feedforward_modules is a subset of target_modules. run the check only if both are sets
        if isinstance(self.feedforward_modules, set) and isinstance(self.target_modules, set):
            if not self.feedforward_modules.issubset(self.target_modules):
                raise ValueError("`feedforward_modules` should be a subset of `target_modules`")

mindnlp.peft.tuners.ia3.model

IA3 Model

mindnlp.peft.tuners.ia3.model.IA3Model

Bases: BaseTuner

Creates a Infused Adapter by Inhibiting and Amplifying Inner Activations ((IA)^3) model from a pretrained transformers model. The method is described in detail in https://arxiv.org/abs/2205.05638

PARAMETER	DESCRIPTION
model	The model to be adapted. TYPE: [`~transformers.PreTrainedModel`]
config	The configuration of the (IA)^3 model. TYPE: [`IA3Config`]
adapter_name	The name of the adapter, defaults to "default". TYPE: `str`

RETURNS	DESCRIPTION
IA3Model	The IA3Lora model. TYPE: [`mindspore.nn.Cell`]

```py
>>> from transformers import AutoModelForSeq2SeqLM, ia3Config
>>> from peft import IA3Model, IA3Config

>>> config = IA3Config(
...     peft_type="IA3",
...     task_type="SEQ_2_SEQ_LM",
...     target_modules=["k", "v", "w0"],
...     feedforward_modules=["w0"],
... )

>>> model = AutoModelForSeq2SeqLM.from_pretrained("t5-base")
>>> ia3_model = IA3Model(config, model)
```

Attributes:

• model ([transformers.PreTrainedModel])— The model to be adapted.

• peft_config ([IA3Config]): The configuration of the (IA)^3 model.

Source code in mindnlp/peft/tuners/ia3/model.py

class IA3Model(BaseTuner):
    """
    Creates a Infused Adapter by Inhibiting and Amplifying Inner Activations ((IA)^3) model from a pretrained
    transformers model. The method is described in detail in https://arxiv.org/abs/2205.05638

    Args:
        model ([`~transformers.PreTrainedModel`]): The model to be adapted.
        config ([`IA3Config`]): The configuration of the (IA)^3 model.
        adapter_name (`str`): The name of the adapter, defaults to `"default"`.

    Returns:
        IA3Model ([`mindspore.nn.Cell`]): The IA3Lora model.

    Example:

        ```py
        >>> from transformers import AutoModelForSeq2SeqLM, ia3Config
        >>> from peft import IA3Model, IA3Config

        >>> config = IA3Config(
        ...     peft_type="IA3",
        ...     task_type="SEQ_2_SEQ_LM",
        ...     target_modules=["k", "v", "w0"],
        ...     feedforward_modules=["w0"],
        ... )

        >>> model = AutoModelForSeq2SeqLM.from_pretrained("t5-base")
        >>> ia3_model = IA3Model(config, model)
        ```
    > **Attributes**:  

    >   - **model** ([`transformers.PreTrainedModel`])— The model to be adapted. 

    >   - **peft_config** ([`IA3Config`]): The configuration of the (IA)^3  model. 
    """

    prefix: str = "ia3_"

    def __init__(self, model, config, adapter_name):
        super().__init__(model, config, adapter_name)

    @staticmethod
    def _create_new_module(ia3_config, adapter_name, target, **kwargs):
        # avoid eager bnb import
        # if is_bnb_available():
        #     import bitsandbytes as bnb

        #     from .bnb import Linear8bitLt

        # if is_bnb_4bit_available():
        #     from .bnb import Linear4bit

        loaded_in_8bit = kwargs.pop("loaded_in_8bit", False)
        loaded_in_4bit = kwargs.pop("loaded_in_4bit", False)
        is_feedforward = kwargs.pop("is_feedforward", False)

        if isinstance(target, BaseTunerLayer):
            target_base_layer = target.get_base_layer()
        else:
            target_base_layer = target

        # if loaded_in_8bit and isinstance(target_base_layer, bnb.nn.Linear8bitLt):
        #     eightbit_kwargs = kwargs.copy()
        #     eightbit_kwargs.update(
        #         {
        #             "has_fp16_weights": target_base_layer.state.has_fp16_weights,
        #             "memory_efficient_backward": target_base_layer.state.memory_efficient_backward,
        #             "threshold": target_base_layer.state.threshold,
        #             "index": target_base_layer.index,
        #         }
        #     )
        #     new_module = Linear8bitLt(target, adapter_name, is_feedforward=is_feedforward, **eightbit_kwargs)
        # elif loaded_in_4bit and isinstance(target_base_layer, bnb.nn.Linear4bit):
        #     fourbit_kwargs = kwargs.copy()
        #     fourbit_kwargs.update(
        #         {
        #             "compute_dtype": target_base_layer.compute_dtype,
        #             "compress_statistics": target_base_layer.weight.compress_statistics,
        #             "quant_type": target_base_layer.weight.quant_type,
        #         }
        #     )
        #     new_module = Linear4bit(target, adapter_name, is_feedforward=is_feedforward, **fourbit_kwargs)
        if isinstance(target, nn.Conv2d):
            new_module = Conv2d(target, adapter_name, is_feedforward=is_feedforward, **kwargs)
        elif isinstance(target_base_layer, nn.Dense):
            if kwargs["fan_in_fan_out"]:
                warnings.warn(
                    "fan_in_fan_out is set to True but the target module is `torch.nn.Linear`. "
                    "Setting fan_in_fan_out to False."
                )
                kwargs["fan_in_fan_out"] = ia3_config.fan_in_fan_out = False
            new_module = Linear(target, adapter_name, is_feedforward=is_feedforward, **kwargs)
        elif isinstance(target_base_layer, Conv1D):
            if not kwargs["fan_in_fan_out"]:
                warnings.warn(
                    "fan_in_fan_out is set to False but the target module is `Conv1D`. "
                    "Setting fan_in_fan_out to True."
                )
                kwargs["fan_in_fan_out"] = ia3_config.fan_in_fan_out = True
            new_module = Linear(
                target, adapter_name, is_feedforward=is_feedforward, is_target_conv_1d_layer=True, **kwargs
            )
        else:
            raise ValueError(
                f"Target module {target} is not supported. "
                f"Currently, only `torch.nn.Linear`, `torch.nn.Conv2d`, and `Conv1D` are supported."
            )
        return new_module

    @staticmethod
    def _check_target_module_exists(ia3_config, key):
        return check_target_module_exists(ia3_config, key)

    def _mark_only_adapters_as_trainable(self, model: nn.Cell) -> None:
        for name, param in model.parameters_and_names():
            if self.prefix not in name:
                param.requires_grad = False

    def _create_and_replace(
        self,
        ia3_config,
        adapter_name,
        target,
        target_name,
        parent,
        **optionnal_kwargs,
    ):
        # check if target module is in feedforward_modules
        current_key = optionnal_kwargs.pop("current_key")
        is_feedforward = self._check_target_module_feedforward(ia3_config, current_key)

        kwargs = {
            "fan_in_fan_out": ia3_config.fan_in_fan_out,
            "init_ia3_weights": ia3_config.init_ia3_weights,
            "is_feedforward": is_feedforward,
        }
        kwargs["loaded_in_8bit"] = optionnal_kwargs.pop("loaded_in_8bit", False)
        kwargs["loaded_in_4bit"] = optionnal_kwargs.pop("loaded_in_4bit", False)
        if isinstance(target, IA3Layer):
            target.update_layer(
                adapter_name,
                ia3_config.init_ia3_weights,
            )
        else:
            new_module = self._create_new_module(ia3_config, adapter_name, target, **kwargs)
            if adapter_name not in self.active_adapters:
                # adding an additional adapter: it is not automatically trainable
                new_module.requires_grad = False
            self._replace_module(parent, target_name, new_module, target)

    @staticmethod
    def _check_target_module_feedforward(ia3_config, key) -> bool:
        """
        A helper private method that checks if the target module `key` matches with a feedforward module specified in
        `ia3_config`
        """
        if isinstance(ia3_config.feedforward_modules, str):
            is_feedforward = bool(re.fullmatch(ia3_config.feedforward_modules, key))
        else:
            is_feedforward = any(key.endswith(target_key) for target_key in ia3_config.feedforward_modules)
        return is_feedforward

    def _replace_module(self, parent, child_name, new_module, child):
        setattr(parent, child_name, new_module)

        # child layer wraps the original module, unpack it
        if hasattr(child, "base_layer"):
            child = child.base_layer

        # layers with base_layer don't need the weight to be copied, as they have a reference already
        if not hasattr(new_module, "base_layer"):
            new_module.weight = child.weight
            if hasattr(child, "bias"):
                new_module.bias = child.bias

        if getattr(child, "state", None) is not None:
            if hasattr(new_module, "base_layer"):
                new_module.base_layer.state = child.state
            else:
                new_module.state = child.state


    def __getattr__(self, name: str):
        """Forward missing attributes to the wrapped module."""
        try:
            return super().__getattr__(name)  # defer to nn.Module's logic
        except AttributeError:
            return getattr(self.model, name)

    def get_peft_config_as_dict(self, inference: bool = False):
        """Get the configuration of the (IA)^3 model as a dictionary."""
        config_dict = {}
        for key, value in self.peft_config.items():
            config = {k: v.value if isinstance(v, Enum) else v for k, v in asdict(value).items()}
            if inference:
                config["inference_mode"] = True
            config_dict[key] = config
        return config

    def _set_adapter_layers(self, enabled=True):
        for module in self.model.modules():
            if isinstance(module, (IA3Layer, ModulesToSaveWrapper)):
                module.enable_adapters(enabled)

    def enable_adapter_layers(self) -> None:
        """Enable all adapters.

        Call this if you have previously disabled all adapters and want to re-enable them.
        """
        self._set_adapter_layers(enabled=True)

    def disable_adapter_layers(self) -> None:
        """Disable all adapters.

        When disabling all adapters, the model output corresponds to the output of the base model.
        """
        self._set_adapter_layers(enabled=False)

    def set_adapter(self, adapter_name: str | list[str]) -> None:
        """Set the active adapter(s).

        Additionally, this function will set the specified adapters to trainable (i.e., requires_grad=True). If this is
        not desired, use the following code.

        ```py
        >>> for name, param in model_peft.named_parameters():
        ...     if ...:  # some check on name (ex. if 'lora' in name)
        ...         param.requires_grad = False
        ```

        Args:
            adapter_name (`str` or `list[str]`): Name of the adapter(s) to be activated.
        """
        for module in self.model.modules():
            if isinstance(module, IA3Layer):
                if module.merged:
                    warnings.warn("Adapter cannot be set when the model is merged. Unmerging the model first.")
                    module.unmerge()
                module.set_adapter(adapter_name)
        self.active_adapter = adapter_name

    def _prepare_adapter_config(self, peft_config, model_config):
        if peft_config.target_modules is None:
            if model_config["model_type"] not in TRANSFORMERS_MODELS_TO_IA3_TARGET_MODULES_MAPPING:
                raise ValueError("Please specify `target_modules` in `peft_config`")
            peft_config.target_modules = TRANSFORMERS_MODELS_TO_IA3_TARGET_MODULES_MAPPING[model_config["model_type"]]
        if peft_config.feedforward_modules is None:
            if model_config["model_type"] not in TRANSFORMERS_MODELS_TO_IA3_FEEDFORWARD_MODULES_MAPPING:
                raise ValueError("Please specify `feedforward_modules` in `peft_config`")
            peft_config.feedforward_modules = TRANSFORMERS_MODELS_TO_IA3_FEEDFORWARD_MODULES_MAPPING[
                model_config["model_type"]
            ]
        return peft_config

    def _unload_and_optionally_merge(
        self, merge: bool = True, safe_merge: bool = False, adapter_names: Optional[list[str]] = None
    ):
        r"""
        This method merges the (IA)^3 layers into the base model. This is needed if someone wants to use the base model
        as a standalone model.

        Args:
            safe_merge (`bool`, `optional`, defaults to `False`):
                If True, the merge operation will be performed in a copy of the original weights and check for NaNs
                before merging the weights. This is useful if you want to check if the merge operation will produce
                NaNs. Defaults to `False`.
            adapter_names (`List[str]`, *optional*):
                The list of adapter names that should be merged. If None, all active adapters will be merged. Defaults
                to `None`.
        """
        if getattr(self.model, "is_loaded_in_8bit", False):
            raise ValueError("Cannot merge ia3 layers when the model is loaded in 8-bit mode")

        if getattr(self.model, "is_loaded_in_4bit", False):
            raise ValueError("Cannot merge ia3 layers when the model is loaded in 4-bit mode")

        self._unloading_checks(adapter_names)
        key_list = [key for key, _ in self.model.name_cells() if self.prefix not in key]
        for key in key_list:
            try:
                parent, target, target_name = _get_submodules(self.model, key)
            except AttributeError:
                continue

            if hasattr(target, "base_layer"):
                if merge:
                    target.merge(safe_merge=safe_merge, adapter_names=adapter_names)
                self._replace_module(parent, target_name, target.get_base_layer(), target)
            elif isinstance(target, ModulesToSaveWrapper):
                # save any additional trainable modules part of `modules_to_save`
                new_module = target.modules_to_save[target.active_adapter]
                if hasattr(new_module, "base_layer"):
                    # check if the module is itself a tuner layer
                    if merge:
                        new_module.merge(safe_merge=safe_merge, adapter_names=adapter_names)
                    new_module = new_module.get_base_layer()
                setattr(parent, target_name, new_module)

        return self.model

    def merge_and_unload(self, safe_merge: bool = False, adapter_names: Optional[list[str]] = None) -> nn.Cell:
        r"""
        This method merges the IA³ layers into the base model. This is needed if someone wants to use the base model as
        a standalone model.

        Args:
            safe_merge (`bool`):
                whether to activate the safe merging check to check if there is any potential Nan in the adapter
                weights
            adapter_names (`List[str]`, *optional*):
                The list of adapter names that should be merged. If None, all active adapters will be merged. Defaults
                to `None`.

        Example:

        ```py
        >>> from transformers import AutoModelForCausalLM
        >>> from peft import PeftModel

        >>> base_model = AutoModelForCausalLM.from_pretrained("tiiuae/falcon-40b")
        >>> peft_model_id = "smangrul/falcon-40B-int4-peft-lora-sfttrainer-sample"
        >>> model = PeftModel.from_pretrained(base_model, peft_model_id)
        >>> merged_model = model.merge_and_unload()
        ```
        """
        return self._unload_and_optionally_merge(safe_merge=safe_merge, adapter_names=adapter_names)

    def unload(self) -> nn.Cell:
        """
        Gets back the base model by removing all the IA³ modules without merging. This gives back the original base
        model.
        """
        return self._unload_and_optionally_merge(merge=False)

    def delete_adapter(self, adapter_name: str) -> None:
        """
        Deletes an existing adapter.

        Args:
            adapter_name (str): Name of the adapter to be deleted.
        """
        if adapter_name not in self.peft_config:
            raise ValueError(f"Adapter {adapter_name} does not exist")
        del self.peft_config[adapter_name]

        key_list = [key for key, _ in self.model.name_cells() if self.prefix not in key]
        new_adapter = None
        for key in key_list:
            _, target, _ = _get_submodules(self.model, key)
            if isinstance(target, IA3Layer):
                target.delete_adapter(adapter_name)
                if new_adapter is None:
                    new_adapter = target.active_adapters[:]

        self.active_adapter = new_adapter or []

mindnlp.peft.tuners.ia3.model.IA3Model.__getattr__(name)

Forward missing attributes to the wrapped module.

Source code in mindnlp/peft/tuners/ia3/model.py

def __getattr__(self, name: str):
    """Forward missing attributes to the wrapped module."""
    try:
        return super().__getattr__(name)  # defer to nn.Module's logic
    except AttributeError:
        return getattr(self.model, name)

mindnlp.peft.tuners.ia3.model.IA3Model.delete_adapter(adapter_name)

Deletes an existing adapter.

PARAMETER	DESCRIPTION
adapter_name	Name of the adapter to be deleted. TYPE: str

Source code in mindnlp/peft/tuners/ia3/model.py

def delete_adapter(self, adapter_name: str) -> None:
    """
    Deletes an existing adapter.

    Args:
        adapter_name (str): Name of the adapter to be deleted.
    """
    if adapter_name not in self.peft_config:
        raise ValueError(f"Adapter {adapter_name} does not exist")
    del self.peft_config[adapter_name]

    key_list = [key for key, _ in self.model.name_cells() if self.prefix not in key]
    new_adapter = None
    for key in key_list:
        _, target, _ = _get_submodules(self.model, key)
        if isinstance(target, IA3Layer):
            target.delete_adapter(adapter_name)
            if new_adapter is None:
                new_adapter = target.active_adapters[:]

    self.active_adapter = new_adapter or []

mindnlp.peft.tuners.ia3.model.IA3Model.disable_adapter_layers()

Disable all adapters.

When disabling all adapters, the model output corresponds to the output of the base model.

Source code in mindnlp/peft/tuners/ia3/model.py

def disable_adapter_layers(self) -> None:
    """Disable all adapters.

    When disabling all adapters, the model output corresponds to the output of the base model.
    """
    self._set_adapter_layers(enabled=False)

mindnlp.peft.tuners.ia3.model.IA3Model.enable_adapter_layers()

Enable all adapters.

Call this if you have previously disabled all adapters and want to re-enable them.

Source code in mindnlp/peft/tuners/ia3/model.py

def enable_adapter_layers(self) -> None:
    """Enable all adapters.

    Call this if you have previously disabled all adapters and want to re-enable them.
    """
    self._set_adapter_layers(enabled=True)

mindnlp.peft.tuners.ia3.model.IA3Model.get_peft_config_as_dict(inference=False)

Get the configuration of the (IA)^3 model as a dictionary.

Source code in mindnlp/peft/tuners/ia3/model.py

def get_peft_config_as_dict(self, inference: bool = False):
    """Get the configuration of the (IA)^3 model as a dictionary."""
    config_dict = {}
    for key, value in self.peft_config.items():
        config = {k: v.value if isinstance(v, Enum) else v for k, v in asdict(value).items()}
        if inference:
            config["inference_mode"] = True
        config_dict[key] = config
    return config

mindnlp.peft.tuners.ia3.model.IA3Model.merge_and_unload(safe_merge=False, adapter_names=None)

This method merges the IA³ layers into the base model. This is needed if someone wants to use the base model as a standalone model.

PARAMETER	DESCRIPTION
safe_merge	whether to activate the safe merging check to check if there is any potential Nan in the adapter weights TYPE: `bool` DEFAULT: False
adapter_names	The list of adapter names that should be merged. If None, all active adapters will be merged. Defaults to None. TYPE: `List[str]`, *optional* DEFAULT: None

>>> from transformers import AutoModelForCausalLM
>>> from peft import PeftModel

>>> base_model = AutoModelForCausalLM.from_pretrained("tiiuae/falcon-40b")
>>> peft_model_id = "smangrul/falcon-40B-int4-peft-lora-sfttrainer-sample"
>>> model = PeftModel.from_pretrained(base_model, peft_model_id)
>>> merged_model = model.merge_and_unload()

Source code in mindnlp/peft/tuners/ia3/model.py

def merge_and_unload(self, safe_merge: bool = False, adapter_names: Optional[list[str]] = None) -> nn.Cell:
    r"""
    This method merges the IA³ layers into the base model. This is needed if someone wants to use the base model as
    a standalone model.

    Args:
        safe_merge (`bool`):
            whether to activate the safe merging check to check if there is any potential Nan in the adapter
            weights
        adapter_names (`List[str]`, *optional*):
            The list of adapter names that should be merged. If None, all active adapters will be merged. Defaults
            to `None`.

    Example:

    ```py
    >>> from transformers import AutoModelForCausalLM
    >>> from peft import PeftModel

    >>> base_model = AutoModelForCausalLM.from_pretrained("tiiuae/falcon-40b")
    >>> peft_model_id = "smangrul/falcon-40B-int4-peft-lora-sfttrainer-sample"
    >>> model = PeftModel.from_pretrained(base_model, peft_model_id)
    >>> merged_model = model.merge_and_unload()
    ```
    """
    return self._unload_and_optionally_merge(safe_merge=safe_merge, adapter_names=adapter_names)

mindnlp.peft.tuners.ia3.model.IA3Model.set_adapter(adapter_name)

Set the active adapter(s).

Additionally, this function will set the specified adapters to trainable (i.e., requires_grad=True). If this is not desired, use the following code.

>>> for name, param in model_peft.named_parameters():
...     if ...:  # some check on name (ex. if 'lora' in name)
...         param.requires_grad = False

PARAMETER	DESCRIPTION
adapter_name	Name of the adapter(s) to be activated. TYPE: `str` or `list[str]`

Source code in mindnlp/peft/tuners/ia3/model.py

def set_adapter(self, adapter_name: str | list[str]) -> None:
    """Set the active adapter(s).

    Additionally, this function will set the specified adapters to trainable (i.e., requires_grad=True). If this is
    not desired, use the following code.

    ```py
    >>> for name, param in model_peft.named_parameters():
    ...     if ...:  # some check on name (ex. if 'lora' in name)
    ...         param.requires_grad = False
    ```

    Args:
        adapter_name (`str` or `list[str]`): Name of the adapter(s) to be activated.
    """
    for module in self.model.modules():
        if isinstance(module, IA3Layer):
            if module.merged:
                warnings.warn("Adapter cannot be set when the model is merged. Unmerging the model first.")
                module.unmerge()
            module.set_adapter(adapter_name)
    self.active_adapter = adapter_name

mindnlp.peft.tuners.ia3.model.IA3Model.unload()

Gets back the base model by removing all the IA³ modules without merging. This gives back the original base model.

Source code in mindnlp/peft/tuners/ia3/model.py

def unload(self) -> nn.Cell:
    """
    Gets back the base model by removing all the IA³ modules without merging. This gives back the original base
    model.
    """
    return self._unload_and_optionally_merge(merge=False)