Attention !

CorDA has been integrated into huggingface/peft.

The code in this repo (iboing/CorDA) only supports LLaMA-2, and you may write mapping files in CorDA/mapping when using other models. So, it is suggested to implement our method by huggingface/peft.

You can find the example to use CorDA in huggingface/peft/examples/corda_finetuning.

Getting Start

Download the repo and install dependencies.

git clone https://github.com/iboing/CorDA.git
cd CorDA
pip install -r requirements.txt

The dataset in json/jsonl formats used to collect covariance matrices (MetaMath for math, CodeFeedback for code, WizardLM_evol_instruct and alpaca for instruction following) can be downloaded from our huggingface repo. The other datasets will be automatically downloaded when runing the code.

Notice: The process of collecting covariance matrices is performed in torch.float32 by default, which consumes about 43G GPU memory for LLaMA-2-7b. You can comment Line 123 of cordalib/act_aware_utils.py to collect covariance matrices in torch.float16 and reduce the memory cost to about 30G. But this may cause numerical instability in a few cases, such that the initialized model after our decomposition and reconstruction (