Bitnet.cpp from Microsoft with NLP and RAG Features

Information

Kernels use in LLM model

• I2_S for x86, x86_64 cpus.
• TL1
• TL2

Model that is supported to use in this branch

• Bitnet-b1.58-2B-4T
• bitnet_b1_58-large
• Bitnet-b1.58-3B
• Llama3-8B-1.58-100B-tokens
• Falcon3 Family
• Falcon-E Family

Installation

Requirements

• python>=3.9
• cmake>=3.22
• clang>=18
• conda (highly recommend) e.g., miniforge, miniconda.

Operating System Instruction for Fulling Requirments

For Windows users, its best use wsl for this thing and install linux distro like.

For Debian/Ubuntu or LTS Linux distro users

• Packages respostory like llvm

    `bash -c "$(wget -O - https://apt.llvm.org/llvm.sh)"`
  

Build from source

1. Clone the repo

git clone --recursive https://github.com/microsoft/BitNet.git
cd BitNet

2. Install the dependencies

# (Recommended) Create a new conda environment
conda create -n bitnet-cpp python=3.9
conda activate bitnet-cpp

pip install -r requirements.txt

3. Build the project

# Manually download the model and run with local path
huggingface-cli download microsoft/BitNet-b1.58-2B-4T-gguf --local-dir models/BitNet-b1.58-2B-4T
python setup_env.py -md models/BitNet-b1.58-2B-4T -q i2_s

## Usage
Simple, run run.py 

### Customize
Simple edit run.py