Skip to main content

​
Table of Contents

​
Importing a fine tuned adapter from Safetensors weights

First, create a Modelfile with a FROM command pointing at the base model you used for fine tuning, and an ADAPTER command which points to the directory with your Safetensors adapter:
Make sure that you use the same base model in the FROM command as you used to create the adapter otherwise you will get erratic results. Most frameworks use different quantization methods, so it’s best to use non-quantized (i.e. non-QLoRA) adapters. If your adapter is in the same directory as your Modelfile, use ADAPTER . to specify the adapter path. Now run ollama create from the directory where the Modelfile was created:
Lastly, test the model:
Ollama supports importing adapters based on several different model architectures including:
  • Llama (including Llama 2, Llama 3, Llama 3.1, and Llama 3.2);
  • Mistral (including Mistral 1, Mistral 2, and Mixtral); and
  • Gemma (including Gemma 1 and Gemma 2)
You can create the adapter using a fine tuning framework or tool which can output adapters in the Safetensors format, such as:

​
Importing a model from Safetensors weights

First, create a Modelfile with a FROM command which points to the directory containing your Safetensors weights:
If you create the Modelfile in the same directory as the weights, you can use the command FROM .. Now run the ollama create command from the directory where you created the Modelfile:
Lastly, test the model:
Ollama supports importing models for several different architectures including:
  • Llama (including Llama 2, Llama 3, Llama 3.1, and Llama 3.2);
  • Mistral (including Mistral 1, Mistral 2, and Mixtral);
  • Gemma (including Gemma 1 and Gemma 2); and
  • Phi3
This includes importing foundation models as well as any fine tuned models which have been fused with a foundation model.

​
Importing a GGUF based model or adapter

If you have a GGUF based model or adapter it is possible to import it into Ollama. You can obtain a GGUF model or adapter by:
  • converting a Safetensors model with the convert_hf_to_gguf.py from Llama.cpp;
  • converting a Safetensors adapter with the convert_lora_to_gguf.py from Llama.cpp; or
  • downloading a model or adapter from a place such as HuggingFace
To import a GGUF model, create a Modelfile containing:
For a GGUF adapter, create the Modelfile with:
When importing a GGUF adapter, it’s important to use the same base model as the base model that the adapter was created with. You can use:
  • a model from Ollama
  • a GGUF file
  • a Safetensors based model
Once you have created your Modelfile, use the ollama create command to build the model.

​
Quantizing a Model

Quantizing a model allows you to run models faster and with less memory consumption but at reduced accuracy. This allows you to run a model on more modest hardware. Ollama can quantize FP16 and FP32 based models into different quantization levels using the -q/--quantize flag with the ollama create command. First, create a Modelfile with the FP16 or FP32 based model you wish to quantize.
Use ollama create to then create the quantized model.

​
Supported Quantizations

  • q8_0

​
K-means Quantizations

  • q4_K_S
  • q4_K_M

​
Sharing your model on ollama.com

You can share any model you have created by pushing it to ollama.com so that other users can try it out. First, use your browser to go to the Ollama Sign-Up page. If you already have an account, you can skip this step. Sign-Up The Username field will be used as part of your model’s name (e.g. jmorganca/mymodel), so make sure you are comfortable with the username that you have selected. Now that you have created an account and are signed-in, go to the Ollama Keys Settings page. Follow the directions on the page to determine where your Ollama Public Key is located. Ollama Keys Click on the Add Ollama Public Key button, and copy and paste the contents of your Ollama Public Key into the text field. To push a model to ollama.com, first make sure that it is named correctly with your username. You may have to use the ollama cp command to copy your model to give it the correct name. Once you’re happy with your model’s name, use the ollama push command to push it to ollama.com.
Once your model has been pushed, other users can pull and run it by using the command: