LLM model binary files irrespective of the format GGUF, SafeTensors, Onnx, Pytorch etc, the
file only contains raw weights, metadata, and tokenizers, not the code required to execute mathematical operations.
You need inference engines like llama.cpp, which will map these weights to CPU/GPU etc.
llamafile combines these engine and gguf file into a single binary so we can run it.
Some of the other popular options to run models locally include ollama, mlx_lm, LM Studio, vLLM etc.
We may run the interactive chat window that often these engines provide to chat or for generative tasks.
But more often we need to interact with these models programatically thru API's so we can make applications utilize the LLM capabilities.
Examine the following code from langchain to interact with openai based models
from langchain_openai import ChatOpenAI
|
model = ChatOpenAI(
model="...",
temperature=0,
max_tokens=None,
timeout=None,
max_retries=2,
# api_key="...",
# base_url="...",
# organization="...",
# other params...
)
Or a native OpenAI, few liner to chat with a local model and get an answer.
from openai import OpenAI
client = OpenAI(base_url="http://HOST:PORT/v1", api_key="None")
response = client.chat.completions.create(
model="...",
messages=[
{"role": "user", "content": QUESTION}
]
)
print(response.choices[0].message.content)
Please go through the following hyper-parameters, that can be set externally, that can control the behaviour of the LLM.
model / model_name: Name of the OpenAI model to use (e.g., gpt-4o).
temperature: Sampling randomness controlling creativity (float).
max_tokens: Maximum number of tokens to generate in the completion.
max_completion_tokens: Maximum upper bound for reasoning and output tokens for newer reasoning models.
seed: Integer random seed for deterministic sampling.
frequency_penalty: Penalizes repeated tokens based on their frequency.
presence_penalty: Penalizes repeated tokens to encourage new topics.
logprobs: Boolean whether to return log probabilities of output tokens.
top_logprobs: Number of most likely tokens to return log probabilities for at each position.