![]() |
|
![]() |
|
![]() |
|
![]() |
|
![]() |
|
![]() |
|
![]() |
|
![]() |
|
![]() |
|
![]() |
|
![]() |
|
![]() |
|
![]() |
|
![]() |
|
![]() |
|
Nachrichtenartikel zu Kryptowährungen
llama.cpp: Writing A Simple C++ Inference Program for GGUF LLM Models
Jan 14, 2025 at 03:04 am
This tutorial will guide you through the process of building a simple C++ program that performs inference on GGUF LLM models using the llama.cpp framework. We will cover the essential steps involved in loading the model, performing inference, and displaying the results. The code for this tutorial can be found here.
Prerequisites
To follow along with this tutorial, you will need the following:
A Linux-based operating system (native or WSL)
CMake installed
GNU/clang toolchain installed
Step 1: Setting Up the Project
Let's start by setting up our project. We will be building a C/C++ program that uses llama.cpp to perform inference on GGUF LLM models.
Create a new project directory, let's call it smol_chat.
Within the project directory, let's clone the llama.cpp repository into a subdirectory called externals. This will give us access to the llama.cpp source code and headers.
mkdir -p externals
cd externals
git clone https://github.com/georgigerganov/llama.cpp.git
cd ..
Step 2: Configuring CMake
Now, let's configure our project to use CMake. This will allow us to easily compile and link our C/C++ code with the llama.cpp library.
Create a CMakeLists.txt file in the project directory.
In the CMakeLists.txt file, add the following code:
cmake_minimum_required(VERSION 3.10)
project(smol_chat)
set(CMAKE_CXX_STANDARD 20)
set(CMAKE_CXX_STANDARD_REQUIRED ON)
add_executable(smol_chat main.cpp)
target_include_directories(smol_chat PUBLIC ${CMAKE_CURRENT_SOURCE_DIR})
target_link_libraries(smol_chat llama.cpp)
This code specifies the minimum CMake version, sets the C++ standard and standard flag, adds an executable named smol_chat, includes headers from the current source directory, and links the llama.cpp shared library to our executable.
Step 3: Defining the LLM Interface
Next, let's define a C++ class that will handle the high-level interactions with the LLM. This class will abstract away the low-level llama.cpp function calls and provide a convenient interface for performing inference.
In the project directory, create a header file called LLMInference.h.
In LLMInference.h, declare the following class:
class LLMInference {
public:
LLMInference(const std::string& model_path);
~LLMInference();
void startCompletion(const std::string& query);
std::string completeNext();
private:
llama_model llama_model_;
llama_context llama_context_;
llama_sampler llama_sampler_;
std::vector
std::vector
std::vector
llama_batch batch_;
};
This class has a public constructor that takes the path to the GGUF LLM model as an argument and a destructor that deallocates any dynamically-allocated objects. It also has two public member functions: startCompletion, which initiates the completion process for a given query, and completeNext, which fetches the next token in the LLM's response sequence.
Step 4: Implementing LLM Inference Functions
Now, let's define the implementation for the LLMInference class in a file called LLMInference.cpp.
In LLMInference.cpp, include the necessary headers and implement the class methods as follows:
#include "LLMInference.h"
#include "common.h"
#include
#include
#include
LLMInference::LLMInference(const std::string& model_path) {
llama_load_model_from_file(&llama_model_, model_path.c_str(), llama_model_default_params());
llama_new_context_with_model(&llama_context_, &llama_model_);
llama_sampler_init_temp(&llama_sampler_, 0.8f);
llama_sampler_init_min_p(&llama_sampler_, 0.0f);
}
LLMInference::~LLMInference() {
for (auto& msg : _messages) {
std::free(msg.content);
}
llama_free_model(&llama_model_);
llama_free_context(&llama_context_);
}
void LLMInference::startCompletion(const std::string& query)
Haftungsausschluss:info@kdj.com
Die bereitgestellten Informationen stellen keine Handelsberatung dar. kdj.com übernimmt keine Verantwortung für Investitionen, die auf der Grundlage der in diesem Artikel bereitgestellten Informationen getätigt werden. Kryptowährungen sind sehr volatil und es wird dringend empfohlen, nach gründlicher Recherche mit Vorsicht zu investieren!
Wenn Sie glauben, dass der auf dieser Website verwendete Inhalt Ihr Urheberrecht verletzt, kontaktieren Sie uns bitte umgehend (info@kdj.com) und wir werden ihn umgehend löschen.
-
-
-
- Lightchain AI (LCAI)-Ein potenzieller langfristiger Herausforderer für BNB, während sich der Kryptowährungsmarkt weiterentwickelt
- Feb 23, 2025 at 07:10 pm
- Die Lightchain AI (LCAI) gewinnt schnell als aufstrebender Konkurrent für etablierte Token wie BNB, während sich der Kryptowährungsmarkt entwickelt.
-
-
-
- Der Bitcoin -Entwickler Peter Todd Pausen erwähnt XRP Token auf der Bergbaukonferenz
- Feb 23, 2025 at 07:10 pm
- Der kanadische Bitcoin -Entwickler Peter Todd, der im vergangenen Jahr von HBO als Top -Kandidat von Satoshi Nakamoto identifiziert wurde, hat kürzlich einige Augenbrauen in den sozialen Medien gezogen, nachdem er das umstrittene XRP -Token erwähnt hatte.
-
-
-