[backend] llama-cpp, C++ gRPC backend

[mudler]

Is your feature request related to a problem? Please describe.
keeping in sync the golang binding can lag behind for certain reasons related to the golang toolchain. It'd be easier to also have a c++ implementation that requires less maitenance and so can be bumped almost automatically

Describe the solution you'd like
At this point we can also have a pure C++ llama-cpp gRPC server binding with just the few little things we need to add on top

Describe alternatives you've considered

Additional context
This would allow us to open up bugs upstream more easily as we have a less heavy implementation on top. We could also upstream the gRPC server

[Aisuko]

Yes, this is a good solution. Sometimes is so hard to follow the log from the golang binding packages.

[mudler]

JFYI I'm currently playing with this

[Aisuko]

Sounds really good. And I believe that the C++ backend can be an example for other languages, like: Rust. I am planing on how to implement the Rust backend. Can I say the basic requirements of our backend are:

• Same proto file to keep same behaviors of API
• async should be as default (Streaming response should be better)
• Only server side gRPC services

Is that right?

[mudler]

Sounds really good. And I believe that the C++ backend can be an example for other languages, like: Rust. I am planing on how to implement the Rust backend. Can I say the basic requirements of our backend are:

* Same `proto` file to keep same behaviors of API

* `async` should be as default (Streaming response should be better)

* Only server side gRPC services

Is that right?

looks reasonable to me! I think also the golang gRPC services are good examples as well, probably C++ is going to be a bit more verbose-ish

[localai-bot]

⚠️⚠️⚠️⚠️⚠️

Hi! I'm a bot running with LocalAI ( a crazy experiment of @mudler ) - please beware that I might hallucinate sometimes!

but.... I can also be funny or helpful 😸 and I can provide generally speaking good tips or places where to look after in the documentation or in the code based on what you wrote in the issue.

Don't engage in conversation with me,
I don't support (yet) replying!

⚠️⚠️⚠️⚠️⚠️

I understand that you are having an issue with the LocalAI project and would like to propose a solution to improve its golang binding. You mentioned that the current implementation can lag behind due to the maintenance required for the toolchain, and that a C++ implementation could be a viable solution.
To implement this solution, we can create a pure C++ llama-cpp gRPC server binding with just the necessary features needed for LocalAI. This would require less maintenance and could be bumped automatically without worrying about compatibility issues.
We can also consider upstreaming the gRPC server to allow for easier bug reporting and collaboration with the community.
As for alternatives, we could explore other languages or frameworks that may offer better performance or ease of use, but C++ seems like a good fit for this specific use case due to its low overhead and high performance.
I hope this helps! Let me know if you have any further questions or concerns.

Sources:

• https://github.com/go-skynet/LocalAI/tree/master/pkg/model/loader.go
• https://github.com/go-skynet/LocalAI/tree/master/pkg/grpc/client.go
• https://github.com/go-skynet/LocalAI/tree/master/pkg/grpc/proto/backend_grpc.pb.go
• https://github.com/go-skynet/LocalAI/tree/master/go.mod