instinct.cpp

instinct.cpp

instinct.cpp is a toolkit for developing LLM-powered applications.

This project is under active development and has not reached to GA stage of first major release. See more at Roadmap section.

Features

What instinct.cpp offer:

• Applications that are working out-of-box.
  • Assistant API server: Agent service that is fully compatible with OpenAI's Assistant API.
    • mini-assistant-api: Single binary for single node deployment with vector database and other dependencies bundled.
    • mighty-assistant-api: (WIP) A cloud native implementation that is highly scalable with distributed components and multi-tenant support.
  • chat-agent: A CLI application that create knowledge index with your docs (PDF,TXT,MD,...) and launch an HTTP server that is fully compatible with OpenAI ChatCompletion.
• Frameworks to build LLM-based applications. Say it langchain.cpp.
  • Integration for privacy-first LLM providers: Built-in support for Ollama and other OpenAI compatible API services like vllm, llama.cpp server, nitro and more.
  • Building blocks for common application patterns like Chatbot, RAG, LLM Agent.
  • Functional chaining components for composable LLM pipelines.
  • Agent patterns: ReACT, OpenAI-based tool agent, LLMCompiler, ...

User Guides

For built-in applications:

• mini-assistant-api
• doc-agent

For library itself:

• instinct.cpp library

Roadmap

Complete project plan is tracked at Project kanban.

Milestone	Features	DDL
v0.1.0	Long-short memory, PDF/TXT/DOCX ingestor, Chain programing paradigm, RAG reference app doc-agent	3.29
v0.1.1	Performance tuning, RAG evaluation, Function calling agent	4.16
v0.1.2	OpenAI Assistant API initial implementation, single-binary reference app mini-assistant	4.30
v0.1.3	* mini-assistant: tool calls with opensourced LLMs	5.17
v0.1.4	* doc-agent : rerank model * mini-assistant: file-search tool support.	6.18 6.14
v0.1.5	Overall optimization	6.30
v0.1.6	code-interpreter in mini-assistant	7.15

Contributions are welcomed! You can join discord server, or contact me via email.

Acknowledgements

This project could not be possible without following awesome projects.

• bshoshany-thread-pool
• base64
• chatllm.cpp
• concurrentqueue
• cpptrace
• corssguid
• cpp-httplib
• duckx
• DuckDB
• exprtk
• fmt
• fmtlog
• hash_library
• icu
• inja
• libcurl
• llama.cpp
• nlohmann_json
• protobuf
• pdfium
• reactiveplusplus
• tsl-ordered-map
• uniparser

And many thanks to the shared training checkpoints from:

• https://huggingface.co/BAAI/bge-m3
• https://huggingface.co/BAAI/bge-reranker-v2-m3

Lists are sorted alphabetically.