next gen rag

Next Generation Retrieval Augmented Generation (RAG)

Introduction

This is a collaborative hub dedicated to advancing the next generation of Retrieval Augmented Generation (RAG) technology. This repository is focused on the dynamic exploration of the evolving landscape of RAG, aiming to uncover, develop, and share the cutting-edge advancements that shape the future of AI-driven information retrieval and generation.

Goals

This repository embarks on a dynamic exploration of the evolving landscape of RAG, aiming to uncover, develop, and share the cutting-edge advancements that shape the future of AI-driven information retrieval and generation.

Goals

The primary goals of this repository are:

• Innovate and Explore: To push the boundaries of current RAG technologies by exploring new concepts, methodologies, and architectures.
• Collaborative Research: To foster a community of researchers, developers, and enthusiasts working together to advance the field of RAG.
• Knowledge Sharing: To serve as a comprehensive resource for state-of-the-art developments, conceptual advancements, and insightful discussions on RAG and its applications.
• Real-world Impact: To translate theoretical advancements into practical solutions that address real-world challenges in information retrieval and text generation.

Topics of Interest

This repository focuses on a wide array of topics within the realm of RAG and related areas, including but not limited to:

• Semantic Routing
• Knowledge Integration
• Advanced Query Processing
• Ranking
  • Strategies for Effective and Efficient Text Ranking Using Large Language Models
  • Training an LLM Re-ranker using Direct Preference Optimization
  • Lost in the Middle: How Language Models Use Long Contexts
    1. Flash Rank
    2. Long Context Retriever
• Security:
  • (todo: add exploration thread)ComPromptMized: Unleashing Zero-click Worms that Target GenAI-Powered Applications

Repository Structure

• notebooks/: Jupyter notebooks containing research, experiments, and demonstrations. Each notebook is self-contained with instructions and explanations.
• docs/: Additional documentation on concepts, techniques, and findings.
• data/: Sample datasets used for experimentation (todo: add datasets).
• scripts/: Utility scripts and code snippets (todo: add scripts).

In Progress

• Best in class ReRanker? Boost Your Search With The Crispy mixedbread Rerank Models

• Matryoshka Embedding Model

• [qdrant / fastembed] (https://github.com/qdrant/fastembed)

Backlog

• CohereForAI/c4ai-command-r-v01 (Cohere's 35 billion parameter model)

Roadmap

• LLM Re-ranker Using Human Feedback
• Security: Gen-AI Zero-click Worms
• Less is More: Specially in the context of preparing a Function Calling Dataset https://arxiv.org/abs/2305.11206