Playbook For Speed & Scale
In the field of generative AI, Retrieval-Augmented Generation (RAG) models overcome the limitation of solely relying on data and patterns that are used to train the models; RAG can retrieve and augment relevant information from the institutional data. For developers and technical architects, the challenge is to access a large corpus of data from institutional data sources. This exercise requires breaking down the input text into manageable pieces, a process known as chunking or text splitting. In this blog post, we will explore different strategies for text chunking and splitting. We’ll provide practical examples and coding tips to help improve the performance of RAG models, focusing on effective data retrieval, processing efficiency, and the relevance and quality of the generated responses.
Chunking strategies improve information retrieval, making sure that each query is both complete and targeted. By splitting text effectively, RAG models can generate a more accurate response that is contextually appropriate. A solution that is designed to have efficient chunking could reduce compute, support better performance, and improve efficiency.To get the best performance from Retrieval-Augmented Generation (RAG) models, it’s important to chunk the text effectively. Next, we’ll look at some strategies to chunk the text that help these models produce more accurate results.
Fixed-length chunking is a simple method of text segmentation where text is divided into equal-sized chunks based on a predetermined number of characters or words. This method is useful for processing large volumes of text where you can create uniform blocks of text. The limitation of this method is that there is no semantic integrity of the text. Fixed-length chunking is a beneficial tool in scenarios where the chunk size is more important than the context.
def split_text_into_fixed_chunks(input_text, chunk_size):return [input_text[i:i + chunk_size] for i in range(0, len(input_text), chunk_size)]
Sentence-aware chunking method segments text into chunks based on sentence boundaries and ensures adherence to the maximum length for each chunk specified during the chunking process. This approach prioritizes that each segment has a full sentence, and the grammatical integrity and contextual flow of the original text are maintained. This technique is useful for texts where the full understanding of each sentence is important.
import nltk# install nltk using “pip install --user -U nltk”nltk.download('punkt')# Punkt Tokenizer Modelsfrom nltk.tokenize import sent_tokenizedef sentence_aware_chunking(input_text, max_len):sentences = sent_tokenize(input_text)chunks = []current_chunk = []current_len = 0for sentence in sentences:if current_length + len(sentence) > max_len:chunks.append(' '.join(current_chunk))current_chunk = [sentence]current_len = len(sentence)else:current_chunk.append(sentence)current_len += len(sentence)if current_chunk:chunks.append(' '.join(current_chunk))return chunks
Semantic chunking leverages advanced natural language processing (NLP) tools to segment text based on semantic boundaries using the meaning and context of the chunk the text. It identifies shifts in topics or themes, ensuring that each chunk represents a coherent idea or narrative thread.
from spacy.lang.en import Englishnlp = English()nlp.add_pipe('sentencizer')def semantic_chunking(input_text, threshold_value):doc = nlp(input_text)chunks = []current_chunk = []for sent in doc.sents:current_chunk.append(sent.input_text)if len(' '.join(current_chunk)) > threshold_value:chunks.append(' '.join(current_chunk))current_chunk = []if current_chunk:chunks.append(' '.join(current_chunk))return chunks
Sliding window chunking is an text processing technique where text is divided into overlapping chunks using a predefined window size and a specific step size. This approach ensures that the end of one chunk overlaps with the start of the next, preserving the continuity of information across chunk boundaries. It maintains the contextual integrity, ensuring that subsequent chunks can leverage context from previous chunk.
def sliding_window_chunking(input_text, chunk_size, step_size):return [input_text[i:i + chunk_size] for i in range(0, len(input_text) - chunk_size + step_size, step_size)]
Adaptive chunking is an advanced text splitting strategy that dynamically adjusts chunk sizes based on the complexity and the content of the text segments. This technique employs advanced natural language processing algorithms to analyze the structure and semantics of the text, identifying logical endpoint and themes where chunks should begin and end. This technique maximizes the relevancy and completeness of the information within each chunk, providing better retrieval and generation in RAG models.
import spacynlp = spacy.load("en_core_web_sm")def adaptive_chunking(text, min_length, max_length):doc = nlp(text)chunks = []current_chunk = []current_length = 0for sentence in doc.sents:sentence_length = len(sentence.text)if current_length + sentence_length > max_length or sentence_length > min_length:if current_chunk:chunks.append(' '.join(current_chunk))current_chunk = [sentence.text]current_length = sentence_lengthelse:# Handle sentence longer than max_lengthchunks.append(sentence.text)current_length = 0else:current_chunk.append(sentence.text)current_length += sentence_lengthif current_chunk:chunks.append(' '.join(current_chunk))return chunks
Effective chunking and text-splitting strategies can significantly enhance the performance of RAG models. By selecting an appropriate method based on the specific needs of your text and application, you can optimize the retrieval process and improve the overall quality of content generation.
