{"id":606560,"date":"2025-07-11T12:56:46","date_gmt":"2025-07-11T17:56:46","guid":{"rendered":"https:\/\/towardsdatascience.com\/?p=606560"},"modified":"2025-07-11T12:57:06","modified_gmt":"2025-07-11T17:57:06","slug":"hitchhikers-guide-to-rag-from-tiny-files-to-tolstoy-with-openais-api-and-langchain","status":"publish","type":"post","link":"https:\/\/towardsdatascience.com\/hitchhikers-guide-to-rag-from-tiny-files-to-tolstoy-with-openais-api-and-langchain\/","title":{"rendered":"Hitchhiker\u2019s Guide to RAG: From Tiny Files to Tolstoy with OpenAI\u2019s API and LangChain"},"content":{"rendered":"\n

In my latest post<\/mdspan>, I walked you through setting up a very simple RAG pipeline in Python<\/a>, using OpenAI’s API, LangChain, and your local files. In that post, I cover the very basics of creating embeddings from your local files with LangChain, storing them in a vector database with FAISS, making API calls to OpenAI’s API, and ultimately generating responses relevant to your files. \ud83c\udf1f<\/p>\n\n\n\n

\"\"
Image by author<\/figcaption><\/figure>\n\n\n\n

Nonetheless, in this simple example, I only demonstrate how to use a tiny .txt file. In this post, I further elaborate on how you can utilize larger files with your RAG pipeline by adding an extra step to the process \u2014 chunking<\/strong><\/em>.<\/p>\n\n\n\n

What about chunking?<\/h2>\n\n\n\n

Chunking refers to the process of parsing a text into smaller pieces of text\u2014chunks\u2014that are then transformed into embeddings. This is very important because it allows us to effectively process and create embeddings for larger files. All embedding models come with various limitations on the size of the text that is passed \u2014 I’ll get into more details about those limitations in a moment. These limitations allow for better performance and low-latency responses. In the case that the text we provide doesn’t meet those size limitations, it\u2019ll get truncated or rejected.<\/p>\n\n\n\n

If we wanted to create a RAG pipeline reading, say from Leo Tolstoy’s War and Peace<\/a><\/em> text (a rather large book), we wouldn’t be able to directly load it and transform it into a single embedding. Instead, we need to first do the chunking <\/em>\u2014 create smaller chunks of text, and create embeddings for each one. Each chunk being below the size limits of whatever embedding model we use allows us to effectively transform any file into embeddings. So, a somewhat more<\/em> realistic landscape of a RAG pipeline would look as follows:<\/p>\n\n\n\n

\"\"
Image by author<\/figcaption><\/figure>\n\n\n\n

There are several parameters to further customize the chunking process and fit it to our specific needs. A key parameter of the chunking process is the chunk size<\/em>, which allows us to specify what the size of each chunk will be (in characters or in tokens). The trick here is that the chunks we create have to be small enough to be processed within the size limitations of the embedding, but at the same time, they should also be large enough to incorporate meaningful information.<\/p>\n\n\n\n

For instance, let’s assume we want to process the following sentence from War and Peace<\/em>, where Prince Andrew contemplates the battle:<\/p>\n\n\n\n

\"\"
Image by author<\/figcaption><\/figure>\n\n\n\n

Let’s also assume we created the following (rather small) chunks :<\/p>\n\n\n\n

\"\"
image by author<\/figcaption><\/figure>\n\n\n\n

Then, if we were to ask something like “What does Prince Andrew mean by ‘all the same now’?”,<\/em> we may not get a good answer because the chunk \u201cBut isn\u2019t it all the same now?\u201d thought he. <\/mark><\/em> does not contain any context and is vague. In contrast, the meaning is scattered across multiple chunks. Thus, even though it is similar to the question we ask and may be retrieved, it does not contain any meaning to produce a relevant response. Therefore, selecting the appropriate chunk size for the chunking process in line with the type of documents we use for the RAG, can largely influence the quality of the responses we’ll be getting. In general, the content of a chunk should make sense for a human reading it without any other information, in order to also be able to make sense for the model. Ultimately, a trade-off for the chunk size exists \u2014 chunks need to be small enough to meet the embedding model’s size limitations, but large enough to preserve meaning.<\/p>\n\n\n\n

\u2022 \u2022 \u2022<\/p>\n\n\n\n

Another significant parameter is the chunk overlap. That is how much overlap we want the chunks to have with one another. For instance, in the War and Peace<\/em> example, we would get something like the following chunks if we chose a chunk overlap of 5 characters.<\/p>\n\n\n\n

\"\"
Image by author<\/figcaption><\/figure>\n\n\n\n

This is also a very important decision we have to make because:<\/p>\n\n\n\n