Project Link: https://dev.azure.com/clarity-ventures/AI-Projects/_git/AI-DataFormatter
¶ Introduction
At Clarity Ventures, we understand the importance of clean, consistent, and well-structured technical documentation. As our internal Nuclino has grown in complexity and scale, ensuring the quality and coherence of the content has become increasingly challenging. To address this, we've developed AI-DataFormatter, a data processing tool that intelligently formats and prepares our wiki data for training a custom chatbot. This article explores the key features and technical implementation of AI-DataFormatter.
¶ Key Features
- Content Cleaning: AI-DataFormatter applies a set of predefined rules to clean and format the wiki content. It preserves critical technical details such as file paths, configuration settings, code snippets, and documentation structures while removing any irrelevant or malformed data.
- Intelligent Text Chunking: The tool employs a text chunking algorithm to break down large wiki pages into smaller, semantically coherent chunks. This ensures that the chatbot can be trained on contextually relevant data snippets, improving its ability to understand and respond to user queries.
- Parallel Processing: To efficiently handle the large volume of wiki data, AI-DataFormatter leverages parallel processing across multiple GPUs. It intelligently distributes the data chunks across a cluster of processing servers, optimizing resource utilization and processing times.
- Database Integration: AI-DataFormatter seamlessly integrates with a SQL Server database to store and retrieve the processed wiki data. It uses Entity Framework Core for efficient database operations and supports bulk data processing for improved performance.
- Error Handling and Logging: The tool incorporates robust error handling mechanisms to identify and handle any issues during the data processing pipeline. It logs detailed information about each processing step, including errors and warnings, facilitating easy debugging and monitoring.
¶ Parallel Processing for Scalability Processing
Processing large volumes of technical documentation requires significant computational resources. AI-DataFormatter addresses this challenge by leveraging parallel processing across multiple GPUs. The system is built around a cluster of vLLM servers, each associated with a specific GPU. When a processing request is received, AI-DataFormatter intelligently distributes the workload across these servers, ensuring optimal utilization of computational resources.
The parallel processing workflow follows these steps:
- Document Chunking: Large documents are intelligently split into smaller, semantically coherent chunks, ensuring each chunk can be processed independently without losing context.
- Load Balancing: The chunks are distributed evenly across the available vLLM servers using a queue-based system.
- Parallel Processing: Each vLLM server independently processes its assigned chunks, cleaning, formatting, and optimizing the content.
- Result Aggregation: The processed chunks are collected and reassembled into the final, cleaned document, with any necessary post-processing performed at this stage.
Figure 1: Sample response and parsing
¶ Intelligent Text Chunking
Maintaining contextual coherence when splitting large documents into smaller chunks is a key challenge. Naively splitting by a fixed word count or character limit can lead to fragmented sentences, broken code snippets, and loss of semantic continuity. AI-DataFormatter addresses this with an intelligent text chunking algorithm that considers the semantic structure of the document:
- Sectioning: The document is first split into top-level sections based on headers and major dividers, respecting major context boundaries.
- Paragraph Chunking: Within each section, the text is further divided into paragraphs, keeping related paragraphs together up to a certain size limit.
- Sentence Splitting: If a paragraph exceeds the size limit, it is carefully split into sentences based on punctuation and grammatical cues to avoid fragmenting thoughts.
- Overlap Retention: Optimizing chunk sizes for context and continuity involves balancing parameters such as
MaxChunkSize,MinChunkSize, and tuningOverlapSizefor better coherence between chunks. To maintain context between chunks, a certain amount of overlap is retained from the previous chunk, chosen to be a complete semantic unit such as a full sentence or code block.
¶ Synthetic Data Generation
To improve the chatbot's knowledge and adaptability, AI-DataFormatter automatically generates diverse question-answer pairs from the cleaned wiki content. It creates various question types, allows customization of generation parameters, integrates with the data pipeline to populate the QA database, and utilizes concurrent generation for efficiency.
For example, from this cleansed text:
"AI-DataFormatter is a powerful tool designed to format and prepare company data for training a custom chatbot. It leverages intelligent text processing techniques, parallel computing, and seamless database integration to ensure the chatbot is trained on high-quality, contextually relevant data. The modular architecture and use of standard libraries and frameworks make the tool maintainable, scalable, and easily extensible."
The module could generate these synthetic QA pairs:
What does AI-DataFormatter do? Formats company data for chatbot training using intelligent techniques and parallel computing.
How does AI-DataFormatter ensure high-quality chatbot training data? Leverages text processing, parallel computing and database integration for contextually relevant data.
What makes AI-DataFormatter maintainable and scalable? Modular architecture and use of standard libraries and frameworks.
Exposing the chatbot to a wide range of potential queries during training through synthetic data generation significantly improves its ability to provide relevant, context-aware responses without manual effort. This intelligent approach to optimizing data preparation is a key innovation of AI-DataFormatter.
¶ Multimedia Content Processing
To further enhance the quality and accessibility of our documentation, AI-DataFormatter includes Multimedia Content Processing. This feature automatically detects and processes images embedded in markdown files, converting them into detailed textual descriptions using vision models like MiniCPM-V. This ensures that visual content is accurately represented and accessible, improving the overall user experience.
By integrating secure Nuclino API access and vision model processing, AI-DataFormatter preserves the context of images, generates automatic alt text when needed, and enriches documentation with meaningful image descriptions. This advancement makes it easier for the chatbot to understand and relay visual information, providing users with more complete and context-aware responses.
For instance, when processing the image: clarity-logo.png
The generated description is:
"The image features a logo that is predominantly in shades of blue and black. The central element of the logo is an uppercase letter "A", which is stylized to resemble a mountain peak. Atop this "A" stands a silhouette of a person, suggesting themes of achievement or reaching new heights. To the right of the "A", there's a smaller triangle, also in blue, adding balance to the design. Below these graphic elements, the word "CLARITY" is written in lowercase letters. The overall impression is one of ascension, clarity, and perhaps a journey towards enlightenment or success."
¶ Prompt Engineering
We provide a specific prompt while dissecting the cleaning rules and JSON result format, identifying technical elements to preserve, and specifying content cleanup and formatting rules. This involves analyzing prompt/completion examples, iterating on prompt design for optimal results, fine-tuning vLLM parameters such as temperature and top_p, and evaluating the impact of prompt changes on real datasets.
¶ Error Handling and Recovery
AI-DataFormatter incorporates a robust error handling mechanism to deal with potential issues during processing. Key components include:
- Retry Mechanism: If a processing request fails due to a transient issue, the system automatically retries the request a configurable number of times, using exponential backoff to avoid overwhelming servers.
- Fallback Strategies: If a vLLM server consistently fails, the system redirects the workload to other available servers or skips it, ensuring processing can continue even if a subset of the infrastructure experiences problems.
- Detailed Logging: All stages of the processing pipeline are meticulously logged, including chunk allocation, processing status, error occurrences, and retry attempts, providing valuable data for debugging, performance analysis, and system optimization.
Figure 2: Console logging
We see in this figure how the tool keeps track of everything during processing. It logs real-time updates like how many items were processed, success rates, and any skipped items. You can also see how it splits large tasks into chunks, uses GPUs for speed, and handles errors if something goes wrong. It’s a clear way to monitor progress and quickly spot issues, making the whole process smooth and efficient.
¶ Future Enhancements
As we continue to develop AI-DataFormatter, we will focus on the following key areas:
- Performance Optimization: We will fine-tune our parallel processing architecture, optimize database queries, and explore caching mechanisms to enhance the tool's speed and efficiency.
- Support for Diverse Data Types: We will extend AI-DataFormatter's capabilities to handle structured data, unstructured data, and multimedia content, providing a comprehensive understanding of our technical documentation.
- Image and Video Processing: We will integrate algorithms to extract information from images and videos, enabling our chatbot to provide more context-rich responses.
- Intelligent Data Fusion: We will develop mechanisms to correlate and combine information from multiple sources, linking text descriptions with relevant images or videos.
¶ Conclusion
AI-DataFormatter is a powerful and efficient tool designed to format and prepare our company's Nuclino data for training a custom chatbot. By leveraging intelligent text processing techniques, parallel computing, and seamless database integration, it ensures that the chatbot is trained on high-quality, contextually relevant data. The modular architecture and use of standard libraries and frameworks make the tool maintainable, scalable, and easily extensible.
Developing AI-DataFormatter in-house represents a significant step towards building a highly accurate and efficient chatbot that can aid employees in accessing and navigating the vast knowledge stored in our company wiki. This tool lays the foundation for more advanced AI applications and showcases our commitment to leveraging cutting-edge technologies to drive productivity and innovation.