Abu Dhabi Open Data Intelligence :

1. Project Overview:

The Abu Dhabi Open Data Intelligence project focuses on the creation of a versatile Chatbot integrated with the Abu Dhabi Open Data website. This Chatbot operates seamlessly, accepting user queries in both text and voice formats. Its core functionality involves providing data visualizations and data summaries, with a primary focus on Excel sheet data. As the Project Coordinator, my role primarily revolved around ensuring effective communication and collaboration among team members, thereby streamlining the project's progress.

2. Dataset Collection:

For this project, we embarked on a web scraping initiative to collect a substantial dataset comprising 3300 datasets, each categorized into Education, Environment, Health, Safety & Security, and to some extent, Energy. These datasets were meticulously extracted from the Abu Dhabi Open Data website, which required establishing a connection and subsequently downloading all 3300+ datasets to our DagsHub repository via API calls.

3. Data Preprocessing:

Data preprocessing was a crucial phase in our project. To streamline this, we effectively employed the AutoClean library for automating data preprocessing tasks. In parallel, we used a Data Profiling library to conduct an in-depth Exploratory Data Analysis (EDA) and subsequently generate HTML reports for each dataset. This EDA phase was instrumental in understanding the structure and characteristics of our data.

4. Model Selection:

In the Abu Dhabi Open Data Intelligence project, the Modeling phase is fundamental to the creation of a cognitive engine capable of effectively responding to user queries and requests. This section provides insight into the processes and components involved, following a structured approach similar to the Instruction Chain.

1. Objective of Modeling: The central objective of the Modeling phase is to fine-tune the Falcon 7B Instruct LLM Model, enhancing its capabilities to generate robust code and provide users with data visualizations based on their requests. This entails training the model on a curated dataset of Python code instructions.
2. Model Selection: During this phase, various models are explored and evaluated. While the baseline model is Falcon 7B Instruct, the team also experimented with alternative models, including CodeLlama 7B Python gptq, CodeLlama 13B Python gptq, and Mistral 7B Instruct gptq. Comparative analysis revealed that the alternative models consistently outperformed Falcon, with CodeLlama 13B ranking as the most proficient. Nonetheless, the Falcon model was retained to support the local open-source community of the UAE.
3. Additional Components: The Modeling phase integrates additional components essential for the success of the project:
   • RAG Integration: This phase marks the inclusion of the RAG (Retrieval-Augmented Generation) technique. To use RAG effectively, SQL Chain is employed, although it comes with a limitation of converting natural language to SQL statements, primarily suitable for data retrieval tasks.
   • Memory Enhancement: Memory integration, particularly through ConversationBufferMemory, plays a vital role. The integration of memory facilitates an iterative learning process for the Falcon 7B Instruct model. The model progressively improves its responses based on past interactions and the contextual understanding of the conversation.

5. Instruction Chain:

To leverage the strengths of Falcon 7B Instruct further, we introduced the concept of the Instruction Chain. This chain plays a vital role in connecting the dataset, user inputs, and the Falcon 7B Instruct model. In this project, we utilized the LangChain for integration. The Instruction Chain essentially operates through the following steps:

1. Prompt Templates: The first step involves creating suitable templates to guide code execution and data visualization. These templates act as the foundation for user interactions, providing a structured format for input and output. After several trials with prompt templates from LangChain, which proved less effective, we incorporated user input using f-Strings. This flexible approach allowed for more dynamic and user-specific prompts.

   

2. RAG for Data Retrieval: To empower the Chatbot with the ability to retrieve relevant information from the extensive Abu Dhabi Open Data repository, we initially adopted SQL Chain. This approach provided highly accurate results, particularly for data retrieval. However, it had a disadvantage: SQL Chains convert natural language to SQL statements, lacking visualization capabilities. Recognizing this limitation, we implemented separate pathways within the framework to address data retrieval (using SQL Chain) and data visualization (via code generation).

3. Memory Integration: Memory enhancement is a key element of the Instruction Chain. We want to Integrate Memory to facilitate the model's learning from past interactions. The model's performance is elevated by integrating ConversationBufferMemory, allowing it to learn from past interactions. This means that the Chatbot becomes progressively more adept at addressing user queries and customizing responses based on the historical context of the conversation.