We are back in office after two weeks of vacation. We have now completed five weeks, and have three weeks remaining. When we left for vacation two weeks ago, we had accomplished the following:

• Deployed backend server to Microsoft Azure App Service in the cloud.
• Configured the API and set up connections to Azure Storage account for temporary storage of the audio files. Also made sure that each device is playing its corresponding audio files, leveraging the opportunity for usage on different devices simultaneously.
• Changed the OpenAI API-keys from “our own” to Azure’s resources through Azure Key Vault.
• Successfully managed to build the frontend client locally as an APK-file for Android devices to download and build the app without having to go through Google Play. This app will then communicate with the online server through the Azure API. We will need an Apple Developer Account to be able to proceed with this task on iPhone.
• Identified and erased as many bugs as possible.

Coming into week 6, with the Translation feature and Community Guide feature completed, we wanted to focus on the last remaining task - the RAG functionality for storing and analyzing the conversations. The goal of this feature is for the admin of the app (for instance the local manager of a Red Cross branch) to be able to keep statistics and analyze the conversational data in order to identify expressed needs and structural barriers, aiding in project evaluation and mapping to improve the operations. This will be done using the RAG (Retrieval-Augmented Generation) technique. In summary, RAG works by storing conversations as vectors, querying a vector database to retrieve relevant conversations, generating a response using a language model, and delivering the response to the user as an answer to the question, based on the context from the stored data. How GDPR, acceptance of terms, and how AI can be applied to identify and anonymize private data in the conversations, is something we will come back to and put focus on later.

At first, we wanted to implement one type of RAG called GraphRAG. GraphRAG is a technique that enhances the retrieval process by leveraging the structure of a graph to improve the relevance and accuracy of the retrieved information. It does this by considering the relationships between different nodes (documents or passages) in the graph and using this information to better match the query with the most relevant documents. This is a technique that we think matches our specific use case best, since we want to gather data from a specific topic over a wide range of vectors. However, the GraphRAG technique comes from very recent research, hence there is very little information and guides on how to implement it. It also requires Python language, and our tech stack is running on Node.js and JavaScript.

During the week, we worked on different approaches on ways to implement the RAG functionality. These approaches included: trying out to use a Function App to run the Python script by calling a separate API, implementing GraphRAG in python, using the LangChain framework to build RAG in both Python and JavaScript, and with the help of different vector databases as ChromaDB, Cosmos and MongoDB. At the end of the week, we managed to successfully implement a fully functional Baseline RAG (i.e. “normal” RAG) by ourselves, without the help of any RAG-framework like LangChain. We felt that this approach was the best for us to learn the most, fully understand the RAG functionality as well as to easily control and manage the RAG pipeline in our code. 

How does our RAG pipeline work?

1. Store conversations as vectors: Conversations are transformed into numerical vectors using a technique called embedding, which captures the semantic meaning of the text. Each talk is put together as a string, and then sent into OpenAI:s Embedding Model. We get a 1536-dimensional vector in return, that has kept the semantics of the string. Then, this vector is uploaded together with the original string and some additional metadata to our resource in Azure called Cosmos MongoDB vCore - an open-source, document-oriented NoSQL database that stores data in a flexible, JSON-like format. The natively integrated vector database in MongoDB enables us to efficiently store, index, and query high-dimensional vector data that's stored directly in Azure Cosmos DB for MongoDB vCore, along with the original data from which the vector data is created.
2. Index the vector database: We have added a Vector Search Index to our collection directly in the MongoDB Shell. We are using the IVF (Inverted File) algorithm, starting with 1 cluster to group the vector data for brute-force search. The similarity is calculated with cosine.
3. Query the vector database: When a new query is received, it is also embedded and compared to the stored vectors in the database. This part of the RAG pipeline starts in the new Admin Search Screen of the app, which has a Q&A-chat interface for the admin to be able to ask questions regarding the data, and get LLM-generated responses in return based on the data. When the query is sent, it gets vectorized by OpenAI:s Embedding model. Then we perform a vector search between the query vector and vector storage.
4. Retrieve relevant conversations: The most similar vectors (conversations) are retrieved from the database based on their similarity to the query. The number of most relevant vectors is now set to 10, but this is a number we will experiment with.
5. Generate a response: The retrieved conversations are used to generate a response using a language model, which can understand and generate human-like text. So from the retrieved vectors, we now extract the original text. This text is sent to OpenAI:s GPT-4 model, together with the question and an instruction for the LLM to follow: “Based on the following information: … Answer this question: …”.
6. Deliver the response: The generated response is then delivered to the user.

Hi,

This week, we focused on finalizing the deployment from last week and preparing the app for organizational testing. One issue we encountered was that our devices were having trouble retrieving and playing processed audio files from Azure blob storage. We resolved this by enhancing the file retrieval function on the client side.

With the deployment now successful, our attention shifted to enabling multiple users to access the app simultaneously, a key reason for hosting the server in the cloud. To achieve this, we needed to ensure that the correct audio files are matched with the appropriate users in the database—preventing the mix-up of translations between users. Below is an overview of the solutions we considered and the approach we implemented:

• React Native’s built-in function for retrieving device_id from devices using the app.
• Generating unique IDs within the client for each chat session.

Next, we concentrated on optimizing the app for both Android and iOS devices. Our client’s primary requirement is that the app be functional on Android, but since our team primarily uses iPhones, we also wanted to ensure compatibility with iOS. However, the process of building apps for these two platforms differs significantly, especially when trying to avoid costs.

For Android, we used the following approach:

• By running Expo Dev from our terminal in VSCode, we initiated the process of building an APK file.
• Once the APK file was generated, we could distribute it as a link or QR code, allowing users to download and install the app on their devices easily.

For iOS, however, the process is more complicated due to Apple's restrictions on free app distribution:

• While it’s possible to build the app directly through Expo, doing so requires an Apple Developer Account ($99 per year), which is also necessary for releasing the app on the App Store. In contrast, Android only charges a $25 fee per app.
• There is a method to directly distribute the app to iPhone devices, but it appears the client must be running during usage.

After thorough consideration, we decided not to proceed with building the app for iOS at this time. The lack of a cost-effective solution that didn’t require manual client operation made it impractical to support iOS alongside Android. As a result, we are focusing our efforts exclusively on optimizing the app for Android devices.

We also transitioned from using OpenAI's APIs for GPT-3.5-turbo, Whisper, and Text-to-Speech (TTS), along with the OpenAI API key, to utilizing these APIs through Azure, using an Azure API key provided by the Swedish Red Cross.

On the design front, we've implemented a homepage that appears as soon as the app is launched. From the homepage, users can navigate to either the translation page or the community guidance page.

Lastly, we have developed a login page to restrict access to authorized Red Cross users only, ensuring that the app is used solely for its intended purpose of communicating with Ukrainian refugees. However, we still need someone from the Red Cross operations team to configure a redirect URI before we can finalize this feature.  

To wrap things up, our team will be taking a two-week vacation, but we're excited to continue developing the app when we return. After the break, our focus will be on creating an admin page that includes a chatbot using Retrieval-Augmented Generation (RAG). This feature will allow admin users to ask for statistics on the conversations, making it easier to track and analyze interactions within the app.

Deployment time!

This week, our focus was to implement our app into the IT infrastructure at the Red Cross, to enable deployment and demo testing in the real world! We also made our app working smoothly on Android devices.

We kicked off the week by enabling Android compatibility. Previously, our App was only tested on iPhones, which caused some problems on Android devices. The main challenge was how we processed audio files and used the embedded audio player within the devices. While iOS uses AVPlayer, Android uses ExoPlayer. The solution was to come up with a new algorithm to ensure one audio file finishes playing before starting the next. So now it works seamlessly on both iPhones and Androids - great stuff!

With Android issues sorted in the beginning of the week, we spent the rest of the week focusing on deploying our app on Microsoft Azure, wanting to shift our backend from running locally to running in the Azure environment instead. This is a necessity to enable the Red Cross to demo test our mobile application within their actual organization. However, this deployment has been trickier than expected - here’s a snapshot:

• Our server deployed to Microsoft Azure App Service via Github Actions, which went very smoothly. This is done by creating a deploy-server.yml file directly in the repo, as well as by adding Repository Action Secrets containing Azure Credentials from the Red Cross IT Deparment on GitHub.
• The client is running locally on the device and successfully connects to the server, and the server handles all our AI functionalities with our connected API keys.
• Audio files are processed through our API:s, and the resulting files with translation or community guidance responses, are uploaded to our Azure Database called Blob Container, which is a cloud storage for binary files such as audio and images. Unfortunately, we have yet to find out how to reliably send these audio files back to the client for playback on the mobile device.

Some tech-nerdy explanations:

The major problem during the deployment to Azure has been the re-building and re-coding of our server functionality to handle the uploads, access and downloads of the audio files. Previously, everything was managed in the local computer's file system, but when moving the server and storage to Azure, a lot of the functions needed to be revised. 

A specific problem is the handling of audio files directly in memory, since the Whisper API needs a proper file to proceed and transcribe the audio. Here we managed to use Buffers (temporary holding spot for data being moved from one place to another). This also lets us skip the process of uploading the file to the Blob Storage before downloading it again and put it through the API. However, Buffers are not seen as files, which the Whisper API expects. To overcome this, we are using Form-Data that are file-like objects which can contain the binary information directly in memory without the need of an actual storage place locally. The Buffer is first converted to .wav format, and then appended to the Form-Data object. With the help of Axios (instead of using OpenAI's official Node.js library) we are then able to send this audio “file” into Whisper API and get the transcription back. Then this transcription is processed through the rest of the API:s, and finally after TTS (Text-to-Speech) is done, the translated audio Buffer is uploaded to Azure Blob Storage in the wait to be played, and then directly deleted after being played. Azure Blob Storage also has a SDK for Node.js which helped the development process. 

Hence, moving into next week, our primary goal is to get our app fully functional in the Azure environment. Once this is achieved, we can fully focus on adding new cool features and getting the app running in the Red Cross organization as soon as possible!

Until next week!

/Team