The internship is now over with the final event successfully held on the 29th of August. 

This summer the team developed a chatbot application covering four use cases for LLM's in healthcare, specified by AI Competence Center at Sahlgrenska University Hospital. They were the following:

• An assitant for a patient to answer questions about their medical records.
• A chatbot that can answer general medical questions using selected sources on the internet.
• An assitant for doctors that can help them to quickly find information about patients.
• A chatbot that can help medical staff with administrative questions, e.g looking up phone numbers, email, policies etc.

Our final product was a  website where the use cases were implemented in a chat interface. 

Key concepts were Retrieval Augmented Generation (RAG), word embeddings, agents (OpenAI function calling) and vector databases.

Even though this application was made as a proof of concept in mind,  we are very satisfied with the result. We are happy to hear that VGR will use our project as a stepping stone for future projects.

Finally we want to thank AI Sweden and VGR for the opportunity to work with and explore the possibilities of LLM's during the summer!

This was our final week working with our project and it was mostly spent adding small features and redesigning the UI. The main feature added was the ability to save chats, enabling multiple conversations at once.

We have also presented our application to groups at VGR where we hope that we gave good directions and ideas to what is possible, both with this application but also with LLMs in general.

Next week we will have our final event and we hope that as many partners as possible will attend!

We want to thank AI Sweden and VGR for the opportunity to explore LLMs in healthcare during the summer and we are excited to see what will happen in the future!

Since the internship is coming to an end, we have been focusing our energy to illustrate our project and its findings. We have created a presentation and a poster for the final event that is being held August 29th .

We also got the opportunity to present our project to a group of people at VGR, where we had an interesting discussion that gave us good insights to what the next steps would be. Another couple of presentations are scheduled to make sure that we can share our findings with as many people as possible.

Finally, our codebase is now public and you are all welcome to take a look. Hopefully you can find something that can help you in your work! Link can be found on this project page.

This week  we worked on integrating the testing via promptfoo as a github-action. This means that every time the prompts are edited in some way, we will run a test script to ensure that the quality was not diminished. It will enable us to more systematically measure improvements in the actual prompts.

For this to work as intended we needed to refactor our codebase to store prompts in a single directory, which we also started working on.

Further, we also worked on some minor UX improvements. For example we now have more nicely formatted links provided by our LangChain bot, as well as further additions to some settings menus for the chatbots.

Overall we are satisfied with our progress, even if there still are some tasks to finish. Next week we will start to work on our final presentation, and keep fixing various issues. We may also explore the possibilities of using a local model like LLaMa-2 rather than an API like with GPT-3/4, which may be preferable in a healthcare context.

These two weeks are summarized in this post since members of the group went in and out of vacation. From now on, all members are back.

Week 5 was spent adding features and fixing several smaller miscellaneous problems. The most prominent feature added was a settings menu where you could choose tools and what type of language to use. 

Week 6 was focused on testing and documentation. We wrote a draft for a set of guidelines when working with LLMs in healthcare with topics that we believe are important to consider. The new test suite was created using promptfoo that enables us to test prompts faster. 

For week 7 we want all currently planned features to be implemented and refine our test suite.

During week 4 we managed to get a fully functioning application with our new frontend/backend-setup. What we also did was to transition our data storage from pkl-files in to a chroma database, speeding up document retrieval and answer generation. 

In parallel with this we also added the following functionalities to our application:

• Possibility to reset chat
• Switch between different clients
• Option to view sources of answer

To finish the week of we had a meeting with a doctor that gave us valuable input on the state of the application, as well as what could be improved. 

We want to continue to refine the product and  improve user experience.  Some time during the coming two weeks we will go on vacation so the progress might slow down a little.

At the beginning of the week we started and finished the development of working versions of our intranet- and internet-bot. The intranet-bot uses similarity search among documents to find relevant information, while the internet-bot uses LangChain agents to access 1177.se, internetmedicin.se and FASS.se. We do not have access to their respective API's, instead we used the DuckDuckGo search engine which might hamper our results slightly.

The rest of the week was dedicated to building a new frontend in Sveltekit as we believe that this would give us more flexibility and is a better long term solution. Paired with this we built a server backend in Flask/Python in order to integrate our models well. 

If we look back at our goals for this week we can say that we achieved what we planned. Looking ahead to next week, we hope to have a fully functioning application in our new setting.

Please note that the title is just a placeholder and may be subject to change.