One possible improvement in the Vinnova database would be the addition of keywords for each project in the form of metadata. Currently, some of the project tags in Qlik Sense, the analytics tool, are given to the projects automatically whereas others are chosen by the person who sends in the project proposal. Therefore, it is likely that the tags or topics have inaccuracies, and the agency would benefit from more representative keywords. This would make classifying and searching for projects in the same professional area easier. Prompting GPT-SW3, one of the functionalities in the app provides a few keywords from an input text. This is easily adaptable to provide and store keywords for every project in the Vinnova portfolio.

Vinnova works with big project populations that answer to their project calls. An initial important step when they start receiving proposals to a project call is to identify the topics present in the population of documents. By using BERTopic, we provide an initial visualization of the possible topic groups or work areas found, as well as titles or headings for them. This last part is done with GPT-SW3, which is prompted to produce representative labels of each of the groups of texts. This way, the user is provided with a visual representation of the projects that they have received and can make decisions based on the general groups found.

A related functionality that we have in the app is one where the user can write their own headings and descriptions of the topics that they expect to find in the population with which they are working. This way, the projects are mapped to their most suitable topics and a visualization of the topic distribution is provided.

This was an experimental development inspired by the fact that there is a lot of data stored as PDFs and analyzing that data is far from being a trivial task. We started experimenting with a slightly easier task than analyzing PDFs, namely question answering based on PDFs. 

We used the popular langchain package. SentenceTransformers was used for identifying relevant text passages that were then fed to GPT-SW3 for constructing the final answer. We used custom prompts and also tried prompt-tuning GPT-SW3 so that it would only base its answers on the available context but noticed that the models were nevertheless hallucinating a lot. 

We decided that it was necessary to understand model hallucinations better before continuing with that project which is why that sub-project was not completed. We are still working on trying to improve our understanding of model hallucinations.

This was an experimental development inspired by the fact that people at Vinnova seemed to have a hard time finding the information they were looking for in the currently available analytics tool. What if there was a virtual assistant that you could just tell which project population you are interested in and ask whatever you want to know and the assistant would then generate a chart that answers your question? The task here was to take the user query and turn it into Python code that reads the necessary information from a data table and that in the end generates a plotly chart that correctly answers the user’s question.

We were not able to finish this sub-project completely but the most promising approach based on experiments - comparing in-context learning and prompt-tuning - seemed to be few-shot prompting, where the few-shot examples were chosen based on semantic similarity with the user query and ordered so that the semantically most similar example to the user query was placed at the end of the prompt.

There are numerous classification-related tasks that are carried frequently at Vinnova. One problem with having many classification tasks is that training a new model for every specific use case would be costly and time consuming. Additionally, there is the possibility that not enough data is available for each and every case. For this reason, this app functionality allows the user to train their own classification model only by providing at least ten examples of each category. It also gives the option to add synthetic data (which uses GPT-SW3 to paraphrase the existing examples) to improve the performance of the model. Once a model is trained, a few examples are presented to the user (the predictions that the model is the most uncertain about), and they can re-label them if they detect any errors and re-train or finish the job if they are happy. The re-labeling can be done iteratively.

This sub-project is intended to make classification tasks easier and more user friendly, with less examples and without the need of an NLP expert in the task, and to provide human evaluation of the model’s performance.

One common need at Vinnova is finding projects that fit certain criteria, such as a query or the description of another project or deciding on the right evaluator (bedömare) for a project. This can be solved by making use of semantic similarity, which involves embedding the texts and obtaining similarities of the vectors that represent them. This way, one can, for example, specify that they want a project related to “mental health and therapy with horses”, and the app will return the projects that are the most similar to the query. It works similarly when the user inputs a project description and wants to find projects that are closely related or needs to decide on the evaluator that is best fit to assess a proposal.

To speed up the processing time of this features, we resorted to using a vector database called Chroma. Chroma stores the project abstracts as vectors, and it can store the text itself and other metadata (such as the original portfolio, the project identification number, etc.), and the vectors are retrieved with a query. A vector database, if not updated regularly or via a pipeline, is static. As project proposals keep coming into the Vinnova database, our Chroma database updates every midnight.