In the fourth week, we began with a cleaned and processed dataset, allowing us to move into the next phase of our project (see previous post for detailed steps). This phase involved exploring various models, including Random Forest, CatBoost, KNN, SVC, GBM, and XGBoost. For each model, we evaluated metrics such as accuracy, precision, recall, F1 score, F1-false, and F1-true. Our initial goal was to establish a fair comparison to narrow down the models for further testing and hyperparameter tuning.

During the tests, CatBoost, GBM and XGBoost emerged as the best-performing models. We also chose to explore a simple Decision Tree due to UNICEF's interest in model explainability and their primary objective of understanding and identifying why people are churning. Despite an exhaustive hyperparameter tuning, we found that while our models performed well on the validation data (approximately 85% accuracy on June data), the results were unsatisfactory on the July data. The models detected only a small fraction of actual churners and misidentified many non-churners as churners. Further investigation revealed high volatility, indicating the model's difficulty in classifying already seen donors in the training set.

In addition to the modelling efforts, we brainstormed with UNICEF about our final solution and the project's conclusion. We considered various tools such as Tableau, PowerBI, Google Looker, and Qlik. Ultimately, we decided to implement our own AI solution, hosted on an AWS instance provided by UNICEF. We began developing our own website and built the tech stack around it. The chosen architecture is depicted in the accompanying image.

Looking ahead, we aim to provide UNICEF with insights into what type of data they should collect to train and obtain better models. We will also work on model explainability to understand which features our models considered most important, so we can report these to UNICEF.

Feel free to reach out if you have any questions, need more information, or have any advice for us!