Authors: Marcus Örtenberg Toftås & Emilie Klefbom

University: Chalmers University of Technology

With the drastic spread of smartphones and other mobile devices capable of continuously collecting data, the concept of responsible mining has become an increasingly important topic. Older data anonymization methods are quickly becoming obsolete as powerful machine learning models, capable of leveraging the exponentially increasing amount of publicly available data to deanonymize sensitive information, emerge. Current anonymization methods such as differential privacy may have unforeseen consequences when training on sensitive machine models, as it utilizes data perturbations, that can significantly affect the performance of the decentralized system. This has led to increasing privacy concerns among users, and more strict regulatory privacy laws such as the European Union's GDPR legislation and the state of California's CCPA act.

One proposed method for overcoming these issues is through distributed ML models such as federated learning. This has great potential as it enables collaborative training without sharing any of the raw data, thus allowing for model training even with small local datasets. The benefits of these systems have also been shown in practice, as some hospitals, retail stores, and even Google have obtained valuable insights into their respective operations through it.

However, current FL models build on training one central model, which introduces a computational bottleneck as well as a single point of vulnerability for potential attackers. These models also generally assume that the data is drawn from a time-independent and stationary distribution, which is seldom the case. Since each unique client typically has some bias to their data, and there is no guarantee that the data belonging to the clients are identically and independently distributed. This results in differing client data distributions (the non-IID data paradigm), which hinders efficient training of deep learning models. Further, user sentiment and preference may also change drastically due to impactful events such as the pandemic or macroeconomic events, further impairing the training. All this leads to what is called concept drift, where data distributions change over time. This causes a dilemma, for when different clients experience data drifts at different times, no single global model can perform well for all clients. And similarly, when multiple concepts exist simultaneously, no centralized training decision works well for all clients.

Decentralized learning's strength is that it removes the necessity for a central node, as clients communicate peer-to-peer and store their own individual model and data. These frameworks also remove the central point of vulnerability that federated frameworks have, making them more robust against attacks, and also personalized for each client. As continual learning for decentralized models is still an unexplored research area, this project aims to explore methods that mitigate catastrophic forgetting in the decentralized setting and develop models that can adapt to distribution shifts in a decentralized setting.