Moving beyond existing green communication projects. SONATA is devoted to making edge learning energy sustainable. Most existing projects are/have been devoted to improving the network performance, either for existing or enabling new applications. In SONATA, we are concerned with turning energy hungry future edge wireless technology into green systems with minimal environmental impact. While the performance aspect is key in all our designs, our main efforts will go into exploring novel computation, communication and learning paradigms to efficiently exploit renewable energy sources and/or to minimize the amount of energy that is drained by edge devices. SONATA is a fundamental research effort, and proposes a paradigm shift in the implementation of edge learning solutions. To the best of our knowledge, this is the first and only project aiming to combine device level design and implementation of ML algorithms with distributed communication and computing algorithms.

To tame the growing carbon footprint of communication networks, it is crucial to devise sustainable communication and computing technologies, particularly tailored towards highly energy-hungry edge AI services. This requires a holistic look at the underlying computing/learning algorithms and their hardware implementations together with intelligent communication and network management protocols, while accounting for distributed energy generation and harvesting technologies. Significant research efforts have been invested in recent years in the exploration of alternative technologies (ASICs, FPGAs, memristors) for energy-efficient implementation of ML algorithms. In parallel, coding and communication theoretic ideas have been put forth to increase the energy efficiency of distributed learning algorithms, in both training and inference stages, and coding, scheduling and networking techniques have been used to most effectively exploit limited ambient energy. The main novelty of our project is to jointly optimize the circuit level implementation of distributed ML algorithms using the novel memristor technology together with accompanying computing, communication and networking algorithms for low-energy edge intelligence. This is a paradigm shift as we diverge from the conventional separate approach, where the design and optimization of ML algorithms (particularly deep learning), the design of wireless communication protocols, and their implementations at the circuit level are considered separately. Another key problem that will be tackled in this project is the exploitation of renewable energy sources for sustainable edge intelligence. While reducing the energy cost of learning and communication is important to achieve this goal, this is not sufficient; erratic and intermittent availability of ambient energy sources must be taken into account to properly operate these networks. Therefore, SONATA’s vision implies the inclusion of new paradigms in learning algorithms, and their implementation using memristive devices to be run on low-complexity edge devices, enabling the learning process to be executed entirely at the edge without accounting for a central server and leveraging the use of energy harvesting and storage hardware to reduce the consumption of resources harmful to the environment.

SONATA’s overall objectives can be summarized as follows:

1. To design, simulate, optimize, and build energy-efficient neuromorphic computing devices for edge applications based on memristor cross-bars. To employ coding theoretic ideas to compensate for low accuracy memristive devices both for in-memory computing and data storage.

2. To exploit synergies between wireless communications and distributed ML algorithms by converting the wireless channel into a joint computing and communication medium through “over-the-air computation (OAC)”.

3. Develop communication/networking protocols to carry out decentralized and sequential learning at the edge with significantly lower energy cost and latency with respect to centralized solutions.

4. Consider energy harvesting technologies at the edge devices for prolonged network operations without battery replacement. Design adaptive communication, coding and computing protocols to exploit the limited and stochastically available energy in the most efficient manner.

5. Implement and evaluate the potential of developed techniques for important use cases that rely on energy-efficient sustainable networking paradigms.

Ideas and technologies towards achieving these objectives will be developed and tested within the framework of the project, and translated into applications up to TRL4.

Target 1: Design and develop a memristor based neural network architecture to reduce the energy consumption of the training process of the network by at least 30% while keeping the same accuracy in the inference phase. Additionally, by combining coding theoretic ideas with our accurate memristor modeling and implementation capabilities, we aim at pushing the analog storage capacity of memristor arrays by at least 20%.

Target 2: Develop FEEL solutions employing energy harvesting technology that can provide at least an order of magnitude increase in the network lifetime. We will combine scheduling based solutions with our recent work on efficient FL with non-iid data [Ozf21]. We will employ hierarchical network architectures as well as low-complexity signal processing techniques to further reduce the transmission energy consumption of wireless networks.

Target 3: Provide extremely energy-efficient OAC solutions for edge learning. Our initial studies indicate up to 50% reduction in transmission energy compared to conventional digital schemes. We will further push this target combining analog transmission in OAC with analog computation and storage capabilities of memristors.

Target 4: Design edge learning solutions based on asynchronous and sequential paradigms that are able to reduce the energy consumption of the system by at least 30%. Realize these gains in real use cases.

CTTC: 

The main goal investigated in SONATA by CTTC is to relax the assumption of having a central server in FL towards more asynchronous, sequential as well as decentralized learning paradigms.

A potential approach is knowledge transfer learning (KTL), where a teacher network is trained with general data and student networks are trained on more specific local datasets. The adoption of KTL for distributed training is mostly unexplored in MEC. In SONATA, KTL will be exploited to learn a feature space transformation to align the source/master and target/student representations, thus reducing the resource demand at the edge devices. Similarly, multi-task Learning (MTL) also results beneficial in terms of used energy when multiple tasks have to be implemented without sacrificing the neural network’s universal function approximation property. In MTL, all the tasks are treated with the same priority and the objective is to improve the performance of all the tasks, while in KTL, the target is to improve the performance of a specific task with the help of a source task. In SONATA, we will investigate multi-task and meta-learning approaches, including energy assessment and considering distributed solutions to orchestrate networking tasks among different base stations.

When considering distributed training, data cannot be considered anymore independent and identically distributed (iid), and this can generate the problem of catastrophic forgetting, especially when using asynchronous and sequential learning. Catastrophic forgetting refers to the problem of neural networks forgetting concepts learned in previous steps, as new ones are learned. Continual learning (CL) deals with this problem of learning from an infinite stream of data with the goal of gradually extending the acquired knowledge, which is also known as stability-plasticity dilemma, where plasticity refers to the ability of assimilate new knowledge, and stability preserving previous knowledge. CL in the distributed setting is mostly unexplored in MEC. We would like to emphasize that the prior works in these domains have mainly focused on the performance limits of aforementinoed solutions without taking into account the energy cost of the proposed solutions. In SONATA, we will put the sustainability aspect to the foreground, and study the performance bounds until strict energy constraints, which may also be stochastic due to ambient energy sources.

Finally, decentralized/ gossip learning (GL) will also be considered to complement the centralized architecture of PS-based FL solutions. GL works by finding the convergence towards a consensus among nodes by exchanging information in a peer-to-peer fashion, over a grap defined by the network topology. While the decentralized and asynchronous nature of GL makes it attractive for low-energy solutions with unreliable nodes, in the wireless setting, network topology depends highly on the communication constraints and resources.  The unreliable and stochastic nature of devices due to random energy availability, as well as inaccurate computations due to memristor devices will be taken into account while optimizing the decentralized communication protocols to enable the optimal network connectivity.

Imperial College London:

In SONATA, we look at a communications perspective on maximisation of the capacity of emerging memory devices, memristors being a prime example, for information storage. Specifically, we focus on mitigating the impact of certain kinds of noise, such as the distortion caused by the phenomenon of resistive drift (where the resistance of a programmed memristor changes in a stochastic way over time following programming), on the quality of data that is stored on the devices and recovered at a later date.

We have identified a trade-off between the energy consumption and permanence of information stored on the devices. We can optimise this trade-off using communication theoretic deep learning techniques, specifically Deep Joint Source-Channel Coding (Deep JSCC). We examine different scenarios for storing information (long term vs. short term), and how techniques such as Deep JSCC can be used to gracefully trade-off the delay we wish to consider for performance in reconstructing semantic information, such as images, that has been stored.

We wish to also consider other kinds of memristive noise in the future, including in more practical settings.  This includes mitigation of the effects of sneak-path-current noise from a communications perspective, to complement or even remove the need for existing techniques involving hardware modifications, such as the inclusion of diodes and transistors to mitigate this noise. The goal is ultimately to minimise the need for additional electronics, which can take up valuable space on devices and limit the design possibilities for future neuromorphic applications.

Bilkent University:

Within the broad umbrella of SONATA, we examine federated learning (FL) with over-the-air (OTA) aggregation, where mobile users (MUs) aim to reach a consensus on a global model with the help of a parameter server (PS) that aggregates the local gradients. In OTA FL, MUs train their models using local data at every training round and transmit their gradients simultaneously using the same frequency band in an uncoded fashion. Based on the received signal of the superposed gradients, the PS performs a global model update. While the OTA FL has a significantly decreased communication cost, it is susceptible to adverse channel effects and noise. Employing multiple antennas at the receiver side can reduce these effects, yet the path-loss is still a limiting factor for users located far away from the PS. To ameliorate this issue, in this paper, we propose a wireless-based hierarchical FL scheme that uses intermediate servers (ISs) to form clusters at the areas where the MUs are more densely located. Our scheme utilizes OTA cluster aggregations for the communication of the MUs with their corresponding IS, and OTA global aggregations from the ISs to the PS. We present a convergence analysis for the proposed algorithm, and show through numerical evaluations of the derived analytical expressions and experimental results that utilizing ISs results in a faster convergence and a better performance than the OTA FL alone while using less transmit power. We also validate the results on the performance using different numbers of cluster iterations with different datasets and data distributions. We conclude that the best choice of cluster aggregations depends on the data distribution among the MUs and the clusters.

Pázmány Péter Catholic University:
In SONATA, our main focus is on the memristor-crossbar implementation-related challenges that make neuromorphic computation possible. Memristors have received significant research attention in recent years as a potential building block of neuromorphic computing systems. In particular, they hold great potential for high-density data storage, in-memory analog computing, and efficient implementation of neural networks. On the other hand, there are many challenges that need to be overcome to bring this technology into widespread use. Specifically, the current memristor mass-production technology is unreliable, typical memristor grids have a 90% yield or less, meaning that every tenth memristor is not working properly, and the number of distinct states that a given memristor device can store may be different.

Choosing the proper parameters of the memristors in crossbar arrays always requires a careful trade-off between size, energy consumption, and resolution. In our previous work, we have made significant progress in this direction. Given that memristors can be produced at the nanoscale and with very high area efficiency, using several memristors interconnected in a subcircuit, implementing a single more reliable device can be beneficial in certain applications. Most of the current implementations can store only two states reliably, making them somewhat similar to NAND Flash or DRAM-based storage, losing the main advantage of memristors. Another novel idea we will explore is to go beyond repetition for combining multiple memristors to increase the efficiency, targeting particular in-memory computation tasks. Recently, the idea of "coded computing" has been introduced for distributed computation across unreliable servers for big data applications. Thanks to the interdisciplinary expertise of our team, we will bring this idea to the memristor level to increase computation efficiency beyond simple repetition. We will utilize realistic memristor models and simulation models we have developed in our prior work for both the training and inference operations of a proof-of-concept memristor-based neural network to minimize the energy consumption.