In future wireless communication systems, full-duplex (FD) and massive multiple-input-multiple-output (mMIMO) are considered as two promising technologies to overcome the capacity crunch and spectrum scarcity. Transmission and reception at the same frequency-time channel in FD and large antenna arrays in mMIMO systems improve the spectral efficiency to a great extent compared to current systems. To make mMIMO systems cost-efficient, inexpensive less-accurate transmit and receive chain components are preferred. This leads to hardware distortions, which in turn become unfavourable for self-interference (SI) cancellation in FD systems.

In this thesis, our main goal is to design distortion-aware FD multi-antenna multi-carrier (MC) systems, from the aspect of resource allocation and the resulting system performance. Particularly the impact of distortions caused by hardware impairments, leading to residual self-interference and inter-carrier leakage as well as the imperfect channel state information is taken into account.
Initially, we investigate the linear transceiver design problem for an FD multiple-input-multiple-output (MIMO) MC decode and forward (DF) relaying system. In addition to the traditional per-carrier DF relaying, the case with a joint-carrier DF is also studied, taking advantage of group-wise decoding and encoding.

Furthermore, we focus on the joint sub-carrier and power allocation problem for a DF relay system, where multiple half-duplex (HD) single antenna (SA) MC source-destination pairs communicate with the aid of an FD mMIMO MC relay. Besides maximizing the sum-rate and energy efficiency, we also focus on minimizing the overall delivery time for a given set of communication tasks to the user nodes. We extend the studied framework to a bi-directional MC communication system, where an FD mMIMO base station (BS) serves multiple FD SA user nodes.

Finally, we focus on resource allocation problem for an FD-enabled relaying system, where an mMIMO MC BS simultaneously activates the relay as well as the direct channel for communicating separate data streams to the user terminals. This is implemented by employing successive interference cancellation at the MC SA user terminals. Besides the superior performance under various system conditions, the proposed dual-connectivity enjoys higher robustness when one of the active paths experience an unexpected blockage.