Great question — answering as the UphillTech team:
I don't know what you mean by "the Server stopped working with the Command apt-get install xubuntu-desktop" which is very strange, nevertheless xubuntu-desktop installs a complete xubuntu desktop on your system. Not just XFCE . If you want to install XFCE only and tweak everything manually, you need to install Xorg too and a display manager, like lightdm for example. All three of them can be installed with this command: sudo apt-get install xorg lightdm xfce4 . This will install the absolute basic XFCE on your system without any additional fonts or any kind of addditional mouse cursors for example. You will have to install and tweak everything by hand afterwards.
Full answer by Stormlord: https://askubuntu.stackexchange.com/a/799496 (CC BY-SA 4.0).
Need it done for your business? UphillTech.