I tested this twice and it worked just fine for me.
Can you please check again and if you still have that problem please let us know details of your system, database, ect. and the exact procedure to reproduce.
Thank you!

In what version it worked properly in your test?

4.4.11

Very strange. I on tested Debian GNU/Linux 10, mysql Ver 15.1 Distrib 10.3.27-MariaDB, for debian-linux-gnu (x86_64) using readline 5.2, PHP 7.3.27-1~deb10u1.

First I installed the system using Finnish as default language and without prefix on tables names. I noticed this, and so deleted everything and installed with English and lime_ -prefix (i.e. default settings) with the same result.

Can you please tell me the exact steps during installation? Screenshots or small movies ( using Peek) also help.

Ah, my bad! I tried this again, and it seems to be OK in Configuration > Global settings > Email settings. But in the Configuration > Global survey settings it is not, and that is what is used as default in new surveys.

So, no bug.

However, what is the meaning of "Default site admin email:"? If it is one sitewide setting, then it can not be "default" value.

Ah, my bad! I tried this again, and it seems to be OK in Configuration > Global settings > Email settings. But in the Configuration > Global survey settings it is not, and that is what is used as default in new surveys.

I think it's an issue here ?

Yep, there seems to be two settings for same thing, other does nothing. That should be corrected first.