OK, I did a deep research:
I have 2 sites on 2 different domains, using own template built on TwentyTen, using 15 own plugins and about 25 public plugins... Admin on primary domain (the same as network admin) is slow, admin on secondary domain is OK. 22sec vs. 1 sec page load.
I have tried to use Twenty Thirteen template and no plugins (i have deleted plugins folder), however still slow. On every page.
There are no errors in apache2 log... It looks like some redirect issue, because secondary domain admin (URL is on this domain) works fine...