Try the recommendations in this guide. https://www.seogrounds.com/how-to/improve-server-response-time-wordpress/
If they still don’t improve your server response time then your issue must be host related. I can offer you super fast hosting if you require it (under 10ms server response time)
Your issue is around caching and cache being purged due to them expiring. Follow this guide to improve your server response time and keep it consistent, without fluctuation. Any fluctuations after implementing this will be due to your host more than anything. See article below:
How to improve Server Response time on WordPress
This is the ultimate guide to improve server response time for WordPress specifically. https://www.seogrounds.com/how-to/improve-server-response-time-wordpress/
If your server response time hasn’t improved after implementing the tutorial. Your hosting provider is most probably at fault.
