Google's Bot Imposes a Large Impact on the Performance of SteemYY.com
In the last 24 hours, I have observed several alarms of the high CPU load average of steemyy.com. I checked the logs and found out there were a huge spikes of the bots from Google (in the User-Agent header, it says Googlebot).
I checked the logs, most visits are the Blockchain Explorer of STEEM where there are many many many pages i.e. each block has a page which will be seen unqiue to Google bots.
There used to be a setting in Google's Webmaster to slow down the crawler access, however this was hidden in the new Version of Google's Web Master.
Anyway, a few solutions:
- Do nothing - the site experiences short downtime and will be back to normal once the bots are gone.
- Tell Googlebots to slow down by sending 429 Too Many Requests (e.g. by implementing a Rate Limiter)
- Buy a much powerful Server (e.g. Dedicated servers)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Thank you for reading ^^^^^^^^^^^^^^^
- Video Downloader
- Steem Blockchain Tools
- Free Cryptos API
- VPS Database
- Computing Technology Blog
- A few useless tools
- And some other online software/tools
- Merge Files/Videos
- LOGO Turtle Programming Chrome Extension
- Teaching Kids Programming - Youtube Channel and All Contents
Steem/Swap to USDT Swap
If you like my work, please:
- Buy Me a Coffee, Thanks!
- Become my Sponsor, Thanks!
- Voting for me:
https://steemit.com/~witnesses type in justyy and click VOTE
- Delegate SP: https://steemyy.com/sp-delegate-form/?delegatee=justyy
- Vote @justyy as Witness: https://steemyy.com/witness-voting/?witness=justyy&action=approve
- Set @justyy as Proxy: https://steemyy.com/witness-voting/?witness=justyy&action=proxy
Alternatively, you can vote witness or set proxy here: https://steemit.com/~witnesses