How do I examine my access.log?

You might find abuse from specific IPs, and often this is due to bots hitting your site. But, you may also find IPs that are not associated with bots over-browsing your site.

This section lists a few commands you can run via SSH to help identify which IPs are hitting your site.

Listing IP hits

Make sure that after you log into the server via SSH that you are in your /logs/example.com/http directory. This is where you’ll run the following commands.

CommandDescription
[server]$ cat access.log| awk '{print $1}' | sort | uniq -c |sort -n
Generates a list of IP address preceded by the number of times it hit a site.
[server]$ tail -10000 access.log| awk '{print $1}' | sort | uniq -c |sort -n
Generates a list that shows the last 10,000 hits to a site.
[server]$ host 66.249.66.167
167.66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-167.googlebot.com
The 'host' command determines the hosting company from which a specific IP is hitting a site. In this example, the IP belongs to Google.
[server]$ tail -f -q access.log
Watches your server logs in real-time to see if the issue presents itself with a specific IP (for intermittent issues).
order allow,deny
deny from 66.249.66.167
allow from all 
Blocks the IP in an .htaccess file. In this example, the .htaccess file blocks the above Google IP.

Listing top files, folders, and domains

CommandDescription
[server]$ awk '{print $7}' access.log|cut -d? -f1|sort|uniq -c|sort -nk1|tail -n10
Generates a list of files or directories on your site being called the most.
[server]$ for k in `ls -S */http/access.log`; do wc -l $k | sort -r -n; done
  • Generates a list of traffic for all domains listed under a specific user (on a shared server).
  • This command must be run in your /logs/ directory.

If you have a VPS or Dedicated plan

CommandDescription
[server]$ for k in `ls -S /home/*/logs/*/http/access.log`; do wc -l $k | sort -r -n; done
  • Generates a list of all traffic for all domains (for multiple domains on a VPS or Dedicated server).
  • You can run this command from within any directory.
[server]$ tail -f -q /home/*/logs/*/http/access.log
  • Watches your server logs in real-time to see if the issue presents itself with a specific IP (for intermittent issues).
  • You can run this command from within any directory.

My Unique IP is making a lot of connections

You may find in your access.log that your site’s Unique IP is making a lot of connections. This is not an issue and can be safely ignored.

This occurs because Apache is internally generating these connections in order to shut down unneeded processes.

You can read more about it here.

See also