How do I examine my access.log?

You might find abuse from specific IPs, and often this is due to bots hitting your site. But, you may also find IPs that are not associated with bots over-browsing your site.

This section lists a few commands you can run via SSH to help identify which IPs are hitting your site.

Listing IP hits

Make sure that after you log into the server via SSH that you are in your /logs/example.com/http directory. This is where you’ll run the following commands.

Command Description
[server]$ cat access.log| awk '{print $1}' | sort | uniq -c |sort -n
Generates a list of IP address preceded by the number of times it hit a site.
[server]$ tail -10000 access.log| awk '{print $1}' | sort | uniq -c |sort -n
Generates a list that shows the last 10,000 hits to a site.
[server]$ host 66.249.66.167
167.66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-167.googlebot.com
The 'host' command determines the hosting company from which a specific IP is hitting a site. In this example, the IP belongs to Google.
[server]$ tail -f -q access.log
Watches your server logs in real-time to see if the issue presents itself with a specific IP (for intermittent issues).
order allow,deny
deny from 66.249.66.167
allow from all 
Blocks the IP in an .htaccess file. In this example, the .htaccess file blocks the above Google IP.

Listing top files, folders, and domains

Command Description
[server]$ awk '{print $7}' access.log|cut -d? -f1|sort|uniq -c|sort -nk1|tail -n10
Generates a list of files or directories on your site being called the most.
[server]$ for k in `ls -S */http/access.log`; do wc -l $k | sort -r -n; done
  • Generates a list of traffic for all domains listed under a specific user (on a shared server).
  • This command must be run in your /logs/ directory.

SHELL users on a VPS or Dedicated plan

The following commands only shows sites under a single 'Shell' user. If you need to view all site logs on your server, view the section below titled 'Admin users on a Dedicated plan'

Command Description
[server]$ for k in `ls -S /home/*/logs/*/http/access.log`; do wc -l $k | sort -r -n; done
  • Generates a list of all traffic for all domains under a single SHELL user.
  • You can run this command from within any directory.
[server]$ tail -f -q /home/*/logs/*/http/access.log
  • Watches your server logs (under a single Shell user) in real-time to see if the issue presents itself with a specific IP (for intermittent issues).
  • You can run this command from within any directory.

Admin users on a Dedicated plan

The following commands only work if you have an 'Admin' user on your Dedicated server. You must then switch to your 'root' user. Only then will you see all logs for all sites on your server.

How to switch to the 'root' user after logging in as an 'admin' user

The following steps detail how to log in as a 'root' user on your server AFTER logging in as your 'admin' user first.

  1. Log into your server as your 'admin' user.
  2. Run 'sudo su':
    [server]$ sudo su
    [sudo] password for exampleuser:
    
  3. Enter your password for your 'admin' user.
  4. You're now logged in as the 'root' user. You can see this in the command prompt:
    root@ds123456#
    

After you're logged in as the 'root' user, run the following commands to view logs for all sites on your server.

Command Description
[server]$ for k in `ls -S /home/*/logs/*/http/access.log`; do wc -l $k | sort -r -n; done
  • Generates a list of all traffic for all domains on a Dedicated server.
  • You can run this command from within any directory.
[server]$ tail -f -q /home/*/logs/*/http/access.log
  • Watches your server logs in real-time to see if the issue presents itself with a specific IP (for intermittent issues).
  • You can run this command from within any directory.

My Unique IP is making a lot of connections

You may find in your access.log that your site’s Unique IP is making a lot of connections. This is not an issue and can be safely ignored.

This occurs because Apache is internally generating these connections in order to shut down unneeded processes.

You can read more about it here.

Troubleshooting

You may see the following error response after running the command to view all traffic for all domains on a server.

[server]$ for k in `ls -S /home/*/logs/*/http/access.log`; do wc -l $k | sort -r -n; done
ls: cannot access /home/*/logs/*/http/access.log: No such file or directory

This error occurs when you run the command as an SFTP user. You must run it as a 'Shell' user on a VPS or 'Shell' or 'Admin' user on a Dedicated server.

See also