Viewing and examining your access.log via SSH

Overview

This article lists a few commands you can run via SSH to help identify which IPs are hitting your site.

Logs are rotated daily which means there are only a few days worth at any time. If you need long-term logs, you must continually back them up. Per GDPR, DreamHost stores HTTP logs by default for 3 days.

Logging into your server

To run the commands below, you must first log into your server via SSH:

SSH overview

After you're logged in, navigate to the correct logs directory.

What logs directory should I view?

DreamPress Plans Only

You do not need to include http/s in the file path (as mentioned in the following examples) to view the log files for DreamPress sites. You can find DreamPress-related log files using the /exampleuser/logs/example.com file path instead.

If you have added an SSL certificate to your website, you will see two folders. One for /http and one for /https. For example:

/logs/example.com/http
/logs/example.com/https

If you visit your site and see it's using HTTPS in the URL (https://example.com), this means you should view the /https directory. If your site is displaying HTTP when visited, you would view the /http directory instead. The following examples assume your site is using HTTPS.

[server]$ cd ~/logs/example.com/https

This is where you’ll run the following commands.

Listing IP hits

Command	Description
[server]$ cat access.log\| awk '{print $1}' \| sort \| uniq -c \|sort -n	Generates a list of IP address preceded by the number of times it hit a site.
[server]$ tail -10000 access.log\| awk '{print $1}' \| sort \| uniq -c \|sort -n	Generates a list that shows the last 10,000 hits to a site.
[server]$ host 66.249.66.167 167.66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-167.googlebot.com	The 'host' command determines the hosting company from which a specific IP is hitting a site. In this example, the IP belongs to Google.
[server]$ tail -f -q access.log	Watches your server logs in real-time to see if the issue presents itself with a specific IP (for intermittent issues).
order allow,deny deny from 66.249.66.167 allow from all	Blocks the IP in an .htaccess file. In this example, the .htaccess file blocks the above Google IP.

Listing top files, folders, and domains

Command	Description
[server]$ awk '{print $7}' access.log\|cut -d? -f1\|sort\|uniq -c\|sort -nk1\|tail -n10	Generates a list of files or directories on your site being called the most.
[server]$ for k in `ls -S */https/access.log`; do wc -l $k \| sort -r -n; done	Generates a list of traffic for all domains listed under a specific user (on a shared server). This command must be run in your /logs/ directory.

SHELL users on a VPS or Dedicated Server

The following commands only show sites under a single Shell user. If you need to view all site logs on your server, view the section below titled Admin users on a Dedicated Server.

Command	Description
[server]$ for k in `ls -S /home//logs//https/access.log`; do wc -l $k \| sort -r -n; done	Generates a list of all traffic for all domains under a single SHELL user. You can run this command from within any directory.
[server]$ tail -f -q /home//logs//https/access.log	Watches your server logs (under a single Shell user) in real-time to see if the issue presents itself with a specific IP (for intermittent issues). You can run this command from within any directory.

Admin users on a Dedicated Server

The following commands only work if you have an 'admin' user on your Dedicated Server. You must then switch to your 'root' user. Only then will you see all logs for all sites on your server.

How to switch to the 'root' user after logging in as an 'admin' user

The following steps detail how to log in as a 'root' user on your server AFTER logging in as your 'admin' user first.

Log into your server as your 'admin' user.

Run sudo su:

[server]$ sudo su
[sudo] password for exampleuser:

Enter your password for your 'admin' user.

You're now logged in as the 'root' user. You can see this in the command prompt:

root@ds123456#

After you're logged in as the 'root' user, run the following commands to view logs for all sites on your server.

Command	Description
[server]$ for k in `ls -S /home//logs//https/access.log`; do wc -l $k \| sort -r -n; done	Generates a list of all traffic for all domains on a Dedicated Server. You can run this command from within any directory.
[server]$ tail -f -q /home//logs//https/access.log	Watches your server logs in real-time to see if the issue presents itself with a specific IP (for intermittent issues). You can run this command from within any directory.

Troubleshooting

My Unique IP is making a lot of connections

You may find in your access.log that your site’s Unique IP is making a lot of connections. This is not an issue and can be safely ignored.

This occurs because Apache is internally generating these connections in order to shut down unneeded processes.

You can read more about it here.

I cannot access my logs

You may see the following error response after running the command to view all traffic for all domains on a server.

[server]$ for k in `ls -S /home/*/logs/*/https/access.log`; do wc -l $k | sort -r -n; done
ls: cannot access /home/*/logs/*/https/access.log: No such file or directory

This error occurs when you run the command as an SFTP user. You must run it as a 'Shell' user on a VPS or 'Shell' or 'Admin' user on a Dedicated Server.