Viewing and examining your access.log via SSH

Overview

You can analyze your website traffic by running commands against your access.log over SSH — listing IP hits, finding the top files and directories, filtering by user agent, and watching traffic in real time.

To run the commands in this article, you must create a Shell user and log into your server via SSH.

How do I locate the logs directory?

[server]$ cd ~/logs/example.com/https

If you have added an SSL certificate to your website, you will see two folders. One for /http and one for /https. For example:

~/logs/example.com/http
~/logs/example.com/https

If you visit your site and see it's using HTTPS in the URL (https://example.com), you should view the /https directory. If your site displays HTTP when visited, you would view the /http directory instead. The following examples assume your site is using HTTPS.

DreamPress Plans Only

There are no http/s directories in the file path for DreamPress sites. Instead, you can locate DreamPress logs in the following directory:

[server]$ cd ~/logs/example.com/

Viewing access.log data

The following commands list various types of information from your website's access.log file.

Logs are rotated daily, which means there are only a few days worth at any time. If you need long-term logs, you must continually back them up. Per GDPR, DreamHost stores HTTP logs by default for 7 days.

How do I list IP hits?

List IPs

List IP addresses preceded by the number of times it hit a site.

[server]$ cat access.log| awk '{print $1}' | sort | uniq -c |sort -n

List the last 10,000 hits to a site.

[server]$ tail -10000 access.log| awk '{print $1}' | sort | uniq -c |sort -n

Watch logs in real-time

Watch your server logs in real-time. This can help identify an issue with a specific IP.

[server]$ tail -f -q access.log

VPS & Dedicated only

You can run this command from any directory on a VPS or Dedicated Server.

[server]$ tail -f -q /home/*/logs/*/https/access.log

View host of IP

The host command displays the hosting company from which a specific IP is originating. In this example, the IP belongs to Google.

[server]$ host 66.249.66.167
167.66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-167.googlebot.com

Block IPs

See this article for instructions on blocking an IP address using an .htaccess file.

How do I list the top files, directories, and domains?

Files and directories

List files and directories being hit the most on a site.

[server]$ awk '{print $7}' access.log|cut -d? -f1|sort|uniq -c|sort -nk1|tail -n10

Domain traffic

List traffic for all domains listed under a specific user (on a Shared Hosting server). This command must be run in your /logs/ directory.

[server]$ for k in `ls -S */https/access.log`; do wc -l $k | sort -r -n; done

VPS & Dedicated Only

List all traffic for all domains under a single user. You can run this command from any directory on a VPS or Dedicated Server.

[server]$ for k in `ls -S /home/*/logs/*/https/access.log`; do wc -l $k | sort -r -n; done

How do I filter logs by user agent?

Filter results by the User-Agent connecting. This can help identify bots/crawlers hitting a site. The example below shows an Amazon bot.

[server]$ awk -F\" '($2 ~ "^GET /"){print $6}' access.log | sort | uniq -c | sort -n
3 Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot) Chrome/119.0.6045.214 Safari/537.36

Dedicated Server commands

These commands display ALL logs for ALL sites on a Dedicated Server.

The following commands must be run under your Dedicated Server's 'root' user.

Switching to the 'root' user

Log into your server as your 'admin' user.

Run sudo su:

[server]$ sudo su
[sudo] password for exampleuser:

Enter your password for your 'admin' user. You are then logged in as the 'root' user, which can be confirmed by how the command prompt changes:
```
root@ds123456#
```

Running commands as the 'root' user

After you're logged in as the 'root' user, run the following commands to view logs for all sites on your server.

List all traffic

List all traffic for all domains. You can run this command from within any directory.

[server]$ for k in `ls -S /home/*/logs/*/https/access.log`; do wc -l $k | sort -r -n; done

Watch logs in real-time

Watch your server logs in real-time. This can help identify an issue with a specific IP. You can run this command from within any directory.

[server]$ tail -f -q /home/*/logs/*/https/access.log

Troubleshooting

My Unique IP is making a lot of connections

If you have added a Unique IP to your website, you may find it is making a lot of connections. This is not an issue and can be safely ignored, as the Apache server internally generates these connections to shut down unneeded processes. You can read more about it here.