Overview
This article explains how to use the Traffic tab in the panel to view and interpret bot metrics.
Background
Bots are automated programs designed to perform specific, repetitive tasks. These tasks can range from beneficial activities like indexing the internet for improved SEO, exemplified by Google's "crawlers", to malicious activities such as spamming, data theft, or executing Distributed Denial of Service (DDoS) attacks.
Detecting bots
DreamHost utilizes a combination of IP addresses and user agent information to detect bots on your website, the following section describes how each method is implemented:
- IP address: Gives an idea of where the visitor is coming from. A large number of requests from a single IP is often an indicator of a bot.
- User agent information: Helps to identify the software the visitor is using to access the website. Bots often use different software than human visitors, providing another clue to their identity.
Using these two pieces of information, DreamHost can more accurately determine whether a visitor is a bot or a human, helping to ensure your website's security and performance.
Understanding different types of bots
This section provides information on common bot types.
Social bots
-
Social bots can be seen as automated tools that can interact with your website's content and social media platforms, mimicking human behavior to execute specific tasks. These tasks can range from marketing and promoting content to engaging with users, spreading awareness about products or services.
-
They can interact by posting, commenting, liking, sharing, or messaging to achieve specific objectives, which can include marketing initiatives, promotional activities, and even disinformation campaigns. A common example would be Facebook bots.
AI bots
-
AI bots can browse a website to extract data. They can fetch various information like web pages, metadata, or specific content, and process it for subsequent uses such as forming replies or developing training datasets.
-
For instance, ChatGPT may gather content from diverse news websites to provide customized summaries to users.
Search engine bots
-
Search engine bots are automated software utilized by search engines to explore, analyze, and index online content. They examine web pages to assist in delivering pertinent search results.
-
Their primary functions include making web content searchable, ranking pages based on relevance and SEO parameters, and maintaining the freshness of search engine databases. A good example are Google bots.
Viewing the Bots Traffic page
Use the following steps to view the Bots Traffic graphs:
- Navigate to the Manage Websites page.
- Click the Manage button to open the Domain Settings page, which allows you to adjust various settings for your site.
- Click the Traffic tab.
-
The Traffic page opens and automatically displays a visitor's graph. To switch to Bots graphs, click on Visitors at the top of the page and select Bots.
-
Analyzing bots traffic graphs
This section provides details on the different bot traffic graphs available.
Recent Traffic
-
Displays unique and total bot counts over time. Click the View More button for an extended view. Once on this page, you can choose to view the 'Last Month', 'Last 7 days', or 'Last 24 hours' reports.
Bot Type Traffic
-
This graph allows you to view and rank bot traffic based on type. By default, the system displays the top 5 Search Engine Bots. However, you can modify the view by selecting either 'Social Bots' or 'AI Bots' from the dropdown menu. Click the View More button for an expanded list of the top 10 bots.
Page Views
-
This graph allows you to analyze bot visit trends on your most viewed pages. You can switch between list and graphic modes. Details for specific pages are displayed when you hover over the graphic. It initially shows the top 8 page visits. Click the View More button to view up to the top 100.
IP Addresses
-
This graph provides a list of recognized bot IP addresses along with their corresponding services or organizations. By default, the top 8 IP addresses are displayed. Click the View More button for a complete list of the top 100.
-
To block or control the frequency of bot visits to your site, you can utilize a robots.txt file. For details to control bots, spiders, and crawlers, see this article.
Visitors By Location
-
This graph provides information on the global distribution of bot traffic and the main source countries. Click the View More button for an in-depth analysis, including region-specific data and an extended list of countries.