Warning!
X-Ray is available out of the box for cPanel, Plesk, and DirectAdmin. For other control panels you should implement integration as described here
X-Ray is a tool developed for website performance monitoring and performance issues detection.
X-Ray can gather and visualize information about top N slowest system functions, external requests, software modules and database queries of the client’s website.
Make sure you have CloudLinux OS+ subscription (only non-reseller accounts apply)
Make sure you have installed LVE Manager version 6.2 or later. You can install or update it with the following commands:
# yum install lvemanager
# yum update lvemanager
X-Ray will be activated on all your servers during 4 hours. You will see the X-Ray tab in the LVE Manager UI.
For instant activation, run the following command:
# rhn_check
If the rhn_check
command is not found, run the following command:
# yum install rhn-check rhn-setup
Then install the alt-php-xray
package
Via user interface
Via SSH by running the following command:
# yum install lvemanager alt-php-xray
After installation, use the Start tracing button to create your first tracing task for a slow site.
X-Ray provides two options for monitoring domain requests speed: Tracing task and Continuous task.
Warning
To use Continuous task, update your LVE Manager and alt-PHP-X-Ray packages to versions lvemanager-6.2.9-1 and alt-php-xray-0.2-1 by running the following command:
yum update lvemanager alt-php-xray
Tracing task is a task created manually for a specific URL to collect server requests. The task will end either after a specified number of requests to the URL or after a specified time (maximum after two days). It is not possible here to automatically email a report but it is possible to export the report in PDF and send to a user.
Continuous task is a task that initiates a daily hourly tracing requests for a specified domain and email a monitoring report. Continuous task can't stop automatically, you need to stop it manually.
In fact, continuous task allows to automatically create a tracing task for each new day, with the ability to get a report for the past day.
The Tracing tasks tab contains a list of all tracing tasks created both manually and automatically via continuous tasks.
The Created column shows how a task was created – automatically (by continuous task) or manually.
Warning
To use Continuous task, update your LVE Manager and alt-PHP-X-Ray packages to versions lvemanager-6.2.9-1 and alt-php-xray-0.2-1 by running the following command:
yum update lvemanager alt-php-xray
The Continuous tracing tab contains a list of continuous tasks for which tracing tasks will be created automatically for a new day for a specific domain.
URL should be a valid URL of the domain which exists on the current hosting server. The URL field supports wildcard matching. To learn more about wildcard matching, click How to use special characters.
Advanced settings allow you to set an IP address and tracing options: by time or by number of queries.
Advanced settings
After creating, the task appears in the list of tracing tasks.
Tasks created Manually are simply tracing tasks.
A tracing task can have the following statuses:
Warning!
Collected requests are available in the UI for two weeks.
Click to open a list of collected requests.
The slowest request is highlighted.
There are filters for the request types and the indicator of a filter used now.
If slow requests were not detected during the tracing task, the following is displayed. Here, you can also view all requests.
X-Ray collects the following data for each request:
The Software modules/plugins section displays the following data:
The Database queries section displays the following data:
The External requests section displays the following data:
The System functions section displays the following data:
Click to stop the tracing task.
The tracing task status will be changed to Stopped. Data will not be collected anymore but you can see already collected information or continue tracing later by clicking .
Click to delete the tracing task.
Warning!
When you have deleted a tracing task, all collected data will be unavailable.
If you stop a continuous tracing task, a new task for the next 24 hours will not be created. The task for the current day will be finished at midnight and the report will be emailed.
If you delete a continuous tracing task, the task for the current day will be finished at midnight and the report will be emailed.
You can find a list of continuous tracing tasks in the Continuous tracing tab.
You can find automatically created tasks in the Tracing tasks tab marked as Automatically in the Created column.
The statuses for automatically created tasks are the same as for tracing task.
To view detailed info about an automatically created task, click . You will get requests grouped by hour.
Click to a group to open a list of the requests.
The following data is collected for each request:
Stopping automatic tracing task (a part of continuous tracing task) affects only the automatic tracing task for the current day. A new task for the next day will be created at the end of the day.
To stop the continuous tracing task completely, see Creating a new continuous task, paragraph 4.
Deleting automatic tracing task (a part of continuous tracing task) affects only the automatic tracing task for the current day. A new task for the next day will be created at the end of the day.
To delete the continuous tracing task completely, see Creating a new continuous task, paragraph 5.
Users get daily reports on their emails. An example of a report is shown below:
Click the link in the email to show the detailed report:
You can view requests grouped by hour:
You can also view the detailed information about request:
Warning
To use the end-user X-Ray plugin, update your LVE Manager and X-Ray packages to the lvemanager-6.3.9-1
(or later) and alt-php-xray-0.3-1
(or later) by running the following command:
# yum update lvemanager alt-php-xray
You can hide or show the end-user X-Ray plugin icon by ticking or unticking the proper checkbox in the LVE Manager.
Go to LVE Manager → Options Tab → User interface settings.
Note
The X-Ray plugin icon in the end-user interface is hidden when the checkbox is ticked.
The web interface of the end-user X-Ray plugin is almost the same as the X-Ray administrator interface.
But there are some differences and they are described further.
You can read about all other basic interface elements and managing tracing tasks in the Managing tracing task section.
Note
Tracing tasks created by an end-user will also be displayed in the administrator interface and administrators can manage the end-user's tasks the same way as they manage their own. At the same time, tasks created by the administrator or other end-users will not be displayed in the UI of the current user.
The end-user X-Ray plugin does not support creating continuous tasks.
The end-user has a limit of tracing tasks running at a time. Before starting the next task, the end-user should wait for the completion of the previous ones or forcefully stop the running ones. Otherwise, the user will get the next error:
Note
The current limit is one tracing task per user.
The administrator and the end-user can’t run the tracing task for the same Domain/URL at the same time. Once, the administrator started a specific tracing task, the end-user will not be able to duplicate it. And the same is true for the administrators – they will just see the running task for the specific domain and see the notification that they're trying to create a tracing task with a duplicated URL.
If continuous tracing is enabled for the domain, the end-user will not be able to create a new task for this domain because the same rule works - it will be a duplicate of the existing tracing tasks. The next warning will appear:
To solve this, the existing running tasks for the same Domain/URL should be stopped or completed. You can find more details about this in the FAQ.
If a user's tracing task was created for a domain which is using the FPM handler there's an additional limitation. To avoid frequent reloads of the particular FPM service, Start tracing , Stop tracing or Continue tracing action would be blocked in case if the latest reload of a corresponding FPM service was done less than 1 minute ago.
If a user gets such an error message - it means that 1 reload in 1 minute for a particular FPM service has been already done. Just try performing the same operation once again in a while.
The X-Ray automated throttling detection system checks if the account exceeds LVE limits by CPU during the HTTP request execution.
If CPU limiting was detected for a particular request, it is indicated in the X-Ray UI that the system itself has slowed down the request processing and this is apparently not a performance issue in the PHP code.
Requests with exceeded LVE limits are indicated in the administrator/user interface of the X-Ray plugin in the following way.
Requests with exceeded LVE limits are also marked if the administrator views the request.
Requests with exceeded LVE limits are marked in the PDF report as well.
X-Ray client is a PHP extension named xray.so
. It analyzes the processing time of the entire request and its parts and then sends the data to the X-Ray agent.
The list of currently supported PHP versions:
ALT PHP: | EA PHP: | Plesk PHP | DirectAdmin PHP | Other panels PHP |
|
|
|
|
|
Warning!
php-zts and custom PHPs, rolled in selector, are not supported
mysql_query
mysql_db_query
mysql_unbuffered_query
mysqli_query
mysqli::query
mysqli_multi_query
mysqli::multi_query
mysqli_real_query
mysqli::real_query
PDO::exec
PDO::query
PDOStatement::execute
It may be any PHP system function which can be related to a PHP engine or other PHP extension, for example fopen()
or json_encode()
. A list of these functions can be found here.
Syntax: xray.enabled=On/Off
Default: On
Changeable: PHP_INI_SYSTEM
Description: Enable or disable X-Ray extension from php.ini
Syntax: xray.database_queries=[number]
Default: 20
Changeable: PHP_INI_SYSTEM
Description: The number of the slowest SQL queries which will be sent to the X-Ray agent. The min value is 0 and the max value is 100. If the variable value is more, the default value will be used.
Syntax: xray.external_requests=[number]
Default: 20
Changeable: PHP_INI_SYSTEM
Description: The number of the slowest external requests (the curl_exec function) which will be sent to the X-Ray agent. The min value is 0 and the max value is 100. If the variable value is more, the default value will be used.
Syntax: xray.system_functions=[number]
Default: 20
Changeable: PHP_INI_SYSTEM
Description: The number of the slowest system functions which will be sent to the X-Ray agent. The min value is 0 and the max value is 100. If the variable value is more, the default value will be used.
Syntax: xray.backtrace_depth=[number]
Default: 10
Changeable: PHP_INI_SYSTEM
Description: The backtrace depth to the main() function which will be sent to the X-Ray agent. The min value is 0 and the max value is 20. If the variable value is more, the default value will be used.
Syntax: xray.processor=[processor_name]
Default: xray
Changeable: PHP_INI_SYSTEM
Description: Tells the X-Ray client which processor to use. The new processors may be added in the future. The default processor is xray which means to send data to the X-Ray agent.
Syntax: xray.tasks=host:uri:ip:id
Default: no value
Changeable: PHP_INI_SYSTEM
Description: The current tracing tasks for the given PHP request. This directive is added automatically by the X-Ray manager when creating a task. It is not allowed to edit manually, as X-Ray may stop working.
Syntax: xray.to_file=On/Off
Default: Off
Changeable: PHP_INI_SYSTEM
Description: Only for debug purposes. Writes to a file data which is sent to the processor.
Syntax: xray.debug=On/Off
Default: Off
Changeable: PHP_INI_SYSTEM
Description: Only for debug purposes. Enables debug output during request processing. In the On mode can slow down the domain.
Syntax: xray.debug_file=[path_to_file]
Default: /tmp/xray-debug.log
Changeable: PHP_INI_SYSTEM
Description: Only for debug purposes. Specifies a file for logging debug information.
This is a service that receives data from the X-Ray client and sends it to the remote storage.
The X-Ray agent is managed by the service
utility.
To start the X-Ray agent, run the following command:
# service xray-agent start
To stop the X-Ray agent, run the following command:
# service xray-agent stop
To restart the X-Ray agent, run the following command:
# service xray-agent restart
X-Ray affects website performance. Our tests show 5-10 % overhead from a website loading time with X-Ray tracing enabled. X-Ray allows you to find website performance issues and should not be enabled permanently. If your website is very slow, you can enable X-Ray to find the cause and then disable it.
This warning means that you already have a task to trace this URL in the list of your tracing tasks. If you see this warning, a new task can be created only with the On hold status and you will be able to run it only when the previous task with the same URL will be completed.
Note that the URL field supports wildcard matching and you can have a case when X-Ray is tracing the URL=domain.com/*
and you are trying to create a new task with URL=domain.com/xray.php
. In this case, you will see that warning because the *
URLs array includes xray.php
.
Check that xray extension is enabled for the domain. To do so, go to the phpinfo()
page and make a request. In the phpinfo output try to find the following section:
If you cannot see that section, try to restart PHP processes for that user (the simplest way is to restart Apache) and check that you can see the xray extension.
If you can see the xray extension in the phpinfo, check that X-Ray agent service is running with the service xray-agent status command. If it is not running, start it with the service xray-agent start
command.
X-Ray may not send data if a site uses a caching plugin, as the caching plugin is outputting HTML, thus there are no PHP scripts to examine. We encountered such issues with sites that use LSCache and WP Super Cache plugins. Check that your site does not use caching plugins. If so, disable it while tracing a site to get information from X-Ray.
If you set a client’s IP when creating the tracing task, check that your requests come to the server with this IP via phpinfo (since there may be NAT between your local machine and the server).
If, after checking the previous items, the issue persists, contact our support team.
If you managed to create a tracing task, this means that the xray.ini
file was created in a system. Therefore, there may be two reasons why it did not appear in the phpinfo page of the domain.
PHP process wasn't reloaded after adding the xray.ini. To solve this, you should restart the Apache or fpm service for the domain on which the tracing was started. At the moment, this is done automatically by the X-Ray manager after creating the task.
Your domain uses a PHP version different from the one which was detected by the X-Ray manager. To solve this, check the scan dir for additional ini files for your domain.
Then check the ini_location
that was passed to the X-Ray manager by running the following command:
# cat /usr/share/alt-php-xray/manager.log | grep ini_location
Find your tracing task in the output and check that the xray.ini
exists in this directory, also check that the ini
path is the same in the phpinfo page output and in the ini_location
directive for your tracing task. If they are the same, you should reload your PHP. If they are different that means that the X-Ray manager could not correctly determine the PHP version your domain uses. In this case, contact our support team at https://cloudlinux.zendesk.com/hc/requests/new.
Check for the CacheLookup on
option in the htaccess
file for your domain.
If the option is there, LiteSpeed processes requests bypassing the PHP X-Ray extension.
In this case, to get tracing information, you should remove the CacheLookup on
option.
All of the examples below are correct:
http://domain.com
http://domain.com/
https://domain.com
https://domain.com/
You can use any of them with a prefix www.
and it is also correct.
Required packages:
lvemanager
>= 6.2.10-1alt-php-xray
>= 0.2-1Centralized Monitoring is a tool that allows hosting administrators to monitor load for all their servers and users.
Centralized Monitoring allows you to:
Note
Make sure that cm.cloudlinux.com
is available on your end server.
yum install lve-utils
yum update lve-utils
# rhn_check
# /usr/share/cloudlinux/cl_plus/manage_clplus enable
rhn_check
command is not found, run the following command:# yum install/update rhn-check rhn-setup
Note
User statistics will be available only for users that were loaded starting from connecting the server to the Centralized Monitoring.
Users can monitor server’s or user’s load for a long time using the mode without session expired.
To turn on the mode without session expired, follow the next steps:
Log in to the cln.cloudlinux.com via your account
Open the cm.cloudlinux.com in a new browser tab/window (please, use the same browser as in step 1)
Use the toggle to turn on/off 10 min auto logout
Your session in the cln.cloudlinux.com will expire in 10 min. But your session in the cm.cloudlinux.com will not expire while your browser tab remains open.
You can access Centralized Monitoring in your CLN account. Click C-Monitoring in the left menu.
This page contains the list of all clients’ end servers. The server appears in the list after finishing Installation. By default, there is a descending sort by CPU usage.
The following values are available for each server:
Note
The values are calculated using a 15 min time period but the metric state is updated automatically every minute by default or you can choose from one of the predefined periods.
There is no pagination on the All servers page and all columns can be sorted by absolute value. Use the search tool to operate with the data.
To get the detailed statistics for the server via charts, click a desired server line in the table. All charts are auto-refreshed and there is an ability to select the period for metrics data to be updated for the chart.
Note
We store the metrics data for one month only.
Note
In the current version, we collect these metrics for the cPanel end servers only. We are planning to add other panels support soon.
Note
In the current version, we collect these metrics only for Apache (NOT for LiteSpeed, Nginx, etc.). The charts will be empty for LiteSpeed, Nginx, etc..
We calculate the user’s load by LVE statistics that we collect on the end server. The idle state for the user means that the LVE statistics were not collected for the last minute for some reason.
In each cell there are current usage/limit values for the basic LVE limits:
In the hint, there is a number of faults for each limit. The values in the columns are underlined (it is red if load-to-limit ratio >=90% and it is yellow if load-to-limit ratio >= 50%). For the current implementation, the only sort by the load-to-limit ratio is available. By default, there is a descending sort by the CPU usage column.
When sorting by a column, the lines with the load-to-limit ratio >=90% for this column will have the red background color, and lines with the load-to-limit ratio >=50% for this column will have the yellow background color.
Note
The users with unlimited resources (∞) will be at the bottom of the table.
This page contains all users for the all server of the client and their LVE statistics for the last minute. You can select the number of users on this page and search by user’s data.
The description of this page is the same as The most loaded server users for the last minute of the top 5 loaded users.
User’s metrics data can be sorted by the load-to-limit ratio and by the absolute value.
The absolute value is used to analyse the load produced by unlimited users.
The value of the load-to-limit ratio is convenient to use in the analysis of how many resources the users consume and whether they need to change the limits.
The values like this means that the resource is unlimited and 500.2 MB is the current usage of it.
Metrics data of Idle users is not used in the sorting, so such users always will be at the end of the list. The sorting can be done for only one metric.
Note
We store the metrics data for one month only.
On the user details page, the admin can find the charts for all LVE limits.
Alert Manager allows you to create a server or user alert for selected metrics and email the triggered events.
The Alert Manager page contains a table with the following:
Color Codes
To create a new alert, click the Create alert button.
Next, fill out the opened popup.
An admin can edit any field in the Alert except the Alert type.
The server alert is used to track the state of the whole server, it does not track user state on the server. The server alert tracks the next list of metrics:
eht0_receive
)eht0_transmit
)ehtN_receive
)ehtN_transmit
)mountpoint: <0>
)mountpoint: <1>
)mountpoint: <N>
)chip<0>
)sensor<0>
)chip<N>
)sensor<N>
)During creating a server alert an admin should select the type of metrics as the first step. The list of servers will be collected according to the availability of these metrics on the server.
For example, for now, we do not collect Apache metrics for non-cPanel servers, so you will get only cPanel servers as a list of servers for these metrics.
We're planning to implement support for other panels/web servers in the next releases.
Small limitation
We collect the server list according to having their statistics in our database (this behavior will be changed in the next releases).
For example, if server state is N/A or idle more than 24 hours, it will not be visible in the list for the alert.
The user alert tracks the next list of LVE metrics:
Small limitation
We collect the server list according to having their statistics in our database (this behavior will be changed in the next releases).
For example, if the user state is N/A or idle more than 24 hours, it will not be visible in the list for the alert.
In this two cases, you will not see the dropdown for selecting users because the metrics will track the server state.
This is the state of an alert that has been active for longer than the configured threshold duration.
Run this command:
/usr/share/cloudlinux/cl_plus/manage_clplus disable
You can find all your servers load in your CM personal account here: https://cm.cloudlinux.com/#/servers or in your CLN personal account here: https://cln.cloudlinux.com/console/cloudlinux/centralized-monitoring.
You can find all your users load in your CM personal account here: https://cm.cloudlinux.com/#/users or in your CLN personal account here: https://cln.cloudlinux.com/console/cloudlinux/centralized-monitoring
Click the desired server in the server list in the UI.
Click the desired user in the user list in the UI.
30 days.
Choose the desired period in the upper right corner or select it directly on the chart.
The user load chart contains three lines:
Limit and current load are drawing regarding the left vertical axis, the count of faults is drawing regarding the right vertical axis. You can focus on a particular line by clicking a required legend.
cl-end-server-tools
>= 1.0.7-1cl-node-exporter
>= 1.1.0-2rhn-client-tools
lve-stats
>= 3.0.7-2lve-utils
>= 4.2.21-2alt-python27-cllib
>= 2.1.13-1lvemanager
>= 6.2.10-1service cl_plus_sender status
/var/log/clplus_sender.log
You can view the events log on the client's server here:
/var/log/clplus_sender.log
Starting from end-server-tools-1.0.7, it supports collecting and sending statistics from the Apache and LiteSpeed web servers.
LiteSpeed is supported on cPanel and DirectAdmin control panels.
Each minute the statistics collection daemon checks which web server is started. If LiteSpeed is started, the daemon will collect data from it, otherwise, it checks if Apache is started.
When the daemon detects that the server is changed, it writes the following line into the statistics collection daemon log /var/log/clplus_sender.log
:
2020-10-09 17:25:31,462: (CL+_sender_daemon) [INFO] Apache/Litespeed collector: Using Apache
or
2020-10-09 18:13:03,897: (CL+_sender_daemon) [INFO] Apache/Litespeed collector: Using Litespeed
If the daemon can't detect either Apache or LiteSpeed, it writes to the log the following:
2020-10-09 17:33:38,399: (CL+_sender_daemon) [INFO] Apache/Litespeed collector: Apache or Litespeed stopped or absent, collector will not work
The statistics collection daemon reacts to the server changing automatically, no need to restart it.
Warning
Please note that the daemon checks the server type once in a minute, so the data sent on a minute of switching can be unreliable.
Starting from cl-end-server-tools
v.1.0.6-1, the statistics collection daemon allows to log data sent to pushgateway to its log /var/log/clplus_sender.log
.
To start logging, run the following command:
touch /var/lve/cmt_debug_logging
To stop logging, run the following command:
rm -f /var/lve/cmt_debug_logging
You don't need to restart the daemon after starting/stopping logging. The presence of a control file is evaluated "on the fly".
Warning
Use this logging with caution because when it is enabled, the size of the daemon log /var/log/clplus_sender.log
will increase each minute minimum on 3-4 KB. The actual increase size depends on the number of active users' processes on a server.