LVE-Stats 2

Why is it needed?

  • Old LVE-statistics store averages as integer numbers, as % of CPU usage. If user used 100% of CPU for 1 second within an hour, it is only 1-2% for a minute, and 0 for 5 minutes. Data in old LVE-statistics is aggregated to 1-hour intervals. So, such peak load will not be recorded and we need to store data with much higher precision.
  • 100% CPU usage in old lve statistics means “all cores”. On 32 core servers usage is not visible for most users (as they are limited to 1 core).
  • Old LVE-statistics does not provide a way to determine a cause of LVE faults, i.e. what processes are running when user hits LVE limits.
  • Notifications in old LVE-statistics are not accurate because they are based on average values for CPU, IO, IOPS.
  • Old LVE-statistics functionality is hard to extend.

Major improvements and features

  • increased precision of statistics;
  • CPU usage is calculated  in terms of % of a single core (100% usage means one core);
  • lvestats-server emulates and tracks faults for CPU, IO, IOPS;
  • lvestats-server saves “snapshots” of user’s processes and queries for each “incident” - added new lve-read-snapshot utility;
  • improved notifications about hitting LVE limits (more informative and without false positives);
  • implemented ability to add custom plugins;
  • MySQL and PostGreSQL support;
  • more pretty, scalable, interactive charts;
  • snapshots include HTTP-requests.

What features will be implemented in the future?

  • Notifications for control panels other than CPanel.
  • Burstable Limits/server health: We are monitoring server health ( LA , memory , idle CPU ) and automatically decreasing/increasing limits based on server health.
  • Reseller Limits: plugin would analyze usage per group of users (reseller’s usage), and do actions.
  • Suspend/notify plugin: would detect that user is being throttled for 10 minutes, and suspend him (just because), or notify, or increase limits.

Installation

To install, please execute:

yum install lve-stats

To update:

yum update lve-stats

Settings of old lve-stats (ver. 0.x) are imported automatically on the first install/update of a new lve-stats package.

SQLite database file is located in /var/lve/lvestats2.db, data from old lve-stats (ver. 0.x) are being migrated automatically in the background. Migrating process can last 2-8 hours (during this time lags are possible when admin is trying to check statistics, at the same time users will not be affected). Migrating the latest 30 days, SQLite DB stable migration is provided.

Currently, the new lve-stats supports all databases available in CloudLinux.

Downgrade

If you have any problems after update, downgrade lve-stats2 to the previous stable version by running:

yum downgrade lve-stats

and contact CloudLinux support at https://cloudlinux.zendesk.com/hc/requests/new

Note

You may need to rename *.rpmsave files to original ones in order to restore settings for old lve-stats (/etc/sysconfig/lvestats, /etc/sysconfig/cloudlinux-notify).

Configuration

Main configuration file /etc/sysconfig/lvestats2 contains the following options:

  • db_type - selects appropriate database type to use;

  • connect-string - connection string for PostGreSQL and MySQL database, has the following form:

    connect_string = USER:PASSWORD@HOST[:PORT]/DATABASE
    

    Default port is used for specific database, if port is not specified (typical port is 3306 for MySQL and 5432 for PostGreSQL). Connection string is not used for sqlite database.

  • server_id - sets the name of the server (at most 10 characters). This option is to use with centralized database ( PostGreSQL or MySQL). For use with sqlite database, value of this option should be "localhost" (without quotes).

  • plugins – path to directory containing custom plugins for lve-stats (default path /usr/share/lve-stats/plugins).

  • db_timeout - period of time to write data to database (in seconds); default value is 60 seconds.

  • timeout - timeout for custom plugins (seconds). If plugin execution does not finish within this period, plugin is terminated. Default value is 5 seconds.

  • interval - duration of one cycle of lvestats-server (seconds). This should be less than total duration of execution of all plugins. Default value is 5 seconds. Increasing this parameter makes precision of statistics worse.

  • keep_history_days - period of time (in days) to keep history in database. Old data is removed from database automatically. Default value is 60 days.

  • mode – sets compatibility output mode (compatibility with older lveinfo version

    • Value v1 enables compatibility with old version of lveinfo.
    • Value v2 enables extended output mode, but can break LVE plugins for control panels (statistics in LVE Manager, Resource Usage, etc). Support of v2 mode will be added to LVE plugins in the recent future. When mode parameter is absent, later version of lveinfo is implied.
  • disable_snapshots - disable snapshots and incidents. Possible values:

    • true
    • false

Configuration files for plugins are located in /etc/sysconfig/lvestats.config directory.

/etc/sysconfig/lvestats.config/SnapshotSaver.cfg contains the following options:

  • period_between_incidents - the minimal interval of time between incidents (in seconds). If minimal interval of time between LVE faults is greater than value specified, than new "incident" will begin and new snapshots will be saved. Default value is 300 seconds.
  • snapshots_per_minute - the maximum number of snapshots saved per minute for specific LVE id (default is 2).
  • max_snapshots_per_incident - the maximum number of snapshots saved for one "incident". Default is 10.

/etc/sysconfig/lvestats.config/StatsNotifier.cfg contains the following options:

  • NOTIFY_ADMIN – enables notification for admin (Y/N, default N);
  • NOTIFY_RESELLER – enables notification for reseller (Y/N, default N);
  • NOTIFY_CUSTOMER - enables notification for customers (Y/N, default N);
  • NOTIFY_INCLUDE_RESELLER_CUSTOMERY=notify all users, N=notify only hoster's users (whos reseller is root), default = N;
  • NOTIFY_CPU – notify about CPU faults when customer hits 100% of his CPU limit (Y/N, default N);
  • NOTIFY_IO - notify about IO faults when customer hits 100% of his IO limit (Y/N, default N);
  • NOTIFY_IOPS - notify about IOPS faults when customer hits 100% of his IOPS limit (Y/N, default N);
  • NOTIFY_MEMORY - notify about memory faults (Y/N, default N);
  • NOTIFY_EP – notify about entry processes faults (Y/N, default N);
  • NOTIFY_NPROC – notify about number of processes faults (Y/N, default N);
  • NOTIFY_MIN_FAULTS_ADMIN – minimum number of faults to notify admin (default 1);
  • NOTIFY_MIN_FAULTS_USER – minimum number of faults to notify customer (default 1);
  • NOTIFY_INTERVAL_ADMIN – period of time to notify admin (default 12h);
  • NOTIFY_INTERVAL_USER – period of time to notify customer (default 12h);
  • NOTIFY_FROM_EMAIL - sender email address. For example: NOTIFY_FROM_EMAIL=main_admin@host.com;
  • NOTIFY_FROM_SUBJECT - email message subject. For example: NOTIFY_FROM_SUBJECT=Message from notifier

These values can also be set using cloudlinux-config CLI utility

Templates of notifications are located here:

  • /usr/share/lve/emails/en_US/admin_notify.txt
  • /usr/share/lve/emails/en_US/reseller_notify.txt
  • /usr/share/lve/emails/en_US/user_notify.txt
  • /usr/share/lve/emails/en_US/admin_notify.html
  • /usr/share/lve/emails/en_US/reseller_notify.html

Note

Notifications about LVE faults are implemented for CPanel only.

After changing any options above, please restart lvestats service:

service lvestats restart

/etc/logrotate.d/lvestats - configuration file for /var/log/lve-stats.log rotation

LVE Stats2 and MySQL DB Server Compatible Work Setup

Note

Run all the commands below under root.

1. MySQL Server Setup

If MySQL Server is not installed, then install it according to control panel documentation.

For non-panel system:

  • CloudLinux 6

    yum install mysql mysql-server
    service mysqld start
    chkconfig mysqld on
    
  • CloudLinux 7

    yum install mariadb mariadb-server
    systemctl start mariadb.service
    systemctl enable mariadb.service
    

2. Database Setup

  • Run MySQL administrative utility: mysql.

  • In utility run the commands:

    • creating server DB. Also, check Note below.
    CREATE DATABASE db_lvestats2;
    
    • creating a user for LVE Stats 2 server to work under. Also, check Note below.
    CREATE USER 'lvestats2'@'localhost' IDENTIFIED BY 'lvestats2_passwd';
    
    • granting all the privileges for all DB tables to the user. Use the username and DB name from the points above.
    GRANT ALL PRIVILEGES ON db_lvestats2.* TO 'lvestats2'@'localhost';
    
    • refreshing privileges information.
    FLUSH PRIVILEGES;
    
    • Exit administrative utility (Ctrl+d).

Note

DB name, username and their passwords above are given for an example - you can use any of your choices. Using old DB from LVE Stats version 1 is also acceptable as LVE Stats2 uses different tables and the old information will not be corrupted.

3. LVE Stats 2 Setup

  • Stop LVE Stats 2 server by running the command:
service lvestats stop
  • In server configuration file /etc/sysconfig/lvestats2, edit the following options:
    • db_type = mysql
    • connect_string = lvestats2:lvestats2_passwd@localhost/db_lvestats2

Note

connect_string option value is used in format: user:pass@host/database. Username, password and DB name must be the same as in point 2 of Database Setup above.

  • After making changes in configuration files run:
/usr/sbin/lve-create-db 

For DB primary initialization (creating tables, indexes, etc). There is no need to create anything in the DB manually.

  • When done, restart server by running:
service lvestats restart

4. Additional Security Settings

If you need to provide access to LVE Stats information utilities (lveinfo, lvechart, lve-read-snapshot) for different users, then we recommend creating one more DB user with read-only privilege to guarantee information security. It can be done by running the following commands in administrative utility:

  • creating a user (check Note above)
CREATE USER 'lvestats2_read'@'localhost' IDENTIFIED BY 'lvestats2_read_passwd';
  • granting read-only privilege to the user
GRANT SELECT ON db_lvestats2.* TO 'lvestats2_read'@'localhost';
  • refreshing privileges information
FLUSH PRIVILEGES;

If LVE Stats2 server is set correctly (see information below), the information utilities will work under this user.

If you need to provide access to information utilities to other users, then in order to guarantee information security you should do the following:

  • Assign permission 600 to the main configuration file (/etc/sysconfig/lvestats2), so that it could be read only by LVE Stats 2 server and by utilities that run under root.
  • Copy /etc/sysconfig/lvestats2 to /etc/sysconfig/lvestats2.readonly, assign permission 644 to the new file, so that it could be read by any user but could only be changed by root.
  • In /etc/sysconfig/lvestats2.readonly file, in the line connect_string, specify DB user with read-only permission, created above.

These steps allow hiding main DB user username/password from other system users.

If there is no need in such access differentiation, then /etc/sysconfig/lvestats2 file access permission should be 644, so that it could be read by all users and could be changed only by root.

5. Using Special Characters in Database Password

Since scheme ://user:password@host[:port]/database_name URI is used in connect_string config option, then usage of special characters in user DB password is not allowed.

To use special symbols in the password, it must be converted to escape-sequence. You can convert a password to escape-sequence in a console as follows:

echo -n '[You_P@$$]:' | perl -MURI::Escape -ne 'print uri_escape($_)."\n"'
%5BYou_P%40%24%24%5D%3A

Or replace the symbols manually:

!    #    $    &    '    (    )    *    +    ,    /    :    ;    =    ?    @   [    ]
%21  %23  %24  %26  %27  %28  %29  %2A  %2B  %2C  %2F  %3A  %3B  %3D  %3F  %40  %5B %5D

After that сonnect_string will look as follows:

сonnect_string=lvestats2:%5BYou_P%40%24%24%5D%3A@localhost/db_lvestats2

LVE Stats 2 and PostgreSQL DB Server Compatible Work Setup

Note

Run all the commands below under root.

1. PostgreSQL Server Installation and Setup

  • PostgreSQL installation and initialization

    For control panels use proper documentation for installation on the links: сPanel, Plesk.

    For non-panel CloudLinux run the following commands:

    • CloudLinux 6
    yum install postgresql-server postgresql
    service postgresql initdb
    service postgresql start
    chkconfig postgresql on
    
    • CloudLinux 7
    yum install postgresql-server postgresql
    postgresql-setup initdb
    systemctl start postgresql
    systemctl enable postgresql
    
  • Setup

    • In /var/lib/pgsql/data/pg_hba.conf config file change user authentication mode. Add the following lines (place before all other authentication parameters):

      # IPv4 local connections for lve-stats-2.x
      host dblvestat all 127.0.0.1/32 password
      # IPv6 local connections for lve-stats-2.x
      host dblvestat all ::1/128 password
      

      These lines enable user authentication by the password for IP4/IP6 connections. You can set other modes if needed.

    • Apply config changes by running:

      service postgresql restart
      

2. DB for lve-stats-2.x - Creating and Setup

  • Run standard PostgreSQL psql administrative utility:
sudo -u postgres psql postgres 

OR for сPanel

psql -w -U postgres
  • In utility run:

    • creating server DB. Also, check Note below.

      CREATE DATABASE dblvestat;
      
    • creating a user for LVE Stats 2 server to work under. Also, check Note below.

      CREATE USER lvestat WITH password 'passw';
      
    • granting lvestat user all privileges for work with dblvestat DB.

      GRANT ALL privileges ON DATABASE dblvestat TO lvestat;
      
    • exit psql utility:

      \q
      

      OR alternatively:

      Ctrl+d
      

Note

DB name, username and their passwords above are given for an example - you can use any of your choices. Using old DB from LVE Stats version 1 is also acceptable as LVE Stats 2 uses different tables and the old information will not be corrupted

3. Lve-stats-2.x Setup

  • Stop lve-stats2 server by running:
service lvestats stop
  • In server config file /etc/sysconfig/lvestats2 edit options for connecting to DB:
db_type = postgresql
connect_string=lvestat:passw@localhost/dblvestat
If DB is going to be used as centralized for multiple hosts then collect_usernames parameter must be changed:
collect_usernames=true

Note

connect_string option value is of the format: user:pass@host/database. Username, password and DB name must be the same as in Database Setup section above.

  • After making changes in configuration files, for DB primary initialization (creating tables, indexes, etc), run:
/usr/sbin/lve-create-db 
  • There is no need to create anything in the DB manually. When done, restart server by running:
service lvestats restart

4. Additional Security Settings

If you need to provide access to LVE Stats information utilities (lveinfo, lve-read-snapshot ) for other users (or if CageFS is disabled), then in order to guarantee DB security the following steps are required:

  • Create a DB user with read-only permission:
CREATE USER lvestat_read WITH password 'passw';
GRANT CONNECT ON DATABASE dblvestat to lvestat_read;
\connect dblvestat;
GRANT SELECT ON lve_stats2_history, lve_stats2_history_gov, lve_stats2_history_x60, lve_stats2_incident, lve_stats2_servers, lve_stats2_snapshot, lve_stats2_user TO lvestat_read;
  • Assign root ownership and permission 600 to the main configuration file (/etc/sysconfig/lvestats2), so that it could be read only by LVE Stats 2 server and by utilities that run under root.

  • Copy /etc/sysconfig/lvestats2 to /etc/sysconfig/lvestats2.readonly, assign permission 644 to the new file, so that it could be read by any user but could be changed only by root.

  • In /etc/sysconfig/lvestats2.readonly file, in the line connect_string, specify DB user with read-only permission, created above.

    These steps allow hiding main DB user username/password from other system users.

    If there is no need in such access differentiation, then /etc/sysconfig/lvestats2 file access permission should be 644, so that it could be read by all users and could be changed only by root.

  • When done, restart server by running:

service lvestats restart

5. Using Special Characters in Database Password

Since scheme ://user:password@host[:port]/database_name URI is used in connect_string config option, then usage of special characters in user DB password is not allowed.

To use special symbols in the password, it must be converted to escape-sequence. You can convert a password to escape-sequence in a console as follows:

echo -n '[You_P@$$]:' | perl -MURI::Escape -ne 'print uri_escape($_)."\n"'
%5BYou_P%40%24%24%5D%3A

OR replace the symbols manually:

!    #    $    &    '    (    )    *    +    ,    /    :    ;    =    ?    @    [    ]
%21  %23  %24  %26  %27  %28  %29  %2A  %2B  %2C  %2F  %3A  %3B  %3D  %3F  %40  %5B  %5D

After that сonnect_string will look as follows:

сonnect_string=lvestats2:%5BYou_P%40%24%24%5D%3A@localhost/db_lvestats2

Customize lve-stats-2 notifications

Jinja2 is used as a template engine for the notifications.

The templates for notifications are located in /usr/share/lve/emails/LOCALE, where LOCALE is the directory with localization name (language codes are formed according to ISO 639-1 and ISO 639-2).

By default the templates for English are set: /usr/share/lve/emails/en_US..

/usr/share/lve/emails/en_US contains the following templates:

  • admin_notify.html admin_notify.txt for administrator;
  • reseller_notify.html reseller_notify.txt for reseller;
  • user_notify.txt for user.

The notification is formed as Multipart content type RFC1341(MIME).

The plain text is taken from the .txt files, html version - from the .html template.

In case when only one template is present (.txt or .html) the notification is sent as a Non-multipart content type notification.

It is better to use Multipart content type notifications because when a mail client can not display an html-format message, then it will be displayed as plain text version.

To localize notifications, copy standard templates into directory with the proper locale name and translate the template. Also you can customize the main template making proper changes into it.

The list of variables that can be used in the template:

Variable Example Description
TONAME Customer Notification receiver user name. Taken from profile in the control panel, by default - Customer for user, Administrator for administrator, Reseller for reseller.
TOMAIL support@cloudlinux.com Notification receiver email address.
DOMAIN wordpress.test247.cloudlinux.com Main domain. Available only for user.
LOCALE en_US Locale in which the notification is sent. Available only for user.
RESELLER root User reseller. Available only for user.
PERIOD 12 hours Verification and notification sending period.
LOGIN wordpress User login in the system.
ID 500 User ID in the system.
lPMem lEP PMemF lVMem anyF IOf VMemF lCPU aIOPS aEP aPMem IOPSf lIO lIOPS aIO EPf aCPU aVMem NprocF aNproc lNproc CPUf See description in lveinfo --help output. Available only for users
STATS_HTML html table with the list of users that exceeded limits. Available for administrator and reseller.
STATS ASCII - table with the list of users that exceeded limits. Available only for admins and resellers.

Sender’s email address by default is administrator email address from control panel settings (root@{hostn_name} if there is no email in the control panel).

It can be changed with NOTIFY_FROM_EMAIL option in the config /etc/sysconfig/lvestats.config/StatsNotifier.cfg

For example:

NOTIFY_FROM_EMAIL=support@hostername.com

To apply changes restart lve-stats service:

service lvestats restart

for CloudLinux 7

systemctl restart lvestats.service

Default subject is Hosting account resources exceeded.  It can be changed for each template (and for localized templates as well). To change subject, in the very beginning of the file (no blank lines allowed in the beginning of the template) add the field Subject:, leave two blank lines after it and add template body.

Customized subjects can be taken only from the templates with the resolution *.txt (admin_notify.txt, reseller_notify.txt, user_notify.txt). Changes apply without lvestats restart.

For backward compatibility the subject can be also changed with the key NOTIFY_FROM_SUBJECT in the config /etc/sysconfig/lvestats.config/StatsNotifier.cfg.

Customized subjects have the higher priority than the key NOTIFY_FROM_SUBJECT.

Example for the file user_notify.txt

Subject: Customized subject example
Dear {{TONAME}},

 
Your {{DOMAIN}} web hosting account exceeded one or more of its resources within the last {{PERIOD}}.
{% if epf %}Exceeded the maximum of {{lep}} concurrent website connections. Your website was not available {{epf}} times because of this problem.
{% endif %}{% if pmemf %}Exceeded the physical memory limit of {{lpmem}}KB. Your website was not available {{pmemf}} times because of this problem.
{% endif %}{% if vmemf %}Exceeded the virtual memory limit of {{lvmem}}KB. Your website was not available {{vmemf}} times because of this problem.
{% endif %}{% if nprocf %}Exceeded the number of processes limit of {{lnproc}}. Your website was not available {{nprocf}} times because of this problem.
{% endif %}{% if cpuf %}You reached limit of {{lcpu}} of total server CPU usage {{cpuf}} times. Your website was forced to load slower to reduce its CPU usage.
{% endif %}{% if iof %}You reached limit of {{lio}}KB/s disk io rate {{iof}} times. The disk io speed for your account was slowed as a result of this problem.
{% endif %}{% if iopsf %}You reached limit of {{liops}} I/O operations {{iopsf}} times. The disk io speed for your account was slowed as a result of this problem.
{% endif %}
 
To view full details about your web hosting account's resource usage, including the time of each incident listed above, please click the link below and log into your cpanel hosting control panel, then click the "Resource Usage" link under the "Logs and Statistics" section.
http://{{DOMAIN}}:2083
 
If your account is regularly exceeding it's available resources, please consider upgrading to a higher level hosting plan that includes more resources. If you have any questions or need help with anything, just reply to this email and let us know.
 
Sincerely,
 
Your Friendly Web Hosting Support Team

Command-line Tools

/usr/sbin/lveinfo utility to display historical information about LVE usage.
/usr/sbin/lvechart creates a chart representing LVE usage for user.
/usr/sbin/dbgovchart creates a chart representing MySQL usage for user.
/usr/sbin/lve-read-snapshot displays information from system state (snapshots) for user.
/usr/sbin/lve-create-db creates/recreates database for lve-stats.
/usr/sbin/cloudlinux-top utility provides information about current MySQL and LVE usage of a running system in JSON format.
/usr/sbin/cloudlinux-statistics utility provides historical information about resource usage.

lveinfo

Note

lve-stats-2.2-2

Usage

lveinfo [-h] [-v] [--dbgov DBGOV] [-f YYYY-MM-DD[HH:MM]]

              [-t YYYY-MM-DD[HH:MM]] [--period PERIOD] [-u USER | --id ID]

              [-d] [-o ALIAS] [-b ALIAS [ALIAS ...]] [-p 0..100]

              [--by-fault ALIAS [ALIAS ...]] [-r FAULTS]

              [--style {user,admin}] [-l LIMIT] [-c [PATH] | -j]

              [--server_id SERVER_ID] [--servers-info]

              [--show-all | --show-columns COLUMN_NAME [COLUMN_NAME ...]]

              [--time-unit TIME_UNIT] [-m {v1,v2}]

              [--blank-value [BLANK_VALUE]]

lveinfo is an utility to display historical information about LVE usage.

Optional arguments

  • -h, --help – show this help message and exit

  • -v, --version – show program's version number and exit

  • --dbgov DBGOV – show MySql Governor statistic

  • -u USER, --user USER – use username instead of LVE id, and show only record for that user

  • --id ID – will display record only for that LVE id

  • -d, --display-username – try to convert LVE id into username when possible

  • -o ALIAS, --order-by ALIAS – orders results by one of the following:

    ALIAS ALIAS DESCRIPTION
    cpu_avg aCPU average CPU usage
    cpu_max mCPU max CPU usage
    total_cpu_faults CPUf total number of max CPU usage faults
    vmem_avg aVMem average virtual memory usage
    mep_avg aEP average number of entry processes (concurrent connections)
    mep_max mEP max number of entry processes (concurrent connections)
    total_ep_faults EPf total number of max entry processes faults
    pmem_avg aPMem average physical memory usage (LVE version >= 6)
    pmem_max mPMem max physical memory usage (LVE version >= 6)
    nproc_avg aNproc average number of processes (LVE version >= 6)
    nproc_max mNproc max number of processes (LVE version >= 6)
    io_avg aIO average io usage (LVE version >= 6)
    io_max mIO max io usage (LVE version >= 6)
    total_pmem_faults PMemF total number of out of physical memory faults (LVE version >= 6)
    total_nproc_faults NprocF total number of max processes faults (LVE version >= 6)
    total_io_faults IOf total number of max io faults (LVE version >= 6)
    iops_avg aIOPS average io operations (LVE version >= 8)
    iops_max mIOPS max io operations (LVE version >= 8)
    total_iops_faults IOPSf total number of max io operations faults (LVE version >= 8)
    any_faults anyF total number of faults of all types
  • -b ALIAS [ALIAS ...] --by-usage ALIAS [ALIAS ...] – show LVEs with usage (averaged) within 90 percent of the limit available values:

    ALIAS ALIAS ALIAS DESCRIPTION
    cpu_avg cpu aCPU average CPU usage
    cpu_max cpu_max mCPU max CPU usage
    vmem_avg vmem aVMem average virtual memory usage
    vmem_max vmem_max mVMem max virtual memory usage
    mep_avg mep aEP average number of entry processes (concurrent connections)
    mep_max mep_max mEP max number of entry processes (concurrent connections)
    pmem_avg pmem aPMem average physical memory usage (LVE version >= 6)
    pmem_max pmem_max mPMem max physical memory usage (LVE version >= 6)
    nproc_avg nproc aNproc average number of processes (LVE version >= 6)
    nproc_max nproc_max mNproc max number of processes (LVE version >= 6)
    io_avg io aIO average io usage (LVE version >= 6)
    io_max io_max mIO max io usage (LVE version >= 6)
    iops_avg iops aIOPS average io operations (LVE version >= 8)
    iops_max iops_max mIOPS max io operations (LVE version >= 8)
  • -p 0..100, --percentage 0..100 – defines percentage for --by-usage option; default 90

  • --style {user,admin} – deprecated, not used.

  • -l LIMIT, --limit LIMIT – max number of results to display, 10 by default, if 0 no limit

  • -c [PATH], --csv [PATH] – save statistics in CSV format; - by default (output to screen)

  • -j, --json – display output in JSON format

  • --server_id SERVER_ID – used with central database for multiple servers, default localhost

  • --servers-info – show servers LVE versions

  • --show-all – full output (show all limits); brief output is default; equivalent --show-columns all

  • -show-columns COLUMN_NAME [COLUMN_NAME ...] – show only the listed columns; all for all supported columns

    COLUMN_NAME DESCRIPTION
    From Show start period statistics
    To Show end period statistics
    ID LVE Id or username
    aCPU Average CPU usage
    uCPU The percentage of user-allocated resource CPU
    mCPU deprecated
    lCPU CPU limit
    CPUf Out Of CPU usage Faults
    aEP Average Entry Processes
    uEP The percentage of user-allocated resource Entry processes
    mEP deprecated
    lEP maxEntryProc limit
    aVMem Average Virtual Memory Usage
    uVMem The percentage of user-allocated resource Virtual Memory
    mVMem deprecated
    lVMem Virtual Memory Limit
    VMemF Out Of Memory Faults
    EPf Entry processes faults
    aPMem Average Physical Memory Usage (LVE version >= 6)
    uPMem The percentage of user-allocated resource Physical Memory (LVE version >= 6)
    mPMem deprecated (LVE version >= 6)
    lPMem Physical Memory Limit (LVE version >= 6)
    aNproc Average Number of processes (LVE version >= 6)
    uNproc The percentage of user-allocated resource Number of processes (LVE version >= 6)
    mNproc deprecated (LVE version >= 6)
    lNproc Limit of Number of processes (LVE version >= 6)
    PMemF Out Of Physical Memory Faults (LVE version >= 6)
    NprocF Number of processes faults (LVE version >= 6)
    aIO Average I/O (LVE version >= 6)
    uIO The percentage of user-allocated resource I/O (LVE version >= 6)
    mIO deprecated (LVE version >= 6)
    lIO I/O Limit (LVE version >= 6)
    IOf Out Of I/O usage Faults (LVE version >= 6)
    aIOPS Average I/O Operations (LVE version >= 8)
    mIOPS deprecated (LVE version >= 8)
    uIOPS The percentage of user-allocated resource I/O Operations (LVE version >= 8)
    lIOPS I/O Operations Limit (LVE version >= 8)
    IOPSf Out Of I/O Operations Faults (LVE version >= 8)
  • --time-unit TIME_UNIT – time step for grouping statistic in minutes; 1 min., by default; can use m\|h\|d suffixes; for example: 1h or 1h30m or 1d12h

  • -m {v1,v2}, --compat {v1,v2}v1 - return old output mode; v2 - new mode; default v1; you can change default in config

  • --blank-value [BLANK_VALUE] – Use to fill unsupported limits; default -

  • -f YYYY-MM-DD[ HH:MM], --from YYYY-MM-DD[ HH:MM] – run report from date and time in [YY]YY-MM-DD[ HH:MM] format; if not present last 10 minutes are assumed

  • -t YYYY-MM-DD[ HH:MM], --to YYYY-MM-DD[ HH:MM] – run report up to date and time in [YY]YY-MM-DD[ HH:MM] format; if not present, reports results up to now

  • --period PERIOD – time period; specify minutes with m, h - hours, days with d, and values: today, yesterday; 5m - last 5 minutes, 4h - last four hours, 2d - last 2 days, as well as today

  • --by-fault ALIAS [ALIAS ...] – show LVEs which failed on max processes limit or memory limit

    ALIAS ALIAS ALIAS DESCRIPTION
    mcpu cpu CPUf total number of max CPU usage faults
    mem vmem VMemF total number of out of virtual memory faults
    mep ep EPf total number of max entry processes faults
    pmem pmem PMemF total number of out of physical memory faults (LVE version >= 6)
    nproc nproc NprocF total number of max processes faults (LVE version >= 6)
    io io IOf total number of max io faults (LVE version >= 6)
    iops iops IOPSf total number of max io operations faults (LVE version >= 8)
    any_faults any anyF total number of faults of all types
  • -r FAULTS, --threshold FAULTS– in combination with --by-fault, shows only LVEs with number of faults above; default 1

Prefixes Kb, Mb and Gb indicates powers of 1024.

Note

All ALIAS options are not case sensitive.

lvechart

/usr/sbin/lvechart - creates a chart representing LVE usage for user.

Usage

/usr/sbin/lvechart [OPTIONS]

Acceptable options

--help This help screen
--version Version number
--from Run report from date and time in YYYY-MM-DD HH:MM format (if not present, last 10 minutes are assumed)
--to= Run report up to date and time in YYYY-MM-DD HH:MM format (if not present, reports results up to now)
--period= Time period: specify minutes with m, h - hours, days with d, and values: today, yesterday; 5m - last 5 minutes, 4h - last four hours, 2d - last 2 days, as well as today
--id= LVE id will display record only for that LVE id
--user= Use username instead of LVE id , and show only record for that user
--server= Server id will display record for that server, instead of default (current)
--output= Filename to save chart as, if not present, output will be sent to STDOUT
--show-all Show all graphs (by default shows graphs for which limits are set)
--style= admin, user set chart style, CPU limits are normalized to 100% in user’s style
--format= svg, png set chart output format

dbgovchart

/usr/sbin/dbgovchart - creates a chart representing MySQL usage for user.`

Usage

/usr/sbin/dbgovchart [OPTIONS]

Acceptable options

--help This help screen
--version Version number
--from= Run report from date and time in YYYY-MM-DD HH:MM format (if not present, last 10 minutes are assumed)
--to= Run report up to date and time in YYYY-MM-DD HH:MM format (if not present, reports results up to now)
--period= Time period: specify minutes with m, h - hours, days with d, and values: today, yesterday; 5m - last 5 minutes, 4h - last four hours, 2d - last 2 days, as well as today`
--user= mysql username
--output= Filename to save chart as, if not present, output will be sent to STDOUT
--show-all Show all graphs (by default shows graphs for which limits are set)
--server= Server id will display record for that server, instead of default (current)
--style= admin, user set chart style, CPU limits are normalized to 100% in user’s style
--format= svg, png set chart output format

lve-read-snapshot

Usage

lve-read-snapshot [-h] [--version] [-f FROM [FROM ...]] [-t TO [TO ...]
                  [ -p PERIOD | --timestamp TIMESTAMP]
                  [-i ID | -u USER] [-l] [-o file] [-j] [--stats]
                  [--unit unit]

Reads LVE system state snapshots for LVE/user.

Optional arguments

  • -h, --help – show this help message and exit
  • --version – version number
  • -f FROM [FROM ...], --from FROM [FROM ...] – run report from date and time in YYYY-MM-DD HH:MM format, if not present last 10 minutes are assumed (default: 2016-10-24 19:28)
  • -t TO [TO ...], --to TO [TO ...] – run report up to date and time in YYYY-MM-DD HH:MM format, if not present, reports results up to now (default: 2016-10-24 19:38)
  • -p PERIOD, --period PERIOD – time period specify minutes with m, h - hours, days with d, and values: today, yesterday, 5m - last 5 minutes, 4h - last four hours, 2d - last 2 days, as well as today (default: 10m)
  • --timestamp TIMESTAMP – time stamp in unix format for get one snapshot (default: None)
  • -i ID, --id ID – LVE id to show records for (default: None)
  • -u USER, --user USER – user account to show records for (default: None)
  • -l, --list – show timestamp list only (default: False)
  • -o file, --output file – filename to save snaphots report to, if not present,output will be sent to STDOUT (default: None)
  • -j, --json – output in json format (default: False)
  • --stats – output stats, instead of snapshots (default: False)
  • --unit unit – group stats by time unit. Example values 3h, 24h, 1d, 1w. Other possible value is auto for grouping by each incident (default: 1d)

One of -u --user or -i --id should be specified.

lve-create-db

Usage

lve-create-db [-h] [--recreate] [--print-sql]
                   [--update-serverid-prompt] [--update-serverid-auto]
                   [--validate]

Creates a database for lve-stats.

Optional arguments

  • -h, --help – show this help message and exit
  • --recreate – drops and recreates database even if tables exists (default: False)
  • --print-sql – prints sql and exits, without creating db (default: False)
  • --update-serverid-prompt – update exist server ID or create new one (default: False)
  • --update-serverid-auto – update exist server ID with uuid (default: False)
  • --validate – check the correctness of the database structure (default: False)

cloudlinux-top

Utility provides information about current MySQL and LVE usage of a running system in JSON format.

Usage

cloudlinux_top [-h] [-v] [-j] [--hide-mysql]
               [-u USERNAME | -r FOR_RESELLER] [-d DOMAIN] [-m MAX]
               [-o ORDER_BY]

Optional arguments

  • -h, --help – show this help message and exit
  • -v, --version – show program version number and exit
  • -j, --json – return data in JSON format
  • --hide-mysql | `don't show MySQL related info
  • -u USERNAME, --username USERNAME – show data only for a specific user. Can be used to filter the output; returns users with username %USERNAME%
  • -r FOR_RESELLER, --for-reseller FOR_RESELLER – get information only about specified reseller and his users
  • -d DOMAIN, --domain DOMAIN – show data only for a specific domain. Can be used to filter the output; returns users with domain %DOMAIN%
  • -m MAX, --max MAX – show up to N records. If --max key is omitted. By default will show top 25 users
  • -o ORDER_BY, --order-by ORDER_BY – sort output by resource usage; available options: cpu, mysql_cpu, io, mysql_io, iops, ep, nproc, pmem

Output format

{
  "mySqlGov": "enabled",              # possible values: enabled, error
  "mySqlGovMode": "abusers",          # see “MySQL Governor > Modes Of Operation”
                                      # if MySQL Governor is not enabled, value is "none"
 
  "resellers": [                      # list of resellers (available only with
                                      # reseller limits feature)
      {
          "id": 1000020005,           # internal record id
          "limit": <lve_section>,     # current limits (last 5 seconds)
          "name": "reseller_name",    # reseller’s login in control panel
          "usage": <lve_section>      # current usage (last 5 seconds)
      }
  ],
  "result": "success",                # see the ‘errors handling’ section
  "timestamp": 1522858537.337549,
  "users": [
      {
          "domain": "domain.com",     # user’s primary domain (from control panel)
          "id": 20005,                # lve_id, same as user id in /etc/passwd file
          "limit": <lve_section>,     # limits for last 5 seconds
          "reseller": "reseler1",     # user’s reseller (from control panel)
          "usage": <lve_section>,     # usage for last 5 seconds
          "username": "user"          # username from /etc/passwd file or “N/A” if user
                                      # with such id does not exist
       }
   ]
 }

The structure * of <lve_section>:

{
"cpu": {
 "all": 50.0,      # CPU usage or limit (LVE only)
 "mysql": 0.0*     # CPU usage or limit (MySQL Governor only)
},
"ep": 1.0,           # number of entry processes
"io": {
 "all": 0.0,       # IO usage or limit (LVE only)
 "mysql": 0.0**     # IO usage or limit (MySQL Governor only)
},
"iops": 0.0,         # IO operations per second
"mem": 258048,       # memory usage or limit
"pno": 1.0           # number of processes
}

Note

  • you can modify this structure using --show option, see usage examples for details.
  • MySQL values are only present when MySQL Governor statistics is available and --hide-mysql options is not used.

Units of measurement

For limits and usage sections we use the following units of measurement.

Value Units of measurement
cpu (lve and mysql) percentage of one CPU core
io (lve and mysql) bytes per second
iops number of IO operations per second
mem bytes
ep number of entry processes
pno number of processes

Errors handling

The format of the error message is the same as in the other cloudlinux- * utilities. When everything is ok, the result value is success. Otherwise, it contains error message. In case of unexpected errors, the output will be as follows.

# cloudlinux-top --json 
{
  "context": {
      "error_text": "Very bad error"
  },
  "result": "An error occured: \"%(error_text)s\"",
  "timestamp": 1523871939.639394
}

Examples

  • get 100 users ordered by CPU usage

    # cloudlinux-top --json --order-by cpu --max=100
    
  • get information about one user

    # cloudlinux-top --json -u username
    
  • get information about reseller and his users

    # cloudlinux-top --json --for-reseller=reseller_name
    
  • show only IO limits and usage

    # cloudlinux-top --json --show=io
    

cloudlinux-statistics

cloudlinux-statistics is a CLI utility that provides historical information about resource usage.

Usage

cloudlinux-statistics [-h] [-j] [-v] [--by-usage BY_USAGE]
                      [--percentage 0..100] [--by-fault BY_FAULT]
                      [--threshold THRESHOLD] [--server_id SERVER_ID]
                      [-f FROM] [-t TO] [--period PERIOD]
                      [--limit LIMIT]
                      [--show COLUMN_NAME [COLUMN_NAME ...]]
                      [-o ORDER_BY] [--id ID] [--time-unit TIME_UNIT]
                      [-r FOR_RESELLER]

Optional arguments

  • -h, --help – show this help message and exit

  • -j, --json – return data in JSON format

  • -v, --version – show program version number and exit

  • --server_id SERVER_ID, --server-id SERVER_ID – can be used with the central database for multiple servers; default ...

  • --limit LIMIT – limit the number of results to display, 0 is unlimited

  • --show COLUMN_NAME [COLUMN_NAME ...] – show only listed columns; all for all supported columns (fields)

    Key Fields to show
    all all available fields
    cpu CPU field
    io IO field
    iops IOPS field
    ep entry processes (concurrent connections) field
    nproc number of processes field
    pmem physical memory field
    vmem virtual memory field
    mysql mysql_cpu & mysql_io field
  • -o ORDER_BY, --order-by ORDER_BY – order results by one of the following keys (fields):

    FIELD DESCRIPTION
    any_faults total number of faults of all types
    cpu average CPU usage`
    mysql_cpu average MySQL CPU usage`
    io average IO usage`
    mysql_io average MySQL IO usage`
    iops average IO operations; (LVE version >= 8)`
    ep average number of entry processes (concurrent connections)`
    nproc average number of processes`
    pmem average physical memory usage`
    vmem average virtual memory usage`
    cpu_faults total number of CPU usage faults`
    io_faults total number of max IO faults`
    iops_fault total number of max IO operations faults; (LVE version >= 8)`
    ep_faults total number of max entry processes faults`
    nproc_faults total number of max processes faults`
    pmem_faults total number of out of physical memory faults`
    vmem_faults total number of out of virtual memory faults`
  • -r FOR_RESELLER, --for-reseller FOR_RESELLER – show statistics only for given reseller and his users

Filter items by resource usage.

  • --by-usage BY_USAGE – show LVEs with usage (averaged) within 90 percent of the limit available values

    FIELD DESCRIPTION
    cpu average CPU usage
    mysql_cpu average MySQL CPU usage
    io average IO usage
    mysql_io average MySQL IO usage
    iops average IO operations; (LVE version >= 8)
    ep average number of entry processes (concurrent connections)
    nproc average number of processes
    pmem average physical memory usage
    vmem average virtual memory usage
  • -percentage 0..100 – define percentage for --by-usage option; default 90

Filter items by the number of faults.

  • --by-fault BY_FAULT – show only accounts that have some faults for the given limit

    FIELD DESCRIPTION
    any faults of all types
    cpu CPU usage faults
    io max IO usage faults
    iops max IO operations faults; (LVE version >= 8)
    ep max entry processes faults
    nproc max processes faults
    pmem out of physical memory faults
    vmem out of virtual memory faults
  • --threshold THRESHOLD – in combination with --by-fault shows only accounts with the number of faults more than given; default 1

Filter items by a time interval.

Allows to get information for the given period of time; you can either set --from and --to options, or just get information for the recent time period using --period option.

Note

--from and --to values are ignored when --period is set.

  • -f FROM, --from FROM – run report from date and time in [YY]YY-MM-DD[ HH:MM] format; if not present, last 10 minutes are assumed
  • -t TO, --to TO – run report up to date and time in [YY]YY-MM-DD[ HH:MM] format; if not present, reports results up to now
  • --period PERIOD – time period; specify minutes with m, hours with h, days with d, and values: today, yesterday; 5m - last 5 minutes, 4h - last four hours, 2d - last 2 days, and today

Get detailed statistics.

  • --id ID – get detailed statistics for database record with the given id
  • --time-unit TIME_UNIT – group statistics using the given time; 1 minute by default. For example: 1h or 1h30m or dynamic; available only in pair with --id

Output format

There are two different JSON formats used for summary statistics and detailed statistics.

Summary statistics

# cloudlinux-statistics --json
 {
 "resellers": [
   {
     "usage": <lve_section>,
     "faults": <lve_section>,
     "name": "reseller",
     "limits": <lve_section>,
     "id": 1000020005
   }
 ],
 "timestamp": 1522920637,
 "mySqlGov": "enabled",            # possible values: ”enabled”, “error”
 "result": "success",
 "users": [
   {
     "username": "username",
     "domain": "example.com",
     "reseller": "reseller",
     "limits": <lve_section>,
     "faults": <lve_section>,
     "usage": <lve_section>,
     "id": 20005
   }
 ]
 }

Detailed statistics

# cloudlinux-statistics --json --id=20001
 {
 "timestamp": 1523011550,
 "mySqlGov": "enabled",           # possible values: ”enabled”, “error”
 "result": "success",
 "user": [
   {
     "usage": <lve_section>,
     "faults": <lve_section>,
     "from": 1523011144,
     "limits": <lve_section>,
     "to": 1523011143
   },
 ...
   {
     "usage": <lve_section>,
     "faults": <lve_section>,
     "from": 1523011204,
     "limits": <lve_section>,
     "to": 1523011203
   }
 ]
 }

For both, summary statistics and detailed statistics, <lve_section> is the same and looks like following *.

{
   "ep": {
     "lve": 1        # number of entry processes
   },
   "vmem": {
     "lve": 2428928  # virtual memory usage or limit (deprecated)
   },
   "iops": {
     "lve": 0        # io operations per second
   },
   "io": {
     "lve": 0.0,     # io usage or limit (lve only)
     "mysql": 0.0**   # io usage or limit (mysql only)
   },
   "nproc": {
     "lve": 1        # number of processes in lve
   },
   "cpu": {
     "lve": 25.6,    # cpu usage (lve only)
     "mysql": 0.0*   # cpu usage (mysql governor only)
   },
   "pmem": {
     "lve": 360448   # physical memory usage or limit
   }
 }

Note

  • you can specify only required fields using --show option;
  • mysql fields are only available with MySQL Governor installed.

Units of measurement

For limits and usage sections we use the following units of measurement.

Value Units of measurement
cpu (LVE and MySQL) percentage of one CPU core
io (LVE and MySQL) bytes per second
iops number of IO operations per second
pmem and vmem bytes
ep number of entry processes
nproc number of processes in LVE

Errors handling

The format of the error message is the same as in the other cloudlinux- * utilities. When everything is ok, the result value is success. Otherwise, it contains error message.

# cloudlinux-statistics --json 
{
  "context": {
      "error_text": "Very bad error"
  },
  "result": "An error occured: \"%(error_text)s\"",
  "timestamp": 1523871939.639394
}

Examples

  • get top 10 users ordered by CPU usage for today
# cloudlinux-statistics --json --order-by=cpu --period=today --limit=10
  • get users that hit IO limit more than 10 times for today
# cloudlinux-statistics --json --period=today --by-fault=io --threshold=10
  • get users that used more than 80% of CPU in last 24 hours
# cloudlinux-statistics --json --by-usage=cpu --percentage=80 --period=24h
  • get information only about reseller and his users
# cloudlinux-statistics --json --for-reseller=reseller_name
  • get information only about CPU and IO usage
# cloudlinux-statistics --json --show=cpu,io

Plugins

LVE Stats 2 comes with a set of generic plugins:

Plugin Name Order Default Period (seconds) Description
LVECollector 1000 Y 5 Collects usage/limits data from /proc/lve/list
CPUInfoCollector 2000 Y 5 collents info about CPU - /proc/cpuinfo
LVEUsernamesCollector 3000 Y 3600 collects usernames & user ids to match uid <-> lve id later on
LVEUsageAnalyzer 4000 Y 5 analyzes usage of LVE
LveUsageAggregator 5000 Y 60 aggregates data by time periods
DBGovSaver 6000 Y 5 Saves data about database governor
FileSaver 7000 Y 5 Saves LVE data into /var/lve/info
CloudLinuxTopFileSaver 8000 Y 60 saves data used by cloudlinux-top to /var/lve/cloudlinux-top.json
DBSaver 9000 Y 60 save LVE data to database
DbUsernamesSaver 10000 Y 3600 saves users name to database
DBSaverX60 11000 Y 3600 saves aggregated hourly data into database
SnapshotSaver 12000 Y 30 collects & saves snapshots data
StatsNotifier 13000 Y varied notify user/admin based on usage
HistoryCleaner 14000 Y 3600 removes old usage
ResMEMCollector 1500 N 30 collects physical memory usage from processes RES field instead of /proc/lve/list
LVEDestroyer - N 5 destroys LVEs that weren't active for X iterations. Number of iterations is passed from config using iterations variable. iterations=0 means plugin disabled

To enable non-default plugin, copy or link it to /usr/share/lve-stats/plugins directory.

For example to enable ResMEMCollector plugin, do:

ln -s /usr/share/lve-stats/plugins.other/res_mem_collector.py /usr/share/lve-stats/plugins/
service lvestats restart

Creating a Plugin for LVE Stats 2

Introduction

LVE Stats 2 complex has scalable architecture, which allows to connect custom plugins.

LVE Stats server searches for plugins in the directory which is specified with plugins parameter of server’s /etc/sysconfig/lvestats2 configuration file. Default directory is /usr/share/lve-stats/plugins.

Each plugin must be of a Python class, must be written in Python language and its file must have .py extension. Files with all other extensions will be ignored. For normal server work access permission 400 is enough; owner – root .

Plugin classes can be of the same name, but better not, because classes' names can affect the set of parameters in set_config method. You can find detailed plugins' configuring information below, in appropriate chapter.

Plugin class must contain execute() method, which is invoked by the server every 5 seconds (by default, can be changed by interval parameter of configuration file). Also set_config method (configuration settings) can be available. You can find more info in Plugins Configuration chapter.

Additionally the following attributes can be set (plugin class instance variable):

  • order (integer) - defines plugin's position in the server's plugin list, (more info in Servers Plugin Arrangement ).
  • timeout (integer or float ) – the longest allowable duration of one launch of the plugin (execute method). Default value of timeout parameter is 5 seconds.
  • period (integer) – sets the interval between two launches of execute plugin method in seconds. If not defined, then plugin runs every 5 seconds ( interval parameter in configuration file).

When execute() method of the plugin is invoked, the server creates an attribute now in it, where launch time is recorded. This value is equal to what a standard Python function time.time() returns. All the plugins launched one after another receive the same  value of now attribute from the server. now is overwritten before execute() method is invoked.

The previous value of now attribute is not saved by the server. If plugin needs it, it has to save it by itself.

Plugin class can be inherited from LveStatsPlugin class, which is the part of the server itself. This is not obligatory, but inheritance can help to avoid different errors in servers work, particularly if a plugin doesn't contain required execute method.

LveStatsPlugin class is defined in the file: /opt/alt/python27/lib/python2.7/site-packages/lvestats/core/plugin.py .

Server Plugin Arrangement

When the server starts, it performs the search of plugins in the directory specified in /etc/sysconfig/lvestats2 configuration file. This directory is scanned only when the server starts, therefore if any plugin was added into the directory, the server has to be restarted with the following command:

service lvestats restart.

After successful restart, the plugins are graded and executed ascending by order attribute. If any plugin order attribute is not set, it is considered as a Python language constant sys.maxint (which is usually 9223372036854775807). This in fact means that such plugins will be executed in the last. If any plugins has similar order meanings, their execution order is unpredictable.

The server invokes execute method of all plugins one after another.

When the server invokes execute() method of any plugin, it transmits a data dictionary ( lve_data argument) into plugin. The dictionary is common for all the plugins. Any plugin can read, write and change any data in this dictionary. LVE Stats 2 server doesn't control this area. That is why one must be careful while developing new plugins, in order not to change or corrupt other plugins' data which can break their functionality.

If an exception occurs in execute() method, its text and python stack trace is recorded into server log /var/log/lve-stats and all the changes made to lve_data dictionary before the exception happened are lost.

The keys of the lve_data dictionary are recommended to look like PluginName_Key , in order the plugins do not corrupt other data accidentally.

Server contains some standard plugins which define and use the following keys in the common dictionary lve_data: LVE_VERSION, stats, old_stats, procs and lve_usage . User plugins can use data from these keys, but it is recommended not to change them if there is no special need, because it can break the next plugins in the execution queue.

Key Content
LVE_VERSION LVE version. The same as console command lvectl lve-version produces.
stats Dictionary, that contains lve id’s as keys and LVEStat class objects as values. Every LVEStat object contains values of usages and limits taken from /proc/lve/list file for every LVE Id . Dictionary keys – integer lve id , including 0 for “ default ” LVE. This dictionary is updated on each iteration of lvestats-server (every 5 seconds by default). LVEStat – is a standard server class, it can be imported with the command from lvestats.core.lvestat import LVEStat. The class is described in the file /opt/alt/python27/lib/python2.7/site-packages/lvestats/core/lvestat.py . Here you can find the whole list of data fields and their functions.
old_stats stats content from the previous iteration. Before the first iteration – empty dictionary.
totalHz When LVE_VERSION is 4, real CPU frequency in Hz multiplied by number of cores. When LVE_VERSION > 4, CPU speed is in conventional units and equals to 1000000000 * cores (1 GHz per core).
procs Quantity of CPU cores.
lve_usages Contains accumulated LVE statistics for each 5-seconds interval in current minute. Cleared each minute.
lve_usage Contains aggregated LVE Statistics for “previous” minute to store to database. Overwritten each minute.

Each plugin’s instance lifetime is from the moment it was loaded till the server stops working. But if execute() method working time exceeds timeout, the plugin will be terminated and restarted in the next iteration. All changes to the lve_data dictionary will be lost.

During servers graceful shutdown (restart, server shutdown, commands service lvestats stop, service lvestats restart ), each plugin receives SIGTERM signal. This is useful to correctly unload the plugin (terminate all subsidiary processes, save data to files etc.). If a plugin doesn't need to “finalize” its execution before termination, then there's no need to implement this signal handler. Below you can see an example of such handler.

Note

If a plugin implements handler for SIGTERM, then this handler must end with sys.exit(0) command. Otherwise plugin process will not be terminated correctly and will become orphaned.

Plugin Configuration

LVE Stats 2 Server allows to configure each plugin separately.

On initialization stage the server invokes set_config() method of the plugin and locates there a dictionary which contains:

  • all parameters from file /etc/sysconfig/lvestats2 (global).
  • plugin's individual configuration file parameters (if one exists). Configuration files must be located in /etc/sysconfig/lvestats.config directory, have .cfg extension and be the same format as /etc/sysconfig/lvestats2 . Files in this directory are matched with the plugins by name. For instance, if plugin class is Plugin1_class , then server will try to find and download /etc/sysconfig/lvestats.config/Plugin1_class.cfg. Names are case sensitive. If any plugin doesn't have an individual configuration file, then it's not an error. In this case plugin will just receive parameters from /etc/sysconfig/lvestats2 .

Note

An individual configuration file of every plugin is loaded after server configuration file. That is why if it contains any parameters with names similar to ones of server config, then plugin will use parameters from its individual config rather than server config parameters.

If a plugin doesn't require any configuration to be done, then set_config method can be skipped.

In addition, plugins can use their own configuration methods.

Types of Plugins

According to server architecture, plugins can be of the following types:

  • collectors
  • analyzers
  • persistors
  • notifiers

Collectors are designed to collect information; analyzers – to analyze it and form some other data on its basis; persistors – to save information from the common dictionary into files, databases, etc.; notifiers - to notify system users about any events.

This division is rather arbitrary. There is an opportunity to execute all the actions on collection, analysis and saving the information in one and only plugin. But at the same time the division into functionally independent parts allows to build flexible and easily configurable system for collecting and processing the data.

Also it is possible to implement the systems of lazy-write, planning of collecting/processing tasks and notifying users about different events.

Examples of Plugins

Here is a practical example of a user plugin.

Specification:

  1. To trace specified file size changes. The name of file being traced must be specified in configuration file, which allows to change it without modifying the plugin itself. If file size has been changed, it has to be written as a message into our log. The name of log must be specified in configuration file as well.

  2. File size must be checked with default interval (5 seconds), and log notification must be held once a minute (to avoid resource expend for possibly regular write).

  3. System administrator must receive emails with file size at the moment the email was sent. These notifications must be sent even if the file size hasn’t been changed. Notification emails must be read periodicity from configuration file as well as sender/receiver emails .

As file size check, fixing the result and notification sending must be held with different periods, then it’s impossible to realize all the tasks by means of one plugin. The fact that one minute (60 seconds) is multiple to 5 seconds doesn’t matter in this case, because default period can be changed in server’s configuration file, but the condition of fixing the result once a minute is a part of the specification, which can not be violated. In addition, notification email period is known in advance, as it is specified by user in configuration file.

That is why we realize 4 plugins: collector, analyzer, persistor and notifier .

Collector

Collector's aim is to determine the size of a proper file.

# FSize_watcher_collector.py
# Example plugin for monitoring file size.
# Part 1. Collector

import os
from lvestats.core.plugin import LveStatsPlugin 

# Key name
COLLECTOR_KEY = 'FSizeWatcher_fsize'
COLLECTOR_KEY_FILENAME = 'FSizeWatcher_fname'  

class FSize_watcher_collector (LveStatsPlugin):
	# this plugin should be first in chain
	order = 0
	# File to monitoring
	file_to_monitoring = None 
	
	def __init__(self):
		pass 
		
	# Sets configuration to plugin
	def set_config(self, config):
		self.file_to_monitoring = config.get('file_to_monitoring', None)
		pass
	# Work method
	def execute(self, lve_data):
		try:
			# if monitoring file absent, do nothing
			if self.file_to_monitoring is None or not os.path.exists(self.file_to_monitoring):
		return 
		
			# Get file size
			stat_info = os.stat(self.file_to_monitoring)
			fsize = stat_info.st_size 
			
			# Place file name and file size to server data dictionary
			lve_data[COLLECTOR_KEY_FILENAME] = self.file_to_monitoring
			lve_data[COLLECTOR_KEY] = fsize
		except (OSError, IOError):
			# file absent or any other error - remove size from dictionary
			del lve_data[COLLECTOR_KEY]

Plugin algorithm is extremely simple – file size is read and written into data dictionary. Files name is read from set_config method configuration. If the name is not specified, then None is written into appropriate variable. All the errors are completely ignored (e.g. if specified file doesn't exist or there's no way to read any of it's information).

order attribute is specified as 0 to make this plugin go the first among three. Data collector must always be the first in plugins logical chain, because it provides all the necessary information for the analyzer which goes the next. Specific values of order can be of any kind, but what is important is that when the server starts, all the plugins line up in proper sequence: collector – analyzer – persistor .

In order to make plugin work, we have to create configuration file /etc/sysconfig/lvestats.config/FSize_watcher_collector.cfg with the following content:

# Config file for FSize_watcher_collector plugin
# Please define monitoring file here
# file_to_monitoring = /usr/local/cpanel/logs/error_log
file_to_monitoring = /usr/local/cpanel/logs/access_log

Note that file name FSize_watcher_collector without .cfg extension matches plugin class name.

file_to_monitoring option is read by plugin in set_config method and contains file’s full name for monitoring.

Files for monitoring, suggested in the actual example - /usr/local/cpanel/logs/error_log and /usr/local/cpanel/logs/access_log - are real, these are cPanel control panel logs.

The first file is errors log; the second is appeal log, is refreshed during common work with panel (e.g. if user email address is changed).

Errors log tracking is more important, but appeal log monitoring allows to illustrate plugins work more in details, because it is refreshed more often.

Note that plugin can monitor one file only.

Analyzer

Analyzer decides if the file's size has changed and gives a command to persistor to refresh log.

# FSize_watcher_analyzer.py
# Example plugin for monitoring file size.
# Part 2. Analyzer 

from lvestats.core.plugin import LveStatsPlugin 

# Key name from collector plugin
COLLECTOR_KEY = 'FSizeWatcher_fsize' 

# Key name 1 for saver plugin
SAVER_KEY = 'FSizeWatcher_fsize_to_store'
# Key name 2 for saver plugin
SAVER_DATA_PRESENCE = 'FSizeWatcher_fsize_present'  

class FSize_watcher_analyzer (LveStatsPlugin):
	# this plugin should be second in chain
	order = 1
	# Last file size
	file_last_size = 0
	# Plugin run period in secondsperiod = 60 
	
	def __init__(self):
		pass 
		
	# work method
	def execute(self, lve_data):
		# Default setting for saver
		lve_data[SAVER_DATA_PRESENCE] = 0
		# Check presence data
		if COLLECTOR_KEY not in lve_data:
		return 
		
		# Get file size from server data dictionary
		fsize = lve_data[COLLECTOR_KEY] 
		
		# Check, if file size changed, store it for saver plugin
		if fsize == self.file_last_size:
			return 
			
		# Put new size for saver plugin
		lve_data[SAVER_KEY] = fsize
		self.file_last_size = fsize
		lve_data[SAVER_DATA_PRESENCE] = 1

This plugin is extremely simple as well. It starts after collector (order=1) , searches for file size in the dictionary and compares it with the previous index. If it has changed, then it writes a sign of presence of a new size into the dictionary. If no changes seen, then sign resets. The previous file size is stored in the plugin itself in file_last_size variable. Note that they are stored during the whole server lve-stats lifetime.

If file size is not found in data dictionary, then plugin just ends.

All the errors are completely ignored.

Analyzer is unconfigurable, that is why it doesn’t require any configuration file and it doesn’t contain set_config method.

Plugin starts every 60 seconds (1 minute), because we need data fixation to be performed one time in a minute.

Persistor

Persistor saves information from the common dictionary into files, databases, etc.

# FSize_watcher_saver.py
# Example plugin for monitoring file size and last modification date-time.
# Part 3. Data saver 

import signal
import sys
import time
from lvestats.core.plugin import LveStatsPlugin 

# Key name 1 for saver plugin
SAVER_KEY = 'FSizeWatcher_fsize_to_store'
# Key name 2 for saver plugin
SAVER_DATA_PRESENCE = 'FSizeWatcher_fsize_present'
# Monitoring file name
COLLECTOR_KEY_FILENAME = 'FSizeWatcher_fname'  

class FSize_watcher_saver (LveStatsPlugin):
	# this plugin should be third in chain
	order = 2
	# Plugin run period in seconds
	period = 60
	# Log filename
	log_file_name = None
	# First run flag
	is_first_run = True 
	
	def __init__(self):
		signal.signal(signal.SIGTERM, self.sigterm_handler) 
		
	# Sets configuration to plugin
	def set_config(self, config):
		# Get log filename
		self.log_file_name = config.get('log_filename', None) 
		
	# work method
	def execute(self, lve_data):
		# do nothing, if log file not defined
		if not self.log_file_name:
			return
		try:
			# Check presence data
			if SAVER_DATA_PRESENCE not in lve_data or lve_data[SAVER_DATA_PRESENCE] == 0:
				# No data
				return
			# Get file size from server data dictionary
			fsize = lve_data[SAVER_KEY]
			
			# Store data to log
			f = open(self.log_file_name, 'a')
			if self.is_first_run:
				f.write('%s - FSize_watcher started. Monitoring file: %s, saving data period=%d sec\n' % (time.asctime(time.localtime()), lve_data[COLLECTOR_KEY_FILENAME], self.period))
				self.is_first_run = False
			f.write('%s - FSize_watcher: file size is %d bytes\n' % (time.asctime(time.localtime()), fsize))
			f.close()
		except:
			# Ignore all errors
			pass 
			
	# Terminate handler
	def sigterm_handler(self, signum, frame):
		if self.log_file_name:
			try:
				# Store data to log file
				f = open(self.log_file_name, 'a')
				f.write('%s - File watcher saver plugin: TERMINATE\n' % time.asctime(time.localtime()))
				f.close()
				pass
			except:
				# Ignore all errors
				pass
		# Terminate process
		sys.exit(0)

Configuration file /etc/sysconfig/lvestats.config/FSize_watcher_saver.cfg :

# Config file for FSize_watcher_saver.py plugin
# Please define log filename here
log_filename = /var/log/FSize_watcher.log

This plugin starts after analyzer (order=2) , checks new file size presence flag, and if positive – writes it into log. If the flag is cleared (which means the size hasn't changed), then plugin simply ends.

Starts once in a minute (period=60).

Also this plugin shows the work of signal handler.

Plugin constructor registers handler-function of a proper signal: signal.signal(signal.SIGTERM, self.sigterm_handler) . This means, that when the server finishes its work, then sigterm_handler method of plugin class will be invoked. In the actual example the function just writes a notification into log, tracing the fact of it's invocation.

Pay attention on sys.exit(0) command in the end of the handler. Find the information on it in Server Plugin Arrangement section.

In addition see into examples of file log /var/log/FSize_watcher.log formed by the plugins above:

Tue Feb  3 13:06:24 2015 - FSize_watcher started. Monitoring file: /usr/local/cpanel/logs/access_log, saving data period=60 sec
Tue Feb  3 13:06:24 2015 - FSize_watcher: file size is 122972890 bytes
Tue Feb  3 13:07:25 2015 - FSize_watcher: file size is 122975507 bytes
Tue Feb  3 13:08:25 2015 - FSize_watcher: file size is 122978124 bytes
Tue Feb  3 13:09:25 2015 - FSize_watcher: file size is 122978997 bytes
Tue Feb  3 13:10:25 2015 - FSize_watcher: file size is 122981033 bytes
Tue Feb  3 13:11:25 2015 - FSize_watcher: file size is 122982052 bytes
Tue Feb  3 13:13:25 2015 - FSize_watcher: file size is 122983798 bytes
Tue Feb  3 13:20:15 2015 - File watcher saver plugin: TERMINATE

and

Thu Feb  5 13:07:27 2015 - FSize_watcher started. Monitoring file: /usr/local/cpanel/logs/error_log, saving data period=60 sec
Thu Feb  5 13:07:27 2015 - FSize_watcher: file size is 14771849 bytes
Thu Feb  5 14:03:32 2015 - FSize_watcher: file size is 14771995 bytes
Thu Feb  5 15:01:36 2015 - FSize_watcher: file size is 14772434 bytes
Thu Feb  5 17:15:47 2015 - FSize_watcher: file size is 14772873 bytes
Thu Feb  5 18:47:54 2015 - FSize_watcher: file size is 14775213 bytes
Thu Feb  5 19:11:56 2015 - FSize_watcher: file size is 14775652 bytes
Thu Feb  5 21:09:05 2015 - FSize_watcher: file size is 14776091 bytes
Thu Feb  5 23:06:14 2015 - FSize_watcher: file size is 14776530 bytes
Fri Feb  6 00:47:23 2015 - FSize_watcher: file size is 14778870 bytes
Fri Feb  6 01:02:24 2015 - FSize_watcher: file size is 14779309 bytes
Fri Feb  6 02:00:28 2015 - FSize_watcher: file size is 14779434 bytes
Fri Feb  6 03:16:34 2015 - FSize_watcher: file size is 14779873 bytes
Fri Feb  6 05:04:42 2015 - FSize_watcher: file size is 14779998 bytes
Fri Feb  6 05:12:43 2015 - FSize_watcher: file size is 14780437 bytes
Fri Feb  6 05:56:50 2015 - FSize_watcher: file size is 14780551 bytes
Fri Feb  6 06:01:50 2015 - FSize_watcher: file size is 14780975 bytes
Fri Feb  6 06:03:51 2015 - FSize_watcher: file size is 14782183 bytes
Fri Feb  6 06:04:51 2015 - FSize_watcher: file size is 14782575 bytes
Fri Feb  6 06:18:52 2015 - FSize_watcher: file size is 14782647 bytes
Fri Feb  6 06:21:52 2015 - FSize_watcher: file size is 14782898 bytes
Fri Feb  6 06:48:54 2015 - FSize_watcher: file size is 14785238 bytes
Fri Feb  6 07:09:56 2015 - FSize_watcher: file size is 14785677 bytes
Tue Feb  6 08:03:15 2015 - File watcher saver plugin: TERMINATE

You can see that log record is being held once a minute (what we actually need), new file size is written.

Also we can notice that handler SIG_TERM was executed, signaling that plugin received the notification about server shut-down.

Notifier

Notifier informs system users about any events.

# FSize_watcher_saver.py
# Example plugin for monitoring file size and last modification date-time.
# Part 4. Notifier 

import time
import smtplib 

from lvestats.lib.commons import dateutil
from lvestats.core.plugin import LveStatsPlugin  


# Key name
COLLECTOR_KEY_FSIZE = 'FSizeWatcher_fsize'
COLLECTOR_KEY_FILENAME = 'FSizeWatcher_fname' 

# email message pattern
EMAIL_MESSAGE_PATTERN = """Hello, administrator!
Size of the file '%s' is %d bytes.
"""  


class FSize_watcher_notifier (LveStatsPlugin):
	# Default period
	DEFAULT_PERIOD_STR = '12h'
	# this plugin should be third in chainorder = 3
	# Timeout
	timeout = 20
	# Notifier Log filename
	log_file_name = '/var/log/FSize_watcher_notifier.log'
	# Email from address
	email_from = None
	# Email to address
	email_to = None
	# Email subject
	email_subject = None
	# Sets configuration to plugin
	def set_config(self, config):
		# Email settings
		self.email_from = config.get('notify_from_email', None)
		self.email_to = config.get('notify_to_email', None)
		self.email_subject = config.get('notify_from_subject', 'Message from FSize_watcher_notifier plugin')
		# Notify period
		s_period = config.get('notify_period', None)
		if s_period:
			self.period = dateutil.parse_period2(s_period)
		else:
			self.period = dateutil.parse_period2(FSize_watcher_notifier.DEFAULT_PERIOD_STR)
		f = open(self.log_file_name, 'a')
		f.write('%s - FSize_watcher_notifier plugin: configure\n' % time.asctime(time.localtime()))
		f.write('       - Period: %s\n' % self.period)
		f.write('       - From: %s\n' % self.email_from)
		f.write('       - To: %s\n' % self.email_to)
		f.write('       - Subject: \'%s\'\n' % self.email_subject)
		f.close() 
		
	# work method
	def execute(self, lve_data):
		if COLLECTOR_KEY_FSIZE not in lve_data or COLLECTOR_KEY_FILENAME not in lve_data:
			return
		if not self.email_from or not self.email_to:
			f = open(self.log_file_name, 'a')
			f.write('%s - FSize_watcher_notifier plugin error: email_from or email_to not set\n')
			f.close()
			return
		try:
			from email.mime.text import MIMEText
			# Send email
			msg = MIMEText(EMAIL_MESSAGE_PATTERN % (lve_data[COLLECTOR_KEY_FILENAME], lve_data[COLLECTOR_KEY_FSIZE]))
		msg['Subject'] = self.email_subject
		msg['From'] = self.email_from
		msg['To'] = self.email_to 
		
		s = smtplib.SMTP('localhost')
		s.sendmail(self.email_from, [self.email_to], msg.as_string())
			s.quit() 
			
		f = open(self.log_file_name, 'a')
			f.write('%s - FSize_watcher_notifier plugin: email message was successfully sent\n' % time.asctime(time.localtime()))
			f.close()
			except Exception as e:
			f = open(self.log_file_name, 'a')
			f.write('%s - FSize_watcher_notifier plugin error:\n%s\n' % (time.asctime(time.localtime()), str(e)))
			f.close()

Configuration file /etc/sysconfig/lvestats.config/FSize_watcher_notifier.cfg :

# Config file for FSize_watcher_notifier.py plugin
# Please define email options here 

NOTIFY_FROM_EMAIL=user@hostname
NOTIFY_FROM_SUBJECT=Message from FSize_watcher_notifier
NOTIFY_TO_EMAIL=admin@hostname
NOTIFY_PERIOD=12h

Plugin’s index number equals 3 ( order=3 ), that is why notifier starts after the rest. But since it uses only data formed by collector , then its order may equal any number bigger that collectors order (>0).

Notifier reads the necessary parameters from the configuration (email address, topic, period) and writes them into its own log as reference.

Plugin’s execute method checks the availability of all the necessary data (email parameters, collectors data) and sends the message. All the notifications are written into the notifier's own log.

If any data is missing, the message is not sent.

Log example:

Thu Feb  5 11:51:34 2015 - FSize_watcher_notifier plugin: configure
       - Period: 60.0
       - From: user@hostname
       - To: admin@hostname
       - Subject: 'Message from FSize_watcher_notifier'
Thu Feb  5 11:51:35 2015 - FSize_watcher_notifier plugin: email message was successfully sent
Thu Feb  5 11:52:35 2015 - FSize_watcher_notifier plugin: email message was successfully sent
Thu Feb  5 11:53:35 2015 - FSize_watcher_notifier plugin: email message was successfully sent
Thu Feb  5 11:54:35 2015 - FSize_watcher_notifier plugin: email message was successfully sent
Thu Feb  5 11:57:00 2015 - FSize_watcher_notifier plugin: configure
       - Period: 43200.0
       - From: user@hostname
       - To: admin@hostname
       - Subject: 'Message from FSize_watcher_notifier'
Thu Feb  5 11:57:00 2015 - FSize_watcher_notifier plugin: email message was successfully sent

File info and format for /var/lve/info file

This file is used by control panels to display to user their 'current' usage. The file is updated every 5 seconds by lve-stats.

When writing to this file we make sure that: average CPU/IOPS/MEM is never greater then LIMIT for that resource.

Example:

0,0,20,0,2500,0,262144,0,0,262144,0,0,100,0,0,0,0,1024,1024,0,0,0,0 600,1,20,2492,2500,70,262144,0,0,262144,33,0,100,1,0,0,0,1024,1024,0,5,0,0 200,0,20,0,2500,0,262144,0,0,262144,0,0,100,0,0,0,0,1024,1024,0,0,0,0 500,0,20,0,2500,0,262144,0,0,262144,0,0,100,0,0,0,0,1024,1024,0,0,0,0

First line of the file is ' default limits '.

Fields:

# 0 - id
# 1 - mep (average entry processes)
# 2 - lep  (limit ...)
# 3 - cpu_usage (average speed)
# 4 - lcpu (limit spped)
# 5 - mem_usage (average virtual memory)
# 6 - lmem (limit ...)
# 7 - mem_fault (number of virtual memory faults)
# 8 - mep_fault (number of entry processes faults)
LVE_VERSION >=6
# 9 - lmemphy (limit physical memory)
# 10 - memphy (average ...)
# 11 - memphy_fault (faults ...)
# 12 - lnproc (limit number of processes)
# 13 - nproc (average ...)
# 14 - nproc_fault (faults ...)
# 15 - lcpuw (CPU weight -- deprecated not used)
# 16 - io_usage (average IO usage)
# 17 - io_limit (limit ...)
LVE_VERSION >=8
#18 - liops  (limit IOPS)
#19 - iops (average IOPS)

Troubleshooting

lvestats service and utilities write fatal errors to system log.

There is /var/log/lve-stats.log file with additional information (warnings, tracebacks for errors)