CloudLinux OS kernel
This documentation describes specific features of the CloudLinux kernel. In other cases the kernel has the same features and innovations as any similar RHEL kernel. More information about the actual kernel changes and releases can be obtained from our changelog.
- Hybrid Kernels
- SecureLinks and Link Traversal Protection
- File change API
- Kernel config variables
- Virtualized /proc filesystem
- Ptrace block
- Xen XVDA
- Umask behavior
- IO limits latency
- Reading LVE usage
- OOM killer for LVE processes
- File system quotas
- Enter LVE when using cPanel utilities
- Proactive reporting kernel crash events with Sentry and Kernel Panic Receiver
Hybrid kernels give you the ability to take advantage of the benefits and features available in newer kernels without having to completely upgrade to another version of the operating system. Example - for the CloudLinux 7 kernel, based on version 3.10, you can install a hybrid kernel (same as on CloudLinux 8), which is based on version 4.18. This provides more kernel options, memory and overall optimization, as well as a positive impact on system performance.
How to migrate from the normal kernel to hybrid one
The system must have an active CloudLinux OS license
If you use
yum-plugin-protectbase, please make sure it is disabled before stating the normal-to-hybrid script.
You should perform the following commands:
yum update normal-to-hybrid
Then if the script execution is completed without errors - perform the server reboot.
You can always migrate from the hybrid kernel to normal one with the following commands:
yum update hybrid-to-normal
Known limitations and issues of hybrid kernels
CloudLinux OS Shared 6 Hybrid kernel
We do not remove Hybrid kernel after migration from Hybrid to the normal channel, but we remove linux-firmware package which is needed to boot Hybrid kernel. This is because CloudLinux OS Shared 6 does not allow to remove the package of currently running kernel. Thus please don't reboot the server back to Hybrid kernel after you remove it.
Kernel module signature isn't checking for now, as 3.10 kernel is using x509 certificates to generate keys and CloudLinux OS Shared 6 cannot detect signatures created in such way.
SecureLinks and Link Traversal Protection
Link Traversal Protection is disabled by default for the CloudLinux OS Shared.
CloudLinux OS Shared and CloudLinux OS Admin provides comprehensive protection against symbolic link attacks popular in shared hosting environment.
The protection requires setting multiple kernel parameters to be enabled:
Link Traversal Protection:
Please reffer to this article in order to know how to setup these parameters.
To protect against symlink attack where attacker tricks Apache web server to read some other user PHP config files, or other sensitive file, enable:
Setting this parameter will deny any process running under gid fs.symlinkown_gid to follow the symlink if owner of the link doesn’t match the owner of the target file.
fs.enforce_symlinksifowner = 1 fs.symlinkown_gid = 48
|fs.enforce_symlinksifowner = 0||do not check symlink ownership|
|fs.enforce_symlinksifowner = 1||deny if symlink ownership doesn’t match target, and process gid matches _symlinkown_gid|
When fs.enforce_symlinksifowner set to 1, processes with GID 48 will not be able to follow symlinks if they are owned by user1 , but point to file owned user2 .
fs.enforce_symlinksifowner = 2 is deprecated and can cause issues for the system operation.
On standard Apache installation, this webserver is usually running under GID 48. On cPanel servers, Apache is running under user nobody, GID 99.
To change GID of processes that cannot follow symlink, edit the file /etc/sysctl.conf , add the line:
fs.symlinkown_gid = XX
$ sysctl -p
To disable symlink owner match protection feature, set fs.enforce_symlinksifowner = 0 in /etc/sysctl.conf , and execute:
$ sysctl -p
fs.symlinkown_gid parameter values for httpd service user and fs.proc_super_gid for nagios service user is written to /etc/sysctl.d/90-cloudlinux.conf.
When used outside CageFS (from cPanel tools for instance),
fs.protected_symlinks_create isn't sufficient for symlink protection. To fully protect symlink access in this case, use
fs.process_symlinks_by_task=1 in addition to
CageFS is extremely powerful at stopping most information disclosure attacks, where a hacker could read sensitive files like /etc/passwd .
Yet, CageFS does not work in each and every situation. For example, on cPanel servers, it is not enabled in WebDAV server, cPanel file manager and webmail, as well as some FTP servers don’t include proper change rooting.
This allows an attacker to create symlink or hardlink to a sensitive file like /etc/passwd and then use WebDAV , filemanager, or webmail to read the content of that file.
You can prevent such attacks by preventing user from creating symlinks and hardlinks to files that they don’t own.
This is done by set following kernel options to 1:
fs.protected_symlinks_create = 1 fs.protected_hardlinks_create = 1
We do not recommend to use protected_symlinks option for cPanel users as it might break some of the cPanel functionality. Please read the known issues section before enabling this feature.
This option only available on CloudLinux 7 Hybrid or on CloudLinux 8 Shared and Admin editions. Also please note that this feature is available for cPanel only.
This parameter is needed to protect against symlink vulnerability where an attacker might get access to files out of the CageFS via cPanel tools: File Manager, WebDAV, Webmail, etc. When a symlink is accessed from cPanel tools (non-root user case) we check whether the current process UID matches the symlink target UID.
To enable the protection for CloudLinux OS Shared , set the
fs.process_symlinks_by_task parameter to 1:
To disable the protection for CloudLinux OS Shared 7 hybrid, set the
fs.process_symlinks_by_task parameter to 0:
Known issues with fs.protected_symlinks_create=1 on cPanel servers
Here are some examples of what may go wrong on cPanel servers. If
fs.protected_symlinks_create=1 is set on the server, it can cause the following issues:
If you use the
rsync to copy/transfer files and they contain symlinks, you'll see the errors like these:
"rsync error: some files could not be transferred (code 23)" and the transfer will be failed.
rsync tasks as well as cPanel transfer tools.
- backup extracting may not work. Errors during the restoration:
Error extracting /home/domain/backups/home.tar.gz : /bin/tar: .cagefs/tmp/mysql.sock: Cannot create symlink to `/var/lib/mysql/mysql.sock'. Permission denied. /bin/tar: .cagefs/tmp/.s.PGSQL.5432: Cannot create symlink to `/var/run/postgres/.s.PGSQL.5432': No such file or directory.
Any backup for accounts (including cPanel backup) cannot be extracted.
dmesgis flooded with the
may_create_sym_link: can't find ea-phpXX in cron may_create_sym_link: can't find ea-phpXX in ea-php-cli
It's popping up each second and may increase the size of the
File change API
One of the main problems on a shared hosting system for file backup operations is to figure out which files have changed. Using INOTIFY on a 1T drive with a large number of small files and directories guarantees slow startup times, and a lot of context switching between kernel and userspace - generating additional load. On the other hand scanning disk for newly modified files is very IO intensive, and can kill the performance of the fastest disks.
CloudLinux OS Shared approach
CloudLinux File Change API is a kernel level technology with the user space interface that buffers lists of modified files in the kernel and then off-loads that list to user space daemon.
After that - any software (with enough permissions) can get a list of files that has been modified for the last 24 hours.
The software is very simple to use and produces the list of modified files. As such we expect file backup software, including integrated cPanel backup system to integrate with this API soon.
Usage and integration
/usr/bin/cloudlinux-backup-helper is a utility for getting the list of changed files.
It is supposed to be run by a super user only.
Command line parameters:
-t | --timestamp retrieve file names for files modified after specified timestamp -u | --uid retrieve file names for particular UID only
protocol version (1 right now), timestamp (in seconds) - up to which time data was collected UID:absolute path to file changed UID:absolute path to file changed …
The timestamp in output is needed so you can clearly identify from which timestamp to get list of changed files next.
[root@localhost ~]# cloudlinux-backup-helper -t 1495533489 -u <UID> 1,1495533925 1001:/home/user2/public_html/output.txt 1001:/home/user2/public_html/info.php [root@localhost ~]# cloudlinux-backup-helper -t 1495533489 1,1495533925 1000:/home/user1/.bashrc 1001:/home/user2/public_html/output.txt 1001:/home/user2/public_html/info.php 1003:/home/user3/logs/data.log
/usr/bin/cloudlinux-backup-helper-uid is a SUID wrapper for the cloudlinux-backup-helper utility that enables an end user to get the list of files changed. It accepts timestamp argument only and retrieves data of the user who is running it only.
[user@localhost ~]$ cloudlinux-backup-helper-uid 1,1495530576 1000:/home/user/.bash_history [user@localhost ~]$ cloudlinux-backup-helper-uid -t 1495547922 1,1495548343 1000:/home/user/file1.txt 1000:/home/user/file2.txt
Installation and configuration
CloudLinux OS 6 (requires Hybrid kernel) or 7 Kernel Version: 3.10.0-427.36.1.lve1.4.47
Installation and configuration
To install cloudlinux-fchange system run:
CloudLinux OS Shared 7:
yum install cloudlinux-fchange --enablerepo=cloudlinux-updates-testing
CloudLinux OS Shared 6 Hybrid:
yum install cloudlinux-fchange --enablerepo=cloudlinux-hybrid-testing
Database containing list of modified files is located at /var/lve/cloudlinux-fchange.db by default.
Starting and stopping
After successful installation the event collecting daemon starts automatically, providing all kernel-exposed data are in place.
To start daemon: CloudLinux OS Shared 7:
systemctl start cloudlinux-file-change-collector
CloudLinux OS Shared 6 Hybrid:
service cloudlinux-file-change-collector start
systemctl stop cloudlinux-file-change-collector
CloudLinux OS Shared 6 Hybrid:
service cloudlinux-file-change-collector stop
To uninstall cloudlinux-fchange run:
yum remove cloudlinux-fchange
Configuration resides in /etc/sysconfig/cloudlinux-fchange. The following is the default configuration (see comments):
# sqlite database file path. If commented out a default value is used #database_path=/var/lve/cloudlinux-fchange.db # If uncommented paths starting with 'include' one are processed only # Pay attention this parameter is a regular string, not a regex # To include more than one item just specify several lines to include: # include=/one # include=/two # If uncommented exclude paths which contain 'exclude' # Pay attention this parameter is a regular string, not a regex # To exclude more than one item just specify several lines to exclude: # exclude=var # exclude=tmp # Daemon polling interval in seconds polling_interval=5 # Time to keep entries in days. Does not clean if commented out or zero time_to_keep=1 # User read-only mode minimal UID # If file change collector stopped, all users with UID >= user_ro_mode_min_uid # are restricted to write to their home directory. This prevents to miss # a file change event. # Value of -1 (default) allows to disable the feature user_ro_mode_min_uid=-1 # Minimal UID of events to be processed. # Events of users with UID less then specified are not handled. # By default 500 (non-system users for redhat-based systems) #minimal_event_uid=500 # SQLite shared lock prevents setting more restrictive locks. That is a # process cannot write to a database table when a concurrent process reads # from the table. As saving data to database is considered far more important # than getting them (data could be reread a second later after all), database # writer could try to terminate concurrent reading processes. Just set # terminate rivals to 'yes' to turn this ability on. # terminate_rivals=no # Events to be handled. Currently the following types of events are processed: # 1. file creation # 2. file deletion # 3. directory creation # 4. directory deletion # 5. file content/metadata modification # 6. file/directory attributes/ownership modification # 7. hardlink creation # 8. symlink creation # 9. file/directory moving/renaming # By default all events are processed. Keep in mind that events for a filepath # are cached, i.e if a file was deleted and then a file with the same absolute # name is created, only the deletion event is triggerred. Changing file # modification timestamp with command 'touch' will trigger modification event # as if a file content is modified. # Currently supported options are: # file_created, file_modified, file_deleted, dir_created, dir_deleted, # owner_changed, attrib_changed, moved, hardlink_created, symlink_created, all # Options that don't have 'file' or 'dir' prefix, applied to both files and # directories. To set more than one options, separate them with commas, # e.g. event_types=file_created,file_deleted,file_modified. Unknown options are # ignored. # # event_types=all
Please keep in mind, that current implementation implies that one process is writing to a database and another is reading from it. As reading sets shared lock to a database table, the writing process cannot write to the table until the lock is released. That’s why passing a timestamp to cloudlinux-backup-helpermatters: this way the number of records to be returned is substantially decreased, lowering the processing time and filtering out old records. Likewise, pay attention to narrowing the scope of events being recorded. Chances are that changing attributes, ownership, directory creation/deletion, symlink events are not relevant and there’s no need to keep them.
Using this options is dangerous, and might cause problems with CloudLinux File Change API.
The kernel exposes the functionality to /proc/sys/fs/datacycle/ folder.
enable - enable/disable the functionality. Write 1 to this file to enable, 0 to disable. If disabled, no events are coming to events file.
events - the modified files log itself. Events in the format <EVENT_ID>:<EVENT_TYPE_ID>:<USER_ID>:<FILE_PATH> are constantly appending to the end of the file if datacycle enabled. File events are never duplicated: if we have file modification event, we would not get file deletion event if the file has been later deleted. This events buffer has limited capacity, therefore from time to time, the events log requires flushing.
flush - a file for clearing events log. For flushing, the last event_id from the events file is written to this file. Right after this, events log is truncated to that event_id .
user_ro_mode - forbidding users with UIDs equal or bigger that set in this file writing to their home directories. At the boot, the file has -1. When it’s written positive value, say 500, the system starts effectively preventing users from modifying their home dirs (on write attempt a user gets ‘read-only filesystem’ error). This feature is designed to prevent users from updating their home dirs when events are not handled.
entries_in_buffer - just counter of log entries of events file.
min_event_uid - this file has minimal UID of events to be handled. Events from users with smaller UID are not handled. By default 500 (non-system users in redhat-based systems).
The tuned-profiles-cloudlinux package brings a range of kernel under-the-hood tunings to address high LA, iowait issues what were detected earlier on particular users deploys. The package also encloses OOM adjustments to prioritize the elimination of overrun PHP, lsphp, Phusion Passenger workers processes over other processes (e.g. ssh, a cron job).
There are three profiles provided by CloudLinux OS Shared:
# tuned-adm list | grep cloudlinux - cloudlinux-default - Default CloudLinux tuned profile - cloudlinux-dummy - Empty CloudLinux tuned profile - cloudlinux-vz - Empty CloudLinux tuned profile
cloudlinux-dummy and cloudlinux-vz are used for internal needs or when Virtuozzo/OpenVZ detected and actually do nothing.
cloudlinux-default is one to be used, it actually does the following:
- Switches CPU power consumption mode to the maximum. CPU operates at maximum performance at the maximum clock rate:
If standard software CPU governors are used.
- Applies the following kernel options:
vm.force_scan_thresh=100 - Improves kernel memory clean-up in case of big number of running LVE.
UBC parameters set the limits for the containers:
ubc.dirty_ratio=100 - Defines maximum RAM percentage for dirty memory pages. dirty_background_ratio=75 - Defines RAM percentage when to allow writing dirty pages on the disk.
- [CloudLinux OS Shared 7 only] Detects used disk types and changes elevator to 'deadline' for HDD and to 'noop' for SSD in /sys/block/[blockname]/queue/scheduler .
The script uses /sys/block/[blockname]/queue/rotational flag, some RAID controllers can not set it properly. For example, SSD used for RAID but rotational is set to 1 by RAID driver. As a workaround add the following to /etc/rc.d/rc.local to make it applied on boot:
echo "noop" > /sys/block/[blockname]/queue/scheduler echo "0" > /sys/block/[blockname]/queue/rotational
Where [blockname] is used device name, like sda/sdb .
And make it executable:
chmod +x /etc/rc.d/rc.local
[CloudLinux OS Shared 7 only] The profile sets I/O scheduler. For the normal discs the Deadline Scheduler is set to improve IO performance and decrease IO latency, for SSD - noop. When configuring scheduler I/O queue is changed and set to the value 1024 which improves overall I/O subsystem performance by caching IO requests in memory.
Disables transparent HugePage .
Provides adjustment group file for OOM-Killer to kill overrun php, lsphp and Phusion Passenger workers first.
yum install tuned-profiles-cloudlinux
To start using a profile:
tuned-adm profile cloudlinux-default
To stop using a profile:
Kernel config variables
Starting from lvemanager 4.0-25.5, lve-utils 3.0-21.2, and cagefs-6.1-26, CloudLinux OS Shared utilities can read/write kernel config variables from a custom config /etc/sysctl.d/90-cloudlinux.conf (earlier, the parameters were read/written only from sysctl.conf ).
CloudLinux OS Shared utilities get parameter by using
sysctl system utility. So for now, even if a config variable is not set in the
sysctl.conf and in the
/etc/sysctl.d config files, this variable will be read by
sysctl utility directly from
If you changed some kernel variables in
/etc/sysctl.d/90-cloudlinux.conf you need to apply these changes to the kernel parameter by running the command:
After that, the variable can be read by the sysctl utility.
Starting from cagefs-6.1-27, fs.proc_can_see_other_uid will be migrated (one time) from /etc/sysctl.conf into /etc/sysctl.d/90-cloudlinux.conf . If this variable is not set in either file, it will default to 0. It is strongly advised against setting this variable in 90-cloudlinux.conf . Define it in /etc/sysctl.conf or in some other config file with an index number greater than 90-cloudlinux.conf , e.g.
Starting from lve-utils-3.0-23.7 fs.proc_super_gid and fs.symlinkown_gid will be migrated (one time) from /etc/sysctl.conf into /etc/sysctl.d/90-cloudlinux.conf .
For lve-utils versions from 3.0-21.2 to 3.0-23.7 the migration was performed the same way, but during every package install/update. Variables setting guidelines are the same as for CageFS (see above).
The CL kernel with the lve module restricts an access to all files in /proc/net/* except the ones in the following whitelist:
tcp, tcp6, udp, udp6, assocs, raw, raw6, unix, dev
If you want to restrict an access even to those that are in the whitelist you should set
kernel.proc_disable_net = 1 (by default it’s disabled).
Virtualized /proc filesystem
You can prevent user from seeing processes of other users (via ps/top command) as well as special files in /proc file system by setting fs.proc_can_see_other_uid sysctl.
To do that, edit /etc/sysctl.conf
# sysctl -p
If fs.proc_can_see_other_uid is set to 0, users will not be able to see special files. If it is set to 1 - user will see other processes IDs in /proc filesystem.
The fs.proc_super_gid sets group ID which will see system files in /proc, add any users to that group so they will see all files in /proc. Usually needed by some monitoring users like nagios or zabbix and cldetect utility can configure few most commonly used monitoring software automatically.
Automatic configuration of a group in the fs.proc_super_gid
Starting from lve-utils v.4.2.0-1, when installing (not updating) the
lve-utils package it automatically creates the
clsupergid group and registers this group in the
fs.proc_super_gid (if this group was not created before). If a group was created in the
fs.proc_super_gid earlier all stays the same.
Virtualized /proc filesystem in CloudLinux OS Shared 8/CloudLinux OS Shared 7 hybrid will display all files to unprivileged users. But unprivileged users will have access only to the following files (as well as directories for PIDs for the user):
/proc/cpuinfo /proc/version /proc/stat /proc/uptime /proc/loadavg /proc/filesystems /proc/stat /proc/cmdline /proc/meminfo /proc/mounts /proc/net/tcp /proc/net/tcp6 /proc/net/udp /proc/net/udp6 /proc/net/assocs /proc/net/raw /proc/net/raw6 /proc/net/unix /proc/net/dev
Starting from lve-utils 3.0-21.2, fs.proc_super_gid parameter in da_add_admin utility is written to /etc/sysctl.d/90-cloudlinux.conf.
Remounting procfs with "hidepid" option
In lve-utils-2.1-3.2 and later
/proc can be remounted with
hidepid=2 option to enable additional protection for procfs. This remount is performed in lve_namespaces service. This option is in sync with
fs.proc_can_see_other_uid kernel parameter described above. When
/etc/sysctl.conf does not contain
fs.proc_can_see_other_uid setting, the protection is off (procfs is remounted with
hidepid=0 option). In this case
fs.proc_super_gid setting is ignored. Users are able to see full
/proc including processes of other users on a server. This is a default behavior.
fs.proc_can_see_other_uid=1 setting, then
/proc will be remounted with
hidepid=0 option (disable
hidepid protection for all users). If
fs.proc_can_see_other_uid=0 setting, then
/proc will be remounted with
hidepid=2 option (enable
hidepid protection for all users). If
fs.proc_super_gid=$GID settings, then
/proc will be remounted with
hidepid=2, gid=$GID options (enable
hidepid for all users except users in group with gid $GID).
/etc/sysctl.conf changes, you should execute
service lve_namespaces restart
So, admin can prevent users from seeing processes of other users via
fs.proc_super_gid settings in
/etc/sysctl.conf, like earlier.
Also, you can override this by specifying desired options for
To disable hidepid, add to
/etc/fstab the following:
proc /proc proc defaults,hidepid=0,gid=0 0 0
proc /proc proc defaults,hidepid=2,gid=clsupergid 0 0
mount -o remount /proc
Nevertheless, we recommend to manage procfs mount options via
/etc/sysctl.conf as described above for backward compatibility.
There is a known issue on CloudLinux OS Shared 6 systems. User cannot see full /proc inside CageFS even when this user is in “super” group, that should see full /proc. This issue does not affect users with CageFS disabled. CloudLinux OS Shared 7 is not affected.
Starting from lve-utils 3.0-21.2, lve_namespaces service can read parameters from the /etc/sysctl.d/90-cloudlinux.conf.
Even if fs.proc_can_see_other_uid and fs.proc_super_gid parameters are not set in config files but specified in /proc/sys, then when restarting lve_namespaces service the parameters from /proc/sys will be used. So, /proc will be remounted according to these parameters.
Remounting procfs on CloudLinux OS Shared 8
On CloudLinux OS Shared 8, starting from the kernel version
4.18.0-193.28.1.lve1.el8.x86_64 remounting is controlled by the kernel.
As before, admins are able to change the settings
fs.proc_super_gid in the
To apply changes, execute the following command:
It is no longer needed to manually run the
/usr/share/cloudlinux/remount_proc.py to remount
On Cloudlinux OS Shared 8, changing the
/etc/fstab file for setting
/proc remounting with the
hidepid= option will not cause any effect as remounting is totally controlled by the kernel and depends on the
Starting with kernel 3.10.0-427.18.s2.lve1.4.21 ( CloudLinux OS Shared 7) and 2.6.32-673.26.1.lve1.4.17 ( CloudLinux OS Shared 6) we re-implemented ptrace block to protect against ptrace family of vulnerabilities. It prevents end user from using any ptrace related functionality, including such commands as strace, lsof or gdb .
By default, CloudLinux OS Shared doesn't prevent ptrace functionality.
kernel.user_ptrace = 1 kernel.user_ptrace_self = 1
The option kernel.user_ptrace disables PTRACE_ATTACH functionality, option kernel.user_ptrace_self disables PTRACE_TRACEME .
To disable all ptrace functionality change both sysctl options to 0, add this section to /etc/sysctl.conf :
## CL. Disable ptrace for users kernel.user_ptrace = 0 kernel.user_ptrace_self = 0 ##
Apply changes with:
$ sysctl -p
Different software could need different access to ptrace , you may need to change only one option to 0 to make them working. In this case, there will be only partial ptrace protection.
ptrace protection is known to break PSA service for Plesk 11
2.6.32 kernels have different mode of naming Xen XVDA drives.
By adding xen_blkfront.sda_is_xvda=0 to kernel boot line in grub.conf you will make sure no naming translation is done, and the drives will be identified as xvde .
By default, this option is set to 1 in the kernel, and drives are detected as xvda . This is needed only for CloudLinux OS Shared 6 and Hybrid kernels.
CloudLinux OS Shared 6, CloudLinux OS Shared 6 hybrid, CloudLinux OS Shared 7, CloudLinux OS Shared 7 hybrid kernels.
Starting from the kernel module lve-kmod-2.0-10, the behavior of umask is changed.
Now, when entering LVE task's original umask value is preserved, instead of using LVE's umask value. This behavior is typical for all kernels: CloudLinux OS Shared 6, CloudLinux OS Shared 6 hybrid, CloudLinux OS Shared 7, CloudLinux OS Shared 7 hybrid kernels.
IO limits latency
When customer reaches IO Limit, the processes that are waiting for IO will be placed to sleep to make sure they don't go over the limit. That could make some processes sleep for a very long time. By defining IO latency, you can make sure that no process sleeps due to IO limit for more then X milliseconds. By doing so, you will also let customers to burst through the limits, and use up more than they were limited too in some instances.
This option is OFF by default.
For CloudLinux OS Shared 6 , CloudLinux OS Shared 7, CloudLinux OS Shared 8 (kmodlve 2.1-2 and later):
To enable IO Limits latency and set it to 10 seconds:
# echo 10000 > /sys/module/kmodlve/parameters/latency
# echo 2000000000 > /sys/module/kmodlve/parameters/latency
It is possible to set, for example, 1000 as a permanent value. To do so, create a file /etc/modprobe.d/kmodlve.conf with the following content:
options kmodlve latency=1000
For CloudLinux OS Shared 5 (OBSOLETE):
To enable IO Limits latency and set it to 10 seconds:
# echo 10000 > /sys/module/iolimits/**parameters/latency
# echo 2000000000 > /sys/module/iolimits/**parameters/latency
Reading LVE usage
CloudLinux OS Shared kernel provides real time usage data in file.
All the statistics can be read from that file in real time. Depending on your kernel version you will get either Version 6 of the file, or version 4 of the file. You can detect the version by reading the first line of the file. It should look like:
6:LVE... for version 6
4:LVE... for version 4
First line presents headers for the data. Second line shows default limits for the server, with all other values being 0. The rest of the lines present limits & usage data on per LVE bases.
Version 6 (CL6 & hybrid kernels):
6:LVE EP lCPU lIO CPU MEM IO lMEM lEP nCPU fMEM fEP lMEMPHY lCPUW lNPROC MEMPHY fMEMPHY NPROC fNPROC 0 0 25 1024 0 0 0 262144 20 1 0 0 262144 100 0 0 0 00 300 0 25 1024 1862407 0 0 262144 20 1 0 0 262144 100 0 31 000
|EP||Number of entry processes||number|
|lCPU||CPU Limit||% relative to total CPU power|
|lIO||IO limits for CL6||KB/s for v6, from 1 to 100 for v4|
|CPU||CPU usage since reboot||in nanoseconds for v6, hertz for v4|
|MEM||Virtual memory usage||number of 4k pages|
|IO||IO usage||KB/s for v6, 0 for v4|
|lMEM||Virtual memory limit||number of 4k pages|
|lEP||Entry Processes limit||number|
|nCPU||Number of cores limit||number of cores|
|fMEM||Virtual memory faults||number of faults|
|fEP||Entry Processes faults||number of faults||v6+|
|lMEMPHY||Physical memory limit||number||v6+|
|lCPUW||CPU weight (not used)||from 1 to 100||v6+|
|lNPROC||Number of processes limit||number||v6+|
|MEMPHY||Physical memory usage||number of 4k pages||v6+|
|fMEMPHY||Physical memory faults||number of faults||v6+|
|NPROC||Number of processes||number||v6+|
|fNPROC||Number of processes faults||number of faults||v6+|
|IOPS||IO operations since reboot||number||v8+|
Available only for x86_64, CloudLinux OS Shared 6 and Hybrid servers
Flashcache is a module originally written and released by Facebook (Mohan Srinivasan, Paul Saab and Vadim Tkachenko ) in April of 2010. It is a kernel module that allows Writethrough caching of a drive on another drive. This is most often used for caching a rotational drive on a smaller solid-state drive for performance reasons. This gives you the speed of an SSD and the size of a standard rotational drive for recently cached files. Facebook originally wrote the module to speed up database I/O , but it is easily extended to any I/O .
To install on CloudLinux OS Shared 6 & Hybrid servers:
$ yum install flashcache
More info on flashcache : https://github.com/facebook/flashcache/
ArchLinux has a good page explaining how to use flashcache : https://wiki.archlinux.org/index.php/Flashcache
OOM killer for LVE processes
When LVE reaches its memory limit, the processes inside that LVE are killed by OOM Killer and appropriate message is written to /var/log/messages . When any LVE hits huge number of memory limits in short period of time, then OOM Killer could cause system overload. Starting from kernel 2.6.32-673.26.1.lve1.4.15 ( CloudLinux OS Shared 6) and from kernel 3.10.0-427.18.2.lve1.4.14 ( CloudLinux OS Shared 7) heavy OOM Killer could be disabled. If so - lightweight SIGKILL will be used instead.
It is recommended to disable OOM killer for LVE processes and use SIGKILL instead
By default OOM Killer is enabled, to disable it please run:
For CloudLinux OS Shared 6 :
# echo 1 > /proc/sys/ubc/ubc_oom_disable
Also, add the following to /etc/sysctl.conf file to apply the same during boot:
For CloudLinux OS Shared 7:
# echo 1 > /proc/sys/kernel/memcg_oom_disable
Also, add the following to
/etc/sysctl.conf file to apply the same during boot:
File system quotas
In Ext4 file system, the process with enabled capability CAP_SYS_RESOURCE is not checked on the quota exceeding by default. It allows userland utilities selectorctl and cagefs to operate without fails even if a user exceeds a quota.
To disable quota checking in XFS file system set
cap_res_quota_disable option to 1 using the following command:
# echo 1 > /proc/sys/fs/xfs/cap_res_quota_disable
Enter LVE when using cPanel utilities
cPanel tools might use more resources than desired, so to limit resource usage, you might want to enter the corresponding LVE when using cPanel tools on-behalf of a non-root user.
This feature is considered experimental, as in this case there might be contention for LVE limits between cPanel tools and web-requests for a given user, which might not be suitable.
lve_setuid_enter parameter controls whether you want to enter the corresponding LVE when using cPanel tools on behalf of a non-root user.
By default, the feature is disabled (0), to enable it, run the following for CloudLinux OS Shared 7 hybrid:
# echo 1 > /sys/module/kmodlve/parameters/lve_setuid_enter
Proactive reporting kernel crash events with Sentry and Kernel Panic Receiver
As updating the kernel is one of the most sensitive tasks, we'd like to achieve the maximum stability along the way. In case something goes wrong, proactive reacting to kernel crashes can help tremendously.
Our new tool, Kernel Panic Receiver, released in June 2020 allows you to to organize kernel panic logs. Kernel Panic Receiver is an open-source component designed to pre-emptively resolve any kernel-related issues.
Visit the Kernel Panic Receiver project GitHub page.
To send required kernel logs from the clients' machines to Kernel Panic Receiver, we configure the default Linux kernel feature called netconsole.
The configuration is done by the
initscripts package, starting from the following versions:
- For CloudLinux OS Shared 6:
- For CloudLinux OS Shared 7:
- For CloudLinux OS Shared 8:
To update the
initscripts package, run the following command:
yum update initscripts --enablerepo=cloudlinux-updates-testing
When a kernel panic occurs, the netconsole module sends logs to our server as plain text via the UDP protocol.
What data is transferred by netconsole?
The netconsole sends only OOPs-related messages from the kernel ring buffer. It doesn't transfer any sensitive data, such as usernames, encryption keys, paths, etc. So, there are no security problems you should worry about.
Disabling the feature
If you don't want to send us the data, you can turn the netconsole service off (we don't recommend it, though). To disable transferring the data, just comment the
SYSLOGADDR parameter in the netconsole config file (
# For more information about this data transmission, check this page: # https://cloudlinux.zendesk.com/hc/en-us/articles/360016481200 # SYSLOGADDR=sentrykernel.cloudlinux.com
And stop the netconsole service by running the following command:
service netconsole stop
Netconsole is used only for Kernel Panic Receiver, so disabling it doesn't lead to issues with other CloudLinux OS Shared services.