sidebar hamburger menu

CloudLinux OS kernel

This documentation describes specific features of the CloudLinux kernel. In other cases the kernel has the same features and innovations as any similar RHEL kernel. More information about the actual kernel changes and releases can be obtained from our changelog.

Hybrid Kernels

Hybrid kernels give you the ability to take advantage of the benefits and features available in newer kernels without having to completely upgrade to another version of the operating system. Example - for the CloudLinux 7 kernel, based on version 3.10, you can install a hybrid kernel (same as on CloudLinux 8), which is based on version 4.18. This provides more kernel options, memory and overall optimization, as well as a positive impact on system performance.

How to migrate from the normal kernel to hybrid one

Note

The system must have an active CloudLinux OS license

Warning

If you use yum-plugin-protectbase, please make sure it is disabled before stating the normal-to-hybrid script.

You should perform the following commands:

yum update
normal-to-hybrid

Then if the script execution is completed without errors - perform the server reboot.

Note

You can always migrate from the hybrid kernel to normal one with the following commands:

yum update
hybrid-to-normal

Known limitations and issues of hybrid kernels

CloudLinux OS Shared 6 Hybrid kernel

  1. We do not remove Hybrid kernel after migration from Hybrid to the normal channel, but we remove linux-firmware package which is needed to boot Hybrid kernel. This is because CloudLinux OS Shared 6 does not allow to remove the package of currently running kernel. Thus please don't reboot the server back to Hybrid kernel after you remove it.

  2. Kernel module signature isn't checking for now, as 3.10 kernel is using x509 certificates to generate keys and CloudLinux OS Shared 6 cannot detect signatures created in such way.

Note

Link Traversal Protection is disabled by default for the CloudLinux OS Shared.

CloudLinux OS Shared and CloudLinux OS Admin provides comprehensive protection against symbolic link attacks popular in shared hosting environment.

The protection requires setting multiple kernel parameters to be enabled:

SecureLinks:

Link Traversal Protection:

Please reffer to this article in order to know how to setup these parameters.

fs.enforce_symlinksifowner

To protect against symlink attack where attacker tricks Apache web server to read some other user PHP config files, or other sensitive file, enable:

fs.enforce_symlinksifowner=1

Setting this parameter will deny any process running under gid fs.symlinkown_gid to follow the symlink if owner of the link doesn’t match the owner of the target file.

Defaults:

fs.enforce_symlinksifowner = 1
fs.symlinkown_gid = 48
fs.enforce_symlinksifowner = 0do not check symlink ownership
fs.enforce_symlinksifowner = 1deny if symlink ownership doesn’t match target, and process gid matches _symlinkown_gid

When fs.enforce_symlinksifowner set to 1, processes with GID 48 will not be able to follow symlinks if they are owned by user1 , but point to file owned user2 .

Note

fs.enforce_symlinksifowner = 2 is deprecated and can cause issues for the system operation.

fs.symlinkown_gid

On standard Apache installation, this webserver is usually running under GID 48. On cPanel servers, Apache is running under user nobody, GID 99.

To change GID of processes that cannot follow symlink, edit the file /etc/sysctl.conf , add the line:

fs.symlinkown_gid = XX

And execute:

sysctl -p

To disable symlink owner match protection feature, set fs.enforce_symlinksifowner = 0 in /etc/sysctl.conf , and execute:

sysctl -p

Note

fs.symlinkown_gid parameter values for httpd service user and fs.proc_super_gid for nagios service user is written to /etc/sysctl.d/90-cloudlinux.conf.

Warning

When used outside CageFS (from cPanel tools for instance), fs.protected_symlinks_create isn't sufficient for symlink protection. To fully protect symlink access in this case, use fs.process_symlinks_by_task=1 in addition to fs.protected_symlinks_create=1.

CageFS is extremely powerful at stopping most information disclosure attacks, where a hacker could read sensitive files like /etc/passwd .

Yet, CageFS does not work in each and every situation. For example, on cPanel servers, it is not enabled in WebDAV server, cPanel file manager and webmail, as well as some FTP servers don’t include proper change rooting.

This allows an attacker to create symlink or hardlink to a sensitive file like /etc/passwd and then use WebDAV , filemanager, or webmail to read the content of that file.

You can prevent such attacks by preventing user from creating symlinks and hardlinks to files that they don’t own.

This is done by set following kernel options to 1:

fs.protected_symlinks_create = 1
fs.protected_hardlinks_create = 1

We do not recommend to use protected_symlinks option for cPanel users as it might break some of the cPanel functionality. Please read the known issues section before enabling this feature.

Note

This option only available on CloudLinux 7 Hybrid or on CloudLinux 8 Shared and Admin editions. Also please note that this feature is available for cPanel only.

This parameter is needed to protect against symlink vulnerability where an attacker might get access to files out of the CageFS via cPanel tools: File Manager, WebDAV, Webmail, etc. When a symlink is accessed from cPanel tools (non-root user case) we check whether the current process UID matches the symlink target UID.

To enable the protection for CloudLinux OS Shared , set the fs.process_symlinks_by_task parameter to 1:

fs.process_symlinks_by_task=1

To disable the protection for CloudLinux OS Shared 7 hybrid, set the fs.process_symlinks_by_task parameter to 0:

fs.process_symlinks_by_task=0

Here are some examples of what may go wrong on cPanel servers. If fs.protected_symlinks_create=1 is set on the server, it can cause the following issues:

  • rsync is failed

If you use the rsync to copy/transfer files and they contain symlinks, you'll see the errors like these:

"rsync error: some files could not be transferred (code 23)" and the transfer will be failed. 

It affects rsync tasks as well as cPanel transfer tools.

  • backup extracting may not work. Errors during the restoration:
Error extracting /home/domain/backups/home.tar.gz : /bin/tar: .cagefs/tmp/mysql.sock: Cannot create symlink to `/var/lib/mysql/mysql.sock'.
Permission denied. /bin/tar: .cagefs/tmp/.s.PGSQL.5432: Cannot create symlink to `/var/run/postgres/.s.PGSQL.5432': No such file or directory.

Any backup for accounts (including cPanel backup) cannot be extracted.

  • dmesg is flooded with the may_create_sym_link messages like:
may_create_sym_link: can't find ea-phpXX in cron
may_create_sym_link: can't find ea-phpXX in ea-php-cli

It's popping up each second and may increase the size of the /var/log/messages file.

File change API

General

General description

One of the main problems on a shared hosting system for file backup operations is to figure out which files have changed. Using INOTIFY on a 1T drive with a large number of small files and directories guarantees slow startup times, and a lot of context switching between kernel and userspace - generating additional load. On the other hand scanning disk for newly modified files is very IO intensive, and can kill the performance of the fastest disks.

CloudLinux OS Shared approach

CloudLinux File Change API is a kernel level technology with the user space interface that buffers lists of modified files in the kernel and then off-loads that list to user space daemon.

After that - any software (with enough permissions) can get a list of files that has been modified for the last 24 hours.

The software is very simple to use and produces the list of modified files. As such we expect file backup software, including integrated cPanel backup system to integrate with this API soon.

Usage and integration

Userland utilities

/usr/bin/cloudlinux-backup-helper is a utility for getting the list of changed files.

It is supposed to be run by a super user only.

Command line parameters:

-t | --timestamp retrieve file names for files modified after specified timestamp
-u | --uid       retrieve file names for particular UID only
If no UID specified, are retrieved for all users. If no timestamp specified, all database events are shown.

Output format

protocol version (1 right now), timestamp (in seconds) - up to which time data was collected
UID:absolute path to file changed
UID:absolute path to file changed
…

Note

The timestamp in output is needed so you can clearly identify from which timestamp to get list of changed files next.

Examples:

[root@localhost ~]# cloudlinux-backup-helper -t 1495533489 -u <UID>
1,1495533925
1001:/home/user2/public_html/output.txt
1001:/home/user2/public_html/info.php
[root@localhost ~]# cloudlinux-backup-helper -t 1495533489
1,1495533925
1000:/home/user1/.bashrc
1001:/home/user2/public_html/output.txt
1001:/home/user2/public_html/info.php
1003:/home/user3/logs/data.log
**Getting changed files by end user**

/usr/bin/cloudlinux-backup-helper-uid is a SUID wrapper for the cloudlinux-backup-helper utility that enables an end user to get the list of files changed. It accepts timestamp argument only and retrieves data of the user who is running it only.

Examples:

[user@localhost ~]$ cloudlinux-backup-helper-uid               
1,1495530576
1000:/home/user/.bash_history 

[user@localhost ~]$ cloudlinux-backup-helper-uid -t 1495547922
1,1495548343
1000:/home/user/file1.txt
1000:/home/user/file2.txt
This command is available within CageFS.

Installation and configuration

cloudlinux-fchange-0.1-5

Requirements

CloudLinux OS 6 (requires Hybrid kernel) or 7 Kernel Version: 3.10.0-427.36.1.lve1.4.47

Installation and configuration

To install cloudlinux-fchange system run:

CloudLinux OS Shared 7:

yum install cloudlinux-fchange --enablerepo=cloudlinux-updates-testing

CloudLinux OS Shared 6 Hybrid:

yum install cloudlinux-fchange --enablerepo=cloudlinux-hybrid-testing
Configuration file can be found in /etc/sysconfig/cloudlinux-fchange

Database containing list of modified files is located at /var/lve/cloudlinux-fchange.db by default.

Starting and stopping

After successful installation the event collecting daemon starts automatically, providing all kernel-exposed data are in place.

To start daemon: CloudLinux OS Shared 7:

systemctl start cloudlinux-file-change-collector

CloudLinux OS Shared 6 Hybrid:

service cloudlinux-file-change-collector start
To stop daemon: _CloudLinux OS Shared 7:_
systemctl stop cloudlinux-file-change-collector

CloudLinux OS Shared 6 Hybrid:

service cloudlinux-file-change-collector stop

To uninstall cloudlinux-fchange run:

yum remove cloudlinux-fchange

Configuration details

Configuration resides in /etc/sysconfig/cloudlinux-fchange. The following is the default configuration (see comments):

# sqlite database file path. If commented out a default value is used
#database_path=/var/lve/cloudlinux-fchange.db

# If uncommented paths starting with 'include' one are processed only
# Pay attention this parameter is a regular string, not a regex
# To include more than one item just specify several lines to include:
# include=/one
# include=/two

# If uncommented exclude paths which contain 'exclude'
# Pay attention this parameter is a regular string, not a regex
# To exclude more than one item just specify several lines to exclude:
# exclude=var
# exclude=tmp

# Daemon polling interval in seconds
polling_interval=5 

# Time to keep entries in days. Does not clean if commented out or zero
time_to_keep=1

# User read-only mode minimal UID
# If file change collector stopped, all users with UID >= user_ro_mode_min_uid
# are restricted to write to their home directory. This prevents to miss
# a file change event.
# Value of -1 (default) allows to disable the feature
user_ro_mode_min_uid=-1

# Minimal UID of events to be processed.
# Events of users with UID less then specified are not handled.
# By default 500 (non-system users for redhat-based systems)
#minimal_event_uid=500
 
# SQLite shared lock prevents setting more restrictive locks. That is a
# process cannot write to a database table when a concurrent process reads
# from the table. As saving data to database is considered far more important
# than getting them (data could be reread a second later after all), database
# writer could try to terminate concurrent reading processes. Just set
# terminate rivals to 'yes' to turn this ability on.
# terminate_rivals=no 

# Events to be handled. Currently the following types of events are processed:
# 1. file creation
# 2. file deletion
# 3. directory creation
# 4. directory deletion
# 5. file content/metadata modification
# 6. file/directory attributes/ownership modification
# 7. hardlink creation
# 8. symlink creation
# 9. file/directory moving/renaming
# By default all events are processed. Keep in mind that events for a filepath
# are cached, i.e if a file was deleted and then a file with the same absolute
# name is created, only the deletion event is triggerred. Changing file
# modification timestamp with command 'touch' will trigger modification event
# as if a file content is modified.
# Currently supported options are:
# file_created, file_modified, file_deleted, dir_created, dir_deleted,
# owner_changed, attrib_changed, moved, hardlink_created, symlink_created, all
# Options that don't have 'file' or 'dir' prefix, applied to both files and
# directories. To set more than one options, separate them with commas,
# e.g. event_types=file_created,file_deleted,file_modified. Unknown options are
# ignored.
#
# event_types=all

Note

Please keep in mind, that current implementation implies that one process is writing to a database and another is reading from it. As reading sets shared lock to a database table, the writing process cannot write to the table until the lock is released. That’s why passing a timestamp to cloudlinux-backup-helpermatters: this way the number of records to be returned is substantially decreased, lowering the processing time and filtering out old records. Likewise, pay attention to narrowing the scope of events being recorded. Chances are that changing attributes, ownership, directory creation/deletion, symlink events are not relevant and there’s no need to keep them.

Low-level access

Note

Using this options is dangerous, and might cause problems with CloudLinux File Change API.

The kernel exposes the functionality to /proc/sys/fs/datacycle/ folder.

  1. enable - enable/disable the functionality. Write 1 to this file to enable, 0 to disable. If disabled, no events are coming to events file.

  2. events - the modified files log itself. Events in the format <EVENT_ID>:<EVENT_TYPE_ID>:<USER_ID>:<FILE_PATH> are constantly appending to the end of the file if datacycle enabled. File events are never duplicated: if we have file modification event, we would not get file deletion event if the file has been later deleted. This events buffer has limited capacity, therefore from time to time, the events log requires flushing.

  3. flush - a file for clearing events log. For flushing, the last event_id from the events file is written to this file. Right after this, events log is truncated to that event_id .

  4. user_ro_mode - forbidding users with UIDs equal or bigger that set in this file writing to their home directories. At the boot, the file has -1. When it’s written positive value, say 500, the system starts effectively preventing users from modifying their home dirs (on write attempt a user gets ‘read-only filesystem’ error). This feature is designed to prevent users from updating their home dirs when events are not handled.

  5. entries_in_buffer - just counter of log entries of events file.

  6. min_event_uid - this file has minimal UID of events to be handled. Events from users with smaller UID are not handled. By default 500 (non-system users in redhat-based systems).

Tuned-profiles-cloudlinux

The tuned-profiles-cloudlinux package brings a range of kernel under-the-hood tunings to address high LA, iowait issues what were detected earlier on particular users deploys. The package also encloses OOM adjustments to prioritize the elimination of overrun PHP, lsphp, Phusion Passenger workers processes over other processes (e.g. ssh, a cron job).

There are three profiles provided by CloudLinux OS Shared:

# tuned-adm list | grep cloudlinux
- cloudlinux-default          - Default CloudLinux tuned profile
- cloudlinux-dummy            - Empty CloudLinux tuned profile
- cloudlinux-vz               - Empty CloudLinux tuned profile

cloudlinux-dummy and cloudlinux-vz are used for internal needs or when Virtuozzo/OpenVZ detected and actually do nothing.

cloudlinux-default is one to be used, it actually does the following:

  1. Switches CPU power consumption mode to the maximum. CPU operates at maximum performance at the maximum clock rate:
governor=performance
energy_perf_bias=performance

Note

If standard software CPU governors are used.

  1. Applies the following kernel options:
    vm.force_scan_thresh=100 - Improves kernel memory clean-up in case of big number of running LVE.

UBC parameters set the limits for the containers:

ubc.dirty_ratio=100 - Defines maximum RAM percentage for dirty memory pages. dirty_background_ratio=75 - Defines RAM percentage when to allow writing dirty pages on the disk.

  1. [CloudLinux OS Shared 7 only] Detects used disk types and changes elevator to 'deadline' for HDD and to 'noop' for SSD in /sys/block/[blockname]/queue/scheduler .

Note

The script uses /sys/block/[blockname]/queue/rotational flag, some RAID controllers can not set it properly. For example, SSD used for RAID but rotational is set to 1 by RAID driver. As a workaround add the following to /etc/rc.d/rc.local to make it applied on boot:

echo "noop" > /sys/block/[blockname]/queue/scheduler  
echo "0" > /sys/block/[blockname]/queue/rotational

Where [blockname] is used device name, like sda/sdb .

And make it executable:

chmod +x /etc/rc.d/rc.local
  1. [CloudLinux OS Shared 7 only] The profile sets I/O scheduler. For the normal discs the Deadline Scheduler is set to improve IO performance and decrease IO latency, for SSD - noop. When configuring scheduler I/O queue is changed and set to the value 1024 which improves overall I/O subsystem performance by caching IO requests in memory.

  2. Disables transparent HugePage .

  3. Provides adjustment group file for OOM-Killer to kill overrun php, lsphp and Phusion Passenger workers first.

To install:

yum install tuned-profiles-cloudlinux

To start using a profile:

tuned-adm profile cloudlinux-default

To stop using a profile:

tuned-adm off

Kernel config variables

Starting from lvemanager 4.0-25.5, lve-utils 3.0-21.2, and cagefs-6.1-26, CloudLinux OS Shared utilities can read/write kernel config variables from a custom config /etc/sysctl.d/90-cloudlinux.conf (earlier, the parameters were read/written only from sysctl.conf ).

CloudLinux OS Shared utilities get parameter by using sysctl system utility. So for now, even if a config variable is not set in the sysctl.conf and in the /etc/sysctl.d config files, this variable will be read by sysctl utility directly from /proc/sys.

If you changed some kernel variables in /etc/sysctl.d/90-cloudlinux.conf you need to apply these changes to the kernel parameter by running the command:

sysctl --system

After that, the variable can be read by the sysctl utility.

Starting from cagefs-6.1-27, fs.proc_can_see_other_uid will be migrated (one time) from /etc/sysctl.conf into /etc/sysctl.d/90-cloudlinux.conf . If this variable is not set in either file, it will default to 0. It is strongly advised against setting this variable in 90-cloudlinux.conf . Define it in /etc/sysctl.conf or in some other config file with an index number greater than 90-cloudlinux.conf , e.g. /etc/sysctl.d/95-custom.conf.

Starting from lve-utils-3.0-23.7 fs.proc_super_gid and fs.symlinkown_gid will be migrated (one time) from /etc/sysctl.conf into /etc/sysctl.d/90-cloudlinux.conf .

For lve-utils versions from 3.0-21.2 to 3.0-23.7 the migration was performed the same way, but during every package install/update. Variables setting guidelines are the same as for CageFS (see above).

The CL kernel with the lve module restricts an access to all files in /proc/net/* except the ones in the following whitelist:

tcp, tcp6, udp, udp6, assocs, raw, raw6, unix, dev

If you want to restrict an access even to those that are in the whitelist you should set kernel.proc_disable_net = 1 (by default it’s disabled).

Virtualized /proc filesystem

You can prevent user from seeing processes of other users (via ps/top command) as well as special files in /proc file system by setting fs.proc_can_see_other_uid sysctl.

To do that, edit /etc/sysctl.conf

fs.proc_can_see_other_uid=0
fs.proc_super_gid=600
And do:
# sysctl -p
fs.proc_can_see_other_uid=0

If fs.proc_can_see_other_uid is set to 0, users will not be able to see special files. If it is set to 1 - user will see other processes IDs in /proc filesystem.

fs.proc_super_gid=XX

The fs.proc_super_gid sets group ID which will see system files in /proc, add any users to that group so they will see all files in /proc. Usually needed by some monitoring users like nagios or zabbix and cldetect utility can configure few most commonly used monitoring software automatically.

Automatic configuration of a group in the fs.proc_super_gid

Starting from lve-utils v.4.2.0-1, when installing (not updating) the lve-utils package it automatically creates the clsupergid group and registers this group in the fs.proc_super_gid (if this group was not created before). If a group was created in the fs.proc_super_gid earlier all stays the same.

Virtualized /proc filesystem in CloudLinux OS Shared 8/CloudLinux OS Shared 7 hybrid will display all files to unprivileged users. But unprivileged users will have access only to the following files (as well as directories for PIDs for the user):

/proc/cpuinfo
/proc/version
/proc/stat
/proc/uptime
/proc/loadavg
/proc/filesystems
/proc/stat
/proc/cmdline
/proc/meminfo
/proc/mounts
/proc/net/tcp
/proc/net/tcp6
/proc/net/udp
/proc/net/udp6
/proc/net/assocs
/proc/net/raw
/proc/net/raw6
/proc/net/unix
/proc/net/dev

Note

Starting from lve-utils 3.0-21.2, fs.proc_super_gid parameter in da_add_admin utility is written to /etc/sysctl.d/90-cloudlinux.conf.

Remounting procfs with "hidepid" option

In lve-utils-2.1-3.2 and later /proc can be remounted with hidepid=2 option to enable additional protection for procfs. This remount is performed in lve_namespaces service. This option is in sync with fs.proc_can_see_other_uid kernel parameter described above. When /etc/sysctl.conf does not contain fs.proc_can_see_other_uid setting, the protection is off (procfs is remounted with hidepid=0 option). In this case fs.proc_super_gid setting is ignored. Users are able to see full /proc including processes of other users on a server. This is a default behavior.

If /etc/sysctl.conf contains fs.proc_can_see_other_uid=1 setting, then /proc will be remounted with hidepid=0 option (disable hidepid protection for all users). If /etc/sysctl.conf contains fs.proc_can_see_other_uid=0 setting, then /proc will be remounted with hidepid=2 option (enable hidepid protection for all users). If /etc/sysctl.conf contains fs.proc_can_see_other_uid=0 and fs.proc_super_gid=$GID settings, then /proc will be remounted with hidepid=2, gid=$GID options (enable hidepid for all users except users in group with gid $GID).

To apply /etc/sysctl.conf changes, you should execute

service lve_namespaces restart
Or
/usr/share/cloudlinux/remount_proc.py

So, admin can prevent users from seeing processes of other users via fs.proc_can_see_other_uid and fs.proc_super_gid settings in /etc/sysctl.conf, like earlier.

Also, you can override this by specifying desired options for /proc in /etc/fstab.

To disable hidepid, add to /etc/fstab the following:

proc /proc proc defaults,hidepid=0,gid=0 0 0
Or you can specify desired hidepid and gid values explicitly:
proc /proc proc defaults,hidepid=2,gid=clsupergid 0 0
You should execute
mount -o remount /proc

to apply /etc/fstab changes.
Nevertheless, we recommend to manage procfs mount options via /etc/sysctl.conf as described above for backward compatibility.

Note

There is a known issue on CloudLinux OS Shared 6 systems. User cannot see full /proc inside CageFS even when this user is in “super” group, that should see full /proc. This issue does not affect users with CageFS disabled. CloudLinux OS Shared 7 is not affected.

Note

Starting from lve-utils 3.0-21.2, lve_namespaces service can read parameters from the /etc/sysctl.d/90-cloudlinux.conf.

Note

Even if fs.proc_can_see_other_uid and fs.proc_super_gid parameters are not set in config files but specified in /proc/sys, then when restarting lve_namespaces service the parameters from /proc/sys will be used. So, /proc will be remounted according to these parameters.

Remounting procfs on CloudLinux OS Shared 8

On CloudLinux OS Shared 8, starting from the kernel version 4.18.0-193.28.1.lve1.el8.x86_64 remounting is controlled by the kernel.

As before, admins are able to change the settings fs.proc_can_see_other_uid and fs.proc_super_gid in the /etc/sysctl.d/90-cloudlinux.conf or /etc/sysctl.conf.

To apply changes, execute the following command:

sysctl --system

It is no longer needed to manually run the /usr/share/cloudlinux/remount_proc.py to remount /proc.

Note

On Cloudlinux OS Shared 8, changing the /etc/fstab file for setting /proc remounting with the hidepid= option will not cause any effect as remounting is totally controlled by the kernel and depends on the fs.proc_can_see_other_uid option.

Ptrace block

Starting with kernel 3.10.0-427.18.s2.lve1.4.21 ( CloudLinux OS Shared 7) and 2.6.32-673.26.1.lve1.4.17 ( CloudLinux OS Shared 6) we re-implemented ptrace block to protect against ptrace family of vulnerabilities. It prevents end user from using any ptrace related functionality, including such commands as strace, lsof or gdb .

By default, CloudLinux OS Shared doesn't prevent ptrace functionality.

Defaults:

kernel.user_ptrace = 1
kernel.user_ptrace_self = 1

The option kernel.user_ptrace disables PTRACE_ATTACH functionality, option kernel.user_ptrace_self disables PTRACE_TRACEME .

To disable all ptrace functionality change both sysctl options to 0, add this section to /etc/sysctl.conf :

## CL. Disable ptrace for users
kernel.user_ptrace = 0
kernel.user_ptrace_self = 0
##

Apply changes with:

$ sysctl -p

Different software could need different access to ptrace , you may need to change only one option to 0 to make them working. In this case, there will be only partial ptrace protection.

ptrace protection is known to break PSA service for Plesk 11

Xen XVDA

2.6.32 kernels have different mode of naming Xen XVDA drives.

By adding xen_blkfront.sda_is_xvda=0 to kernel boot line in grub.conf you will make sure no naming translation is done, and the drives will be identified as xvde .

By default, this option is set to 1 in the kernel, and drives are detected as xvda . This is needed only for CloudLinux OS Shared 6 and Hybrid kernels.

Umask behavior

Note

CloudLinux OS Shared 6, CloudLinux OS Shared 6 hybrid, CloudLinux OS Shared 7, CloudLinux OS Shared 7 hybrid kernels.

Starting from the kernel module lve-kmod-2.0-10, the behavior of umask is changed.

Now, when entering LVE task's original umask value is preserved, instead of using LVE's umask value. This behavior is typical for all kernels: CloudLinux OS Shared 6, CloudLinux OS Shared 6 hybrid, CloudLinux OS Shared 7, CloudLinux OS Shared 7 hybrid kernels.

IO limits latency

[lve1.2.29+]

When customer reaches IO Limit, the processes that are waiting for IO will be placed to sleep to make sure they don't go over the limit. That could make some processes sleep for a very long time. By defining IO latency, you can make sure that no process sleeps due to IO limit for more then X milliseconds. By doing so, you will also let customers to burst through the limits, and use up more than they were limited too in some instances.

This option is OFF by default.

For CloudLinux OS Shared 6 , CloudLinux OS Shared 7, CloudLinux OS Shared 8 (kmodlve 2.1-2 and later):

To enable IO Limits latency and set it to 10 seconds:

echo 10000 > /sys/module/kmodlve/parameters/latency
To disable latency:
echo 2000000000 > /sys/module/kmodlve/parameters/latency

It is possible to set, for example, 1000 as a permanent value. To do so, create a file /etc/modprobe.d/kmodlve.conf with the following content:
options kmodlve latency=1000

For CloudLinux OS Shared 5 (OBSOLETE):

To enable IO Limits latency and set it to 10 seconds:

echo 10000 > /sys/module/iolimits/**parameters/latency
To disable latency:
echo 2000000000 > /sys/module/iolimits/**parameters/latency

Reading LVE usage

CloudLinux OS Shared kernel provides real time usage data in file.

All the statistics can be read from that file in real time. Depending on your kernel version you will get either Version 6 of the file, or version 4 of the file. You can detect the version by reading the first line of the file. It should look like:

6:LVE... for version 6
4:LVE... for version 4

First line presents headers for the data. Second line shows default limits for the server, with all other values being 0. The rest of the lines present limits & usage data on per LVE bases.

Version 6 (CL6 & hybrid kernels):

6:LVE        EP        lCPU        lIO        CPU        MEM        IO        lMEM        lEP        nCPU        fMEM        fEP        lMEMPHY        lCPUW        lNPROC        MEMPHY        fMEMPHY        NPROC        fNPROC
0        0        25        1024        0        0        0        262144        20        1        0        0        262144        100        0        0        0        00
300        0        25        1024        1862407        0        0        262144        20        1        0        0        262144        100        0        31        000
LabelDescriptionValueSupported versions
LVE LVE ID number
EP Number of entry processes number
lCPU CPU Limit% relative to total CPU power
lIO IO limits for CL6KB/s for v6, from 1 to 100 for v4
CPU CPU usage since rebootin nanoseconds for v6, hertz for v4
MEM Virtual memory usagenumber of 4k pages
IO IO usageKB/s for v6, 0 for v4
lMEM Virtual memory limitnumber of 4k pages
lEP Entry Processes limitnumber
nCPU Number of cores limitnumber of cores
fMEM Virtual memory faultsnumber of faults
fEP Entry Processes faultsnumber of faultsv6+
lMEMPHY Physical memory limitnumberv6+
lCPUW CPU weight (not used)from 1 to 100v6+
lNPROC Number of processes limitnumberv6+
MEMPHY Physical memory usagenumber of 4k pagesv6+
fMEMPHY Physical memory faultsnumber of faultsv6+
NPROC Number of processes numberv6+
fNPROC Number of processes faultsnumber of faultsv6+
IOPS IO operations since rebootnumberv8+

Flashcache

Note

Available only for x86_64, CloudLinux OS Shared 6 and Hybrid servers

Flashcache is a module originally written and released by Facebook (Mohan Srinivasan, Paul Saab and Vadim Tkachenko ) in April of 2010. It is a kernel module that allows Writethrough caching of a drive on another drive. This is most often used for caching a rotational drive on a smaller solid-state drive for performance reasons. This gives you the speed of an SSD and the size of a standard rotational drive for recently cached files. Facebook originally wrote the module to speed up database I/O , but it is easily extended to any I/O .

To install on CloudLinux OS Shared 6 & Hybrid servers:

yum install flashcache

More info on flashcache : https://github.com/facebook/flashcache/

ArchLinux has a good page explaining how to use flashcache : https://wiki.archlinux.org/index.php/Flashcache

OOM killer for LVE processes

When LVE reaches its memory limit, the processes inside that LVE are killed by OOM Killer and appropriate message is written to /var/log/messages . When any LVE hits huge number of memory limits in short period of time, then OOM Killer could cause system overload. Starting from kernel 2.6.32-673.26.1.lve1.4.15 ( CloudLinux OS Shared 6) and from kernel 3.10.0-427.18.2.lve1.4.14 ( CloudLinux OS Shared 7) heavy OOM Killer could be disabled. If so - lightweight SIGKILL will be used instead.

Note

It is recommended to disable OOM killer for LVE processes and use SIGKILL instead

By default OOM Killer is enabled, to disable it please run:

For CloudLinux OS Shared 6 :

echo 1 > /proc/sys/ubc/ubc_oom_disable

Also, add the following to /etc/sysctl.conf file to apply the same during boot:

ubc.ubc_oom_disable=1

For CloudLinux OS Shared 7:

echo 1 > /proc/sys/kernel/memcg_oom_disable

Also, add the following to /etc/sysctl.conf file to apply the same during boot:

kernel.memcg_oom_disable=1

File system quotas

In Ext4 file system, the process with enabled capability CAP_SYS_RESOURCE is not checked on the quota exceeding by default. It allows userland utilities selectorctl and cagefs to operate without fails even if a user exceeds a quota.

To disable quota checking in XFS file system set cap_res_quota_disable option to 1 using the following command:

echo 1 > /proc/sys/fs/xfs/cap_res_quota_disable

Enter LVE when using cPanel utilities

cPanel tools might use more resources than desired, so to limit resource usage, you might want to enter the corresponding LVE when using cPanel tools on-behalf of a non-root user.

This feature is considered experimental, as in this case there might be contention for LVE limits between cPanel tools and web-requests for a given user, which might not be suitable.

The lve_setuid_enter parameter controls whether you want to enter the corresponding LVE when using cPanel tools on behalf of a non-root user.

By default, the feature is disabled (0), to enable it, run the following for CloudLinux OS Shared 7 hybrid:

echo 1 > /sys/module/kmodlve/parameters/lve_setuid_enter

Proactive reporting kernel crash events with Sentry and Kernel Panic Receiver

As updating the kernel is one of the most sensitive tasks, we'd like to achieve the maximum stability along the way. In case something goes wrong, proactive reacting to kernel crashes can help tremendously.

Our new tool, Kernel Panic Receiver, released in June 2020 allows you to to organize kernel panic logs. Kernel Panic Receiver is an open-source component designed to pre-emptively resolve any kernel-related issues.

Visit the Kernel Panic Receiver project GitHub page.

Usage

To send required kernel logs from the clients' machines to Kernel Panic Receiver, we configure the default Linux kernel feature called netconsole.

The configuration is done by the initscripts package, starting from the following versions:

  • For CloudLinux OS Shared 6: 9.03.61-1.cloudlinux
  • For CloudLinux OS Shared 7: 9.49.49-1.cloudlinux
  • For CloudLinux OS Shared 8: 10.00.4-1.cloudlinux

To update the initscripts package, run the following command:

yum update initscripts --enablerepo=cloudlinux-updates-testing

When a kernel panic occurs, the netconsole module sends logs to our server as plain text via the UDP protocol.

What data is transferred by netconsole?

The netconsole sends only OOPs-related messages from the kernel ring buffer. It doesn't transfer any sensitive data, such as usernames, encryption keys, paths, etc. So, there are no security problems you should worry about.

Disabling the feature

If you don't want to send us the data, you can turn the netconsole service off (we don't recommend it, though). To disable transferring the data, just comment the SYSLOGADDR parameter in the netconsole config file (/etc/sysconfig/netconsole):

# For more information about this data transmission, check this page:
# https://cloudlinux.zendesk.com/hc/en-us/articles/360016481200
# SYSLOGADDR=sentrykernel.cloudlinux.com

And stop the netconsole service by running the following command:

service netconsole stop

Note

Netconsole is used only for Kernel Panic Receiver, so disabling it doesn't lead to issues with other CloudLinux OS Shared services.