VirtualBox update / kernel update

The VirtualBox GUI usually annoys with information on new versions of the VirtualBox (rpm).

To upgrade VirtualBox, follow the link provided in the message window:

1. http://download.virtualbox.org/virtualbox/4.3.20/ and then get correct rpm and extpack VirtualBox-4.3-4.3.20_96996_el6-1.x86_64.rpm
   Oracle_VM_VirtualBox_Extension_Pack-4.3.20.vbox-extpack 
2. stop all virtual machines
3. upgrade
   

   rpm -Uvh VirtualBox-4.3-4.3.20_96996_el6-1.x86_64.rpm 
   VBoxManage extpack install Oracle_VM_VirtualBox_Extension_Pack-4.3.20.vbox-extpack 

   (rpm rebuilds the new kernel module and
   extpack installs /usr/share/virtualbox/VBoxGuestAdditions.iso )
4. start virtual machines

When kernel is to be updated then

1. update kernel
2. reboot
3. rebuild kernel modules
   

   /etc/init.d/vboxdrv setup
   /etc/init.d/vboxdrv restart

4. start virtual machines

3ware 9500S-8: replace failing drive from raid 5

I have failing drive at port 3 of my 3ware sata controler. From /var/log/messages:

Dec 15 00:01:25 kernel: 3w-9xxx: scsi2: AEN: WARNING (0x04:0x000F): SMART threshold exceeded:port=3.

What we actually have:

]# tw_cli  show

Ctl   Model        Ports   Drives   Units   NotOpt   RRate   VRate   BBU
------------------------------------------------------------------------
c2    9500S-8      8       8        2       0        5       5       -     

and configuration:

]# tw_cli /c2 show

Unit  UnitType  Status         %RCmpl  %V/I/M  Stripe  Size(GB)  Cache  AVrfy
------------------------------------------------------------------------------
u0    RAID-5    VERIFY-PAUSED  -       22      64K     11175.8   ON     ON     
u1    SPARE     VERIFY-PAUSED  -       98      -       1863.01   -      ON     

Port   Status           Unit   Size        Blocks        Serial
---------------------------------------------------------------
p0     OK               u0     1.82 TB     3907029168    Z1E0AH83            
p1     OK               u0     1.82 TB     3907029168    Z1E5AL3Q            
p2     OK               u0     1.82 TB     3907029168    Z1E0BBE6            
p3     SMART-FAILURE    u0     1.82 TB     3907029168    5XW28M3A            
p4     OK               u0     1.82 TB     3907029168    W1H2T46V            
p5     OK               u0     1.82 TB     3907029168    W1E1SL7X            
p6     VERIFYING        u1     1.82 TB     3907029168    W2411LK9            
p7     OK               u0     1.82 TB     3907029168    W2411SSY            

Despite of the SMART-falure the array seems to be working fine (no degraded regime). However,  I'll remove the drive and use spare instead. Then I can test the failing drive and/or replace it.

]# tw_cli /c2 remove p3

Removing port /c2/p3 ... Done.

]#  tw_cli /c2 rescan

Rescanning controller /c2 for units and drives ...Done.
Found the following unit(s): [none].
Found the following drive(s): [/c2/p3].

]# tw_cli /c2/u0 show

Unit     UnitType  Status         %RCmpl  %V/I/M  Port  Stripe  Size(GB)
------------------------------------------------------------------------
u0       RAID-5    REBUILD-PAUSED 0       -       -     64K     11175.8   
u0-0     DISK      OK             -       -       p0    -       1862.63   
u0-1     DISK      OK             -       -       p2    -       1862.63   
u0-2     DISK      DEGRADED       -       -       p6    -       1862.63   
u0-3     DISK      OK             -       -       p1    -       1862.63   
u0-4     DISK      OK             -       -       p5    -       1862.63   
u0-5     DISK      OK             -       -       p4    -       1862.63   
u0-6     DISK      OK             -       -       p7    -       1862.63   

u0 should be rebuilding now, but it is not. To start rebuild it is necessary to disable scheduled actions (rebuild and verify):

]# tw_cli sched rebuild c2 disable

Disabling scheduled rebuilds on controller /c2 ...Done.

]# tw_cli /c2 show

Unit  UnitType  Status         %RCmpl  %V/I/M  Stripe  Size(GB)  Cache  AVrfy
------------------------------------------------------------------------------
u0    RAID-5    REBUILDING     0       -       64K     11175.8   ON     ON     

Now we can export the failing disk for testing:

]# tw_cli /c2 add type=single disk=3 name=drive-test

Creating new unit on controller /c2 ...  Done. The new unit is /c2/u1.
Naming unit /c2/u1 to [drive-test] ... Done.
Setting write cache=ON for the new unit ... Done.

The failing disk will be present as a separate device (/dev/sdd)

 tw_cli /c2 show 

Unit  UnitType  Status         %RCmpl  %V/I/M  Stripe  Size(GB)  Cache  AVrfy
------------------------------------------------------------------------------
u0    RAID-5    REBUILDING     1       -       64K     11175.8   ON     ON     
u1    SINGLE    OK             -       -       -       1862.63   ON     OFF    

Port   Status           Unit   Size        Blocks        Serial
---------------------------------------------------------------
p0     OK               u0     1.82 TB     3907029168    Z1E0AH83            
p1     OK               u0     1.82 TB     3907029168    Z1E5AL3Q            
p2     OK               u0     1.82 TB     3907029168    Z1E0BBE6            
p3     SMART-FAILURE    u1     1.82 TB     3907029168    5XW28M3A            
p4     OK               u0     1.82 TB     3907029168    W1H2T46V            
p5     OK               u0     1.82 TB     3907029168    W1E1SL7X            
p6     DEGRADED         u0     1.82 TB     3907029168    W2411LK9            
p7     OK               u0     1.82 TB     3907029168    W2411SSY    

What does smartctl say:

]# smartctl -a -d 3ware,3 /dev/twa0

Model Family:     Seagate Barracuda LP
Device Model:     ST32000542AS
Serial Number:    5XW28M3A
...
  5 Reallocated_Sector_Ct   0x0033   014   014   036    Pre-fail  Always   FAILING_NOW 3556
  9 Power_On_Hours          0x0032   067   067   000    Old_age   Always       -       29287
197 Current_Pending_Sector  0x0012   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age   Always       -       0

This Seagate Barracuda Drive is intended for NAS, thus it should be capable of 24/7 load. It is not too old (29287/24/365 = 3.33 years), but definitely older than its 3 years warranty. However, lack of pending and uncorrectable records gives some hope. First I try the long selftest, then multiple rewriting using badblocks.

]# time smartctl -C -t select,0-max -d 3ware,3 /dev/twa0

In this case the test fails immediately (I have not seen such behaviour yet):

Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Selective captive   Completed: unknown failure    90%     29287         0

and badblocks:

]# time badblocks -svw -o bbl-sdd.out  /dev/sdd 

Checking for bad blocks in read-write mode
From block 0 to 1953114111
...                                
Reading and comparing: done                                
Pass completed, 1184 bad blocks found.

real    6803m14.142s
user    234m16.798s
sys     120m22.010s

The drive is really ready for replacement. I will use a WD RE2-GP (pdf) drive instead (I have one in the shelf) even though I have some bad experiences with WD drives and 3ware 9500S controller.

tw_cli /c2 remove u1
tw_cli /c2 rescan

Rescanning controller /c2 for units and drives ...Done.
Found the following unit(s): [/c2/u1].
Found the following drive(s): [none].

]# tw_cli  /c2 show 

Unit  UnitType  Status         %RCmpl  %V/I/M  Stripe  Size(GB)  Cache  AVrfy
------------------------------------------------------------------------------
u0    RAID-5    VERIFY-PAUSED  -       51      64K     11175.8   ON     ON     
u1    JBOD      OK             -       -       -       1863.02   OFF    OFF    

Port   Status           Unit   Size        Blocks        Serial
---------------------------------------------------------------
...
p2     OK               u0     1.82 TB     3907029168    Z1E0BBE6            
p3     OK               u1     1.82 TB     3907029168    WD-WCAVY5487741     
p4     OK               u0     1.82 TB     3907029168    W1H2T46V            
...         

Use u1 as a spare:

tw_cli maint deleteunit c2 u1
tw_cli /c2 add type=spare disk=3

Creating new unit on controller /c2 ...  Done. The new unit is /c2/u1.

]# tw_cli  /c2 show 

Unit  UnitType  Status         %RCmpl  %V/I/M  Stripe  Size(GB)  Cache  AVrfy
------------------------------------------------------------------------------
u0    RAID-5    VERIFYING      -       53      64K     11175.8   ON     ON     
u1    SPARE     OK             -       -       -       1863.01   -      OFF    

Port   Status           Unit   Size        Blocks        Serial
---------------------------------------------------------------
...          
p3     OK               u1     1.82 TB     3907029168    WD-WCAVY5487741     
...       

]# tw_cli /c2 set verify=enable
]# tw_cli /c2/u1 set autoverify=on

smartctl shows fairly good values:

smartctl -a -d 3ware,4 /dev/twa0

Model Family:     Western Digital RE4-GP
Device Model:     WDC WD2002FYPS-02W3B0
Firmware Version: 04.01G01
User Capacity:    2,000,398,934,016 bytes [2.00 TB]

  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       24
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   093   093   000    Old_age   Always       -       5829
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       21
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   200   200   000    Old_age   Offline      -       0

Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed without error       00%      5431         -

Enable writeback cache, if necessary

# tw_cli /c2/u0 set cache=on 

Long live array!
Links:
 http://wiki.hetzner.de/index.php/3Ware_RAID_Controller/en

---

update 2016/07/01: 3Ware controller is stuck in REBUILD-PAUSED status.

To force the rebuild to start, toggle the setting for the rebuild schedule. The rebuild should then start.

tw_cli sched rebuild c2 enable
tw_cli sched rebuild c2 disable

Java plugin installation

1. check old version	rpm -qa |grep jre
2. remove old version	rpm -e jre-1.7.0_55
3. install new version	rpm -ivh jre-8u25-linux-x64.rpm
4. find plugin library	rpm -qal | grep libnpjp2.so
5. create link in browser's plugin repository	ln -sf ...path.../libnpjp2.so /usr/lib64/mozilla/plugins/
6. configure java (control panel)	jcontrol

Open-AudIT on CentOS 7 linux

Adapted form http://sathisharthars.wordpress.com/2014/03/19/installing-open-audit-in-centos-6/
The new versions of Open-AudIT require usage of it's "installator". That does not support CentOS 7.

Check dependency: 
Check hostname and timezone: 

uname -n
ls -l /etc/localtime 

epel-release is necessary

yum -y install epel-release

Install components:
Apache, mysql (MariaDB is default in CentOS 7). Set root password of mysql to 'PASSWORD'

yum -y install mariadb-server mariadb httpd
systemctl start mariadb.service
systemctl enable mariadb.service
systemctl start httpd.service
systemctl enable httpd.service
mysql_secure_installation 

yum -y install php php-cli php-mysql php-ldap php-mbstring php-mcrypt php-snmp \
               php-xml nmap zip curl wget sshpass screen samba-client          \
               perl-version php-process perl-Time-modules

We also need to install winexe. It is not in repositiories, but available for most distributions via the SuSe Build Server. Go to the URL http://download.opensuse.org/repositories/home:/ahajda:/winexe/ and download the relevant package for your distribution.
I am not sure, if it is necessary.  There is no CentOS 7 version available, but older version installs without complaints

wget http://download.opensuse.org/repositories/home:/ahajda:/winexe/CentOS_CentOS-6/x86_64/winexe-1.00-2.4.x86_64.rpm
rpm -ivh winexe-1.00-2.4.x86_64.rpm

setup php.ini:

TIMEZONE=`ll /etc/localtime |sed -e 's+.*/\([^/]*/[^/]*$\)+\1+'`
sed -i -e 's/memory_limit/;memory_limit/g' /etc/php.ini
echo "memory_limit = 512M" >> /etc/php.ini
sed -i -e 's/max_execution_time/;max_execution_time/g' /etc/php.ini
echo "max_execution_time = 300" >> /etc/php.ini
sed -i -e 's/max_input_time/;max_input_time/g' /etc/php.ini
echo "max_input_time = 600" >> /etc/php.ini
sed -i -e 's/error_reporting/;error_reporting/g' /etc/php.ini
echo "error_reporting = E_ALL" >> /etc/php.ini
sed -i -e 's/display_errors/;display_errors/g' /etc/php.ini
echo "display_errors = On" >> /etc/php.ini
sed -i -e 's/upload_max_filesize/;upload_max_filesize/g' /etc/php.ini
echo "upload_max_filesize = 10M" >> /etc/php.ini
sed -i -e 's/date.timezone/;date.timezone/g' /etc/php.ini
echo "date.timezone = $TIMEZONE" >> /etc/php.ini

Set the server name and shell (used for scripts) for Apache and restart daemon

HOSTNAME=`uname -n`
echo "ServerName $HOSTNAME" >> /etc/httpd/conf/httpd.conf
chsh -s /bin/bash apache
systemctl restart httpd.service

Set the SUID for the nmap binary (so we can use the apache front end to run scripts which call nmap).

chmod u+s /usr/bin/nmap

You should be able to determine the IP Address of your Open-AudIT server via the following command:

ifconfig | grep -Eo 'inet (addr:)?([0-9]*\.){3}[0-9]*' | grep -Eo '([0-9]*\.){3}[0-9]*' | grep -v '127.0.0.1'

Install OpenAudit using standard installer. The errors are normal in this case.

wget http://dl-openaudit.opmantek.com/OAE-Linux-x86_64-release_1.4.1.tar.gz
tar xvf OAE-Linux-x86_64-release_1.4.1.tar.gz 
cd Open-AudIT*
./installer -t /var/www/oa

Note that the installed software resides in three separate places:
/var/www/oa ,
/var/www/html/open-audit,
/usr/local/open-audit
It seems to be better to let open-audIT to install to the default location: /usr/local.

Open ports 80, 443, 8042 in firewall or disable it completely. Disable selinux.

systemctl stop firewalld
setenforce 0

Selinux experiments:
enable httpd to exec scripts:

semanage fcontext -a -t httpd_sys_script_exec_t '/usr/local/open-audit(/.*)?'

restore context

restorecon -R /usr/local/open-audit/

set:

setsebool -P nis_enabled 1
setsebool -P httpd_can_network_connect 1

Let the selinux to to teach itselves:

setenforce permissive

do some work...

grep httpd /var/log/audit/audit.log | audit2allow -m httpdopenaudit > httpdopenaudit.te
less httpdopenaudit.te 

grep httpd /var/log/audit/audit.log | audit2allow -M httpdopenaudit 
semodule -i httpdopenaudit.pp

and enable selinux:

setenforce enforcing

 

SSH based VPN for virtual machine

My virtual machine is hidden behind a host-based NAT.  It should work as a (headless) test server. I need to see and access all its ports. The virtual machine is a fresh system, accessible via ssh (VirtualBox port forwarding: host:2222 -> guest:22 ).

1. virtual (as root):

vi /etc/ssh/sshd_config 
 PermitTunnel yes
:wq 

systemctl restart sshd 
systemctl stop firewalld

2. host (as root):

ssh -w5:5 -p 2222 root@localhost ifconfig tun5 10.0.1.1 netmask 255.255.255.252 up &
sleep 3
ifconfig tun5 10.0.1.2 netmask 255.255.255.252 up
ping 10.0.1.1

Links:
Thanks to: http://sleepyhead.de/howto/?href=vpn

CentOs: rsyncd + selinux

Preliminary info - not sure if all steps are necessary.
symptoms: client cannot connect with @ERROR: chroot failed
rsyncd.log on server:
 rsync: chdir /data/rsync_direcory failed: Permission denied (13)


first try (no visible effect):
setsebool -P allow_rsync_anon_write on

next (agaiin, no effect):
semanage fcontext -a -t public_content_t /data/rsync_directory

does not make sense, as we are on server:
setsebool -P rsync_client on

finally - after this it works:
cat /var/log/audit/audit.log | audit2allow -M rsync
semodule -i rsync-.pp

grub repair

1. boot CentOS rescue USB/CD

2. allow the rescue manager to mount root (and boot) partition (as /mnt/sysimage) - not necessary.
The disk setup is as follows:

/dev/sda = (hd0) = usb boot drive 
/dev/sdb = (hd1) = normal boot drive with failed grub.

3. start grub

server:# grub

4. I have separate boot and root partitions: /dev/sda1 (boot), /dev/sda3 (root). In a single root/boot partition setup try to find /boot/grub/stage1

grub> find /grub/stage1
(hd1,0)
grub> root (hd1,0)
grub> setup (hd1)
grub> quit 

In this case grub uses grub/menu.lstto setup the configuration. Even if it refers to (HD0)

5. exit, reboot