Category Archives: Linux

CentOS / CloudLinux 6 locale issue

When you use Terminal to ssh into a freshly installed CentOS / CloudLinux 6, you would encounter this:

-bash: warning: setlocale: LC_CTYPE: cannot change locale (UTF-8): No such file or directory

There are a few solution out there, but the best and simplest is this:

echo ‘LC_CTYPE=”en_US.UTF-8″‘ >> /etc/sysconfig/i18n



iptables mangle and NAT notes, etc.


PREROUTING in mangle table: alter routing (e.g. source-based routing)
FORWARD in mangle table: traffic shaping for tc (flowid)

In MikroTik RouterOS, /ip firewall nat:
srcnat chain = PREROUTING
dstnat chain = POSTROUTING

/ip firewall mangle:
prerouting chain = global-in
forward chain = global-out (support NAT traffic shaping, higher load on router)

Update: I used to think that global-in literally means global input, while global-out literally means global output.  Apparently I was mistaken.  The parent queues MUST be matched to the right parent, either global-in, global-out, global-total, or one of the network interfaces depending on the mangle rules.  Top-most queue parent (global-in, global-out, global-total) doesn’t decide whether it’s up/down.  The mangle rules do the direction decision trick, whether a packet is incoming/download or outgoing/upload.  This makes perfect sense now! (doh)  I kept wondering why half of my queues (download queues’ top-most parent was global-in) weren’t working, when all my mangle rules were in forward chain.  Obviously they wouldn’t, because global-in marks both direction in prerouting chain.

Update 2: mangle: download rules first then upload rules. global-in, connection mark: prerouting; packet mark: prerouting. global-out, connection mark: forward, packet mark: postrouting.


This didn’t make sense until I read the URL listed below under References.  Silly me!  Assumption is the root of most, if not all, problems indeed.


VPS (HyperVM) initial setup

I just purchased a VPS account from a local VPS provider (using HyperVM). The first thing I was to rebuild the VPS because it has LXAdmin installed by default. I want a lightweight VPS with everything minimal.

I was excited to see some minimal OS templates on the web Control Panel. I have a Debian 4.0 UK VPS. This time I choose Ubuntu 6.06, because there are lots of Ubuntu repository mirrors in Indonesia. 🙂

Once I rebuilt the VPS with Ubuntu template, the VPS cannot be instantly ssh’d into. sshd fails to start because  udev is installed and we need to remove that. Quickly remove udev by typing the following on Command Center: apt-get -y remove udev

Once it’s removed, reboot the VPS. Voila! You should be able to login using SSH now. 🙂

No need to do mknod /dev/random c 1 8, mknod /dev/urandom c 1 9, /sbin/MAKEDEV tty, /sbin/MAKEDEV ttyp, nor /sbin/MAKEDEV pts.


errno.h problem

If you see an error like the following when compiling, then most likely it’s the errno.h problem:

/usr/bin/ld: errno: TLS definition in /lib64/ section .tbss mismatches non-TLS reference in envdir.o
/lib64/ could not read symbols: Bad value
collect2: ld returned 1 exit status
make: *** [envdir] Error 1

If there is a conf-cc file, then add “–include /path/to/errno.h” to the gcc line. Normally the file is at /usr/include/errno.h.


ext3 write barriers and write caching

I was reading on ext4 articles just now and read its advantages compared to ext3, which is the most popular Linux file system today. I read that performance-wise, ext4 is faster at handling large files but does not provide significant improvements over real world tasks. ext4 is also backward-compatible to be mounted as ext3 as long as extents are not used in that particular file system.

Out of the few advantages ext4 has over ext3, there is one feature which is very useful to ensure integrity of the file system, journal checksumming.

Quoting Wikipedia’s article on ext3:

Ext3 does not do checksumming when writing to the journal. If barrier=1 is not enabled as a mount option (in /etc/fstab), and if the hardware is doing out-of-order write caching, one runs the risk of severe filesystem corruption during a crash.

Consider the following scenario: If hard disk writes are done out-of-order (due to modern hard disks caching writes in order to amortize write speeds), it is likely that one will write a commit block of a transaction before the other relevant blocks are written. If a power failure or unrecoverable crash should occur before the other blocks get written, the system will have to be rebooted. Upon reboot, the file system will replay the log as normal, and replay the “winners” (transactions with a commit block, including the invalid transaction above which happened to be tagged with a valid commit block). The unfinished disk write above will thus proceed, but using corrupt journal data. The file system will thus mistakenly overwrite normal data with corrupt data while replaying the journal. There is a test program available to trigger the problematic behavior. If checksums had been used, where the blocks of the “fake winner” transaction were tagged with a mutual checksum, the file system could have known better and not replayed the corrupt data onto the disk. Journal checksumming has been added to EXT4.

The ext3 barrier option is not enabled by default on almost all popular Linux distributions, and thus most distributions are at risk. In addition, filesystems going through the device mapper interface (including software RAID and LVM implementations) may not support barriers, and will issue a warning if that mount option is used. There are also some disks that do not properly implement the write cache flushing extension necessary for barriers to work, which causes a similar warning. In these situations, where barriers are not supported or practical, reliable write ordering is possible by turning off the disk’s write cache and using the data=journal mount option.

ext3 apparently has a feature called write barriers which can help maintain integrity without journal checksumming, but there is a conflicting report over its performance impact.

Quoted from Andreas Dilger in a mail to ext3-users mailing list in May 2007:

Ideally, the jbd layer could be notified when the transaction blocks are flushed from device cache before writing the commit block, but the current linux mechanism to do this (write barriers) sucks perforance-wise (it sent throughput from 180MB/s to 7MB/s when enabled in our test systems). It was better to just turn off write cache entirely than to use barriers.

contradicts this post made by

Well, I think I see where ext3 gets its reputation for slow deletes. With the write cache off the delete performance is terrible, nearly 70% lower. It’s clear that enabling write barriers does something as the numbers are lower on a number of items (though not all). However it’s clear that write barriers is minor loss of performance compared to turning the write cache off. I think this leads me to consider how many servers I can run with just ext3 and md raid1 so as to keep the write cache enabled and the filesystem safe. I’ll have to weigh the performance gains against the benefits of using LVM (especially snapshots) and dm-crypt (which might have limited benefits on a server anyway).

Maybe some improvements had been done at the write barriers feature of ext3 since Andreas Dilger’s post, but I couldn’t find anything on Google about it. I couldn’t even find a website which has proper documentation on ext3’s write barriers. At least now I have specialj’s Bonnie++ test results from 2 months ago and it’s good enough to convince me enabling write barriers on all of my servers which run md raid1.

Thanks, specialj! 🙂


Painfully slow CentOS system

Today I was installing CentOS 5.2 to a Tyan barebone server with Intel Xeon X3220 processor, 2GB of RAM, and 2 Western Digital SATA II 250GB HDD. I chose to use software RAID 1 for my /boot and /. I didn’t create RAID 1 for swap partition because it is useless and in fact it might slow down performance. It took more than 2 hours to complete the whole installation process even when I used bare minimum packages selection. The format process of / partition itself took quite a while to complete.

Once the installation was complete, it rebooted and I found a painfully slow newly-installed CentOS system. The keyboard response was slow as if I were connected to a remote server with 1 second latency. I wasn’t happy because this is a quad-core processor server with acceptable amount of RAM. When I looked at /proc/mdstat to see how the software RAID is doing, I noticed that the sync speed is only around 1000KB/s. I tried looking for solutions on Google and found this and this. The solution I got didn’t help increase the sync speed, nor the slow response I was getting typing on the console or remotely.

I then somehow managed to find another possible solution. I can’t remember how I finally got this solution (thanks to nixCraft/CyberCiti). I didn’t notice that the hard drives were actually detected as hdX (IDE/PATA) instead of sdX (SATA). Now this makes sense because PATA is much slower than SATA. It turned out to be an issue with the hardware detection process. Somehow CentOS detected the HDDs and used PATA driver instead of SATA driver, so the devices were named hdX and treated as PATA hence the slow sync speed. I made the BIOS changes as mentioned in the solution and attempted another install, it was REALLY fast this time and the software RAID 1 sync speed was over 70000KB/s.


Extending PPPoE access network with network bridge

In a situation, I had to extend my PPPoE network over a wireless bridge. I do not want to have nor maintain 2 PPPoE servers. I didn’t bother failed to understand how PPPoE works and made a wireless bridge without WDS. When one user log in from the bridged side of the network, it works flawlessly. When more users are trying to log in, the user which logged in earlier gets disconnected. Apparently this is caused by MAC address problem[1]. From the PPPoE server’s side, users logging in from the bridged side of the network have the same MAC address which is the bridge device’s MAC address. From the users’ side, they are able to see correct MAC address of every device on the other side of the network. The PPPoE server gets confused when it sends PPPoE packets because multiple users have the same MAC address and there is no way for the server to direct reply to individual user.

[1]Wireless AP is connected to the side of the network where the PPPoE server is connected to, and wireless client is connected to the other side of the network. Every device connected to the same side of the wireless AP gets the wireless client’s MAC address for every device connected over the wireless bridge, however every device over the bridge gets to see the real MAC address of every device connected to the same side of the wireless AP. I believe that if I interchange the wireless AP and client (so now PPPoE server is connected to the wireless client instead), it may work properly since the PPPoE server will get the correct MAC address of all devices over the bridge. Devices connected to the wireless AP obviously will get the same MAC address for the PPPoE server and PPPoE users connected to the same side of the wireless client, but the most important thing is that the PPPoE users over the bridge can communicate properly with the PPPoE server. Since this is a one-to-many and many-to-one situation, this should work, but not for many-to-many situation. This explanation sounds quite confusing due to my limited English. If you could rewrite this part, please let me know.

Historical CPU statistics utility

Previously I wrote a post which discusses real-time CPU statistics utility. Now if you need historical data of CPU utilization statistics, use sysstat’s sar. For Fedora and CentOS, there is an official RPM package for sysstat.

Once you have installed sysstat, leave it for a few minutes until you get some records. The data collector runs from /etc/cron.d every 10 minutes (by default) and records all CPU usage. If you have SMP systems, then it will record each CPU’s utilization statistics. The default behaviour of sar is to print ALL CPU statistics combined, use -p to select individual CPU. Have fun watching your CPU utilization stats! 🙂