You should never rely on the environment variables, unless you explicitly specify that in the documentation (which is still bad way in most cases). You script must work if there’s no PATH env. variable, or no EDITOR set, etc. Yes, it may say error to the end user in unresolvable conditions, as `svn ci` do if no EDITOR and no -m specified, but generally it shouldn’t fail.

Not `sh`, but `/bin/sh`, not `cat`, but `/bin/cat`! Remember it and you’ll save a lot of nerves for you and all those people that will use your tools.

Thank you.

No, seriously, I understand why there’s ‘autoconf’ and ‘suggest’ options – they’re really rarely called, maybe on node installation or when module first added to the configuration. But ‘config’ should be (and it is) called before each data fetch. Because graph legend or graph options may change between calls. So, if we always call ‘config’ and ‘fetch’ pair why do we have separate commands for them?

So, it is not an option and should be merged into the ‘fetch’ command.

Most of users will have gaps in df munin graphs as the result. Remote filesystem info may be not accessible for dozens of reasons and cause the whole df_ plugin to timeout, which will report “unknown” values to munin and you will get alert email into your inbox and gap in the graph.

To resolve the problem you will need to add smbfs and cifs into the ignore list in two lines of both /usr/local/share/munin/plugins/df_ and df_inode files:

/bin/df -P -t noprocfs,devfs,fdescfs,linprocfs,nfs,smbfs | tail +2 | grep -v “//” | while read i; do

This should be fixed in two lines in each file. You may remove `grep` if you want, I keep it just to be safe.

P.S.: As far as I know the problem exists also in Linux. In the RedHat’s and Ubuntu’s repos at least.

Let’s add /etc/periodic/hourly as the example:

#> mkdir /etc/periodic/hourly
#> # creating the crontab entry to run scripts at 25 minutes every hour
#> echo "25	*	*	*	*	root	periodic hourly" >> /etc/crontab

into /etc/periodic.conf:

hourly_output="root"				# user or /file
weekly_show_success="NO"			# scripts returning 0
weekly_show_info="YES"				# scripts returning 1
weekly_show_badconfig="NO"			# scripts returning 2

That’s all. We are done! Now you can place executable scripts into /etc/periodic/hourly and they will run every hour under root account.

I also use nginx as reverse-proxy and load balancer in my project.

mbuf clusters

FreeBSD stores the network data in the mbuf clusters 2Kb each, but only 1500B are used in each cluster (the size of the Ethernet packet)

mbufs

For each mbuf cluster there is “mbuf” structure needed, which have 256B in size and used to organize mbuf clusters in chains. There’s possibility to store some additional useful 100B data into the mbuf, but it is not always used.

If server have the RAM of 1Gb or more 25 thousands of mbuf clusters will be created by default but it is not enough in some cases.

When there’s no any free mbuf clusters available FreeBSD enters the zonelimit state and stops to answer to any network requests. You can see it as the `zoneli` state in the output of the `top` command.

To fix this problem the only solution is to log in through the local console and reboot the system. It is impossible to kill the process in `zoneli` state. This problem is also actual for Linux 2.6.x but even local console will not work in this state for Linux.

There is the patch that fixes the problem, it returns ENOBUFS error, which indicates entering the `zoneli` state and the program may close some connections when receives the error. Unfortunately this patch have not been merged into FreeBSD yet.

The state of used mbuf clusters can be checked by the following command:

> netstat -m
1/1421/1425 mbufs in use (current/cache/total)

0/614/614/25600 mbufs clusters in use (current/cache/total/max)

You can increase quantity of the mbufs clusters through the kern.ipc.nmbclusters parameter:

> sysctl kern.ipc.nmbclusters=65536

For earlier versions of FreeBSD mbuf clusters can be configured only in boot time:

/boot/loader.conf:

kern.ipc.mbclusters=65536

25000 mbuf clusters takes bout a 50Mb in the memory, 32000 – 74Mb, 65000 – 144 Mb (raises by the power of 2). 65000 is the boundary value and I can’t recommend to exceed it without increasing address space of the kernel first.

Increasing the amount of memory available for kernel

The default space for the kernel in memory is 1Gb for i386 architecture. To set it to 2Gb specify the following line in the kernel configuration file:

options KVA_PAGES=512

On the amd64 the the KVA is always 2Gb and there’s no possibility to increase it yet.

In addition to increasing the address space there’s the possibility to increase the limit of the physical memory available for kernel (320Mb by default). Let’s increase it to 1Gb:

/boot/loader.conf:

vm.kmem_size=1G

And reserve 275Mb for mbuf cluster from that space:

sysctl kern.ipc.nmbclusters=262144

Establishing the connection. syncache and syncookies

There’s approximately 100 bytes needed to serve one single connection.
Approximatelly 100 bytes space is used for single unfinished connection in syncache.
There’s possibility to store information about 15000 connections in memory. Approximately.

Snyncache parameters can bee seen by “sysctl net.inet.tcp.syncache” command (read-only).

Syncache parameters can be changed only during boot time:

/boot/loader.conf:
net.inet.tcp.syncache.hashsize=1024
net.inet.tcp.syncache.bucketlimit=100

when the new connection does not fit into overfull syncache FreeBSD enters the `syncookies` state (TCP SYN cookies). This possibility is enabled with:

sysctl net.inet.tcp.syncookies=1

The syncache population and the syncookies stats can be seen with `ntestat -s -p tcp` command.

When the connection is accepted it comes to the “listen socket queue”

Their’s stats can be seen with the `netstat -Lan` command.

Inreasing of the queue is possible with the `sysctl kern.ipc.somaxconn=4096` command

Whan the connection is accepted FreeBSD creates the sockets structures.

To increase the limit of the open sockets:

sysctl kern.ipc.maxsockets=204800

In earlier versions:

/boot/loader.conf:
kern.ipc.maxsockets=204800

The current state can be seen with the following command:

> vmstat -z

tcb hash

If the server processes several tens of thousands connections the tcb hash allows to detect the target connection for each incoming tcp packet quickly.

The tcb hash is 512 bytes by default.

The current size can be seen with:

sysctl net.inet.tcp.tcbhashsize

It is changeable in the boot time:

/boot/loader.conf:|
sysctl net.inet.tcp.tcbhashsize=4096

Files

Applicatios are working not with the sockets but with files. And there’s file descriptor needed for each socket because of that. To increase:

sysctl kern.maxfiles=204800
sysctl kern.maxfilesperproc=200000

These options can be changed on the live system but they will not affect already running processes. nginx have the ability to change the open files limit on the fly:

nginx.conf:
worker_limit_nofile 200000;
events {
worker_connections 200000;
}

receive buffers

Buffers for incoming data. 64Kb by default, if there’s no large uploads can be decreased to 8Kb (decreases the probability of overflow during a DDoS attack):

sysctl net.inet.tcp.recvspace=8192

For nginx:

nginx.conf:
listen 80 default rcvbuf=8k;

send buffers

Buffers for outgoing data. 32K by default. If data have a small size usually or there’s a lack of mbuf clusters it may be decreased:

sysctl net.inet.tcp.sendspace=16384

For nginx:

nginx.conf:
listen 80 default sendbuf=16k;

In the case when server has written some data to the socket but the client do not want to receive it the data will live in the kernel for several minutes even after the connection will be closed by timeout. Nginx have the option to erase all data after the timeout:

nginx.conf:
reset_timedout_connections on;

sendfile

Another way to save some mbuf clusters is the sendfile. It uses the kernel file buffers memory to send the data to the network interface without any intermediate buffers usage.

To enable in nginx:

nginx.conf:
sendfile on;

(you should explicitly switch it off if you’re sending files from the partition mounted via smbfs or cifs – ReRePi)

On the i386 platform with 1Gb and more memory 6656 sendfile buffers will be allocated which is usually enough. On the amd64 platform more optimal implementation is used and there’s no need in sendbufs at all.

On the sendbuf overflow the process stucks in the `sfbufa` state, but things turns ok after the buffer size is increased:

/boot/loader.conf:
kern.ipc.nsfbufs=10240

TIME_WAIT

After the connection was closed the socket enters the TIME_WAIT state. In this state it can live for 60 seconds by default. This time can be changed with sysctl (in milliseconds divided by 2. 2×30000 MSL = 60 seconds):

sysctl net.inet.tcp.msl=30000

TCP/IP ports

Outgoing connection are bind to the ports from the 49152 – 65535 range (16 thousands). It is better to be increased (1024 – 65535):

sysctl net.inet.ip.portrange.first=1024
sysctl net.inet.ip.portrange.last=65535

To use ports in natural order instead of random (to make the second connection for the same port impossible before TIME_WAIT):

sysctl net.inet.ip.portrange.randomized=0

In FreeBSD 6.2 the possibility to not create TIME_WAIT state for localhost connections was added:

sysctl net.inet.tcp.nolocaltimewait=1

Some time ago I bought Latitude D620 and was completelly happy about it – it was fast, relatively small and not too heavy. And everything was working. Until I’ve re-installed the operating system. Even with Dell drivers I don’t have bluetooth now, I had approximatelly a week of hardcode fucking with wi-fi, and it is overheating now. Even when idle. If I’ll install Linux will it explode?

I can’t believe things can be such bad in 21th century!

But for now only the whole system layout:

I have six servers at the moment. Two of them is isolated set-up and runs live installation of the application. They will run it until I’ll get the next release online on new servers set-up.

Remaining four servers:

Two absolutelly same relatively powerfull servers used for web front-end. Almost typical Apache+mod_php installation and nginx reverse-proxy for load-balancing.

One server if dedicated for the MySQL and some background tasks like images processing.

One server (unlike others it is not very powerfull) is now doing absolutely nothing. I’m planning to put system monitoring, logs, backups, staging installation, and maybe some other non-critical stuff there.

All four servers runs under FreeBSD 6.2, shared storage required by application is located on third-party windows server (client’s data-center) and mounted through smb protocol.

Most of developers also know what is database denormalization for – for improving performance in most cases.

Btw because of cheap hardware and relatively expensive development services for most applications maximum performance is not the main goal. So, seems like no reason for denormalizaition?

Nope. There’s no reason for denormalization only if the data never changes.

Let’s see the simple example. We have a shop and we’re selling blankets with custom art printed on them. There are several steps from order submit to shipping – the art should at least be rendered, approved by quality control team and printed. What will happen if we will change some properties of the product when order is already submitted but not yet rendered? The size of printable area for example (we’re going to buy some new printing equipment with smaller printable area and don’t want to allow users to use old large area anymore).

With perfectly normalized database structure we will take the size of the printable area from the product properties and we will finally render smaller image. Will it be cropped or resized – doesn’t matter because it will differ from what user expected to get and what he saw on the preview anyway.

There are several possible approaches to avoid this problem. First is to make a clone of the product record on every change, so all old orders will point to unchanged data. In some cases it is acceptable way – if product info changed rarely we will not have too much garbage in the database, but if it is changed relatively frequently, or we expect the system to keep data for several years, then the second way is better I think.

The second way is to keep all information necessary for printing within the order record even if it will duplicate some product’s properties. It is actually not the ‘denormalization’ if we’ll call that ‘extra’ fields in order record not the “product’s editable area size” but “image size this order was submitted with”. This simple trick allows us to consider the whole database still perfectly normalized.