Linux Real Time Patch Review - Vanilla vs. RT patch comparison
Hardly any standard operating system kernel is hard real time capable. At best,
they're soft real time capable, with latencies up to milliseconds.
RTAI and Xenomai provide hard real time, which means that those operating systems
(inserted between the hardware and Linux) can assure interrupt response times
in the microsecond range. Not so with that standard Linux kernel. The 2.6 kernel
- in comparison to the 2.4 kernel -
has been much improved in regard to latencies, but still has some bottlenecks.
Ingo Molnar provides a patch to compensate for those bottlenecks.
Ingo Molnar RealTime Preempt Patch
This patch improves the responsiveness of the kernel dramatically.
The tests below have been conducted with a tool provided by Andrew Morton. Within the
amlat package, there is a tool called "realfeel" which also works on 2.6 kernels, despite
the "amlat" tool which, well, doesn't even work on 2.4 kernels on recent systems.
Linux scheduling latency (Andrew Morton)
Here is a small tar.gz which provides just the "realfeel" tool from the AKPM package
and the famous "hackbench" (tweaked to acually compile):
captest.tar.gz (2k)
Compile with "make".
# ./realfeel histogramfilename.hist
1400.038 MHz
secondsPerTick=0.000000
ticksPerSecond=1400037759.748198
2048 Hz
0.451 msec
0.361 msec
-0.004 msec
-0.029 msec
-0.146 msec
-0.205 msec
For the hackbench stress test call it like this:
# ./hackbench 100
Adjust the parameter - WARNING: if the parameter is too high, your system might freeze!
If you get something like this with "realfeel":
ioctl(RTC_IRQP_SET) failed: Invalid argument
do this:
# rmmod genrtc
# modprobe rtc
This removes the module "genrtc" and loads the "rtc" (real time clock driver).
In any case, you should unload the module "genrtc" and load the module "rtc".
THE RESULTS
The result with a vanilla 2.6.15 kernel:
(first column is in milliseconds; second column is the number of RTC (real time clock) interrupts received)
0.0 54410
0.1 32
0.2 8
0.3 5
0.4 3
0.5 1
0.6 1
0.8 1
1.0 1
1.4 1
1.5 1
2.0 1
3.7 1
As you can see, the max. latency of the RTC interrupt was about 3.7 milliseconds.
(UPDATE: with "hackbench" I've even got more than ONE SECOND!!!)
Results with the rt16 RT patch:
0.0 308821
0.1 832
0.2 104
As you can see, the max. latency was just about 200 microseconds. (UPDATE: with "hackbench" I got a max.
of about 400us)
If there is no or almost no load on your system, take into account that
SMI (System Maintenance Interrupts on e.g. Intel 82845 845 (Brookdale) motherboards)
can and will produce latencies of > 200us. Also see: Xenomai: High latencies with SMI not disabled
CONCLUSIONS:
If you need to have a low latency system for audio, music or video processing, the Molnar
low latency patch might be the right thing for you.
But still, if you need hard real time performance, where latencies must not be higher than
a few tens of microseconds, you'll better off with RTAI or XENOMAI.
Appendix:
Here are a few stress testing routines proven to be good for determining the worst
case latencies.
# dd if=/dev/zero of=bigfile bs=1024000 count=1024
# ping -l 100000 -q -s 10 -f localhost
# ping -f -s 1400 & dd if=/dev/urandom of=/dev/null & ls -lR /
# - a=0; while "true"; do cat /proc/interrupts; a=`expr $a + 1`; echo $a; done
# while :; do dd if=/dev/zero of=/tmp/foo bs=1M count=500; sync; rm /tmp/foo; done
If you want to combine some of those:
#!/bin/sh
dd if=/dev/zero of=bigfile1 bs=1024000 count=1000 &
ping -l 100000 -q -s 10 -f localhost &
du / &
-or one from Paolo Mantegazza (RTAI.org)
a=0; while "true"; do dd if=/dev/hda1 of=/dev/null bs=1M count=1000;
a=`expr $a + 1`; date; echo $a; done
(this is just one line!)
If you use "hackbench" you can try this:
#!/bin/sh
while true; do dd if=/dev/zero of=bigfile bs=1024000 count=1024; done &
while true; do killall hackbench; sleep 5; done &
while true; do ./hackbench 20; done &
ping -l 100000 -q -s 10 -f localhost &
du / &
Most hardcore test
Philippe Gerum (Ingo Molnar):
http://lkml.org/lkml/2005/6/22/347
while true; do dd if=/dev/zero of=bigfile bs=1024000 count=1024; done &
while true; do killall hackbench; sleep 5; done &
while true; do ./hackbench 20; done &
( cd ltp-full-20040707; while true; do ./runalltests.sh -x 40; done; ) &
ping -l 100000 -q -s 10 -f localhost &
du / &
And last but not least, here is a process listing of the processes on a RT preempt patch
patched kernel and a vanilla kernel just after booting on a debian sarge system ...
... to illustrate the differences between the rt-patch and vanilla kernel system:
2.6.15-rt16
# ps -A
PID TTY TIME CMD
1 ? 00:00:00 init
2 ? 00:00:00 softirq-high/0
3 ? 00:00:00 softirq-timer/0
4 ? 00:00:00 softirq-net-tx/
5 ? 00:00:00 softirq-net-rx/
6 ? 00:00:00 softirq-scsi/0
7 ? 00:00:00 softirq-tasklet
8 ? 00:00:00 desched/0
9 ? 00:00:03 events/0
10 ? 00:00:00 khelper
11 ? 00:00:00 kthread
12 ? 00:00:00 kblockd/0
13 ? 00:00:00 pdflush
14 ? 00:00:00 pdflush
16 ? 00:00:00 aio/0
15 ? 00:00:00 kswapd0
17 ? 00:00:00 kseriod
18 ? 00:00:00 IRQ 12
19 ? 00:00:00 IRQ 14
20 ? 00:00:00 kjournald
21 ? 00:00:00 IRQ 1
378 ? 00:00:00 IRQ 19
389 ? 00:00:00 IRQ 4
392 ? 00:00:00 IRQ 3
544 ? 00:00:00 syslogd
547 ? 00:00:00 klogd
579 ? 00:00:00 exim4
584 ? 00:00:00 inetd
592 ? 00:00:00 sshd
597 ? 00:00:00 atd
600 ? 00:00:00 cron
606 tty1 00:00:00 getty
607 tty2 00:00:00 getty
608 tty3 00:00:00 getty
609 tty4 00:00:00 getty
610 tty5 00:00:00 getty
611 tty6 00:00:00 getty
612 ? 00:00:00 sshd
614 pts/0 00:00:00 bash
618 pts/0 00:00:00 ps
2.6.15 vanilla
# ps -A
PID TTY TIME CMD
1 ? 00:00:00 init
2 ? 00:00:00 ksoftirqd/0
3 ? 00:00:04 events/0
4 ? 00:00:00 khelper
5 ? 00:00:00 kthread
6 ? 00:00:00 kblockd/0
7 ? 00:00:00 pdflush
8 ? 00:00:00 pdflush
10 ? 00:00:00 aio/0
9 ? 00:00:00 kswapd0
11 ? 00:00:00 kseriod
12 ? 00:00:00 kjournald
532 ? 00:00:00 syslogd
535 ? 00:00:00 klogd
567 ? 00:00:00 exim4
572 ? 00:00:00 inetd
580 ? 00:00:00 sshd
585 ? 00:00:00 atd
588 ? 00:00:00 cron
594 tty1 00:00:00 getty
595 tty2 00:00:00 getty
596 tty3 00:00:00 getty
597 tty4 00:00:00 getty
598 tty5 00:00:00 getty
599 tty6 00:00:00 getty
602 ? 00:00:00 sshd
604 pts/0 00:00:00 bash
609 pts/0 00:00:00 ps
Last-Modified: Thu, 09 Feb 2006 19:14:41 GMT
