The Ghost in the Shell series were about efficient working in the shell environment but one of the feats of any sysadmin profession is the shell scripting. It is often needed to ‘glue’ various solutions and technologies to work as ‘Business’ requires or to fill the gap where any solution is not available – or at least not for free. It also serves a growing role in the automation of various tasks. Today I will try to show you the basics of writing POSIX /bin/sh compatible shell scripts.
You may want to check other articles in the Ghost in the Shell series on the Ghost in the Shell β Global Page where you will find links to all episodes of the series along with table of contents for each episodeβs contents.
Basics
In your own ‘yard’ you can use any shell language you want – there are many good interactive shells like zsh(1)/bash(1)/fish(1)/ksh(1) to name a few. Just keep in mind to stay away from csh(1)/tcsh(1) shells as they are mediocre at most in interactive mode and terrible for scripting. Its really pity that csh(1)/tcsh(1) shells are still used as the default FreeBSD shells today knowing that zsh(1) is available under MIT license and could be painlessly integrated into the FreeBSD Base System – but who I am to fix all the world’s problems … I just install zsh(1) from packages and live on.
By writing POSIX /bin/sh scripts you are making sure that they will run not only on bash(1) in Linux but also on all BSD systems and all other UNIX systems out there. Even the really old dinosaurs like HP-UX or AIX.
I always struggled to find good example for learning the shell scripting but recently I got one idea and we will follow it today.
For our ‘target’ I have chosen the kldstat(8) command from FreeBSD. Its output is far from perfect (from my perspective) with showing the Size column in hexadecimal values – while sysadmin expects values in (mega/giga/tera)bytes. Our task will be to parse that kldstat(8) output into something more human readable.
Lets check that kldstat(8) output then.
% kldstat | head
Id Refs Address Size Name
1 133 0xffffffff80200000 1f11f28 kernel
2 1 0xffffffff82112000 67feb0 zfs.ko
3 1 0xffffffff82792000 1abe8 geom_eli.ko
4 3 0xffffffff82a3c000 56ec0 vboxdrv.ko
5 2 0xffffffff82a93000 4240 vboxnetflt.ko
6 3 0xffffffff82a98000 aac8 netgraph.ko
7 1 0xffffffff82aa3000 31c8 ng_ether.ko
8 1 0xffffffff82aa7000 55e0 vboxnetadp.ko
9 1 0xffffffff82aad000 158458 i915kms.ko
Now what does 1f11f28 tell me about kernel for the Size column. Not much.
For a start I would like to print just the Size and Name columns in our script – we will call it kld.sh for the lack of better name and I will add version ‘tag’ to its name for each of our steps like kld.0.1.sh for first and ./kld.0.2.sh for the second one and so on.
There are many ways to parse that kldstat(8) output in our script but I will discuss two approaches here.
First is to get the /bin/sh output into variable and then parse it in a loop.
Second one to parse it in a loop in pipe after the command directly. I will use the second one here because the first one – with keeping then /bin/sh output in a variable – my be useful if we want to parse it more then once and as we will parse it only once then its pointless to ‘waste’ memory for that variable. Below you will find the first draft or kld.sh.
0.1
Our first 0.1 version has only the interpreter set at the beginning (the #!/bin/sh shebang) and the simple while read loop to get output of the kldstat(8) command and print it on the screen with shell builtin echo(1) command.
% cat ./kld.0.1.sh
#!/bin/sh
kldstat \
| while read LINE
do
echo "${LINE}"
done
Here is our script output – its generally identical as the kldstat(8) command.
% ./kld.0.1.sh | head
Id Refs Address Size Name
1 133 0xffffffff80200000 1f11f28 kernel
2 1 0xffffffff82112000 67feb0 zfs.ko
3 1 0xffffffff82792000 1abe8 geom_eli.ko
4 3 0xffffffff82a3c000 56ec0 vboxdrv.ko
5 2 0xffffffff82a93000 4240 vboxnetflt.ko
6 3 0xffffffff82a98000 aac8 netgraph.ko
7 1 0xffffffff82aa3000 31c8 ng_ether.ko
8 1 0xffffffff82aa7000 55e0 vboxnetadp.ko
9 1 0xffffffff82aad000 158458 i915kms.ko
0.2
As we know that kldstat(8) has fixed number of columns we can read its more intelligently with variables names as its columns and print only Size and Name columns as we wanted it in the first place. We should also omit the first line of kldstat(8) output as we will be printing our own header for just Size and Name columns. We will achieve that with sed(1) command.
Here is out script after our improvements.
% cat kld.0.2.sh
#!/bin/sh
echo "SIZE NAME"
kldstat \
| sed 1d \
| while read ID REFS ADDRESS SIZE NAME
do
echo "${SIZE} ${NAME}"
done
Here is its output at current early stage.
% ./kld.0.2.sh | head
SIZE NAME
1f11f28 kernel
67feb0 zfs.ko
1abe8 geom_eli.ko
56ec0 vboxdrv.ko
4240 vboxnetflt.ko
aac8 netgraph.ko
31c8 ng_ether.ko
55e0 vboxnetadp.ko
158458 i915kms.ko
As you can see the columns are not aligned so we can use column(1) command to make it look more like original command.
% ./kld.0.2.sh | column -t | head
SIZE NAME
1f11f28 kernel
67feb0 zfs.ko
1abe8 geom_eli.ko
56ec0 vboxdrv.ko
4240 vboxnetflt.ko
aac8 netgraph.ko
31c8 ng_ether.ko
55e0 vboxnetadp.ko
158458 i915kms.ko
But typing that each time we execute our script can be PITA so we will now use printf(1) instead of echo(1) to print our output. We will also alight the first Size column to the right to make the command output even more human readable. We will sacrifice 8 places of width for the Size column (%8s) and the rest with aligned to left (%-s) for Name column.
0.3
Here is our improved script.
% cat kld.0.3.sh
#!/bin/sh
printf "%8s %-s\n" SIZE NAME
kldstat \
| sed 1d \
| while read ID REFS ADDRESS SIZE NAME
do
printf "%8s %-s\n" ${SIZE} ${NAME}
done
Our output now looks like that one below.
% ./kld.0.3.sh | head
SIZE NAME
1f11f28 kernel
67feb0 zfs.ko
1abe8 geom_eli.ko
56ec0 vboxdrv.ko
4240 vboxnetflt.ko
aac8 netgraph.ko
31c8 ng_ether.ko
55e0 vboxnetadp.ko
158458 i915kms.ko
Better. Now we will improve two things. First we will start keeping our output format ("%8s %-s\n") in a separate variable and we will finally convert that hexadecimal values into decimal ones – to bytes – there are many ways to do that but I am leaning to use the printf(1) builtin because both of speed and it being available in the shell (builtin).
0.4
Here is the script.
% cat kld.0.4.sh
#!/bin/sh
FORMAT="%8s %-s\n"
printf "${FORMAT}" SIZE NAME
kldstat \
| sed 1d \
| while read ID REFS ADDRESS SIZE NAME
do
SIZE=$( printf "%d" 0x${SIZE} )
printf "${FORMAT}" ${SIZE} ${NAME}
done
And its output with bytes instead of hexadecimal values.
% ./kld.0.4.sh | head
SIZE NAME
32579368 kernel
6815408 zfs.ko
109544 geom_eli.ko
356032 vboxdrv.ko
16960 vboxnetflt.ko
43720 netgraph.ko
12744 ng_ether.ko
21984 vboxnetadp.ko
1410136 i915kms.ko
Now we have output in bytes and its nicely formatted. Its even easily sortable by the sort(1) command so its leaning nicely with UNIX principles.
% ./kld.0.4.sh | sort -n | head
SIZE NAME
8432 coretemp.ko
8504 cd9660_iconv.ko
8504 msdosfs_iconv.ko
8504 udf_iconv.ko
8576 smbus.ko
8736 cpuctl.ko
8800 pty.ko
9000 lindebugfs.ko
9024 uhid.ko
The next step would be to print that information in megabytes instead of just plain bytes. To convert bytes into kilobytes we need to divide our bytes value by 1024. To get the megabytes we need to do it twice. We will use the $(( ... )) syntax to use the shell builtin for simple math calculations instead of dropping that task to a subshell with $( ... ) syntax and external commands.
0.5
This is our ‘show in megabytes only’ script looks like.
% cat kld.0.5.sh
#!/bin/sh
FORMAT="%8s %-s\n"
printf "${FORMAT}" SIZE NAME
kldstat \
| sed 1d \
| while read ID REFS ADDRESS SIZE NAME
do
SIZE=$( printf "%d" 0x${SIZE} )
SIZE=$(( ${SIZE} / 1024 / 1024 ))
printf "${FORMAT}" ${SIZE} ${NAME}
done
And here is its output.
% ./kld.0.5.sh | head
SIZE NAME
31 kernel
6 zfs.ko
0 geom_eli.ko
0 vboxdrv.ko
0 vboxnetflt.ko
0 netgraph.ko
0 ng_ether.ko
0 vboxnetadp.ko
1 i915kms.ko
That did not wend too well, didn’t it? Because many module use less then 1 megabytes of memory after being rounded to natural numbers its 0 megabytes value for many modules. We will try to use bc(1) calculator instead with up to tenths precision.
0.6
Here is out script after using bc(1) instead of using the $(( ... )) syntax with dividing.
% cat kld.0.6.sh
#!/bin/sh
FORMAT="%8s %-s\n"
printf "${FORMAT}" SIZE NAME
kldstat \
| sed 1d \
| while read ID REFS ADDRESS SIZE NAME
do
SIZE=$( printf "%d" 0x${SIZE} )
SIZE=$( echo "scale=1; ${SIZE} / 1024 / 1024" | bc -l )
printf "${FORMAT}" ${SIZE} ${NAME}
done
And here is its output.
% ./kld.0.6.sh | head
SIZE NAME
31.0 kernel
6.4 zfs.ko
.1 geom_eli.ko
.3 vboxdrv.ko
0 vboxnetflt.ko
0 netgraph.ko
0 ng_ether.ko
0 vboxnetadp.ko
1.3 i915kms.ko
Far from ideal. The bc(1) output omits the leading zero if value is less then one. Seems that we can fix that with different printf(1) formatting. Lets try that. We will change from %8s (string) into %8.1f (float). That will also force us to use different formats for header and values so will stop using single FORMAT variable and we will use separate ones.
0.7
This is our current script state.
% cat kld.0.7.sh
#!/bin/sh
HEAD_FORMAT="%8s %-s\n"
LOOP_FORMAT="%8.1f %-s\n"
printf "${HEAD_FORMAT}" SIZE NAME
kldstat \
| sed 1d \
| while read ID REFS ADDRESS SIZE NAME
do
SIZE=$( printf "%d" 0x${SIZE} )
SIZE=$( echo "scale=1; ${SIZE} / 1024 / 1024" | bc -l )
printf "${LOOP_FORMAT}" ${SIZE} ${NAME}
done
And its output.
% ./kld.0.7.sh | head
SIZE NAME
31.0 kernel
6.4 zfs.ko
0.1 geom_eli.ko
0.3 vboxdrv.ko
0.0 vboxnetflt.ko
0.0 netgraph.ko
0.0 ng_ether.ko
0.0 vboxnetadp.ko
1.3 i915kms.ko
Works as advertised. We can now think of something different. How about we will also add an argument to include the kernel and modules file sizes as well? Not very useful I think but for the the purpose of shell scripting learning process we will do it anyway. The first caveat here is that kernel modules are on two locations on FreeBSD. The Base System modules are kept at /boot/kernel location and the modules that were installed by pkg(8) packages (or from FreeBSD Ports) are located at /boot/modules place. To get their size we will use the stat(1) command. Similarly like with memory usage – we would like to have the output of kernel and its modules size in megabytes.
There are of course several ways to achieve that. Lets start with the longest most educational example below. I will just paste the fragment that gets that kernel or module size for the FILE column.
if [ -f /boot/modules/${NAME} ]
then
FILE=$( stat -f %z /boot/modules/${NAME} )
fi
if [ -f /boot/kernel/${NAME} -a -z ${NAME} ]
then
FILE=$( stat -f %z /boot/kernel/${NAME} )
fi
if [ "${FILE}" = "" ]
then
FILE=-
fi
FILE=$( echo "scale=1; ${FILE} / 1024 / 1024" | bc -l )
One note about the [ "${FILE}" = "" ] syntax – in all old POSIX shells out there that I used /bin/sh always worked well with that syntax when FILE variable was empty or non existing. In a extreme example this one – [ "" = "" ] – works as desired. In case if you find yourself in a situation when this does not work in some POSIX /bin/sh implementation then use the most secure variant with additional same word added to both sides like that – [ "${FILE}test" = "test" ] – this way even the most badly written POSIX /bin/sh implementation will work π
It first checks the /boot/modules location for the module because I know a period of FreeBSD history in which the i915kms.ko module existed in both of these places and if you had them both then there is 99% percent chance that you are using the one installed by packages – that is why we try the third party modules first – then the ones from the Base System place. We also make sure that if for some reason the file will not be found the stat(1) command would not yield about its missing with 2> /dev/null at the end of command.
If we fail to find it under the third party modules then we will try the Base System location – but only when we did not find anything in the third party place – hence the additional test with -z ${NAME}.
For the record the syntax for these tests is:
- for single test its like that: [ TEST ]
- to test for both parameters (AND operator) its like that: [ TEST1 -a TEST ]
- for only one of tests to pass (OR operator) its like that: [ TEST1 -o TEST ]
If we fail to find the file size then we set that to ‘–‘ value.
At the end we divide by 1024 two times so we get megabytes from bytes.
This can be shortened to to take less place (and writing) into something like that.
[ -f /boot/modules/${NAME} ] && FILE=$( stat -f %z /boot/modules/${NAME} 2> /dev/null )
[ -f /boot/kernel/${NAME} -a -z ${NAME} ] && FILE=$( stat -f %z /boot/kernel/${NAME} 2> /dev/null )
[ ${FILE} = "" ] && FILE=-
FILE=$( echo "scale=1; ${FILE} / 1024 / 1024" | bc -l )
The end result is the same but it requires less space and writing. I also added some spaces for ‘logical formatting’ to make it more readable.
There is also more extreme way to shorten this up while keeping the same logic – here it is.
FILE=$( stat -f %z /boot/kernel/${NAME} 2> /dev/null \
|| stat -f %z /boot/modules/${NAME} 2> /dev/null \
|| FILE=- )
FILE=$( echo "scale=1; ${FILE} / 1024 / 1024" | bc -l )
We use then || OR operator in the subshell to make that shorter and still keep it readable. This is the version that we will use in our script.
0.8
Lets see now how it looks after modifications.
% cat kld.0.8.sh
#!/bin/sh
HEAD_FORMAT="%8s %8s %-s\n"
LOOP_FORMAT="%8.1f %8.1f %-s\n"
printf "${HEAD_FORMAT}" SIZE FILE NAME
kldstat \
| sed 1d \
| while read ID REFS ADDRESS SIZE NAME
do
FILE=$( stat -f %z /boot/kernel/${NAME} 2> /dev/null \
|| stat -f %z /boot/modules/${NAME} 2> /dev/null \
|| FILE=- )
FILE=$( echo "scale=1; ${FILE} / 1024 / 1024" | bc -l )
SIZE=$( printf "%d" 0x${SIZE} )
SIZE=$( echo "scale=1; ${SIZE} / 1024 / 1024" | bc -l )
printf "${LOOP_FORMAT}" ${SIZE} ${FILE} ${NAME}
done
And here is its output.
% ./kld.0.8.sh | head
SIZE FILE NAME
31.0 27.7 kernel
6.4 5.0 zfs.ko
0.1 0.1 geom_eli.ko
0.3 0.4 vboxdrv.ko
0.0 0.0 vboxnetflt.ko
0.0 0.1 netgraph.ko
0.0 0.0 ng_ether.ko
0.0 0.0 vboxnetadp.ko
1.3 2.2 i915kms.ko
Its interesting to see that used memory and file size are different.
Another step would be printing also the summary of the used RAM for each column. This is where things get more interesting. The while loop is created in a pipe which means its in a subshell. This has some serious implications. Normally we would add two variables like SIZE_TOTAL and FILE_TOTAL to add each module size there and then after the loop ends just print the summary. Because the while loop is spawned as subshell these variables will vanish as soon as the loop will end its life and these variables would not exist (they existed only in that while subshell).
But fear not – there is very clever way with file descriptor to have these variables exist with their values after the while loop ends. Below you will find the shortened prototypes of our currently used ‘pipe’ way and the ‘descriptor’ way.
This is the way you already know.
kldstat \
| sed 1d \
| while read LINE
do
echo "${LINE}"
TOTAL="Now You Don't."
done
echo ${TOTAL}
When you will execute that you will NOT see the "Now You Don't." string.
Now this is the way to overcome that subshell limitation.
while read LINE
do
echo "${LINE}"
TOTAL="Now You See Me."
done << BSD
$( kldstat | sed 1d )
BSD
echo ${TOTAL}
As you try it you will see the "Now You See Me." sign at the end.
This way we will provide summary for each column.
0.9
This is our code after our effort to add summary for the columns. You may noticed that we added the FILE_TOTAL and SIZE_TOTAL before the FILE and SIZE values are converted to megabytes. That ensures we are as accurate as possible. If we would just sum up the SIZE and FILE after they were converted to megabytes we would lost several bytes in the process.
% cat kld.0.9.sh
#!/bin/sh
HEAD_FORMAT="%8s %8s %-s\n"
LOOP_FORMAT="%8.1f %8.1f %-s\n"
printf "${HEAD_FORMAT}" SIZE FILE NAME
while read ID REFS ADDRESS SIZE NAME
do
FILE=$( stat -f %z /boot/kernel/${NAME} 2> /dev/null \
|| stat -f %z /boot/modules/${NAME} 2> /dev/null \
|| FILE=- )
FILE_TOTAL=$(( ${FILE_TOTAL} + ${FILE} ))
FILE=$( echo "scale=1; ${FILE} / 1024 / 1024" | bc -l )
SIZE=$( printf "%d" 0x${SIZE} )
SIZE_TOTAL=$(( ${SIZE_TOTAL} + ${SIZE} ))
SIZE=$( echo "scale=1; ${SIZE} / 1024 / 1024" | bc -l )
printf "${LOOP_FORMAT}" ${SIZE} ${FILE} ${NAME}
done << BSD
$( kldstat | sed 1d )
BSD
FILE_TOTAL=$( echo "scale=1; ${FILE_TOTAL} / 1024 / 1024" | bc -l )
SIZE_TOTAL=$( echo "scale=1; ${SIZE_TOTAL} / 1024 / 1024" | bc -l )
printf "${LOOP_FORMAT}" ${SIZE_TOTAL} ${FILE_TOTAL} TOTAL
This is how its execution looks like.
% ./kld.0.9.sh | (head -5; echo '(...)'; tail -5)
SIZE FILE NAME
31.0 27.7 kernel
6.4 5.0 zfs.ko
0.1 0.1 geom_eli.ko
0.3 0.4 vboxdrv.ko
(...)
0.0 0.0 linsysfs.ko
0.0 0.0 fdescfs.ko
0.0 0.0 nullfs.ko
0.0 0.0 acpi_ibm.ko
40.9 39.5 TOTAL
As you can see I also used shell feature to pipe output into many commands at once – this allows us to show information that is most important to use – beginning and ending – for the summary.
We even can do nested piping as shown on the screenshot below.
I deliberately used head(1) for entire guide because I have total of 42 kernel modules loaded. I did not wanted these outputs to overshadow our objective here. Here at the end I will show you complete output for the sake of it.
% kldstat | wc -l
42
% ./kld.0.9.sh
SIZE FILE NAME
31.0 27.7 kernel
6.4 5.0 zfs.ko
0.1 0.1 geom_eli.ko
0.3 0.4 vboxdrv.ko
0.0 0.0 vboxnetflt.ko
0.0 0.1 netgraph.ko
0.0 0.0 ng_ether.ko
0.0 0.0 vboxnetadp.ko
1.3 2.2 i915kms.ko
0.4 0.8 drm.ko
0.0 0.0 linuxkpi_gplv2.ko
0.0 0.0 lindebugfs.ko
0.0 0.1 fusefs.ko
0.0 0.0 coretemp.ko
0.0 0.0 sem.ko
0.0 0.0 cpuctl.ko
0.0 0.0 ichsmb.ko
0.0 0.0 smbus.ko
0.0 0.0 cuse.ko
0.0 0.0 libiconv.ko
0.0 0.0 cd9660_iconv.ko
0.0 0.0 msdosfs_iconv.ko
0.0 0.0 udf_iconv.ko
0.0 0.0 udf.ko
0.0 0.0 acpi_wmi.ko
0.0 0.0 uhid.ko
0.0 0.0 usbhid.ko
0.0 0.0 hidbus.ko
0.0 0.0 wmt.ko
0.0 0.0 ums.ko
0.1 0.2 ng_btsocket.ko
0.0 0.0 ng_bluetooth.ko
0.2 0.6 linux.ko
0.0 0.1 linux_common.ko
0.1 0.5 linux64.ko
0.0 0.0 pty.ko
0.0 0.0 linprocfs.ko
0.0 0.0 linsysfs.ko
0.0 0.0 fdescfs.ko
0.0 0.0 nullfs.ko
0.0 0.0 acpi_ibm.ko
40.9 39.5 TOTAL
% kldstat
Id Refs Address Size Name
1 133 0xffffffff80200000 1f11f28 kernel
2 1 0xffffffff82112000 67feb0 zfs.ko
3 1 0xffffffff82792000 1abe8 geom_eli.ko
4 3 0xffffffff82a3c000 56ec0 vboxdrv.ko
5 2 0xffffffff82a93000 4240 vboxnetflt.ko
6 3 0xffffffff82a98000 aac8 netgraph.ko
7 1 0xffffffff82aa3000 31c8 ng_ether.ko
8 1 0xffffffff82aa7000 55e0 vboxnetadp.ko
9 1 0xffffffff82aad000 158458 i915kms.ko
10 1 0xffffffff82c06000 7f548 drm.ko
11 2 0xffffffff82c86000 cbc8 linuxkpi_gplv2.ko
12 2 0xffffffff82c93000 2328 lindebugfs.ko
13 1 0xffffffff82c96000 11f10 fusefs.ko
14 1 0xffffffff82ca8000 20f0 coretemp.ko
15 1 0xffffffff82cab000 39e8 sem.ko
16 1 0xffffffff82caf000 2220 cpuctl.ko
17 1 0xffffffff82cb2000 3250 ichsmb.ko
18 1 0xffffffff82cb6000 2180 smbus.ko
19 1 0xffffffff82cb9000 6730 cuse.ko
20 4 0xffffffff82cc0000 4798 libiconv.ko
21 1 0xffffffff82cc5000 2138 cd9660_iconv.ko
22 1 0xffffffff82cc8000 2138 msdosfs_iconv.ko
23 1 0xffffffff82ccb000 2138 udf_iconv.ko
24 1 0xffffffff82cce000 5a00 udf.ko
25 1 0xffffffff82cd4000 3378 acpi_wmi.ko
26 1 0xffffffff82cd8000 2340 uhid.ko
27 1 0xffffffff82cdb000 3380 usbhid.ko
28 1 0xffffffff82cdf000 31f8 hidbus.ko
29 1 0xffffffff82ce3000 3320 wmt.ko
30 1 0xffffffff82ce7000 4350 ums.ko
31 1 0xffffffff82cec000 1ce48 ng_btsocket.ko
32 1 0xffffffff82d09000 25a8 ng_bluetooth.ko
33 1 0xffffffff82d0c000 388f8 linux.ko
34 4 0xffffffff82d45000 db70 linux_common.ko
35 1 0xffffffff82d53000 30ac8 linux64.ko
36 1 0xffffffff82d84000 2260 pty.ko
37 1 0xffffffff82d87000 639c linprocfs.ko
38 1 0xffffffff82d8e000 3284 linsysfs.ko
39 1 0xffffffff82d92000 3530 fdescfs.ko
40 1 0xffffffff82d96000 4700 nullfs.ko
41 1 0xffffffff82d9b000 41d8 acpi_ibm.ko
Summary
This concludes this Ghost in the Shell episode.
Feel free to share your scripting habits and spells π
EOF
