Tag Archives: fish

Ghost in the Shell – Part 6 – Learn Shell Scripting

The Ghost in the Shell series were about efficient working in the shell environment but one of the feats of any sysadmin profession is the shell scripting. It is often needed to ‘glue’ various solutions and technologies to work as ‘Business’ requires or to fill the gap where any solution is not available – or at least not for free. It also serves a growing role in the automation of various tasks. Today I will try to show you the basics of writing POSIX /bin/sh compatible shell scripts.

You may want to check other articles in the Ghost in the Shell series on the Ghost in the Shell – Global Page where you will find links to all episodes of the series along with table of contents for each episode’s contents.

Basics

In your own ‘yard’ you can use any shell language you want – there are many good interactive shells like zsh(1)/bash(1)/fish(1)/ksh(1) to name a few. Just keep in mind to stay away from csh(1)/tcsh(1) shells as they are mediocre at most in interactive mode and terrible for scripting. Its really pity that csh(1)/tcsh(1) shells are still used as the default FreeBSD shells today knowing that zsh(1) is available under MIT license and could be painlessly integrated into the FreeBSD Base System – but who I am to fix all the world’s problems … I just install zsh(1) from packages and live on.

By writing POSIX /bin/sh scripts you are making sure that they will run not only on bash(1) in Linux but also on all BSD systems and all other UNIX systems out there. Even the really old dinosaurs like HP-UX or AIX.

I always struggled to find good example for learning the shell scripting but recently I got one idea and we will follow it today.

For our ‘target’ I have chosen the kldstat(8) command from FreeBSD. Its output is far from perfect (from my perspective) with showing the Size column in hexadecimal values – while sysadmin expects values in (mega/giga/tera)bytes. Our task will be to parse that kldstat(8) output into something more human readable.

Lets check that kldstat(8) output then.

% kldstat | head
Id Refs Address                Size Name
 1  133 0xffffffff80200000  1f11f28 kernel
 2    1 0xffffffff82112000   67feb0 zfs.ko
 3    1 0xffffffff82792000    1abe8 geom_eli.ko
 4    3 0xffffffff82a3c000    56ec0 vboxdrv.ko
 5    2 0xffffffff82a93000     4240 vboxnetflt.ko
 6    3 0xffffffff82a98000     aac8 netgraph.ko
 7    1 0xffffffff82aa3000     31c8 ng_ether.ko
 8    1 0xffffffff82aa7000     55e0 vboxnetadp.ko
 9    1 0xffffffff82aad000   158458 i915kms.ko

Now what does 1f11f28 tell me about kernel for the Size column. Not much.

For a start I would like to print just the Size and Name columns in our script – we will call it kld.sh for the lack of better name and I will add version ‘tag’ to its name for each of our steps like kld.0.1.sh for first and ./kld.0.2.sh for the second one and so on.

There are many ways to parse that kldstat(8) output in our script but I will discuss two approaches here.

First is to get the /bin/sh output into variable and then parse it in a loop.

Second one to parse it in a loop in pipe after the command directly. I will use the second one here because the first one – with keeping then /bin/sh output in a variable – my be useful if we want to parse it more then once and as we will parse it only once then its pointless to ‘waste’ memory for that variable. Below you will find the first draft or kld.sh.

0.1

Our first 0.1 version has only the interpreter set at the beginning (the #!/bin/sh shebang) and the simple while read loop to get output of the kldstat(8) command and print it on the screen with shell builtin echo(1) command.

% cat ./kld.0.1.sh
#!/bin/sh

kldstat \
  | while read LINE
    do
      echo "${LINE}"
    done

Here is our script output – its generally identical as the kldstat(8) command.

% ./kld.0.1.sh | head
Id Refs Address                Size Name
1  133 0xffffffff80200000  1f11f28 kernel
2    1 0xffffffff82112000   67feb0 zfs.ko
3    1 0xffffffff82792000    1abe8 geom_eli.ko
4    3 0xffffffff82a3c000    56ec0 vboxdrv.ko
5    2 0xffffffff82a93000     4240 vboxnetflt.ko
6    3 0xffffffff82a98000     aac8 netgraph.ko
7    1 0xffffffff82aa3000     31c8 ng_ether.ko
8    1 0xffffffff82aa7000     55e0 vboxnetadp.ko
9    1 0xffffffff82aad000   158458 i915kms.ko

0.2

As we know that kldstat(8) has fixed number of columns we can read its more intelligently with variables names as its columns and print only Size and Name columns as we wanted it in the first place. We should also omit the first line of kldstat(8) output as we will be printing our own header for just Size and Name columns. We will achieve that with sed(1) command.

Here is out script after our improvements.

% cat kld.0.2.sh
#!/bin/sh

echo "SIZE NAME"
kldstat \
  | sed 1d \
  | while read ID REFS ADDRESS SIZE NAME
    do
      echo "${SIZE} ${NAME}"
    done

Here is its output at current early stage.

% ./kld.0.2.sh | head
SIZE NAME
1f11f28 kernel
67feb0 zfs.ko
1abe8 geom_eli.ko
56ec0 vboxdrv.ko
4240 vboxnetflt.ko
aac8 netgraph.ko
31c8 ng_ether.ko
55e0 vboxnetadp.ko
158458 i915kms.ko

As you can see the columns are not aligned so we can use column(1) command to make it look more like original command.

% ./kld.0.2.sh | column -t | head
SIZE     NAME
1f11f28  kernel
67feb0   zfs.ko
1abe8    geom_eli.ko
56ec0    vboxdrv.ko
4240     vboxnetflt.ko
aac8     netgraph.ko
31c8     ng_ether.ko
55e0     vboxnetadp.ko
158458   i915kms.ko

But typing that each time we execute our script can be PITA so we will now use printf(1) instead of echo(1) to print our output. We will also alight the first Size column to the right to make the command output even more human readable. We will sacrifice 8 places of width for the Size column (%8s) and the rest with aligned to left (%-s) for Name column.

0.3

Here is our improved script.

% cat kld.0.3.sh
#!/bin/sh

printf "%8s %-s\n" SIZE NAME
kldstat \
  | sed 1d \
  | while read ID REFS ADDRESS SIZE NAME
    do
      printf "%8s %-s\n" ${SIZE} ${NAME}
    done

Our output now looks like that one below.

% ./kld.0.3.sh | head
      SIZE NAME
   1f11f28 kernel
    67feb0 zfs.ko
     1abe8 geom_eli.ko
     56ec0 vboxdrv.ko
      4240 vboxnetflt.ko
      aac8 netgraph.ko
      31c8 ng_ether.ko
      55e0 vboxnetadp.ko
    158458 i915kms.ko

Better. Now we will improve two things. First we will start keeping our output format ("%8s %-s\n") in a separate variable and we will finally convert that hexadecimal values into decimal ones – to bytes – there are many ways to do that but I am leaning to use the printf(1) builtin because both of speed and it being available in the shell (builtin).

0.4

Here is the script.

% cat kld.0.4.sh
#!/bin/sh

FORMAT="%8s %-s\n"
printf "${FORMAT}" SIZE NAME
kldstat \
  | sed 1d \
  | while read ID REFS ADDRESS SIZE NAME
    do
      SIZE=$( printf "%d" 0x${SIZE} )
      printf "${FORMAT}" ${SIZE} ${NAME}
    done

And its output with bytes instead of hexadecimal values.

% ./kld.0.4.sh | head
      SIZE NAME
  32579368 kernel
   6815408 zfs.ko
    109544 geom_eli.ko
    356032 vboxdrv.ko
     16960 vboxnetflt.ko
     43720 netgraph.ko
     12744 ng_ether.ko
     21984 vboxnetadp.ko
   1410136 i915kms.ko


Now we have output in bytes and its nicely formatted. Its even easily sortable by the sort(1) command so its leaning nicely with UNIX principles.

% ./kld.0.4.sh | sort -n | head
      SIZE NAME
      8432 coretemp.ko
      8504 cd9660_iconv.ko
      8504 msdosfs_iconv.ko
      8504 udf_iconv.ko
      8576 smbus.ko
      8736 cpuctl.ko
      8800 pty.ko
      9000 lindebugfs.ko
      9024 uhid.ko

The next step would be to print that information in megabytes instead of just plain bytes. To convert bytes into kilobytes we need to divide our bytes value by 1024. To get the megabytes we need to do it twice. We will use the $(( ... )) syntax to use the shell builtin for simple math calculations instead of dropping that task to a subshell with $( ... ) syntax and external commands.

0.5

This is our ‘show in megabytes only’ script looks like.

% cat kld.0.5.sh
#!/bin/sh

FORMAT="%8s %-s\n"
printf "${FORMAT}" SIZE NAME
kldstat \
  | sed 1d \
  | while read ID REFS ADDRESS SIZE NAME
    do
      SIZE=$( printf "%d" 0x${SIZE} )
      SIZE=$(( ${SIZE} / 1024 / 1024 ))
      printf "${FORMAT}" ${SIZE} ${NAME}
    done

And here is its output.

% ./kld.0.5.sh | head
      SIZE NAME
        31 kernel
         6 zfs.ko
         0 geom_eli.ko
         0 vboxdrv.ko
         0 vboxnetflt.ko
         0 netgraph.ko
         0 ng_ether.ko
         0 vboxnetadp.ko
         1 i915kms.ko

That did not wend too well, didn’t it? Because many module use less then 1 megabytes of memory after being rounded to natural numbers its 0 megabytes value for many modules. We will try to use bc(1) calculator instead with up to tenths precision.

0.6

Here is out script after using bc(1) instead of using the $(( ... )) syntax with dividing.

% cat kld.0.6.sh
#!/bin/sh

FORMAT="%8s %-s\n"
printf "${FORMAT}" SIZE NAME
kldstat \
  | sed 1d \
  | while read ID REFS ADDRESS SIZE NAME
    do
      SIZE=$( printf "%d" 0x${SIZE} )
      SIZE=$( echo "scale=1; ${SIZE} / 1024 / 1024" | bc -l )
      printf "${FORMAT}" ${SIZE} ${NAME}
    done

And here is its output.

% ./kld.0.6.sh | head
      SIZE NAME
      31.0 kernel
       6.4 zfs.ko
        .1 geom_eli.ko
        .3 vboxdrv.ko
         0 vboxnetflt.ko
         0 netgraph.ko
         0 ng_ether.ko
         0 vboxnetadp.ko
       1.3 i915kms.ko

Far from ideal. The bc(1) output omits the leading zero if value is less then one. Seems that we can fix that with different printf(1) formatting. Lets try that. We will change from %8s (string) into %8.1f (float). That will also force us to use different formats for header and values so will stop using single FORMAT variable and we will use separate ones.

0.7

This is our current script state.

% cat kld.0.7.sh
#!/bin/sh

HEAD_FORMAT="%8s %-s\n"
LOOP_FORMAT="%8.1f %-s\n"
printf "${HEAD_FORMAT}" SIZE NAME
kldstat \
  | sed 1d \
  | while read ID REFS ADDRESS SIZE NAME
    do
      SIZE=$( printf "%d" 0x${SIZE} )
      SIZE=$( echo "scale=1; ${SIZE} / 1024 / 1024" | bc -l )
      printf "${LOOP_FORMAT}" ${SIZE} ${NAME}
    done

And its output.

% ./kld.0.7.sh | head
      SIZE NAME
      31.0 kernel
       6.4 zfs.ko
       0.1 geom_eli.ko
       0.3 vboxdrv.ko
       0.0 vboxnetflt.ko
       0.0 netgraph.ko
       0.0 ng_ether.ko
       0.0 vboxnetadp.ko
       1.3 i915kms.ko

Works as advertised. We can now think of something different. How about we will also add an argument to include the kernel and modules file sizes as well? Not very useful I think but for the the purpose of shell scripting learning process we will do it anyway. The first caveat here is that kernel modules are on two locations on FreeBSD. The Base System modules are kept at /boot/kernel location and the modules that were installed by pkg(8) packages (or from FreeBSD Ports) are located at /boot/modules place. To get their size we will use the stat(1) command. Similarly like with memory usage – we would like to have the output of kernel and its modules size in megabytes.

There are of course several ways to achieve that. Lets start with the longest most educational example below. I will just paste the fragment that gets that kernel or module size for the FILE column.

if [ -f /boot/modules/${NAME} ]
then
  FILE=$( stat -f %z /boot/modules/${NAME} )
fi

if [ -f /boot/kernel/${NAME} -a -z ${NAME} ]
then
  FILE=$( stat -f %z /boot/kernel/${NAME} )
fi

if [ "${FILE}" = "" ]
then
  FILE=-
fi

FILE=$( echo "scale=1; ${FILE} / 1024 / 1024" | bc -l )

One note about the [ "${FILE}" = "" ] syntax – in all old POSIX shells out there that I used /bin/sh always worked well with that syntax when FILE variable was empty or non existing. In a extreme example this one – [ "" = "" ] – works as desired. In case if you find yourself in a situation when this does not work in some POSIX /bin/sh implementation then use the most secure variant with additional same word added to both sides like that – [ "${FILE}test" = "test" ] – this way even the most badly written POSIX /bin/sh implementation will work πŸ™‚

It first checks the /boot/modules location for the module because I know a period of FreeBSD history in which the i915kms.ko module existed in both of these places and if you had them both then there is 99% percent chance that you are using the one installed by packages – that is why we try the third party modules first – then the ones from the Base System place. We also make sure that if for some reason the file will not be found the stat(1) command would not yield about its missing with 2> /dev/null at the end of command.

If we fail to find it under the third party modules then we will try the Base System location – but only when we did not find anything in the third party place – hence the additional test with -z ${NAME}.

For the record the syntax for these tests is:

  • for single test its like that: [ TEST ]
  • to test for both parameters (AND operator) its like that: [ TEST1 -a TEST ]
  • for only one of tests to pass (OR operator) its like that: [ TEST1 -o TEST ]

If we fail to find the file size then we set that to ‘‘ value.

At the end we divide by 1024 two times so we get megabytes from bytes.

This can be shortened to to take less place (and writing) into something like that.

[ -f /boot/modules/${NAME} ]              && FILE=$( stat -f %z /boot/modules/${NAME} 2> /dev/null )
[ -f /boot/kernel/${NAME} -a -z ${NAME} ] && FILE=$( stat -f %z /boot/kernel/${NAME}  2> /dev/null )
[ ${FILE} = "" ]                          && FILE=-
FILE=$( echo "scale=1; ${FILE} / 1024 / 1024" | bc -l )

The end result is the same but it requires less space and writing. I also added some spaces for ‘logical formatting’ to make it more readable.

There is also more extreme way to shorten this up while keeping the same logic – here it is.

FILE=$( stat -f %z /boot/kernel/${NAME}  2> /dev/null \
     || stat -f %z /boot/modules/${NAME} 2> /dev/null \
     || FILE=- )
FILE=$( echo "scale=1; ${FILE} / 1024 / 1024" | bc -l )

We use then || OR operator in the subshell to make that shorter and still keep it readable. This is the version that we will use in our script.

0.8

Lets see now how it looks after modifications.

% cat kld.0.8.sh
#!/bin/sh

HEAD_FORMAT="%8s %8s %-s\n"
LOOP_FORMAT="%8.1f %8.1f %-s\n"
printf "${HEAD_FORMAT}" SIZE FILE NAME
kldstat \
  | sed 1d \
  | while read ID REFS ADDRESS SIZE NAME
    do
      FILE=$( stat -f %z /boot/kernel/${NAME}  2> /dev/null \
           || stat -f %z /boot/modules/${NAME} 2> /dev/null \
           || FILE=- )
      FILE=$( echo "scale=1; ${FILE} / 1024 / 1024" | bc -l )
      SIZE=$( printf "%d" 0x${SIZE} )
      SIZE=$( echo "scale=1; ${SIZE} / 1024 / 1024" | bc -l )
      printf "${LOOP_FORMAT}" ${SIZE} ${FILE} ${NAME}
    done

And here is its output.

% ./kld.0.8.sh | head
    SIZE     FILE NAME
    31.0     27.7 kernel
     6.4      5.0 zfs.ko
     0.1      0.1 geom_eli.ko
     0.3      0.4 vboxdrv.ko
     0.0      0.0 vboxnetflt.ko
     0.0      0.1 netgraph.ko
     0.0      0.0 ng_ether.ko
     0.0      0.0 vboxnetadp.ko
     1.3      2.2 i915kms.ko

Its interesting to see that used memory and file size are different.

Another step would be printing also the summary of the used RAM for each column. This is where things get more interesting. The while loop is created in a pipe which means its in a subshell. This has some serious implications. Normally we would add two variables like SIZE_TOTAL and FILE_TOTAL to add each module size there and then after the loop ends just print the summary. Because the while loop is spawned as subshell these variables will vanish as soon as the loop will end its life and these variables would not exist (they existed only in that while subshell).

But fear not – there is very clever way with file descriptor to have these variables exist with their values after the while loop ends. Below you will find the shortened prototypes of our currently used ‘pipe’ way and the ‘descriptor’ way.

This is the way you already know.

kldstat \
  | sed 1d \
  | while read LINE
    do
      echo "${LINE}"
      TOTAL="Now You Don't."
    done

echo ${TOTAL}

When you will execute that you will NOT see the "Now You Don't." string.

Now this is the way to overcome that subshell limitation.

while read LINE
do
  echo "${LINE}"
  TOTAL="Now You See Me."
done << BSD
  $( kldstat | sed 1d )
BSD

echo ${TOTAL}

As you try it you will see the "Now You See Me." sign at the end.

This way we will provide summary for each column.

0.9

This is our code after our effort to add summary for the columns. You may noticed that we added the FILE_TOTAL and SIZE_TOTAL before the FILE and SIZE values are converted to megabytes. That ensures we are as accurate as possible. If we would just sum up the SIZE and FILE after they were converted to megabytes we would lost several bytes in the process.

% cat kld.0.9.sh
#!/bin/sh

HEAD_FORMAT="%8s %8s %-s\n"
LOOP_FORMAT="%8.1f %8.1f %-s\n"
printf "${HEAD_FORMAT}" SIZE FILE NAME
while read ID REFS ADDRESS SIZE NAME
do
  FILE=$( stat -f %z /boot/kernel/${NAME}  2> /dev/null \
       || stat -f %z /boot/modules/${NAME} 2> /dev/null \
       || FILE=- )
  FILE_TOTAL=$(( ${FILE_TOTAL} + ${FILE} ))
  FILE=$( echo "scale=1; ${FILE} / 1024 / 1024" | bc -l )
  SIZE=$( printf "%d" 0x${SIZE} )
  SIZE_TOTAL=$(( ${SIZE_TOTAL} + ${SIZE} ))
  SIZE=$( echo "scale=1; ${SIZE} / 1024 / 1024" | bc -l )
  printf "${LOOP_FORMAT}" ${SIZE} ${FILE} ${NAME}
done << BSD
  $( kldstat | sed 1d )
BSD
FILE_TOTAL=$( echo "scale=1; ${FILE_TOTAL} / 1024 / 1024" | bc -l )
SIZE_TOTAL=$( echo "scale=1; ${SIZE_TOTAL} / 1024 / 1024" | bc -l )
printf "${LOOP_FORMAT}" ${SIZE_TOTAL} ${FILE_TOTAL} TOTAL

This is how its execution looks like.

% ./kld.0.9.sh | (head -5; echo '(...)'; tail -5)
    SIZE     FILE NAME
    31.0     27.7 kernel
     6.4      5.0 zfs.ko
     0.1      0.1 geom_eli.ko
     0.3      0.4 vboxdrv.ko
(...)
     0.0      0.0 linsysfs.ko
     0.0      0.0 fdescfs.ko
     0.0      0.0 nullfs.ko
     0.0      0.0 acpi_ibm.ko
    40.9     39.5 TOTAL

As you can see I also used shell feature to pipe output into many commands at once – this allows us to show information that is most important to use – beginning and ending – for the summary.

We even can do nested piping as shown on the screenshot below.

lolcat

I deliberately used head(1) for entire guide because I have total of 42 kernel modules loaded. I did not wanted these outputs to overshadow our objective here. Here at the end I will show you complete output for the sake of it.

% kldstat | wc -l
      42

% ./kld.0.9.sh
    SIZE     FILE NAME
    31.0     27.7 kernel
     6.4      5.0 zfs.ko
     0.1      0.1 geom_eli.ko
     0.3      0.4 vboxdrv.ko
     0.0      0.0 vboxnetflt.ko
     0.0      0.1 netgraph.ko
     0.0      0.0 ng_ether.ko
     0.0      0.0 vboxnetadp.ko
     1.3      2.2 i915kms.ko
     0.4      0.8 drm.ko
     0.0      0.0 linuxkpi_gplv2.ko
     0.0      0.0 lindebugfs.ko
     0.0      0.1 fusefs.ko
     0.0      0.0 coretemp.ko
     0.0      0.0 sem.ko
     0.0      0.0 cpuctl.ko
     0.0      0.0 ichsmb.ko
     0.0      0.0 smbus.ko
     0.0      0.0 cuse.ko
     0.0      0.0 libiconv.ko
     0.0      0.0 cd9660_iconv.ko
     0.0      0.0 msdosfs_iconv.ko
     0.0      0.0 udf_iconv.ko
     0.0      0.0 udf.ko
     0.0      0.0 acpi_wmi.ko
     0.0      0.0 uhid.ko
     0.0      0.0 usbhid.ko
     0.0      0.0 hidbus.ko
     0.0      0.0 wmt.ko
     0.0      0.0 ums.ko
     0.1      0.2 ng_btsocket.ko
     0.0      0.0 ng_bluetooth.ko
     0.2      0.6 linux.ko
     0.0      0.1 linux_common.ko
     0.1      0.5 linux64.ko
     0.0      0.0 pty.ko
     0.0      0.0 linprocfs.ko
     0.0      0.0 linsysfs.ko
     0.0      0.0 fdescfs.ko
     0.0      0.0 nullfs.ko
     0.0      0.0 acpi_ibm.ko
    40.9     39.5 TOTAL

% kldstat
Id Refs Address                Size Name
 1  133 0xffffffff80200000  1f11f28 kernel
 2    1 0xffffffff82112000   67feb0 zfs.ko
 3    1 0xffffffff82792000    1abe8 geom_eli.ko
 4    3 0xffffffff82a3c000    56ec0 vboxdrv.ko
 5    2 0xffffffff82a93000     4240 vboxnetflt.ko
 6    3 0xffffffff82a98000     aac8 netgraph.ko
 7    1 0xffffffff82aa3000     31c8 ng_ether.ko
 8    1 0xffffffff82aa7000     55e0 vboxnetadp.ko
 9    1 0xffffffff82aad000   158458 i915kms.ko
10    1 0xffffffff82c06000    7f548 drm.ko
11    2 0xffffffff82c86000     cbc8 linuxkpi_gplv2.ko
12    2 0xffffffff82c93000     2328 lindebugfs.ko
13    1 0xffffffff82c96000    11f10 fusefs.ko
14    1 0xffffffff82ca8000     20f0 coretemp.ko
15    1 0xffffffff82cab000     39e8 sem.ko
16    1 0xffffffff82caf000     2220 cpuctl.ko
17    1 0xffffffff82cb2000     3250 ichsmb.ko
18    1 0xffffffff82cb6000     2180 smbus.ko
19    1 0xffffffff82cb9000     6730 cuse.ko
20    4 0xffffffff82cc0000     4798 libiconv.ko
21    1 0xffffffff82cc5000     2138 cd9660_iconv.ko
22    1 0xffffffff82cc8000     2138 msdosfs_iconv.ko
23    1 0xffffffff82ccb000     2138 udf_iconv.ko
24    1 0xffffffff82cce000     5a00 udf.ko
25    1 0xffffffff82cd4000     3378 acpi_wmi.ko
26    1 0xffffffff82cd8000     2340 uhid.ko
27    1 0xffffffff82cdb000     3380 usbhid.ko
28    1 0xffffffff82cdf000     31f8 hidbus.ko
29    1 0xffffffff82ce3000     3320 wmt.ko
30    1 0xffffffff82ce7000     4350 ums.ko
31    1 0xffffffff82cec000    1ce48 ng_btsocket.ko
32    1 0xffffffff82d09000     25a8 ng_bluetooth.ko
33    1 0xffffffff82d0c000    388f8 linux.ko
34    4 0xffffffff82d45000     db70 linux_common.ko
35    1 0xffffffff82d53000    30ac8 linux64.ko
36    1 0xffffffff82d84000     2260 pty.ko
37    1 0xffffffff82d87000     639c linprocfs.ko
38    1 0xffffffff82d8e000     3284 linsysfs.ko
39    1 0xffffffff82d92000     3530 fdescfs.ko
40    1 0xffffffff82d96000     4700 nullfs.ko
41    1 0xffffffff82d9b000     41d8 acpi_ibm.ko

Summary

This concludes this Ghost in the Shell episode.

Feel free to share your scripting habits and spells πŸ™‚

EOF

Ghost in the Shell – Part 5

The Ghost in the Shell series were quite neglected while I was busy writing about other things. Its about time to continue the series. I hope you are not mad at me because of it. Here are another few things that I think some of you may find useful.

You may want to check other articles in the Ghost in the Shell series on the Ghost in the Shell – Global Page where you will find links to all episodes of the series along with table of contents for each episode’s contents.

Less More Useful

From all less(1) command line options I find these very handy.

Often when you pass some command output to less(1) you loose color. To keep color in the less(1) output use --raw-control-chars (or -r for short equivalent) option.

The other useful less(1) option I find useful is --chop-long-lines (or -S for short equivalent) which prevents line wrapping. You can of course scroll horizontally to see what does not fit on the screen.

While less(1) is a command line program it also has a very nice --mouse option – with this option you can scroll its output with your mouse wheel. How cool is that? You can even specify how many lines you want to scroll with --wheel-lines=n options where ‘n is as you probably guessed the number of scrolled lines.

It may be also useful to make less(1) quit if you want to display file that its contents fits in the current console screen – use --quit-if-one-screen for that.

Often people when they want to just view some config files they use vi(1) (or their other favorite ${EDITOR} that they use) – even if they do not intend to edit the file. Its better to open such file in less(1) and if you find out that you would want to edit the file hit the ‘v char while being at less(1) – it will open that file in your ${EDITOR} for editing.

You can also display line number in less(1) with --LINE-NUMBERS option (or use -N for shorter equivalent).

Detox These Filenames

Often when copying files from various sources the filenames may become corrupt in the process – mostly because of differences in encodings. To fix that very fast one may use detox(1) command. On FreeBSD systems its available as sysutils/detox package. Because the FILE you will be renaming almost for sure contains some special characters or spaces then its best to add quotation marks as shown below.

% detox "FILE"

Of course detox(1) renames one file at a time so to rename all files in the current directory we will use simple loop.

% for FILE in *; do detox "${FILE}"; done

If you want to also include subdirectories the do the following.

% find . -type f -exec detox {} ';'

If you do not want to limit yourself to files only (fix directories names also) then – as Mandalorian would say – this is the way.

% find . -exec detox {} ';'

Man Up the Info Pages

In the learning process of mastering UNIX systems one has to get used to reading man(1) pages and often getting back to re-reading them when needed. Like with many other things the GNU folks wanted to do things in their own way – seems they did not liked the man(1) pages that much as they created info(1) pages as an alternative. I dunno about you but IMHO info(1) pages does not feel like the UNIX way … maybe it’s because GNU is a recursive acronym for GNU IS NOT UNIX πŸ™‚

However there is an elegant way to convert any info(1) page into man(1) page by piping the info(1) page output into less(1) command – or other ${PAGER} that you use.

% info ls | less

Real UNIX Sorting

After you setup your UNIX environment the LC_ALL environment variable is mostly set to some UTF variant – like en_us.utf-8 for example. That has implications as names of files and directories are now sorted case insensitively. To get back to original case sensitive UNIX sorting you can use the LC_ALL variable set ‘C‘. You can use that on the fly or make it permanent by adding it to your shell configuration. For example with ls(1) command shown below.

% ls -1
FreeBSD.org
kernel.org
Linux.com
NetBSD.org
openbsd.org
X11.org
xorg.conf

% env LC_ALL=C ls -1
FreeBSD.org
Linux.com
NetBSD.org
X11.org
kernel.org
openbsd.org
xorg.conf

Faster Better Uptime

When you want check for how long system was running we usually use uptime(1) command.

% uptime
8:15PM  up 5 days,  4:42, 4 users, load averages: 0.71, 0.76, 0.82

But you can type just one letter instead of six and get even more info – the w(1) command. It also includes information about other active sessions to this system – which comes handy because you want to know if someone else can try to fix or configure the same things as you intend to.

% w
8:15PM  up 5 days,  4:42, 4 users, load averages: 0.77, 0.78, 0.83
USER       TTY      FROM    LOGIN@  IDLE WHAT
vermaden   pts/0    :0     Thu10PM  3:09 -zsh (zsh)
szasstam   pts/1    :0     Sun08PM 1day  -zsh (zsh)
edwin      pts/2    :0      7:04PM     - -zsh (zsh)
larloch    pts/3    :0      7:56PM     - w

Filter Huge Files

When you start grep(1) to filter really big file – like several gigabytes in size for example – the grep(1) command uses locale from LC_ALL and LANG variables – which as you probably guess right know from the context of this sentence – slows things down.

You can modify both LC_ALL and LANG on the fly to ‘C‘ value to make that grep(1) really fast – and when I mean fast I mean sometimes you will gain several orders of magnitude.

% env LC_ALL=C LANG=C grep string HUGEFILE

That is all for this episode. Hope you liked it.

EOF

Ghost in the Shell – Part 3

Time to bring some life into the Ghost in the Shell series with Part 3 article.

You may want to check other articles in the Ghost in the Shell series on the Ghost in the Shell – Global Page where you will find links to all episodes of the series along with table of contents for each episode’s contents.

Query Functions

I haven’t found better name for that solution. There are generally two types of UNIX people. These that prefer to navigate and operate with basic ls/cd/mv/mkdir/rm commands and those who use some file manager like Midnight Commander (mc) or ranger or vifm or … you get the idea. I have tried various CLI file managers but always came back to navigate without them. If you are one of those people then these Query Functions are for you πŸ™‚

The so called Query Functions are for filter the information you look for. For example if you have directory with large number of files, then you would probably do something like that.

% ls | grep QUERY

… or if you also want to include subdirectories then something like that.

% find . | grep QUERY

For both of these examples you would also probably want to sometimes search case sensitive or insensitive depending on the need.

That leads us to four Query Functions:

  • q is an equivalent of ls | grep -i QUERY command.
  • Q is an equivalent of ls | grep QUERY command.
  • qq is an equivalent of find . | grep -i QUERY command.
  • QQ is an equivalent of find . | grep QUERY command.

Thus if I need to query the contents of directory while searching for something is very fast with q SOMETHING.

These are definitions of these Query Functions:

# SHORT QUERY FUNCTIONS q()
  q() {
    if [ ${#} -eq 1 ]
    then
      ls | grep --color -i ${1} 2> /dev/null
    else
      echo "usage: q string"
    fi
  }
     
# SHORT QUERY FUNCTIONS Q()
  Q() {
    if [ ${#} -eq 1 ]
    then
      ls | grep --color ${1} 2> /dev/null
    else
      echo "usage: Q string"
    fi
  }

# SHORT QUERY FUNCTIONS qq()
  qq() {
    if [ ${#} -eq 1 ]
    then
      find . \
        | grep -i ${1} 2> /dev/null \
        | cut -c 3-999 \
        | grep --color -i ${1} 2> /dev/null
    else
      echo "usage: qq string"
    fi
  }

# SHORT QUERY FUNCTIONS QQ()
  QQ() {
    if [ ${#} -eq 1 ]
    then
      find . \
        | grep ${1} 2> /dev/null \
        | cut -c 3-999 \
        | grep ${1} 2> /dev/null
    else
      echo "usage: QQ string"
    fi
  }

The qq and QQ functions uses grep(1) two times to make sure the output is colored.

I assume that You use colored grep(1) described in Ghost in the Shell – Part 2 article.

If you prefer to use alias(1) instead then they would look like that.

# SHORT QUERY FUNCTIONS q() Q() qq() QQ()
  alias q="ls | grep --color -i"
  alias Q="ls | grep --color"
  alias qq="find . | grep -i"
  alias QQ="find . | grep"

The qq and QQ will be little more limited as with functions its possible to trim the output to the exact needs with cut(1).

q.png

qq.png

Lots of people use recursive history search which also helps, but what if you used/typed needed command long ago with the arguments you need now? You would probably search the command with history(1) command and then using grep(1) to limit the results to what you look for. I keep enormous large list of commands to keep in history – with my current setting of 655360 the ~/.zhistory (ZSH) file takes about 2.7 MB size. I also wanted to be sure that two identical commands would not be kept in history hence the setopt hist_ignore_all_dups ZSH option enabled. When I wc -l my ~/.zhistory file it currently has 75695 lines of commands.

% grep HISTSIZE /usr/local/etc/zshrc
export HISTSIZE=655360
export SAVEHIST=${HISTSIZE}

% grep dups /usr/local/etc/zshrc
setopt hist_ignore_all_dups

Now back to Query Functions for history:

  • h is an equivalent of cat ~/.zhistory | grep -i QUERY command.
  • H is an equivalent of cat ~/.zhistory | grep QUERY command.

They fit in aliases this time. In alias(1) we will use just grep(1) to not ‘do’ Useless Use of Cat.

Here are the Query Functions for history.

# SHORT HISTORY ALIASES h() H()
  alias h='< ~/.zhistory grep -i'
  alias H='< ~/.zhistory grep'

h

… but what if we would like to filter the outputs of q family and h family Query Functions? The obvious response is using grep(1) like q QUERY | grep ANOTHER or h QUERY | grep ANOTHER for example. To make that faster we will make g and G shortcuts.

  • g is an equivalent of grep -i command.
  • G is an equivalent of just grep command.

Here they are.

# SHORT GREP FUNCTIONS g() G()
  alias g='grep -i'
  alias G='grep'

Now it will be just q QUERY | g ANOTHER and h QUERY | G ANOTHER for example.

To clear terminal output you may use clear(1) command, some prefer [CTRL]-[L] shortcut but I find ‘c‘ alias to be the fastest solution.

# SHORT GREP FUNCTIONS c()
  alias c='clear'

To make the solution complete I would also add exa(1) here with an alias of ‘e‘.

# SHORT LISTING WITH e()
  alias e='exa --time-style=long-iso --group-directories-first'

Why exa(1) will you ask while there is BSD ls(1) and GNU ls(1) (installed as gls(1) on FreeBSD to not confuse). To add GNU ls(1) to FreeBSD system use the coreutils package.

Well, the BSD ls(1) has two major cons:

  • It is not able to sort directories first.
  • It selects width for ALL columns based on single longest file name.

BSD-ls.png

The BSD ls(1) was used as following alias:

alias ls='ls -p -G -D "%Y.%m.%d %H:%M"'

The GNU ls(1) does not have these two problems but it does color the output only on the very limited pattern like:

  • Not executable file.
  • Executable file.
  • Directory.
  • Link.
  • Device.

GNU-ls.png

The GNU ls(1) was used as following alias:

gls -p -G --color --time-style=long-iso --group-directories-first --quoting-style=literal

Here is where exa(1) comes handy as it does not have any cons like FreeBSD’s ls(1) and it colors a lot more types of files.

e.png

exa --time-style=long-iso --group-directories-first

Its still very simple coloring based on file extension and not magic number as plain (empty) text file SOME-NOT-FILE.pdf is colored like PDF document.

e-pdf.png

But even this ‘limited’ coloring helps in 99% of the cases and while with BSD ls(1) and GNU ls(1) all of these files ‘seem’ like plain text files with exa(1) its obvious from the start which are plain files, which are images and which are ‘documents’ like PDF files for example.

Where Is My Space

On all UNIX and Linux systems there exists du(1) command. Combined with sort(1) it is universal way of searching for space eaters. Example for the / root directory with -g flag to display units in gigabytes.

# cd /
# du -sg * | sort -n
1       bin
1       boot
1       compat
1       COPYRIGHT
1       data
1       dev
1       entropy
1       etc
1       lib
1       libexec
1       media
1       mnt
1       net
1       proc
1       rescue
1       root
1       sbin
1       sys
1       tmp
1       var
2       jail
8       usr
305     home

Contents of UNIX System Resources directory with -m flag to display unit in megabytes.

# cd /usr
# du -sm * | sort -n
1       libdata
1       obj
1       tests
3       libexec
11      sbin
13      include
45      lib32
56      lib
58      share
105     bin
1080    ports
1343    src
5274    local

But its PITA to type cd and du all the time, not to mention that some oldschool UNIX systems does not provide -g or -m flags so on HP-UX you are limited to kilobytes at most.

You may also try -h (human readable) with sort -h (sort human readable) du(1) variant.

# du -smh * | sort -h
512B    data
512B    net
512B    proc
512B    sys
4.5K    COPYRIGHT
4.5K    entropy
5.5K    dev
6.5K    mnt
 53K    media
143K    tmp
205K    libexec
924K    bin
2.2M    etc
3.9M    root
4.6M    sbin
6.2M    rescue
6.6M    lib
 90M    boot
117M    compat
564M    jail
667M    var
5.4G    usr
297G    home

This is where ncdu(1) comes handy. Its ncurses based disk usage analyzer which helps finding that space eaters in very fast time without typing the same commands over and over again. Here is ncdu(1) in action.

First it calculates the sizes of the files.

ncdu.png

After a while you get the output sorted by size.

ncdu-usr.png

If you hit [ENTER] on the directory you will be instantly moved into that directory.

ncdu-usr-local.png

If you delete something with ‘d‘ then remember to recalculate the output with ‘r‘ letter.

It also has great options such as spawning shell ‘b‘ in the current directory or toggle between apparent size and disk usage with ‘a‘ option. The latter is very useful when you use filesystem with builtin compression like ZFS.

       up, k  Move cursor up
     down, j  Move cursor down
 right/enter  Open selected directory
  left, <, h  Open parent directory
           n  Sort by name (ascending/descending)
           s  Sort by size (ascending/descending)
           C  Sort by items (ascending/descending)
           d  Delete selected file or directory
           t  Toggle dirs before files when sorting
           g  Show percentage and/or graph
           a  Toggle between apparent size and disk usage
           c  Toggle display of child item counts
           e  Show/hide hidden or excluded files
           i  Show information about selected item
           r  Recalculate the current directory
           b  Spawn shell in current directory
           q  Quit ncdu

The apparent size using the du(1) command.

Disk usage.

% du -sm books
39145   books

Apparent size.

% du -smA books
44438   books

So I have 1.13 compression ratio on the ZFS filesystem. More then 5 GB saved just in that directory πŸ™‚

Where Are My Files

Once I got some space back I also wanted to know if there are some directories with enormous amount of very small files.

First I came up with my own files-count.sh script solution which is not that long.

#! /bin/sh

export LC_ALL=C

if [ ${#} -eq 0 ]
then
  DIR=.
else
  DIR="${1}"
fi

find "${DIR}" -type d -maxdepth 1 -mindepth 1 \
  | cut -c 3- \
  | while read I
    do
      find "${I}" | wc -l | tr -d '\n'
      echo " ${I}"
    done | sort -n

It works reliably but same as with du | sort tandem you have to retype it (or at least use cd(1) and hit [UP] arrow again) … but then I discovered that ncdu(1) also counts files! It does not provide ‘startup’ argument to start in this count files mode but when you hit ‘c‘ letter it will instantly display count of files in each scanned directory. To sort this output by the count of files hit the ‘C‘ letter (large ‘C‘ letter).

ncdu-files.png

The files-count.sh script still has one advantage over ncdu(1) – the latter stops counting files at 100k which is shown on the screenshot so if You need to search for really big amount of files or just about 100k then files-count.sh script will be more accurate/adequate.

% cd /usr
% files-count.sh 
       1 obj
      36 libdata
     299 sbin
     312 libexec
     390 tests
     498 bin
     723 lib32
     855 lib
    2127 include
   16936 share
  159945 src
  211854 ports
  266021 local

… but what if there were some very big files hidden somewhere deep in the directories tree? The du(1) or ncdu(1) will not help here. As usual I though about short files-big.sh script that will do the job.

#! /bin/sh

export LC_ALL=C

if [ ${#} -eq 0 ]
then
  DIR=.
else
  DIR="${1}"
fi

find "${DIR}" -type f -exec stat -f "%16z; doas rm -f \"%N\"" {} ';' | sort -n

An example usage on the /var directory.

# cd /var
# files-big.sh | tail
        10547304; doas rm -f "./tmp/kdecache-vermaden/icon-cache.kcache"
        29089823; doas rm -f "./db/clamav/clamav-2671b72fce703c2133c61e5bf85aad19.tmp/clamav-373e311ca7f610a39c7cf5c5c5a4fd83.tmp/daily.hdb"
        30138884; doas rm -f "./tmp/pkg-provides-wyK2"
        48271360; doas rm -f "./db/pkg/repo-HardenedBSD.sqlite"
        54816768; doas rm -f "./db/pkg/repo-FreeBSD.sqlite"
        66433024; doas rm -f "./db/pkg/local.sqlite"
        82313216; doas rm -f "./db/clamav/clamav-2671b72fce703c2133c61e5bf85aad19.tmp/clamav-373e311ca7f610a39c7cf5c5c5a4fd83.tmp/daily.hsb"
       117892267; doas rm -f "./db/clamav/main.cvd"
       132431872; doas rm -f "./db/clamav/daily.cld"
       614839082; doas rm -f "./db/pkg/provides/provides.db"

The output is in ‘executable’ format so if you select whole line and paste it into terminal, then this file will be deleted. By default it uses doas(1) but nothing can stop you from putting sudo(8) there. Not sure if you will find it useful but it helped me at least dozen times.

How Many Copies Do You Keep

I often find myself keeping the same files in several places which also wastes space (unless you use ZFS deduplication of course).

The dedup.sh script I once made is little larger so I will not paste it here and just put a link to it.

It has the following options available. You may search/compare files by name or size (fast) or by its MD5 checksum (slow).

% dedup.sh
usage: dedup.sh OPTION DIRECTORY
  OPTIONS: -n   check by name (fast)
           -s   check by size (medium)
           -m   check by md5  (slow)
           -N   same as '-n' but with delete instructions printed
           -S   same as '-s' but with delete instructions printed
           -M   same as '-m' but with delete instructions printed
  EXAMPLE: dedup.sh -s /mnt

Simple usage example.

% cd misc/man
% cp zfs-notes zfs-todo
% dedup.sh -M .
count: 2 | md5: 4ff4be66ab7e5484de2bf7c168ff995a
  doas rm -rf "./zfs-notes"
  doas rm -rf "./zfs-todo"

count: 2 | md5: 6d87f5b1317ea189165fcdc71380735c
  doas rm -rf "./x11"
  doas rm -rf "./xinit"

By copying the zfs-notes file into the zfs-todo file I wanted to show you what dedup.sh will print on the screen, but accidentally I also found another duplicate πŸ™‚

The output of dedup.sh is simple and like with files-big.sh script selecting the while line and pasting it into the terminal will remove the duplicate. By default it uses doas(1) but you can change it into sudo(8) if that works better for you.

Unusual cron(1) Intervals

Most of us already remember what the five fields of crontab(5) file mean, but what if you would like to run command every second … or after reboot only? The answer lies in the man 5 crontab page. Here are these exotic options.

string          meaning
------          -------
@reboot         Run once, at startup of cron.
@yearly         Run once a year, "0 0 1 1 *".
@annually       (same as @yearly)
@monthly        Run once a month, "0 0 1 * *".
@weekly         Run once a week, "0 0 * * 0".
@daily          Run once a day, "0 0 * * *".
@midnight       (same as @daily)
@hourly         Run once an hour, "0 * * * *".
@every_minute   Run once a minute, "*/1 * * * *".
@every_second   Run once a second.

Check cron(1) Environment

Many times I found myself lost lots of time debugging what went wrong when my script was run by the crontab(5) file. Often it was some variable missing or some command or script I used was not in the PATH variable.

To make that debugging faster You can use ENV.sh script to just store the cron(1) environment.

% cat ENV.sh
env > /tmp/ENV.out

The ENV.sh script will write current environment in the /tmp/ENV.out file.

Lets put it into the crontab(5) for a test.

% crontab -l | grep ENV
@every_second ~/ENV.sh

Now after at most a second you can check for the contents of the /tmp/ENV.out file.

% cat /tmp/ENV.out
LOGNAME=vermaden
PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin
PWD=/home/vermaden
HOME=/home/vermaden
USER=vermaden
SHELL=/bin/sh

Now you can easily debug the scripts run by the crontab(5) … at least on the environment part πŸ™‚

Simple HTTP Server

I found myself many times in a situation that I would want to allow download of some files from my machine and SSH could not be used.

This is when python(1) comes handy. It has SimpleHTTPServer (or http.server in Python 3 version) so you can instantly start HTTP server in any directory!

Here are the commands for both Python versions.

  • Python 2.x – python -m SimpleHTTPServer PORT
  • Python 3.x – python -m http.server PORT

I even made a simple http.sh wrapper script to make it even more easy.

#! /bin/sh

if ${#} -ne 1 ]
then
  echo "usage: ${0##*/} PORT"
  exit 1
fi

python -m SimpleHTTPServer ${1}

Example usage.

% cd misc/man
% http.sh 8080
Serving HTTP on 0.0.0.0 port 8080 ...
127.0.0.1 - - [14/Sep/2018 23:06:50] "GET / HTTP/1.1" 200 -
127.0.0.1 - - [14/Sep/2018 23:06:50] code 404, message File not found
127.0.0.1 - - [14/Sep/2018 23:06:50] "GET /favicon.ico HTTP/1.1" 404 -
127.0.0.1 - - [14/Sep/2018 23:09:15] "GET /bhyve HTTP/1.1" 200 -

To stop it simply hit [CTRL]-[C] interrupt sequence.

Here is how it looks in the Epiphany browser.

http.png

Simple FTP Server

Similarly with FTP service, another Python goodie called pyftpdlib (Python FTP Server Library) provides that.

Mine ftp.py wrapper is little bigger as you can write quite comlicated setups with pyftpdlib but mine is simple, it starts in the current directory and adds read only anonymous user and read/write user named writer with WRITER password.

#! /usr/bin/env python

from sys                   import argv,exit
from pyftpdlib.authorizers import DummyAuthorizer
from pyftpdlib.handlers    import FTPHandler
from pyftpdlib.servers     import FTPServer

if len(argv) != 2:
  print "usage:", argv[0], "PORT"
  print
  exit(1)
  
authorizer = DummyAuthorizer()
authorizer.add_user("writer", "WRITER", ".", perm="elradfmw")
authorizer.add_anonymous(".")
handler = FTPHandler
handler.authorizer = authorizer
handler.passive_ports = range(60000, 60001)
address = ("0.0.0.0", argv[1])
ftpd = FTPServer(address, handler)
ftpd.serve_forever()

The ftp.py is handy if you want to enable someone to upload something for you (or you are doing it o the other machine) when SSH/SCP is not possible for some reason.

To stop it simply hit [CTRL]-[C] interrupt sequence.

Here is its terminal startup and logs.

% cd misc/man
% ftp.py 2121
[I 2018-09-14 23:21:53] }}} starting FTP server on 0.0.0.0:2121, pid=64399 {{{
[I 2018-09-14 23:21:53] concurrency model: async
[I 2018-09-14 23:21:53] masquerade (NAT) address: None
[I 2018-09-14 23:21:53] passive ports: 60000->60000

… and how Firefox renders its contents.

ftp.png

Hope you will find some of these useful, see you at Part 4 some day.

EOF

Ghost in the Shell – Part 2

The article in the Ghost in the Shell series was the first post on my blog, so while I was busy by writing various server related articles and recently the FreeBSD Desktop series its about time for the Part 2 of the Ghost in the Shell series.

You may want to check other articles in the Ghost in the Shell series on the Ghost in the Shell – Global Page where you will find links to all episodes of the series along with table of contents for each episode’s contents.

Lets start with something simple – yet powerful and time saving.

Alias with Arguments

One may of course write any function to do similar job, but keeping track and ‘maintaining’ all those functions becomes complicated and one has to organize itself. This partially applies to aliases, but they are smaller and easier to maintain then whole functions. In any modern shell an alias(1) can also have arguments, while You will not be able to parse them as appropriate as with functions, they do the job for their basic use.

Here is an example of such alias(1) with arguments.

% ls
gfx/ info/ misc/ scripts/ tmp/

% alias lsg='ls | grep'

% lsg gfx
gfx/

Color grep(1) Patterns

As we already ‘touched’ the grep(1) command topic, lets make it more usable by highlighting the found results in color. The ${GREP_COLOR} variable is used for that purpose and it expects a number for a color, here is the table with number-color format.

Color    Number
Black    30
Red      31
Green    32
Yellow   33
Blue     34
Magenta  35
Cyan     36
White    37

You may as well use ‘bold’ output by adding ‘1;‘ before the number, for example.

% echo ${GREP_COLOR}
1;31

You will also have to make an alias(1) to grep(1) with --color argument, like that:

% alias grep='grep --color'

Here is how it looks in practice.

% export GREP_COLOR=31
% alias grep='grep --color'
% dmesg | grep SMP
FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
FreeBSD/SMP: 1 package(s) x 2 core(s)
SMP: AP CPU #1 Launched!

Here is how it looks on the xterm(1) terminal.

ghost-terminal

Process Management

This one is very useful on any UNIX system, does not matter if its server or desktop.

These are commands and operands that will help us manage processes started by hand:

  • &
  • fg
  • bg
  • jobs
  • kill
  • disown
  • nohup
  • [CTRL]+[Z]
  • [CTRL]+[C]

As you probably already know to start command ‘in the background’ – which means do what I tell you but do not block the terminal – you have to add ‘&‘ (ampersand) at the end of such command. That command does not magically go away and as long as its running its visible by the jobs(1) command. You may use ‘-l‘ switch to also show the PID of background processes.

% galculator &
[1] 8449

% jobs
[1]  + running    galculator

% jobs -l
[1]  + 8449 running    galculator

Now, what of you forget to add ‘&‘ (ampersand) at the end of command but you wanted to put it into the background? Hit [CTRL]+[Z] shortcut (Control key with ‘small’ Z letter) and the process will be put into the suspended state. Now you have several options, you can out that process into the background with bg(1) command – by default it uses last suspended job – %1, you can also bring it back into the foreground blocking the terminal with fg(1) command. You can also list its state with jobs(1) and of course kill(1) it either with PID showed by jobs -l command or by specifying the process number – %1 in that case.

Here is an example.

% galculator
^Z
zsh: suspended  galculator

% jobs
[1]  + suspended  galculator

% bg
[1]  + continued  galculator

% jobs -l
[1]  + 72892 running    galculator

% kill %1
[1]  + terminated  galculator

%

While fg(1) and bg(1) allow you to put command in the background or foreground respectively when the process is in suspended state, one may ask how to ‘switch’ a process to suspended state while its already running in the background. Its done with kill -17 signal called SIGSTOP. You can also bring back such suspended process to running state with kill -19 signal called SIGCONT … or just again use fg(1) or bg(1) command. Other difference between fg(1)/bg(1) commands and more ‘direct’ kill -17/kill -19 commands are that kill(1) does not inform the user what has changed to the process. You may as well use kill -SIGCONT syntax or kill -s SIGCONT if that is more readable for you.

% galculator
^Z
zsh: suspended  galculator

% bg
[1]  + continued  galculator

% xcalc
^Z
zsh: suspended  xcalc

% jobs -l
[1]  - 19537 running    galculator
[2]  + 20563 suspended  xcalc

% kill -17 %1
[1]  + suspended (signal)  galculator

% jobs -l
[1]  + 19537 suspended (signal)  galculator
[2]  - 20563 suspended  xcalc

% kill -SIGCONT %1
% bg %2
[2]  - continued  xcalc

% jobs -l
[1]  + 19537 running    galculator
[2]  - 20563 running    xcalc

Also check man kill and man signal for more information.

What about disown(1) then? Its a ‘magic’ helper when you start some long running jobs directly at the terminal without Screen or Tmux and you need to disconnect that terminal, for example because you are taking your laptop with you. When you do this – depending on the settings of the current shell – the processes in the background may be killed or ‘moved’ to PID 1 (the init(1) of course) as the PPID (Parent PID). To achieve that we will used that disown(1) command. Once you ‘disown’ a process it will no longer be show by the jobs(1) command, but it will run ‘pinned’ to the init(1) process after you disconnect the terminal session.

% galculator
^Z
zsh: suspended  galculator

% bg
[1]  + continued  galculator

% jobs -l
[1]  + 98556 running    galculator

% disown %1

% jobs -l

% pgrep galculator
98556

% pstree -p 98556
─┬◆ 00001 root /sbin/init --
 └─┬─ 48708 vermaden xterm
   └─┬◆ 52463 vermaden -zsh (zsh)
     └──◆ 98556 vermaden galculator

Now its still pinned to the shell in the xterm(1) terminal. After we close the xterm(1) window (or kill that zsh(1) shell) it will switch to init(1) as PPID (Parent PID).

% pstree -p 98556
─┬◆ 00001 root /sbin/init --
 └──◆ 98556 vermaden galculator

% pgrep -P 1 galculator
98556

We are left with nohup(1) then, when and why to use it as we already has great disown(1) magic? Well, disown(1) is not always available, so when You need to put some command into the long background run and disconnect after it its the best possible option. By default the nohup(1) command will log the output of started command into the nohup.out file. Remember that nohup(1) will still run the process in the foreground, to put it into the background use ‘&‘ (ampersand) or [CTRL]+[Z] with bg(1) combo.

% nohup galculator
appending output to nohup.out
^Z
zsh: suspended  nohup galculator

% bg
[1]  + continued  nohup galculator

% jobs -l
[1]  + 22322 running    nohup galculator

% pstree -p 22322
─┬◆ 00001 root /sbin/init --
 └─┬─ 89568 vermaden xterm
   └─┬◆ 91486 vermaden -zsh (zsh)
     └──◆ 22322 vermaden galculator

… and after disconnect out process switched to init(1) as PPID.

% pstree -p 22322
─┬◆ 00001 root /sbin/init --
 └──◆ 22322 vermaden galculator

You may of course end a running process in the foreground with [CTRL]+[C] shortcut, but that is probably already known to you. I just mention it for the ‘completeness’ of the guide.

% galculator
^C

%

Which Which

While the which(1) command shows the full path of the executable found in the first directory of the ${PATH} variable, it also shows what alias is used for that command it there is one. One may ask how then to find information about absolute executable path if it shows and alias(1) instead. Well, you have to use unalias(1) on that command, so which(1) would be showing full path again.

% which caja
caja: aliased to caja --browser --no-desktop

% unalias caja

% which caja
/usr/local/bin/caja

Also be sure to check Smylers comment below about the difference between shell builtin which and /usr/bin/which command.

The difference is that by typing which you are executing your shell builtin command (ZSH in my case) which also takes aliases into account. If you want to omit the unalias part then use /usr/bin/which which will ignore any existing aliases.

% which caja
caja: aliased to caja --browser --no-desktop

% /usr/bin/which caja
/usr/local/bin/caja

Record Session

If you have used PuTTY or MobaXterm in your work, then you appreciate the possibility of saving the terminal output to a file, foe example for the documentation purposes. This is also available ‘natively’ in the shell by using the script(1) command. Remember that script(1) will record also ‘special’ characters like colors, so to properly ‘replay’ the session you may want to either use script(1) or cat(1) commands for that or use less with -R argument.

Here is example recorded script(1) session.

% script script.out
Script started, output file is script.out

% ls
gfx info misc scripts tmp unix.png

% uname -spr
FreeBSD 11.2-RELEASE amd64

% exit
Script done, output file is script.out

% cat script.out
Script started on Sun Jul  8 08:24:06 2018
You have mail.
% ls | grep gfx
gfx
% uname -spr
FreeBSD 11.2-RELEASE amd64
% exit
exit

Script done on Sun Jul  8 08:24:20 2018

% less -R script.out
Script started on Sun Jul  8 08:24:06 2018
You have mail.
% ls | grep gfx
gfx
% uname -spr
FreeBSD 11.2-RELEASE amd64
% exit
exit

Script done on Sun Jul  8 08:24:20 2018

% less script.out
Script started on Sun Jul  8 08:24:06 2018
You have mail.
% ls | grep gfx
ESC[1;31mgfxESC[00mESC[K
% uname -spr
FreeBSD 11.2-RELEASE amd64
% exit
exit

Script done on Sun Jul  8 08:24:20 2018


Edit Command Before Executing

Sometimes you have long multi-line command to execute, so often it is crafted in you favorite ${EDITOR} and then pasted into the terminal. To omit copying and pasting yo may want to check fc(1) command which serves similar purpose. After you type a command, for example simple ls(1) command, and then you type fc(1) command, then fc(1) will take that ls(1) command into your favorite text editor from ${EDITOR} variable, will allow you to edit it and if you save and exit the that editor, it will execute it.

Lets see how it behave by example.

% ls
gfx   books   download   scripts

% fc

Now you are taken into the ${EDITOR} which is vi(1) in my case.

      1 ls
~
~
~
/tmp/zsh999EQ6: unmodified: line 1

Lets made some changes.

      1 ls -l \
      2    -h
~
~
~
~

:wq

After you hit [ENTER] it will exit from ${EDITOR} and execute that command.

total 6181
drwxr-xr-x    87 vermaden  vermaden    87B 2017.12.18 15:30 books/
drwxr-xr-x    12 vermaden  vermaden    12B 2018.06.19 16:02 download/
drwxr-xr-x    19 vermaden  vermaden    20B 2018.05.24 11:52 gfx/
drwx------    12 vermaden  vermaden   310B 2018.07.07 03:23 scripts/

You may show that command by pressing [Up] key to check what has been executed.

% ls -l -h

Edit or Just View

When working in multi-admin environment – especially while debugging – one admin may block other admin’s work by using vi(1) – or just their favorite editor to ‘browse’ the file contents. Good practice in that case is using more(1) or less(1) instead of vi(1), but that frustrates some admins to type vi(1) again if they need to change something.

… and by the way, on FreeBSD more(1) is less(1) πŸ™‚

% uname -spr
FreeBSD 11.2-RELEASE amd64

% ls -i `which less` `which more`
492318 /usr/bin/less  492318 /usr/bin/more

A blocked ‘example’ is shown below when the second admin wanted to browse the /etc/rc.conf file while the first one already did that.

# vim /etc/rc.conf

E325: ATTENTION
Found a swap file by the name "/etc/.rc.conf.swp"
          owned by: root   dated: Sun Jul  8 08:38:35 2018
         file name: /etc/rc.conf
          modified: no
         user name: root   host name: t420s.local
        process ID: 54219 (still running)
While opening file "/etc/rc.conf"
             dated: Fri Jul  6 00:51:11 2018

(1) Another program may be editing the same file.  If this is the case,
    be careful not to end up with two different instances of the same
    file when making changes.  Quit, or continue with caution.
(2) An edit session for this file crashed.
    If this is the case, use ":recover" or "vim -r /etc/rc.conf"
    to recover the changes (see ":help recovery").
    If you did this already, delete the swap file "/etc/.rc.conf.swp"
    to avoid this message.

Swap file "/etc/.rc.conf.swp" already exists!
[O]pen Read-Only, (E)dit anyway, (R)ecover, (Q)uit, (A)bort:

This is where less(1) comes handy because of you open a file in it, you do not ‘block’ access to it and if you need to edit something just hi [V] key (small ‘v’ letter). It will open that file in your ${EDITOR} editor and you can make any changes now.

Reset

Last but not least, often when you paste ‘too much’ into the terminal it becomes ‘fragile’ or ‘broken’. To reset it into the ‘stable’ and ‘proper’ state just use the reset(1) command.

% reset

Hope You find it useful, see you at the Part 3 sometime πŸ˜‰

EOF

Ghost in the Shell – Part 1

I wanted to post this earlier, but the busy daily life does not help πŸ˜‰

This will be first article in the series about efficient working in the shell environment. There are actually a lot articles and blog posts about efficient working in the terminal, but a lot of them are biased towards very specific uses, like hints only for Bash shell or only for specific terminal emulator. For example Moving efficiently in the CLI.

These series are about universal knowledge that would work on most shells and environments. Lets start with hint that I use many times a day that saves a lot time for not having to type …

You may want to check other articles in the Ghost in the Shell series on the Ghost in the Shell – Global Page where you will find links to all episodes of the series along with table of contents for each episode’s contents.

Recall Last Argument of Previous Command

Imagine most simple scenario, creating directory and entering it. Typically its like that:

% mkdir clear-place-for-new-work
% cd clear-place-for-new-work
%

The longer the name, the bigger the chance that You would type mkdir, then hit the [UP] arrow, then [HOME] or [CTRL]+[A] keys and then put cd in the place of mkdir.

With the use of !$ You can recall last argument of the precious command, so it will now look like that.

% mkdir clear-place-for-new-work
% cd !$
cd clear-place-for-new-work
%

Faster isn’t it?

Swap First Occurrence of a Word

The upper example can be used for the next advice as well. By typing ^fromwhat^towhat in the terminal You will swap the first occurrence of word fromwhat word to towhat word in the previous command, lets see how its working.

% mkdir clear-place-for-new-work
% ^mkdir^cd
cd clear-place-for-new-work
%

It still takes more time to write then using the !$ so its useful mostly when there are short things to swap, like numbers, for example ^3^4 to ‘move’ from one target to another. … or also if You can not recall to the last argument of previous command.

There and Back Again

A lot people does not know, that You can go back to previous working directory with dash. Lets assume that You need to get to /tmp directory for one command and get back to where You were to continue the work. Here is an example.

% pwd
/usr/local/etc/bareos/bareos-dir.d/jobdefs
% cd /tmp
% pwd
/tmp% (do needed work in /tmp dir)
% cd -
/usr/local/etc/bareos/bareos-dir.d/jobdefs
% pwd
/usr/local/etc/bareos/bareos-dir.d/jobdefs

You can even create entire directory stack with pushd/popd commands if needed, check Wikipedia article on that for more information. You can also use ${OLDPWD} variable. Useful with umount command for example.

% pwd
/media/backup-pendrive-key
% cd ~
% umount $OLDPWD
% pwd
/home/vermaden

Repeat Command from History

With exclamation mark (!) You can re-invoke the command from history with all its arguments (which sometimes can be risky). For example.

% !pkg
pkg update -f
(runs actual command)
%

Its better to first check what arguments have been used in that command, that is where :p comes handy. Here is its example usage.

% !pkg:p
pkg update -f
(just prints command without running it)
% !pkg
pkg update -f
(runs actual command)
%

Now, as arguments are known its safe to re-invoke the command with arguments. When this can be dangerous? Can ls command can be dangerous, that depends what You have on Your history, check the example below.

% ls | while read I; do rm -f ${I}; done

This command first lists the contents of the current working directory with ls command, then the output is piped to the while loop which invokes rm -f command for each item listed by ls command, which efficiently removes all non-hidden files in current working directory … which probably is not what we mean by typing !ls on the command prompt ;). That is why its valuable to first check what arguments were used with !ls:p syntax.

Enough for now, I will write more parts with more hints on how to efficiently work in the shell/terminal environment.

UPDATE 1

The Ghost in the Shell – Part 1 article was included in the BSD Now 241 – Bowling in the LimeLight episode.

Thanks for mentioning!

UPDATE 2

About Recall last argument of previous command section … there is also $_ that does similar thing as !$ but there is little difference. The !$ is ‘line oriented’ while $_ is ‘previous command oriented’. Below is an example that shows the difference in the behavior.

The $! takes value from last command in ‘previous line’ which means that '-l' value will be used from line 001 and not 'asd' from the current line 002 from previously executed command.

001 % ls -l
002 % echo asd; ls !$ | tail -2
echo asd; ls -l | tail -2
asd
// ls output //

The $_ takes value from last executed command, thus it points at 'asd' used on line 002 and not at '-l' used at previous 001 line.

001 % ls -l
002 % echo asd; ls $_ | tail -2
asd
ls: asd: No such file or directory

On BASH shell there is also [ALT]-[.] shortcut that switches between $! from previous lines. To achieve the same shortcut on ZSH use this line below in ZSH config.

bindkey '\e.' insert-last-word

Thank you Zachery Purnell for pointing that out.

EOF