Tag Archives: xterm

FreeBSD Desktop – Part 18 – Configuration – Global Dashboard

Many times I have found myself watching the various ‘debug’ commands like top/ps/mount/df or various log files like /var/log/messages or /var/log/automount.log when I thought something went wrong … or just takes little too long. I needed to open several terminal xterm(1) sessions (which is quite fast as I open them with [WIN]+[SPACE] and then [ENTER] but still …) and check what went wrong.

These actions tired my so I created a thing called Global Dashboard with all information I would ever need for such debugging.

You may want to check other articles in the FreeBSD Desktop series on the FreeBSD Desktop – Global Page where you will find links to all episodes of the series along with table of contents for each episode’s contents.

From all the commands that FreeBSD contains I have chosen these 12 ones:

  • mount -p
  • /var/log/automount.log
  • /var/log/messages
  • vmstat -i
  • usbconfig
  • ps axwww -o %cpu,rss,command
  • sockstat -l -4
  • top -m io -o total
  • gstat -p
  • df -g
  • pciconf -l
  • ifconfig

Make sure you have doas(1) installed and configured. The most basic way to do it is below. You will have to be in wheel group to make it work properly.

# pkg install doas
# echo 'permit nopass :wheel as root' > /usr/local/etc/doas.conf
# chmod 400 /usr/local/etc/doas.conf

Let me show you how it looks.

Here is the typical empty desktop with Global Dashboard disabled.

conky-off.png

… and here is the Global Dashboard enabled.

conky-on.png

For the sake of comfort I will use [Scroll Lock] key with xbindkeys to toggle between this ‘debug’ session on and off as I already use [Pause Break] key to Pause Any Application described in the Part 16 – Configuration – Pause Any Application episode of FreeBSD Desktop series.

scroll-lock.jpg

Conky

We will have to use older (1.9) version of Conky as the current one (1.10/1.11) are broken for anything serious.

We will use portdowngrade tool for that job.

First, lets install needed packages.

# pkg install portdowngrade conky xbindkeys

Assuming that you have up to date FreeBSD Ports tree in the /usr/ports directory – we see that current Conky version in the Ports is 1.11.

% cd /usr/ports/sysutils/conky
% cat distinfo 
TIMESTAMP = 1550919299
SHA256 (brndnmtthws-conky-v1.11.3_GH0.tar.gz) = 0140e749537d4d05bf33fbac436e54756faa26021e16f2bca418e9eeea724eb4
SIZE (brndnmtthws-conky-v1.11.3_GH0.tar.gz) = 2390099

We will now downgrade the Conky port to usable 1.9 version with portdowngrade utility. I already tried various Conky Port versions and the one that you are looking for is r419144 revision.

# cd /usr/ports/sysutils
# mv conky conky-1.11
# portdowngrade sysutils/conky | grep -C 17 r419144
------------------------------------------------------------------------
r422880 | madpilot | 2016-09-28 18:55:38 +0200 (Wed, 28 Sep 2016) | 13 lines

- Update conky and conky-awesome to 1.10.4
- Take maintainership [1]
- Options adapted to new version
- Removed LUA option since it's a mandatoory requirement now
- Use project own install target
- Fix installation of lua helper libraries
- Project moved to github
- in conky-awesome, properly use OPTIONS_EXCLUDE

PR:           212629
Submitted by: me
Approved by:  ntarmos@ceid.upatras.gr (former maintainer) [1]

------------------------------------------------------------------------
r419144 | pawel | 2016-07-26 20:57:23 +0200 (Tue, 26 Jul 2016) | 2 lines

Fix typo

------------------------------------------------------------------------
r419142 | pawel | 2016-07-26 20:40:20 +0200 (Tue, 26 Jul 2016) | 8 lines

- Add explicit IMPLIES between dependencies and simplify option handling [1]
- Convert to USES=localbase
- Switch some options helpers from LIB_DEPENDS to USE=xorg and USE=gnome

PR:           210414 [1] (based on)
Submitted by: elferdo@gmail.com
Approved by:  maintainer timeout

------------------------------------------------------------------------
r418767 | mat | 2016-07-19 13:04:13 +0200 (Tue, 19 Jul 2016) | 11 lines

We will now fetch the Conky port from r419144 revision – working 1.9 version.

# portdowngrade sysutils/conky r419144
A    conky/files
A    conky/Makefile
A    conky/files/patch-configure
A    conky/files/patch-lua-cairo.pkg
A    conky/files/patch-src-conky.c
A    conky/files/patch-src-freebsd.c
A    conky/files/patch-src-freebsd.h
A    conky/files/patch-src-fs.c
A    conky/pkg-descr
A    conky/distinfo
Checked out revision 419144.
You should be done-- now cd into conky and you can run
# make deinstall install clean

Please note that portdowngrade no longer modifies the ports tree; the
checked out port is at
/usr/ports/sysutils/conky

Done. Let’s verify that its the version we need.

% pwd
/usr/ports/sysutils
% cat conky-1.11/distinfo 
TIMESTAMP = 1550919299
SHA256 (brndnmtthws-conky-v1.11.3_GH0.tar.gz) = 0140e749537d4d05bf33fbac436e54756faa26021e16f2bca418e9eeea724eb4
SIZE (brndnmtthws-conky-v1.11.3_GH0.tar.gz) = 2390099

% cat conky/distinfo 
SHA256 (conky-1.9.0.tar.bz2) = baf1b550f135fbfb53e5e286a33aadc03a667d63bf6c4d52ba7637366295bb6f
SIZE (conky-1.9.0.tar.bz2) = 626555

Yup. We will now build a Conky 1.9 package (may be handy later).

# pwd
/usr/ports/sysutils
# cd conky
# pwd
/usr/ports/sysutils/conky
# make package
===>   conky-1.9.0_6 depends on file: /usr/local/sbin/pkg - found
=> conky-1.9.0.tar.bz2 doesn't seem to exist in /usr/ports/distfiles/.
=> Attempting to fetch https://downloads.sourceforge.net/project/conky/conky/1.9.0/conky-1.9.0.tar.bz2
conky-1.9.0.tar.bz2                           100% of  611 kB  216 kBps 00m03s
===> Fetching all distfiles required by conky-1.9.0_6 for building
===>  Extracting for conky-1.9.0_6
=> SHA256 Checksum OK for conky-1.9.0.tar.bz2.
===>  Patching for conky-1.9.0_6
===>  Applying FreeBSD patches for conky-1.9.0_6
===>   conky-1.9.0_6 depends on executable: gmake - found
===>   conky-1.9.0_6 depends on package: libiconv>=1.14_11 - found
===>   conky-1.9.0_6 depends on package: pkgconf>=1.3.0_1 - found
===>   conky-1.9.0_6 depends on file: /usr/local/libdata/pkgconfig/x11.pc - found
===>   conky-1.9.0_6 depends on file: /usr/local/libdata/pkgconfig/xext.pc - found
===>   conky-1.9.0_6 depends on file: /usr/local/libdata/pkgconfig/xdamage.pc - found
===>   conky-1.9.0_6 depends on file: /usr/local/libdata/pkgconfig/xfixes.pc - found
===>   conky-1.9.0_6 depends on file: /usr/local/libdata/pkgconfig/xft.pc - found
===>  Configuring for conky-1.9.0_6
===>   FreeBSD 10 autotools fix applied to /usr/ports/obj/usr/ports/sysutils/conky/work/conky-1.9.0/config.rpath
(...)
====> Compressing man pages (compress-man)
===>  Building package for conky-1.9.0_6
===>  Cleaning for conky-1.9.0_6

… but where is our package, its not in the /usr/ports/sysutils/conky directory. Its not in the /usr/ports/distfiles dir either.

As I use WRKDIRPREFIX=${PORTSDIR}/obj option in the /etc/make.conf file it should be somewhere in the /usr/ports/obj then.

% grep WRKDIRPREFIX /etc/make.conf 
WRKDIRPREFIX=${PORTSDIR}/obj

Let’s find(1) it.

% find /usr/ports/obj -name conky\*txz
/usr/ports/obj/usr/ports/sysutils/conky/work/pkg/conky-1.9.0_6.txz

There. I will move it to /root directory to keep it.

# mv /usr/ports/obj/usr/ports/sysutils/conky/work/pkg/conky-1.9.0_6.txz /root

We will not clean up after the port/package building.

# make -C /usr/ports/sysutils/conky clean distclean
===>  Cleaning for conky-1.9.0_6
# 

We will now delete installed Conky 1.11 version and install our working 1.9 version.

# pkg delete conky
Checking integrity... done (0 conflicting)
Deinstallation has been requested for the following 1 packages (of 0 packages in the universe):

Installed packages to be REMOVED:
        conky-1.11.3

Number of packages to be removed: 1

Proceed with deinstalling packages? [y/N]: y
[1/1] Deinstalling conky-1.11.3...
[1/1] Deleting files for conky-1.11.3: 100%

# pkg add /root/conky-1.9.0_6.txz
Installing conky-1.9.0_6...
Extracting conky-1.9.0_6: 100%

Last check for the Conky version.

% conky --version
Conky 1.9.0 compiled Tue Mar 19 12:55:55 CET 2019 for FreeBSD 11.2-RELEASE-p9 (amd64)

Compiled in features:

System config file: /usr/local/etc/conky/conky.conf
Package library path: /usr/local/lib/conky

 X11:
  * Xdamage extension
  * XDBE (double buffer extension)
  * Xft
  * ARGB visual

 Music detection:

 General:
  * math
  * config-output

Great. We have needed Conky version.

By the way – did you thought how much work will it take to make the same on Debian or CentOS without the FreeBSD Ports infrastructure? πŸ™‚

Xbindkeys

The only needed configuration in the ~/.xbindkeysrc is this one below – it may be different for your keyboard so make sure to ‘catch’ needed key event.

% cat ~/.xbindkeysrc
# SCROLL LOCK | Scroll Lock
"~/scripts/desktop-debug.sh"
  m:0x0 + c:78

If you need more information about how Xbindkeys work then read the FreeBSD Desktop – Part 9 – Key Components – Keyboard/Mouse Shortcuts episode.

Scripts and Configs

This is the ~/scripts/desktop-debug.sh script.

#! /bin/sh

pgrep -q conky

case ${?} in
  (0) killall -9 conky ;;
  (1) ~/scripts/__openbox_restart_conky.sh ;;
esac

… and the ~/scripts/__openbox_restart_conky.sh script.

#! /bin/sh

VERSION=1.9
PROFILE=T420s

killall -9 conky

nice -n 20 conky -c ~/.conkyrc.${VERSION}.${PROFILE}.LOG.1 &
nice -n 20 conky -c ~/.conkyrc.${VERSION}.${PROFILE}.LOG.2 &
nice -n 20 conky -c ~/.conkyrc.${VERSION}.${PROFILE}.LOG.3 &
nice -n 20 conky -c ~/.conkyrc.${VERSION}.${PROFILE}.LOG.4 &
nice -n 20 conky -c ~/.conkyrc.${VERSION}.${PROFILE}.LOG.5 &
nice -n 20 conky -c ~/.conkyrc.${VERSION}.${PROFILE}.LOG.6 &
nice -n 20 conky -c ~/.conkyrc.${VERSION}.${PROFILE}.LOG.7 &
nice -n 20 conky -c ~/.conkyrc.${VERSION}.${PROFILE}.LOG.8 &
nice -n 20 conky -c ~/.conkyrc.${VERSION}.${PROFILE}.LOG.9 &
nice -n 20 conky -c ~/.conkyrc.${VERSION}.${PROFILE}.LOG.a &
nice -n 20 conky -c ~/.conkyrc.${VERSION}.${PROFILE}.LOG.b &
nice -n 20 conky -c ~/.conkyrc.${VERSION}.${PROFILE}.LOG.c &

I use have several laptops so I need to distinguish which config files are used on which laptop, that is why I use PROFILE field – which is set to ThinkPad T420s in that example.

Here are the commands defined in these ~/.conkyrc.1.9.T420s.LOG.* files.

% grep exec ~/.conkyrc.1.9.T420s.LOG.*
.conkyrc.1.9.T420s.LOG.1:${color #eeeeee}${exec mount -p | awk '{print $1, $2, $3}' | column -t}
.conkyrc.1.9.T420s.LOG.2:${color #eeeeee}${exec tail -n 16 /var/log/automount.log}
.conkyrc.1.9.T420s.LOG.3:${color #eeeeee}${exec grep -v -E 'pulseaudio|message repeated|null_update_chw|route failed:|send_packet: |gen6_gt_|feeder_|cdce0: (Su|Re)' /var/log/messages | tail -16}
.conkyrc.1.9.T420s.LOG.4:${color #eeeeee}${exec vmstat -i}
.conkyrc.1.9.T420s.LOG.5:${color #eeeeee}${exec doas usbconfig}
.conkyrc.1.9.T420s.LOG.6:${color #eeeeee}${exec ps axwww -o %cpu,rss,command | head -1; ps axwww -o %cpu,rss,command | grep -v conky | grep -v '%CPU' | sort -n -r | head -15 }
.conkyrc.1.9.T420s.LOG.7:${color #eeeeee}${exec sockstat -l -4 | cut -c 1-50}
.conkyrc.1.9.T420s.LOG.8:${color #eeeeee}${exec top -m io -o total -b -s 1 -d 2 | grep -A 15 'PID USERNAME' | tail -n 16}
.conkyrc.1.9.T420s.LOG.9:${color #eeeeee}${exec gstat -p -I 345678}
.conkyrc.1.9.T420s.LOG.a:${color #eeeeee}${exec df -g | awk '{print $5,$6}' | column -t}
.conkyrc.1.9.T420s.LOG.b:${color #eeeeee}${exec pciconf -l}
.conkyrc.1.9.T420s.LOG.c:${color #eeeeee}${exec ifconfig -l -u | sed s/lo0//g | while read I; do ifconfig ${I}; done}

… and here is the diagram showing where these commands are placed.

I will use twelve (12) Conky configuration files for this purpose, each with one of the commands from above list.


 a df(1)       | b pciconf(8)             | c ifconfig(8)
---------------+--------------------------+---------------------
 7 sockstat(1) | 8 top(1)                 | 9 gstat(8)
---------------+--------------------------+---------------------
 4 vmstat(8)   | 5 usbconfig(8)           | 6 ps(1)
---------------+--------------------------+---------------------
 1 mount(8)    | 2 /var/log/automount.log | 3 /var/log/messages

Next are the full Conky configuration files.

~/.conkyrc.1.9.T420s.LOG.1

alignment                bottom_left
background               yes
gap_x                    3
gap_y                    3
minimum_size             279 193
maximum_width            280
double_buffer            yes
draw_outline             no
draw_shades              no
default_outline_color    444444
default_shade_color      444444
own_window               yes
own_window_class         conky
own_window_colour        222222
own_window_type          override
own_window_transparent   no
update_interval          2.1
use_xft                  yes
xftfont                  ubuntu mono-10
border_inner_margin      0
border_outer_margin      0
border_width             2

TEXT
${color #ee0000}% /sbin/mount -p
${color #eeeeee}${exec mount -p | awk '{print $1, $2, $3}' | column -t}

~/.conkyrc.1.9.T420s.LOG.2

alignment                bottom_left
background               yes
gap_x                    288
gap_y                    3
minimum_size             513 193
maximum_width            514
double_buffer            yes
draw_outline             no
draw_shades              no
default_outline_color    444444
default_shade_color      444444
own_window               yes
own_window_class         conky
own_window_colour        222222
own_window_type          override
own_window_transparent   no
update_interval          2.2
use_xft                  yes
xftfont                  ubuntu mono-10
border_inner_margin      0
border_outer_margin      0
border_width             2

TEXT
${color #ee0000}% /var/log/automount.log
${color #eeeeee}${exec tail -n 16 /var/log/automount.log}

~/.conkyrc.1.9.T420s.LOG.3

alignment                bottom_left
background               yes
gap_x                    807
gap_y                    3
minimum_size             789 193
maximum_width            790
double_buffer            yes
draw_outline             no
draw_shades              no
default_outline_color    444444
default_shade_color      444444
own_window               yes
own_window_class         conky
own_window_colour        222222
own_window_type          override
own_window_transparent   no
update_interval          2.3
use_xft                  yes
xftfont                  ubuntu mono-10
border_inner_margin      0
border_outer_margin      0
border_width             2

TEXT
${color #ee0000}% /var/log/messages
${color #eeeeee}${exec grep -v -E 'pulseaudio|message repeated|null_update_chw|route failed:|send_packet: |gen6_gt_|feeder_|cdce0: (Su|Re)' /var/log/messages | tail -16}

~/.conkyrc.1.9.T420s.LOG.4

alignment                bottom_left
background               yes
gap_x                    3
gap_y                    201
minimum_size             279 193
maximum_width            280
double_buffer            yes
draw_outline             no
draw_shades              no
default_outline_color    444444
default_shade_color      444444
own_window               yes
own_window_class         conky
own_window_colour        222222
own_window_type          override
own_window_transparent   no
update_interval          2.4
use_xft                  yes
xftfont                  ubuntu mono-10
border_inner_margin      0
border_outer_margin      0
border_width             2

TEXT
${color #ee0000}% /usr/bin/vmstat -i
${color #eeeeee}${exec vmstat -i}

~/.conkyrc.1.9.T420s.LOG.5

alignment                bottom_left
background               yes
gap_x                    288
gap_y                    201
minimum_size             513 193
maximum_width            514
double_buffer            yes
draw_outline             no
draw_shades              no
default_outline_color    444444
default_shade_color      444444
own_window               yes
own_window_class         conky
own_window_colour        222222
own_window_type          override
own_window_transparent   no
update_interval          2.5
use_xft                  yes
xftfont                  ubuntu mono-10
border_inner_margin      0
border_outer_margin      0
border_width             2

TEXT
${color #ee0000}% /usr/sbin/usbconfig
${color #eeeeee}${exec doas usbconfig}

~/.conkyrc.1.9.T420s.LOG.6

alignment                bottom_left
background               yes
gap_x                    807
gap_y                    201
minimum_size             789 193
maximum_width            790
double_buffer            yes
draw_outline             no
draw_shades              no
default_outline_color    444444
default_shade_color      444444
own_window               yes
own_window_class         conky
own_window_colour        222222
own_window_type          override
own_window_transparent   no
update_interval          2.6
use_xft                  yes
xftfont                  ubuntu mono-10
border_inner_margin      0
border_outer_margin      0
border_width             2

TEXT
${color #ee0000}% /bin/ps axwww -o %cpu,rss,command
${color #eeeeee}${exec ps axwww -o %cpu,rss,command | head -1; ps axwww -o %cpu,rss,command | grep -v conky | grep -v '%CPU' | sort -n -r | head -15 }

~/.conkyrc.1.9.T420s.LOG.7

alignment                bottom_left
background               yes
gap_x                    3
gap_y                    399
minimum_size             279 193
maximum_width            280
double_buffer            yes
draw_outline             no
draw_shades              no
default_outline_color    444444
default_shade_color      444444
own_window               yes
own_window_class         conky
own_window_colour        222222
own_window_type          override
own_window_transparent   no
update_interval          2.7
use_xft                  yes
xftfont                  ubuntu mono-10
border_inner_margin      0
border_outer_margin      0
border_width             2

TEXT
${color #ee0000}% /usr/bin/sockstat -l -4
${color #eeeeee}${exec sockstat -l -4 | cut -c 1-50}

~/.conkyrc.1.9.T420s.LOG.8

alignment                bottom_left
background               yes
gap_x                    288
gap_y                    399
minimum_size             513 193
maximum_width            514
double_buffer            yes
draw_outline             no
draw_shades              no
default_outline_color    444444
default_shade_color      444444
own_window               yes
own_window_class         conky
own_window_colour        222222
own_window_type          override
own_window_transparent   no
update_interval          2.8
use_xft                  yes
xftfont                  ubuntu mono-10
border_inner_margin      0
border_outer_margin      0
border_width             2

TEXT
${color #ee0000}% /usr/bin/top -m io -o total
${color #eeeeee}${exec top -m io -o total -b -s 1 -d 2 | grep -A 15 'PID USERNAME' | tail -n 16}

~/.conkyrc.1.9.T420s.LOG.9

alignment                bottom_left
background               yes
gap_x                    807
gap_y                    399
minimum_size             789 193
maximum_width            790
double_buffer            yes
draw_outline             no
draw_shades              no
default_outline_color    444444
default_shade_color      444444
own_window               yes
own_window_class         conky
own_window_colour        222222
own_window_type          override
own_window_transparent   no
update_interval          2.9
use_xft                  yes
xftfont                  ubuntu mono-10
border_inner_margin      0
border_outer_margin      0
border_width             2

TEXT
${color #ee0000}% /usr/sbin/gstat -p -I 300000
${color #eeeeee}${exec gstat -p -I 345678}

~/.conkyrc.1.9.T420s.LOG.a

alignment                bottom_left
background               yes
gap_x                    3
gap_y                    597
minimum_size             279 272
maximum_width            280
double_buffer            yes
draw_outline             no
draw_shades              no
default_outline_color    444444
default_shade_color      444444
own_window               yes
own_window_class         conky
own_window_colour        222222
own_window_type          override
own_window_transparent   no
update_interval          2.7
use_xft                  yes
xftfont                  ubuntu mono-10
border_inner_margin      0
border_outer_margin      0
border_width             2

TEXT
${color #ee0000}% /bin/df -g
${color #eeeeee}${exec df -g | awk '{print $5,$6}' | column -t}

~/.conkyrc.1.9.T420s.LOG.b

alignment                bottom_left
background               yes
gap_x                    288
gap_y                    597
minimum_size             513 272
maximum_width            514
double_buffer            yes
draw_outline             no
draw_shades              no
default_outline_color    444444
default_shade_color      444444
own_window               yes
own_window_class         conky
own_window_colour        222222
own_window_type          override
own_window_transparent   no
update_interval          2.8
use_xft                  yes
xftfont                  ubuntu mono-10
border_inner_margin      0
border_outer_margin      0
border_width             2

TEXT
${color #ee0000}% /usr/sbin/pciconf -l
${color #eeeeee}${exec pciconf -l}

~/.conkyrc.1.9.T420s.LOG.c

alignment                bottom_left
background               yes
gap_x                    807
gap_y                    597
minimum_size             789 272
maximum_width            790
double_buffer            yes
draw_outline             no
draw_shades              no
default_outline_color    444444
default_shade_color      444444
own_window               yes
own_window_class         conky
own_window_colour        222222
own_window_type          override
own_window_transparent   no
update_interval          2.9
use_xft                  yes
xftfont                  ubuntu mono-10
border_inner_margin      0
border_outer_margin      0
border_width             2

TEXT
${color #ee0000}% /sbin/ifconfig wlan0/em0/tun0
${color #eeeeee}${exec ifconfig -l -u | sed s/lo0//g | while read I; do ifconfig ${I}; done}

Thats a quite a lot configuration files but I think that this configuration done once will serve many many times in the future πŸ™‚

These Conky configuration files are suited for the 1600×900 resolution, you will have to modify values of the gap_x/gap_y/minimum_size/maximum_width parameters to make it fit into other resolution.

Initially I wanted to write a script/generator for that, but lets face it – I will not be able to properly cover each possible resolution πŸ™‚

UPDATE 2 – Latest Conky 1.11 Also Works

When I wrote this article I wrote that older Conky 1.9 version is needed (The conky-1.9.0_6 exactly which could be retrieved using portdowngrade sysutils/conky r419144 command).

Conky 1.10 introduced many bugs along with entirely new configuration format.

Latest Conky 1.11 (its conky-1.11.4_1 package on my box to be exact) works like a charm with Conky 1.9 configuration. It still has bug of NOT passing the mouse clicks to the desktop so of you want to make a left/middle/right click on the desktop aim on the place other then the Conky Dashboard space.

You can of course still follow the original article and fetch/build Conky with 1.9 version and have working left/middle/right mouse clicks on the desktop.

EOF

Ghost in the Shell – Part 3

Time to bring some life into the Ghost in the Shell series with Part 3 article.

You may want to check other articles in the Ghost in the Shell series on the Ghost in the Shell – Global Page where you will find links to all episodes of the series along with table of contents for each episode’s contents.

Query Functions

I haven’t found better name for that solution. There are generally two types of UNIX people. These that prefer to navigate and operate with basic ls/cd/mv/mkdir/rm commands and those who use some file manager like Midnight Commander (mc) or ranger or vifm or … you get the idea. I have tried various CLI file managers but always came back to navigate without them. If you are one of those people then these Query Functions are for you πŸ™‚

The so called Query Functions are for filter the information you look for. For example if you have directory with large number of files, then you would probably do something like that.

% ls | grep QUERY

… or if you also want to include subdirectories then something like that.

% find . | grep QUERY

For both of these examples you would also probably want to sometimes search case sensitive or insensitive depending on the need.

That leads us to four Query Functions:

  • q is an equivalent of ls | grep -i QUERY command.
  • Q is an equivalent of ls | grep QUERY command.
  • qq is an equivalent of find . | grep -i QUERY command.
  • QQ is an equivalent of find . | grep QUERY command.

Thus if I need to query the contents of directory while searching for something is very fast with q SOMETHING.

These are definitions of these Query Functions:

# SHORT QUERY FUNCTIONS q()
  q() {
    if [ ${#} -eq 1 ]
    then
      ls | grep --color -i ${1} 2> /dev/null
    else
      echo "usage: q string"
    fi
  }
     
# SHORT QUERY FUNCTIONS Q()
  Q() {
    if [ ${#} -eq 1 ]
    then
      ls | grep --color ${1} 2> /dev/null
    else
      echo "usage: Q string"
    fi
  }

# SHORT QUERY FUNCTIONS qq()
  qq() {
    if [ ${#} -eq 1 ]
    then
      find . \
        | grep -i ${1} 2> /dev/null \
        | cut -c 3-999 \
        | grep --color -i ${1} 2> /dev/null
    else
      echo "usage: qq string"
    fi
  }

# SHORT QUERY FUNCTIONS QQ()
  QQ() {
    if [ ${#} -eq 1 ]
    then
      find . \
        | grep ${1} 2> /dev/null \
        | cut -c 3-999 \
        | grep ${1} 2> /dev/null
    else
      echo "usage: QQ string"
    fi
  }

The qq and QQ functions uses grep(1) two times to make sure the output is colored.

I assume that You use colored grep(1) described in Ghost in the Shell – Part 2 article.

If you prefer to use alias(1) instead then they would look like that.

# SHORT QUERY FUNCTIONS q() Q() qq() QQ()
  alias q="ls | grep --color -i"
  alias Q="ls | grep --color"
  alias qq="find . | grep -i"
  alias QQ="find . | grep"

The qq and QQ will be little more limited as with functions its possible to trim the output to the exact needs with cut(1).

q.png

qq.png

Lots of people use recursive history search which also helps, but what if you used/typed needed command long ago with the arguments you need now? You would probably search the command with history(1) command and then using grep(1) to limit the results to what you look for. I keep enormous large list of commands to keep in history – with my current setting of 655360 the ~/.zhistory (ZSH) file takes about 2.7 MB size. I also wanted to be sure that two identical commands would not be kept in history hence the setopt hist_ignore_all_dups ZSH option enabled. When I wc -l my ~/.zhistory file it currently has 75695 lines of commands.

% grep HISTSIZE /usr/local/etc/zshrc
export HISTSIZE=655360
export SAVEHIST=${HISTSIZE}

% grep dups /usr/local/etc/zshrc
setopt hist_ignore_all_dups

Now back to Query Functions for history:

  • h is an equivalent of cat ~/.zhistory | grep -i QUERY command.
  • H is an equivalent of cat ~/.zhistory | grep QUERY command.

They fit in aliases this time. In alias(1) we will use just grep(1) to not ‘do’ Useless Use of Cat.

Here are the Query Functions for history.

# SHORT HISTORY ALIASES h() H()
  alias h='< ~/.zhistory grep -i'
  alias H='< ~/.zhistory grep'

h

… but what if we would like to filter the outputs of q family and h family Query Functions? The obvious response is using grep(1) like q QUERY | grep ANOTHER or h QUERY | grep ANOTHER for example. To make that faster we will make g and G shortcuts.

  • g is an equivalent of grep -i command.
  • G is an equivalent of just grep command.

Here they are.

# SHORT GREP FUNCTIONS g() G()
  alias g='grep -i'
  alias G='grep'

Now it will be just q QUERY | g ANOTHER and h QUERY | G ANOTHER for example.

To clear terminal output you may use clear(1) command, some prefer [CTRL]-[L] shortcut but I find ‘c‘ alias to be the fastest solution.

# SHORT GREP FUNCTIONS c()
  alias c='clear'

To make the solution complete I would also add exa(1) here with an alias of ‘e‘.

# SHORT LISTING WITH e()
  alias e='exa --time-style=long-iso --group-directories-first'

Why exa(1) will you ask while there is BSD ls(1) and GNU ls(1) (installed as gls(1) on FreeBSD to not confuse). To add GNU ls(1) to FreeBSD system use the coreutils package.

Well, the BSD ls(1) has two major cons:

  • It is not able to sort directories first.
  • It selects width for ALL columns based on single longest file name.

BSD-ls.png

The BSD ls(1) was used as following alias:

alias ls='ls -p -G -D "%Y.%m.%d %H:%M"'

The GNU ls(1) does not have these two problems but it does color the output only on the very limited pattern like:

  • Not executable file.
  • Executable file.
  • Directory.
  • Link.
  • Device.

GNU-ls.png

The GNU ls(1) was used as following alias:

gls -p -G --color --time-style=long-iso --group-directories-first --quoting-style=literal

Here is where exa(1) comes handy as it does not have any cons like FreeBSD’s ls(1) and it colors a lot more types of files.

e.png

exa --time-style=long-iso --group-directories-first

Its still very simple coloring based on file extension and not magic number as plain (empty) text file SOME-NOT-FILE.pdf is colored like PDF document.

e-pdf.png

But even this ‘limited’ coloring helps in 99% of the cases and while with BSD ls(1) and GNU ls(1) all of these files ‘seem’ like plain text files with exa(1) its obvious from the start which are plain files, which are images and which are ‘documents’ like PDF files for example.

Where Is My Space

On all UNIX and Linux systems there exists du(1) command. Combined with sort(1) it is universal way of searching for space eaters. Example for the / root directory with -g flag to display units in gigabytes.

# cd /
# du -sg * | sort -n
1       bin
1       boot
1       compat
1       COPYRIGHT
1       data
1       dev
1       entropy
1       etc
1       lib
1       libexec
1       media
1       mnt
1       net
1       proc
1       rescue
1       root
1       sbin
1       sys
1       tmp
1       var
2       jail
8       usr
305     home

Contents of UNIX System Resources directory with -m flag to display unit in megabytes.

# cd /usr
# du -sm * | sort -n
1       libdata
1       obj
1       tests
3       libexec
11      sbin
13      include
45      lib32
56      lib
58      share
105     bin
1080    ports
1343    src
5274    local

But its PITA to type cd and du all the time, not to mention that some oldschool UNIX systems does not provide -g or -m flags so on HP-UX you are limited to kilobytes at most.

You may also try -h (human readable) with sort -h (sort human readable) du(1) variant.

# du -smh * | sort -h
512B    data
512B    net
512B    proc
512B    sys
4.5K    COPYRIGHT
4.5K    entropy
5.5K    dev
6.5K    mnt
 53K    media
143K    tmp
205K    libexec
924K    bin
2.2M    etc
3.9M    root
4.6M    sbin
6.2M    rescue
6.6M    lib
 90M    boot
117M    compat
564M    jail
667M    var
5.4G    usr
297G    home

This is where ncdu(1) comes handy. Its ncurses based disk usage analyzer which helps finding that space eaters in very fast time without typing the same commands over and over again. Here is ncdu(1) in action.

First it calculates the sizes of the files.

ncdu.png

After a while you get the output sorted by size.

ncdu-usr.png

If you hit [ENTER] on the directory you will be instantly moved into that directory.

ncdu-usr-local.png

If you delete something with ‘d‘ then remember to recalculate the output with ‘r‘ letter.

It also has great options such as spawning shell ‘b‘ in the current directory or toggle between apparent size and disk usage with ‘a‘ option. The latter is very useful when you use filesystem with builtin compression like ZFS.

       up, k  Move cursor up
     down, j  Move cursor down
 right/enter  Open selected directory
  left, <, h  Open parent directory
           n  Sort by name (ascending/descending)
           s  Sort by size (ascending/descending)
           C  Sort by items (ascending/descending)
           d  Delete selected file or directory
           t  Toggle dirs before files when sorting
           g  Show percentage and/or graph
           a  Toggle between apparent size and disk usage
           c  Toggle display of child item counts
           e  Show/hide hidden or excluded files
           i  Show information about selected item
           r  Recalculate the current directory
           b  Spawn shell in current directory
           q  Quit ncdu

The apparent size using the du(1) command.

Disk usage.

% du -sm books
39145   books

Apparent size.

% du -smA books
44438   books

So I have 1.13 compression ratio on the ZFS filesystem. More then 5 GB saved just in that directory πŸ™‚

Where Are My Files

Once I got some space back I also wanted to know if there are some directories with enormous amount of very small files.

First I came up with my own files-count.sh script solution which is not that long.

#! /bin/sh

export LC_ALL=C

if [ ${#} -eq 0 ]
then
  DIR=.
else
  DIR="${1}"
fi

find "${DIR}" -type d -maxdepth 1 -mindepth 1 \
  | cut -c 3- \
  | while read I
    do
      find "${I}" | wc -l | tr -d '\n'
      echo " ${I}"
    done | sort -n

It works reliably but same as with du | sort tandem you have to retype it (or at least use cd(1) and hit [UP] arrow again) … but then I discovered that ncdu(1) also counts files! It does not provide ‘startup’ argument to start in this count files mode but when you hit ‘c‘ letter it will instantly display count of files in each scanned directory. To sort this output by the count of files hit the ‘C‘ letter (large ‘C‘ letter).

ncdu-files.png

The files-count.sh script still has one advantage over ncdu(1) – the latter stops counting files at 100k which is shown on the screenshot so if You need to search for really big amount of files or just about 100k then files-count.sh script will be more accurate/adequate.

% cd /usr
% files-count.sh 
       1 obj
      36 libdata
     299 sbin
     312 libexec
     390 tests
     498 bin
     723 lib32
     855 lib
    2127 include
   16936 share
  159945 src
  211854 ports
  266021 local

… but what if there were some very big files hidden somewhere deep in the directories tree? The du(1) or ncdu(1) will not help here. As usual I though about short files-big.sh script that will do the job.

#! /bin/sh

export LC_ALL=C

if [ ${#} -eq 0 ]
then
  DIR=.
else
  DIR="${1}"
fi

find "${DIR}" -type f -exec stat -f "%16z; doas rm -f \"%N\"" {} ';' | sort -n

An example usage on the /var directory.

# cd /var
# files-big.sh | tail
        10547304; doas rm -f "./tmp/kdecache-vermaden/icon-cache.kcache"
        29089823; doas rm -f "./db/clamav/clamav-2671b72fce703c2133c61e5bf85aad19.tmp/clamav-373e311ca7f610a39c7cf5c5c5a4fd83.tmp/daily.hdb"
        30138884; doas rm -f "./tmp/pkg-provides-wyK2"
        48271360; doas rm -f "./db/pkg/repo-HardenedBSD.sqlite"
        54816768; doas rm -f "./db/pkg/repo-FreeBSD.sqlite"
        66433024; doas rm -f "./db/pkg/local.sqlite"
        82313216; doas rm -f "./db/clamav/clamav-2671b72fce703c2133c61e5bf85aad19.tmp/clamav-373e311ca7f610a39c7cf5c5c5a4fd83.tmp/daily.hsb"
       117892267; doas rm -f "./db/clamav/main.cvd"
       132431872; doas rm -f "./db/clamav/daily.cld"
       614839082; doas rm -f "./db/pkg/provides/provides.db"

The output is in ‘executable’ format so if you select whole line and paste it into terminal, then this file will be deleted. By default it uses doas(1) but nothing can stop you from putting sudo(8) there. Not sure if you will find it useful but it helped me at least dozen times.

How Many Copies Do You Keep

I often find myself keeping the same files in several places which also wastes space (unless you use ZFS deduplication of course).

The dedup.sh script I once made is little larger so I will not paste it here and just put a link to it.

It has the following options available. You may search/compare files by name or size (fast) or by its MD5 checksum (slow).

% dedup.sh
usage: dedup.sh OPTION DIRECTORY
  OPTIONS: -n   check by name (fast)
           -s   check by size (medium)
           -m   check by md5  (slow)
           -N   same as '-n' but with delete instructions printed
           -S   same as '-s' but with delete instructions printed
           -M   same as '-m' but with delete instructions printed
  EXAMPLE: dedup.sh -s /mnt

Simple usage example.

% cd misc/man
% cp zfs-notes zfs-todo
% dedup.sh -M .
count: 2 | md5: 4ff4be66ab7e5484de2bf7c168ff995a
  doas rm -rf "./zfs-notes"
  doas rm -rf "./zfs-todo"

count: 2 | md5: 6d87f5b1317ea189165fcdc71380735c
  doas rm -rf "./x11"
  doas rm -rf "./xinit"

By copying the zfs-notes file into the zfs-todo file I wanted to show you what dedup.sh will print on the screen, but accidentally I also found another duplicate πŸ™‚

The output of dedup.sh is simple and like with files-big.sh script selecting the while line and pasting it into the terminal will remove the duplicate. By default it uses doas(1) but you can change it into sudo(8) if that works better for you.

Unusual cron(1) Intervals

Most of us already remember what the five fields of crontab(5) file mean, but what if you would like to run command every second … or after reboot only? The answer lies in the man 5 crontab page. Here are these exotic options.

string          meaning
------          -------
@reboot         Run once, at startup of cron.
@yearly         Run once a year, "0 0 1 1 *".
@annually       (same as @yearly)
@monthly        Run once a month, "0 0 1 * *".
@weekly         Run once a week, "0 0 * * 0".
@daily          Run once a day, "0 0 * * *".
@midnight       (same as @daily)
@hourly         Run once an hour, "0 * * * *".
@every_minute   Run once a minute, "*/1 * * * *".
@every_second   Run once a second.

Check cron(1) Environment

Many times I found myself lost lots of time debugging what went wrong when my script was run by the crontab(5) file. Often it was some variable missing or some command or script I used was not in the PATH variable.

To make that debugging faster You can use ENV.sh script to just store the cron(1) environment.

% cat ENV.sh
env > /tmp/ENV.out

The ENV.sh script will write current environment in the /tmp/ENV.out file.

Lets put it into the crontab(5) for a test.

% crontab -l | grep ENV
@every_second ~/ENV.sh

Now after at most a second you can check for the contents of the /tmp/ENV.out file.

% cat /tmp/ENV.out
LOGNAME=vermaden
PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin
PWD=/home/vermaden
HOME=/home/vermaden
USER=vermaden
SHELL=/bin/sh

Now you can easily debug the scripts run by the crontab(5) … at least on the environment part πŸ™‚

Simple HTTP Server

I found myself many times in a situation that I would want to allow download of some files from my machine and SSH could not be used.

This is when python(1) comes handy. It has SimpleHTTPServer (or http.server in Python 3 version) so you can instantly start HTTP server in any directory!

Here are the commands for both Python versions.

  • Python 2.x – python -m SimpleHTTPServer PORT
  • Python 3.x – python -m http.server PORT

I even made a simple http.sh wrapper script to make it even more easy.

#! /bin/sh

if ${#} -ne 1 ]
then
  echo "usage: ${0##*/} PORT"
  exit 1
fi

python -m SimpleHTTPServer ${1}

Example usage.

% cd misc/man
% http.sh 8080
Serving HTTP on 0.0.0.0 port 8080 ...
127.0.0.1 - - [14/Sep/2018 23:06:50] "GET / HTTP/1.1" 200 -
127.0.0.1 - - [14/Sep/2018 23:06:50] code 404, message File not found
127.0.0.1 - - [14/Sep/2018 23:06:50] "GET /favicon.ico HTTP/1.1" 404 -
127.0.0.1 - - [14/Sep/2018 23:09:15] "GET /bhyve HTTP/1.1" 200 -

To stop it simply hit [CTRL]-[C] interrupt sequence.

Here is how it looks in the Epiphany browser.

http.png

Simple FTP Server

Similarly with FTP service, another Python goodie called pyftpdlib (Python FTP Server Library) provides that.

Mine ftp.py wrapper is little bigger as you can write quite comlicated setups with pyftpdlib but mine is simple, it starts in the current directory and adds read only anonymous user and read/write user named writer with WRITER password.

#! /usr/bin/env python

from sys                   import argv,exit
from pyftpdlib.authorizers import DummyAuthorizer
from pyftpdlib.handlers    import FTPHandler
from pyftpdlib.servers     import FTPServer

if len(argv) != 2:
  print "usage:", argv[0], "PORT"
  print
  exit(1)
  
authorizer = DummyAuthorizer()
authorizer.add_user("writer", "WRITER", ".", perm="elradfmw")
authorizer.add_anonymous(".")
handler = FTPHandler
handler.authorizer = authorizer
handler.passive_ports = range(60000, 60001)
address = ("0.0.0.0", argv[1])
ftpd = FTPServer(address, handler)
ftpd.serve_forever()

The ftp.py is handy if you want to enable someone to upload something for you (or you are doing it o the other machine) when SSH/SCP is not possible for some reason.

To stop it simply hit [CTRL]-[C] interrupt sequence.

Here is its terminal startup and logs.

% cd misc/man
% ftp.py 2121
[I 2018-09-14 23:21:53] }}} starting FTP server on 0.0.0.0:2121, pid=64399 {{{
[I 2018-09-14 23:21:53] concurrency model: async
[I 2018-09-14 23:21:53] masquerade (NAT) address: None
[I 2018-09-14 23:21:53] passive ports: 60000->60000

… and how Firefox renders its contents.

ftp.png

Hope you will find some of these useful, see you at Part 4 some day.

EOF

Ghost in the Shell – Part 2

The article in the Ghost in the Shell series was the first post on my blog, so while I was busy by writing various server related articles and recently the FreeBSD Desktop series its about time for the Part 2 of the Ghost in the Shell series.

You may want to check other articles in the Ghost in the Shell series on the Ghost in the Shell – Global Page where you will find links to all episodes of the series along with table of contents for each episode’s contents.

Lets start with something simple – yet powerful and time saving.

Alias with Arguments

One may of course write any function to do similar job, but keeping track and ‘maintaining’ all those functions becomes complicated and one has to organize itself. This partially applies to aliases, but they are smaller and easier to maintain then whole functions. In any modern shell an alias(1) can also have arguments, while You will not be able to parse them as appropriate as with functions, they do the job for their basic use.

Here is an example of such alias(1) with arguments.

% ls
gfx/ info/ misc/ scripts/ tmp/

% alias lsg='ls | grep'

% lsg gfx
gfx/

Color grep(1) Patterns

As we already ‘touched’ the grep(1) command topic, lets make it more usable by highlighting the found results in color. The ${GREP_COLOR} variable is used for that purpose and it expects a number for a color, here is the table with number-color format.

Color    Number
Black    30
Red      31
Green    32
Yellow   33
Blue     34
Magenta  35
Cyan     36
White    37

You may as well use ‘bold’ output by adding ‘1;‘ before the number, for example.

% echo ${GREP_COLOR}
1;31

You will also have to make an alias(1) to grep(1) with --color argument, like that:

% alias grep='grep --color'

Here is how it looks in practice.

% export GREP_COLOR=31
% alias grep='grep --color'
% dmesg | grep SMP
FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
FreeBSD/SMP: 1 package(s) x 2 core(s)
SMP: AP CPU #1 Launched!

Here is how it looks on the xterm(1) terminal.

ghost-terminal

Process Management

This one is very useful on any UNIX system, does not matter if its server or desktop.

These are commands and operands that will help us manage processes started by hand:

  • &
  • fg
  • bg
  • jobs
  • kill
  • disown
  • nohup
  • [CTRL]+[Z]
  • [CTRL]+[C]

As you probably already know to start command ‘in the background’ – which means do what I tell you but do not block the terminal – you have to add ‘&‘ (ampersand) at the end of such command. That command does not magically go away and as long as its running its visible by the jobs(1) command. You may use ‘-l‘ switch to also show the PID of background processes.

% galculator &
[1] 8449

% jobs
[1]  + running    galculator

% jobs -l
[1]  + 8449 running    galculator

Now, what of you forget to add ‘&‘ (ampersand) at the end of command but you wanted to put it into the background? Hit [CTRL]+[Z] shortcut (Control key with ‘small’ Z letter) and the process will be put into the suspended state. Now you have several options, you can out that process into the background with bg(1) command – by default it uses last suspended job – %1, you can also bring it back into the foreground blocking the terminal with fg(1) command. You can also list its state with jobs(1) and of course kill(1) it either with PID showed by jobs -l command or by specifying the process number – %1 in that case.

Here is an example.

% galculator
^Z
zsh: suspended  galculator

% jobs
[1]  + suspended  galculator

% bg
[1]  + continued  galculator

% jobs -l
[1]  + 72892 running    galculator

% kill %1
[1]  + terminated  galculator

%

While fg(1) and bg(1) allow you to put command in the background or foreground respectively when the process is in suspended state, one may ask how to ‘switch’ a process to suspended state while its already running in the background. Its done with kill -17 signal called SIGSTOP. You can also bring back such suspended process to running state with kill -19 signal called SIGCONT … or just again use fg(1) or bg(1) command. Other difference between fg(1)/bg(1) commands and more ‘direct’ kill -17/kill -19 commands are that kill(1) does not inform the user what has changed to the process. You may as well use kill -SIGCONT syntax or kill -s SIGCONT if that is more readable for you.

% galculator
^Z
zsh: suspended  galculator

% bg
[1]  + continued  galculator

% xcalc
^Z
zsh: suspended  xcalc

% jobs -l
[1]  - 19537 running    galculator
[2]  + 20563 suspended  xcalc

% kill -17 %1
[1]  + suspended (signal)  galculator

% jobs -l
[1]  + 19537 suspended (signal)  galculator
[2]  - 20563 suspended  xcalc

% kill -SIGCONT %1
% bg %2
[2]  - continued  xcalc

% jobs -l
[1]  + 19537 running    galculator
[2]  - 20563 running    xcalc

Also check man kill and man signal for more information.

What about disown(1) then? Its a ‘magic’ helper when you start some long running jobs directly at the terminal without Screen or Tmux and you need to disconnect that terminal, for example because you are taking your laptop with you. When you do this – depending on the settings of the current shell – the processes in the background may be killed or ‘moved’ to PID 1 (the init(1) of course) as the PPID (Parent PID). To achieve that we will used that disown(1) command. Once you ‘disown’ a process it will no longer be show by the jobs(1) command, but it will run ‘pinned’ to the init(1) process after you disconnect the terminal session.

% galculator
^Z
zsh: suspended  galculator

% bg
[1]  + continued  galculator

% jobs -l
[1]  + 98556 running    galculator

% disown %1

% jobs -l

% pgrep galculator
98556

% pstree -p 98556
─┬◆ 00001 root /sbin/init --
 └─┬─ 48708 vermaden xterm
   └─┬◆ 52463 vermaden -zsh (zsh)
     └──◆ 98556 vermaden galculator

Now its still pinned to the shell in the xterm(1) terminal. After we close the xterm(1) window (or kill that zsh(1) shell) it will switch to init(1) as PPID (Parent PID).

% pstree -p 98556
─┬◆ 00001 root /sbin/init --
 └──◆ 98556 vermaden galculator

% pgrep -P 1 galculator
98556

We are left with nohup(1) then, when and why to use it as we already has great disown(1) magic? Well, disown(1) is not always available, so when You need to put some command into the long background run and disconnect after it its the best possible option. By default the nohup(1) command will log the output of started command into the nohup.out file. Remember that nohup(1) will still run the process in the foreground, to put it into the background use ‘&‘ (ampersand) or [CTRL]+[Z] with bg(1) combo.

% nohup galculator
appending output to nohup.out
^Z
zsh: suspended  nohup galculator

% bg
[1]  + continued  nohup galculator

% jobs -l
[1]  + 22322 running    nohup galculator

% pstree -p 22322
─┬◆ 00001 root /sbin/init --
 └─┬─ 89568 vermaden xterm
   └─┬◆ 91486 vermaden -zsh (zsh)
     └──◆ 22322 vermaden galculator

… and after disconnect out process switched to init(1) as PPID.

% pstree -p 22322
─┬◆ 00001 root /sbin/init --
 └──◆ 22322 vermaden galculator

You may of course end a running process in the foreground with [CTRL]+[C] shortcut, but that is probably already known to you. I just mention it for the ‘completeness’ of the guide.

% galculator
^C

%

Which Which

While the which(1) command shows the full path of the executable found in the first directory of the ${PATH} variable, it also shows what alias is used for that command it there is one. One may ask how then to find information about absolute executable path if it shows and alias(1) instead. Well, you have to use unalias(1) on that command, so which(1) would be showing full path again.

% which caja
caja: aliased to caja --browser --no-desktop

% unalias caja

% which caja
/usr/local/bin/caja

Also be sure to check Smylers comment below about the difference between shell builtin which and /bin/which command.

Record Session

If you have used PuTTY or MobaXterm in your work, then you appreciate the possibility of saving the terminal output to a file, foe example for the documentation purposes. This is also available ‘natively’ in the shell by using the script(1) command. Remember that script(1) will record also ‘special’ characters like colors, so to properly ‘replay’ the session you may want to either use script(1) or cat(1) commands for that or use less with -R argument.

Here is example recorded script(1) session.

% script script.out
Script started, output file is script.out

% ls
gfx info misc scripts tmp unix.png

% uname -spr
FreeBSD 11.2-RELEASE amd64

% exit
Script done, output file is script.out

% cat script.out
Script started on Sun Jul  8 08:24:06 2018
You have mail.
% ls | grep gfx
gfx
% uname -spr
FreeBSD 11.2-RELEASE amd64
% exit
exit

Script done on Sun Jul  8 08:24:20 2018

% less -R script.out
Script started on Sun Jul  8 08:24:06 2018
You have mail.
% ls | grep gfx
gfx
% uname -spr
FreeBSD 11.2-RELEASE amd64
% exit
exit

Script done on Sun Jul  8 08:24:20 2018

% less script.out
Script started on Sun Jul  8 08:24:06 2018
You have mail.
% ls | grep gfx
ESC[1;31mgfxESC[00mESC[K
% uname -spr
FreeBSD 11.2-RELEASE amd64
% exit
exit

Script done on Sun Jul  8 08:24:20 2018


Edit Command Before Executing

Sometimes you have long multi-line command to execute, so often it is crafted in you favorite ${EDITOR} and then pasted into the terminal. To omit copying and pasting yo may want to check fc(1) command which serves similar purpose. After you type a command, for example simple ls(1) command, and then you type fc(1) command, then fc(1) will take that ls(1) command into your favorite text editor from ${EDITOR} variable, will allow you to edit it and if you save and exit the that editor, it will execute it.

Lets see how it behave by example.

% ls
gfx   books   download   scripts

% fc

Now you are taken into the ${EDITOR} which is vi(1) in my case.

      1 ls
~
~
~
/tmp/zsh999EQ6: unmodified: line 1

Lets made some changes.

      1 ls -l \
      2    -h
~
~
~
~

:wq

After you hit [ENTER] it will exit from ${EDITOR} and execute that command.

total 6181
drwxr-xr-x    87 vermaden  vermaden    87B 2017.12.18 15:30 books/
drwxr-xr-x    12 vermaden  vermaden    12B 2018.06.19 16:02 download/
drwxr-xr-x    19 vermaden  vermaden    20B 2018.05.24 11:52 gfx/
drwx------    12 vermaden  vermaden   310B 2018.07.07 03:23 scripts/

You may show that command by pressing [Up] key to check what has been executed.

% ls -l -h

Edit or Just View

When working in multi-admin environment – especially while debugging – one admin may block other admin’s work by using vi(1) – or just their favorite editor to ‘browse’ the file contents. Good practice in that case is using more(1) or less(1) instead of vi(1), but that frustrates some admins to type vi(1) again if they need to change something.

… and by the way, on FreeBSD more(1) is less(1) πŸ™‚

% uname -spr
FreeBSD 11.2-RELEASE amd64

% ls -i `which less` `which more`
492318 /usr/bin/less  492318 /usr/bin/more

A blocked ‘example’ is shown below when the second admin wanted to browse the /etc/rc.conf file while the first one already did that.

# vim /etc/rc.conf

E325: ATTENTION
Found a swap file by the name "/etc/.rc.conf.swp"
          owned by: root   dated: Sun Jul  8 08:38:35 2018
         file name: /etc/rc.conf
          modified: no
         user name: root   host name: t420s.local
        process ID: 54219 (still running)
While opening file "/etc/rc.conf"
             dated: Fri Jul  6 00:51:11 2018

(1) Another program may be editing the same file.  If this is the case,
    be careful not to end up with two different instances of the same
    file when making changes.  Quit, or continue with caution.
(2) An edit session for this file crashed.
    If this is the case, use ":recover" or "vim -r /etc/rc.conf"
    to recover the changes (see ":help recovery").
    If you did this already, delete the swap file "/etc/.rc.conf.swp"
    to avoid this message.

Swap file "/etc/.rc.conf.swp" already exists!
[O]pen Read-Only, (E)dit anyway, (R)ecover, (Q)uit, (A)bort:

This is where less(1) comes handy because of you open a file in it, you do not ‘block’ access to it and if you need to edit something just hi [V] key (small ‘v’ letter). It will open that file in your ${EDITOR} editor and you can make any changes now.

Reset

Last but not least, often when you paste ‘too much’ into the terminal it becomes ‘fragile’ or ‘broken’. To reset it into the ‘stable’ and ‘proper’ state just use the reset(1) command.

% reset

Hope You find it useful, see you at the Part 3 sometime πŸ˜‰

EOF

FreeBSD Desktop – Part 3 – X11 Window System

In this article I would like to cover setting up the X11 Window System on FreeBSD.

x11

You may want to check other articles in the FreeBSD Desktop series on the FreeBSD Desktop – Global Page where you will find links to all episodes of the series along with table of contents for each episode’s contents.

I always get the impression that the X11 configuration/setup seems to be the seen as hard or weird. I will try to gather all useful information about that topic regarding the FreeBSD operating system. A lot of that is covered in the FreeBSD Handbook – 5.4. Xorg Configuration – section.

BIOS or UEFI

First things first, if You find a device that is not supported by any ‘accelerated’ driver like ‘intel‘ or ‘nvidia‘ You would use ‘vesa‘ driver (Video Electronics Standards Association) while booting in BIOS mode and You will use ‘scfb‘ driver (System Console Frame Buffer) while booting on UEFI mode. This can be checked by machdep.bootmethod sysctl(8) parameter.

% sysctl machdep.bootmethod
machdep.bootmethod: BIOS

This way You will know if you will use ‘scfb‘ driver or ‘vesa‘ driver when troubleshooting or when just running the ARM system on which ‘scfb‘ will be used.

Packages

We will install the X11 server, the XDM login manager, the XTERM terminal emulator and the Openbox window manager. We will also install the Intel graphics card driver. To achieve that type the command below.

# pkg install xorg xdm xterm openbox xf86-video-intel

Of course You will need to have network connection configured but we covered that in earlier parts.

Group

To have hardware accelerated X11 You will also need to add your regular user to the ‘video‘ group.

# pw groupmod video -m username

Login Class

Remember to also add UTF-8 as the default encoding (charset and lang parameters) in the /etc/login.conf file as show below. Also, after adding them, run the cap_mkdb /etc/login.conf command as root and relogin with your regular user.

Below are shown the lines that need to be added to the default: profile in the /etc/login.conf file.

(...)

 default:\
         :passwd_format=sha512:\
         :copyright=/etc/COPYRIGHT:\
         :welcome=/etc/motd:\
         :setenv=MAIL=/var/mail/$,BLOCKSIZE=K:\
         :path=/sbin /bin /usr/sbin /usr/bin /usr/local/sbin /usr/local/bin ~/bin:\
         :nologin=/var/run/nologin:\
         :cputime=unlimited:\
         :datasize=unlimited:\
         :stacksize=unlimited:\
         :memorylocked=64K:\
         :memoryuse=unlimited:\
         :filesize=unlimited:\
         :coredumpsize=unlimited:\
         :openfiles=unlimited:\
         :maxproc=unlimited:\
         :sbsize=unlimited:\
         :vmemoryuse=unlimited:\
         :swapuse=unlimited:\
         :pseudoterminals=unlimited:\
         :kqueues=unlimited:\
         :umtxp=unlimited:\
         :priority=0:\
         :ignoretime@:\
+        :charset=UTF-8:\
+        :lang=en_US.UTF-8:\
         :umask=022:
(...)

The diff(1) after/before modification looks like that.

# diff -u login.conf.ORG login.conf
--- login.conf.ORG      2018-05-22 07:10:09.864831000 +0200
+++ login.conf  	2017-07-07 04:47:04.643412000 +0200
@@ -46,6 +46,8 @@
        :umtxp=unlimited:\
        :priority=0:\
        :ignoretime@:\
+       :charset=UTF-8:\
+       :lang=en_US.UTF-8:\
        :umask=022:

The UTF-8 encoding – same as in the lang parameter in the /etc/login.conf file – must be added at the begining of your ~/.xinitrc file, like that:

% grep LC_ALL ~/.xinitrc
# SET PROPER locale(1) with LC_ALL VARIABLE.
  export LC_ALL=en_US.UTF-8

Kernel Module

In 2010 Intel introduced Nehalem architecture which have integrated GPU in the CPU (and memory controller which AMD had in CPU since 2003). This means that most computers/laptops/servers (Intel has about 80% market share) will have integrated GPU which should be supported by the X11 ‘intel‘ driver and:

  • /boot/kernel/i915.ko from base system (odler)
  • /boot/kernel/i915kms.ko from base system (newer)
  • /boot/modules/i915kms.ko from ‘drm-next-kmod‘ package

The ‘drm-next-kmod‘ package is available for FreeBSD 12-CURRENT development branch and upcoming FreeBSD 11.2-RELEASE (currently at BETA2 stage) about which You can read here on FreeBSD Mailing Lists on https://lists.freebsd.org/pipermail/freebsd-stable/2018-February/088406.html and https://svnweb.freebsd.org/ports?view=revision&revision=462202 accordingly.

You will probably be using the builtin KMS module which is the /boot/kernel/i915kms.ko file. You may configure loading of this module in either /boot/loader.conf (loaded at boot menu) or in the /etc/rc.conf main FreeBSD configuration file – which will load modules during the startup process. I prefer the second option as it is faster for FreeBSD system to boot while loading the kernel modules from the /etc/rc.conf file along with starting needed services then lading them one by one from /boot/loader.conf file. To keep system startup time to minimum we will have only one kernel module configured to load from the /boot/loader.conf file – the ZFS module with zfs_load=YES line.

If you found yourself with situation that KMS modules from the base system will not support your graphics card then You will have to try the ‘drm-next-kmod‘ package for latest support of Intel graphics chips, that sometimes requires running FreeBSD CURRENT version which is not a problem, you will get all the new stuff instantly as a bonus πŸ˜‰

To add the KMS (Kernel Mode Setting) Intel module loading into the /etc/rc.conf file put the line below into it.

# KERNEL MODULES
  kld_list="${kld_list} i915kms"

We can also add other useful kernel module loading such as fuse (for alien filsystems), aesni (for encryption acceleration if You CPU has these extensions) geom_eli (for the encryption itself) or coretemp (for the temperature sensor). I will cover all of these kernel module in the future series with exact explanation what are they for.

With all of them the /etc/rc.conf kld_list variables will look like that:

# grep kld_list /etc/rc.conf 
  kld_list="${kld_list} i915kms aesni geom_eli"
  kld_list="${kld_list} fuse coretemp cpuctl"

In case of need to use ‘drm-next-kmod‘ module instead of the base system one, your /etc/rc.conf file would look like that one below.

# grep kld_list /etc/rc.conf
  kld_list="${kld_list} /boot/modules/i915kms.ko aesni geom_eli"
  kld_list="${kld_list} fuse coretemp cpuctl"

If you find yourself in the situation that the ‘drm-next-kmod‘ would still not support your latest graphics card then you may try the latest version of the ‘drm-next-kmod‘. Current ‘drm-next-kmod‘ is on par with Linux 4.11 when it comes to graphics support, the latest WIP (work in progress) tree is at 4.15, so that may enable some graphics cards, anyway, here it is – https://github.com/FreeBSDDesktop/freebsd-base-graphics/tree/drm-v4.15-WIP – use at your own risk. πŸ˜‰

X11 Window System Configuration

Historically You would create entire /etc/X11/xorg.conf file which would include complete X11 Window System configuration. Recently to comply with FreeBSD hier(7) directory structure and logic this can be also configured as /usr/local/etc/X11/xorg.conf file … and even more recently You can just configure these parts of X11 server that You need without touching other parts. This ‘individual’ configuration is done in the /usr/local/etc/X11/xorg.conf.d directory with individual files for each setting, like card.conf for graphics card configuration.

As the latter method is the most modern one we will use that in this guide for the sake of future installations.

# cat /usr/local/etc/X11/xorg.conf.d/flags.conf
Section "ServerFlags"
  Option "DontZap" "off"
EndSection

The DontZap option is to enable the X11 server abort with [CTRL] + [ALT] + [BACKSPACE] shortcut. I found it useful several times when none other option worked. Only X11 server required restart. I could of course login to this laptop via SSH and kill needed processes, but I do not carry two laptops with me to be prepared for such situations πŸ˜‰

Next we will configure the keyboard with Polish layout and two additional options, terminate:ctrl_alt_bksp to allow termination of the X11 server with [CTRL] + [ALT] + [BACKSPACE] shortcut and ctrl:nocaps to ignore the Caps Lock key as I also find that useful, if you not, then do not enable that option.

# cat /usr/local/etc/X11/xorg.conf.d/keyboard.conf
Section "InputDevice"
  Identifier "Keyboard0"
  Driver "kbd"
  Option "XkbLayout" "pl"
  Option "XkbOptions" "terminate:ctrl_alt_bksp,ctrl:nocaps"
EndSection

The last one is the graphics card configuration, we will put ‘intel‘ here as our graphics is fully supported by it.

# cat /usr/local/etc/X11/xorg.conf.d/card.conf
Section "Device"
  Identifier "Card0"
  Option "DPMS"
  Driver "intel"
EndSection

Remember to NOT enable the SNA acceleration method – Option "AccelMethod" "sna" – on ‘intel‘ driver cards as it prevents suspend/resume from working. We will also enable DPMS to save power when its possible.

XDM or XINIT/STARTX

You will also have to decide how You want to start your X11 Window Server, you may login in plan text console and then type xinit(1) or startx(1) to read your ~/.xinitrc configuration and daemons or You may want to use X11 Login manager such as xdm/sddm/slim with ~/.xsession configuration to load after successful login. The difference between xinit and startx is that startx command executes xinit command with arguments like -nolisten tcp to not listen for the external connections for example. Its pointless to start another layer of binary (startx) just to pass an argument to command, so we will omit startx and go directly to xinit command with suitable alias.

Which is better – some login manager or directly starting X11 with xinit? Both are good, it only depends on your preferences. For example I used text login and then typed ‘x‘ as an alias to xinit command and recently switched to xdm along with symlink of ~/.xsession which leads to the very same ~/.xinitrc file and also like it.

When I use xinit to directly start the X11 server I use an alias to xinit with these parameters:

alias x='xinit ~/.xinitrc -- -dpi 75 -nolisten tcp'

Of course I keep that alias in the ~/.zshrc file (for the ZSH shell) so everytime I login in the text console I just type ‘x‘ and X11 session starts reading contents of my ~/.xinitrc file.

While xinit run commands based on the ~/.xinitrc file the XDM login manager looks for the ~/.xsession file. As You will be loading same stuff regardless of the startup method we will create a link of ~/.xsession pointing to the ~/.xinitrc file. This way either method You choose You will always end with started X11 session.

% ln -s ~/.xinitrc ~/.xsession

% ls -l ~/.xsession
lrwxr-xr-x  1 vuk  vuk  8 2018.05.22 00:27 .xsession -> .xinitrc

One more case about the ~/.xinitrc (or ~/.xsession) file. It is interpreted as a shell script (and yes you can do if/then/else/fi and case/esac or for/while POSIX shell scripting in it) but it does not need to be executable. The last command in this file MUST NOT to be put in the background (must be without the & char at the end) because the X11 session will end.

XDM Black Theme

I prepared two nice looking themes for XDM login manager, the first one uses black background and grey/white fonts.

To configure this black XDM theme You will need to modify contents of two files as show below.

# cat /usr/local/etc/X11/xdm/Xsetup_0
xsetroot -solid black

This we will set black background on the login screen on the :0 X11 server – the default one.

# cat /usr/local/etc/X11/xdm/Xresources
xlogin.Login.greeting:
xlogin.Login.unsecureGreeting:
xlogin.Login.fail:                Fail.
xlogin.Login.changePasswdMessage: Change.
xlogin.Login.namePrompt:          Username:
xlogin.Login.passwdPrompt:        Password:
xlogin.Login.echoPasswd:          true
xlogin.Login.background:          black
xlogin.Login.foreground:          white
xlogin.Login.failColor:           #cccccc
xlogin.Login.inpColor:            black
xlogin.Login.promptColor:         #aaaaaa
xlogin.Login.face:                courier:size=13:style=Bold
xlogin.Login.failFace:            courier:size=13:style=Bold
xlogin.Login.promptFace:          courier:size=13:style=Normal
xlogin.Login.greetFace:           courier:size=13
xlogin.Login.width:               400

… and the rest of the XDM login manager configuration is kept in the Xresources file, at least the appearance part πŸ˜‰

Wonder what font will be chosen when we put ‘courier‘ in the *face options? Lets check with fc-match(1) command.

% fc-match courier  
c0419bt_.pfb: "Courier 10 Pitch" "Regular"

You have the answer, You do not always have to type whole font name in the configuration files.

Here is the screenshot of this black theme in action.

xdm.BLACK

XDM Grey Theme

If You did not liked the black XDM theme maybe the grey one will suit your taste better.

# cat /usr/local/etc/X11/xdm/Xsetup_0
xsetroot -solid gray50

We will use the gray50 predefined X11 server color for the background.

# cat /usr/local/etc/X11/xdm/Xresources
xlogin.Login.greeting:
xlogin.Login.unsecureGreeting:
xlogin.Login.fail:                Fail.
xlogin.Login.changePasswdMessage: Change.
xlogin.Login.namePrompt:          Username:
xlogin.Login.passwdPrompt:        Password:
xlogin.Login.echoPasswd:          true
xlogin.Login.background:          gray50
xlogin.Login.foreground:          white
xlogin.Login.failColor:           #dddddd
xlogin.Login.inpColor:            gray50
xlogin.Login.promptColor:         #cccccc
xlogin.Login.greetColor:          gray50
xlogin.Login.promptColor          gray50
xlogin.Login.hiColor              gray50
xlogin.Login.shdColor             gray50
xlogin.Login.face:                courier:size=13:style=Bold
xlogin.Login.failFace:            courier:size=13:style=Bold
xlogin.Login.promptFace:          courier:size=13:style=Bold
xlogin.Login.greetFace:           courier:size=13:style=Bold
xlogin.Login.frameWidth:          0
xlogin.Login.innerFramesWidth:    0
xlogin.Login.sepWidth:            0
xlogin.Login.width:               32768
xlogin.Login.height:              32768
xlogin.Login.x:                   0
xlogin.Login.y:                   0

… and knowing the XDM theming limitations we will customize to mimic the fullscreen of grey color. The only downside of that approach is that Username: and Password: fields will be placed in the top/left side of the screen.

Here is the screenshot of this grey theme in action.

xdm.GREY

Client Configuration

Last but not least, we will configure the simple ~/.xinitrc file with Openbox window manager, prestarted xterm terminal and black background for our simple session.

Here are the contents of the ~/.xinitrc file (to which ~/.xsession links of course).

% cat ~/.xinitrc
# SET PROPER locale(1) with LC_ALL VARIABLE.
  export LC_ALL=en_US.UTF-8

# PRESTARTED APPS
  xterm &
  xsetroot -solid black &

# WINDOW MANAGER
  openbox

It will not be very pretty – yet – but we will cover that later in the series.

desktop

That’s all for today’s X11 configuration, hope that was useful.

UPDATE 1 – The devd(8) Backend and AutoAddDevices Option

As user seschwar on Lobsters suggested:


“Xorg now has a devd(8) backend it can use to get informed about hotplugged devices instead of hald(8): https://lists.freebsd.org/pipermail/freebsd-x11/2017-March/018978.html That’s working fine for me, even without moused(8).”

Thank you for that information, this may be also verified by checking the X11 log /var/log/Xorg.0.log file.

# grep devd /var/log/Xorg.0.log
[ 96890.425] (II) The server relies on devd to provide the list of input devices.
        If no devices become available, reconfigure devd or disable AutoAddDevices.
[ 96890.955] (II) config/devd: probing input devices...
[ 96890.955] (II) config/devd: adding input device (null) (/dev/kbdmux)
[ 96890.956] (**) Option "config_info" "devd:kbdmux"
[ 96890.960] (II) config/devd: kbdmux is enabled, ignoring device ukbd0
[ 96890.960] (II) config/devd: kbdmux is enabled, ignoring device atkbd0
[ 96890.960] (II) config/devd: adding input device (null) (/dev/sysmouse)
[ 96890.961] (**) Option "config_info" "devd:sysmouse"
[ 96890.962] (II) config/devd: device /dev/ums0 already opened
[ 96890.962] (II) config/devd: device /dev/psm0 already opened

I will modify the original post to not confuse future readers.

UPDATE 2 – Using modesetting Driver

As user alx82 on Reddit suggested:


“If you use i915kms kernel module, you don’t need to install xf86-video-intel, the modesetting generic DDX driver that is built in the X server, will do just well. The xf86-video-intel might even cause you troubles, as you mentioned, for example my x230 was rebooting instead of suspending when I had installed the xf86-video-intel (I had installed it to solve the tearing problem with xfwm4 using the xf86-video-intel’s SNA acceleration). Now removed, my x230 suspends no problem.”

Thank you for that information.

I have tried below configuration with ‘modesetting’ driver and it works without a problem. I did not had any issues with ‘intel’ driver either but I will stick with ‘modesetting’ driver for a while see how it behaves on my machine.

# cat /usr/local/etc/X11/xorg.conf.d/card.conf
Section "Device"
  Identifier "Card0"
  Option "DPMS"
  Driver "modesetting"
EndSection
# cat /usr/local/etc/X11/xorg.conf.d/flags.conf
Section "ServerFlags"
  Option "DontZap" "off"
EndSection
# cat /usr/local/etc/X11/xorg.conf.d/keyboard.conf
Section "InputDevice"
  Identifier "Keyboard0"
  Driver "kbd"
  Option "XkbLayout" "pl"
  Option "XkbOptions" "terminate:ctrl_alt_bksp,ctrl:nocaps"
EndSection

To verify which driver You use You can check the X11 log /var/log/Xorg.0.log file.

Here is ‘intel‘ driver being used:

# grep -c intel /var/log/Xorg.0.log
98

# grep -c modeset /var/log/Xorg.0.log
8

# grep DRI /var/log/Xorg.0.log
[ 95305.483] (II) intel(0): [DRI2] Setup complete
[ 95305.483] (II) intel(0): [DRI2]   DRI driver: i965
[ 95305.483] (II) intel(0): [DRI2]   VDPAU driver: va_gl
[ 95305.484] (II) intel(0): DRI2: Enabled
[ 95305.484] (II) intel(0): DRI3: Disabled
[ 95305.549] (II) GLX: Initialized DRI2 GL provider for screen 0

Here is ‘modesetting‘ driver being used:

# grep -c intel /var/log/Xorg.0.log
0

# grep -c modeset /var/log/Xorg.0.log
80

# grep DRI /var/log/Xorg.0.log
[ 95556.446] (II) glamor: EGL version 1.4 (DRI2):
[ 95556.671] (II) modeset(0): [DRI2] Setup complete
[ 95556.671] (II) modeset(0): [DRI2]   DRI driver: i965
[ 95556.671] (II) modeset(0): [DRI2]   VDPAU driver: i965
[ 95556.714] (II) GLX: Initialized DRI2 GL provider for screen 0

UPDATE 3 – Using Dedicated video: Login Class Instead of Global One

As Lorenzo suggested in the comments below.


“If you modify /etc/login.conf, you are modifying the systemwide locale for the default login class to which all users belong (also user www for Apache, for example).”

We can of course created dedicated login class for the X11 Window System, as we use group video already we will create video: login class for the X11 Window System.

I assume that the default: login class is untouched – if You already modified it with lang and charset options, remove them.

Add this login class to the /etc/login.conf file.

video:\
        :charset=UTF-8:\
        :lang=en_US.UTF-8:\
        :tc=default:

Rebuild the login class database.

# cap_mkdb /etc/login.conf

How the account looks before setting the login class.

# grep vuk /etc/master.passwd
vuk:{REMOVED}:1000:1000::0:0:vuk:/home/vuk:/bin/sh

Lets set the login class to video for the vuk user.

# pw usermod -L video -n vuk

How the account looks after setting the login class.

# grep vuk /etc/master.passwd
vuk:{REMOVED}:1000:1000:video:0:0:vuk:/home/vuk:/bin/sh

Now logout and login again to make that work.

Hope that helps.

EOF

Ghost in the Shell – Part 1

I wanted to post this earlier, but the busy daily life does not help πŸ˜‰

This will be first article in the series about efficient working in the shell environment. There are actually a lot articles and blog posts about efficient working in the terminal, but a lot of them are biased towards very specific uses, like hints only for Bash shell or only for specific terminal emulator. For example Moving efficiently in the CLI.

These series are about universal knowledge that would work on most shells and environments. Lets start with hint that I use many times a day that saves a lot time for not having to type …

You may want to check other articles in the Ghost in the Shell series on the Ghost in the Shell – Global Page where you will find links to all episodes of the series along with table of contents for each episode’s contents.

Recall Last Argument of Previous Command

Imagine most simple scenario, creating directory and entering it. Typically its like that:

% mkdir clear-place-for-new-work
% cd clear-place-for-new-work
%

The longer the name, the bigger the chance that You would type mkdir, then hit the [UP] arrow, then [HOME] or [CTRL]+[A] keys and then put cd in the place of mkdir.

With the use of !$ You can recall last argument of the precious command, so it will now look like that.

% mkdir clear-place-for-new-work
% cd !$
cd clear-place-for-new-work
%

Faster isn’t it?

Swap First Occurrence of a Word

The upper example can be used for the next advice as well. By typing ^fromwhat^towhat in the terminal You will swap the first occurrence of word fromwhat word to towhat word in the previous command, lets see how its working.

% mkdir clear-place-for-new-work
% ^mkdir^cd
cd clear-place-for-new-work
%

It still takes more time to write then using the !$ so its useful mostly when there are short things to swap, like numbers, for example ^3^4 to ‘move’ from one target to another. … or also if You can not recall to the last argument of previous command.

There and Back Again

A lot people does not know, that You can go back to previous working directory with dash. Lets assume that You need to get to /tmp directory for one command and get back to where You were to continue the work. Here is an example.

% pwd
/usr/local/etc/bareos/bareos-dir.d/jobdefs
% cd /tmp
% pwd
/tmp% (do needed work in /tmp dir)
% cd -
/usr/local/etc/bareos/bareos-dir.d/jobdefs
% pwd
/usr/local/etc/bareos/bareos-dir.d/jobdefs

You can even create entire directory stack with pushd/popd commands if needed, check Wikipedia article on that for more information. You can also use ${OLDPWD} variable. Useful with umount command for example.

% pwd
/media/backup-pendrive-key
% cd ~
% umount $OLDPWD
% pwd
/home/vermaden

Repeat Command from History

With exclamation mark (!) You can re-invoke the command from history with all its arguments (which sometimes can be risky). For example.

% !pkg
pkg update -f
(runs actual command)
%

Its better to first check what arguments have been used in that command, that is where :p comes handy. Here is its example usage.

% !pkg:p
pkg update -f
(just prints command without running it)
% !pkg
pkg update -f
(runs actual command)
%

Now, as arguments are known its safe to re-invoke the command with arguments. When this can be dangerous? Can ls command can be dangerous, that depends what You have on Your history, check the example below.

% ls | while read I; do rm -f ${I}; done

This command first lists the contents of the current working directory with ls command, then the output is piped to the while loop which invokes rm -f command for each item listed by ls command, which efficiently removes all non-hidden files in current working directory … which probably is not what we mean by typing !ls on the command prompt ;). That is why its valuable to first check what arguments were used with !ls:p syntax.

Enough for now, I will write more parts with more hints on how to efficiently work in the shell/terminal environment.

UPDATE 1

The Ghost in the Shell – Part 1 article was included in the BSD Now 241 – Bowling in the LimeLight episode.

Thanks for mentioning!

UPDATE 2

About Recall last argument of previous command section … there is also $_ that does similar thing as !$ but there is little difference. The !$ is ‘line oriented’ while $_ is ‘previous command oriented’. Below is an example that shows the difference in the behavior.

The $! takes value from last command in ‘previous line’ which means that '-l' value will be used from line 001 and not 'asd' from the current line 002 from previously executed command.

001 % ls -l
002 % echo asd; ls !$ | tail -2
echo asd; ls -l | tail -2
asd
// ls output //

The $_ takes value from last executed command, thus it points at 'asd' used on line 002 and not at '-l' used at previous 001 line.

001 % ls -l
002 % echo asd; ls $_ | tail -2
asd
ls: asd: No such file or directory

On BASH shell there is also [ALT]-[.] shortcut that switches between $! from previous lines. To achieve the same shortcut on ZSH use this line below in ZSH config.

bindkey '\e.' insert-last-word

Thank you Zachery Purnell for pointing that out.

EOF