Tag Archives: unix

FreeBSD Desktop – Part 25 – Configuration – Random Terminal Theme

Some time ago when I was mostly writing about Openbox setup I also showed how to setup xterm(1) so it will start with new random theme with each start. Since then I reworked that feature a little and also added random theme and background selection for urxvt(1) terminal. This post will guide you through the needed steps to make that setup working.

terminal

You may want to check other articles in the FreeBSD Desktop series on the FreeBSD Desktop – Global Page where you will find links to all episodes of the series along with table of contents for each episode’s contents.

I already once wrote about random xterm(1) themes in the FreeBSD Desktop – Part 12 – Configuration – Openbox part – but I really wanted to expand that topic and also include other terminals.

After trying many terminal emulators – some more bulky like Konsole/GNOME Terminal/XFCE Terminal/MATE Terminal – some more lightweight like st(1)/rox-term(1)/eterm(1)/… – I always went back to the most old-school and basic one – good old xterm(1) terminal. Mostly because of its good compatibility with all UNIX systems – especially the older ones like IBM AIX or HP-UX. I do not remember last time when I had to manage these archaic systems but the respect for xterm(1) remains.

xterm

The other one that always got my attention was urxvt(1) terminal. Unfortunately it has some issues with fonts rendering – making larger spaces between the lines and making fonts bigger for example – but as I also like bitmap fonts like CLEAN or FIXED – so I use bitmap fonts for urxvt(1).

Recently also sakura(1) got my attention – but it’s theming possibilities are even more limited then xterm(1) with themes/colors hardcoded directly into the source code. That is why I will omit it in this article – but I mention it since its also nice terminal.

This is the Table of Contents for this article.

  • xterm(1)
    • Proper Font Selection
    • Selecting Text for Copy/Paste
    • Selection Buffers and Keyboard Shortcuts
    • Increase/Decrease Font Size on the Fly
    • Copy/Paste with [CTRL]+[SHIFT]+[C/V]
    • Interactive Menus
    • Random xterm(1) Theme
  • urxvt(1)
    • Tabbed Interface
    • Daemon and Client Mode
    • Random urxvt(1) Theme and Background
  • sakura(1)
    • Different sakura(1) Themes
  • RAM Usage Comparison
  • CPU Time Usage Comparison

xterm(1)

First lest start with some reasonable configuration in the ~/.Xdefaults file. Alternatively some people use ~/.Xresources file. You can use any of them. Just pick one and stick to it.

! XTERM
! -----------------------------------------------------------------------------
  xterm*allowBoldFonts:     true
  xterm*allowWindowOps:     true
  xterm*boldMode:           false
  xterm*charClass:          33:48,35:48,37:48,43:48,45-47:48,64:48,95:48,126:48,35:48,58:48,63:48,61:48,44:48,38:48,59:48
  xterm*cursorBlink:        false
  xterm*cutNewline:         true
  xterm*faceName:           consolas
  xterm*faceSize:           11
  xterm*fastScroll:         true
  xterm*fullscreen:         false
  xterm*iconHint:           /home/vermaden/.icons/vermaden/xterm.xpm
  xterm*internalBorder:     1
  xterm*jumpScroll:         true
  xterm*keepSelection:      true
  xterm*loginShell:         true
  xterm*metaSendsEscape:    true
  xterm*multiScroll:        true
  xterm*omitTranslation:    fullscreen
  xterm*on4Clicks:          group
  xterm*on5Clicks:          page
  xterm*saveLines:          1024000
  xterm*scaleHeight:        1.0
  xterm*scrollKey:          true
  xterm*scrollTtyOutput:    false
  xterm*selectToClipboard:  true
  xterm*SimpleMenu*font:    -*-clean-*-*-*-*-*-*-*-*-*-*-iso8859-2
  xterm*termName:           xterm-256color
  xterm*title:              xterm
  xterm*veryBoldColors:     14
  xterm*VT100*geometry:     150x40
  xterm*VT100*translations: #override                                             \n\
                            <btn1up>: select-end(PRIMARY, CLIPBOARD, CUT_BUFFER0) \n\
                            Ctrl <key> minus: smaller-vt-font()                   \n\
                            Ctrl <key> plus: larger-vt-font()                     \n\
                            Ctrl Shift <key> C: copy-selection(CLIPBOARD)         \n\
                            Ctrl Shift <key> V: insert-selection(CLIPBOARD)

I will not discuss all possible settings as they are well described in the xterm(1) man page but I will comment some more useful and interesting ones.

Proper Font Selection

This advice is not limited to xterm(1) but its worth to mention it. Many times after adding fonts to my system – and renaming them to my ‘standard’ which looks like that one below I was asking myself how to properly specify the variant I need.

% ls -1 ~/.fonts/ubuntu-mono*
/home/vermaden/.fonts/ubuntu-mono-bold-italic.ttf
/home/vermaden/.fonts/ubuntu-mono-bold.ttf
/home/vermaden/.fonts/ubuntu-mono-italic.ttf
/home/vermaden/.fonts/ubuntu-mono.ttf

The answer to that question comes with fc-match(1) from fontconfig package. Check my ‘queries’ below.

% fc-match consolas:bold
consolas-bold.ttf: "Consolas" "Bold"

% fc-match consolas     
consolas.ttf: "Consolas" "Regular"

% fc-match consolas:bold:italic
consolas-bold-italic.ttf: "Consolas" "Bold Italic"

Selecting Text for Copy/Paste

The xterm*charClass defines which sets of characters should be treated the same when doing cut and paste. Especially with double-clicking the text. The setting above I use is based on 15 years of experience and seems to work best. You are of course encouraged to investigate the CHARACTER CLASSES section of the xterm(1) man page to read more on this topic.

The xterm*on4Clicks and xterm*on5Clicks are not used by default while xterm*on2Clicks are predefined as word and xterm*on3Clicks as line values. This is why I added them so you can select entire group with xterm*on4Clicks and entire page with xterm*on5Clicks option. Alternatively you can also use some fancy regex for some of these ‘CLICKS’ but I never thought about a REGEX that would be useful here – maybe you will come with something sensible.

Here are these ‘CLICKS’ in action.

First the xterm*on2Clicks with word selection. This is when the xterm*charClass is taken into account – what is word and that is not πŸ™‚

xterm.2.clicks

Then xterm*on3Clicks with line selection.

xterm.3.clicks

Now xterm*on4Clicks with group selection.

xterm.4.clicks

Finally the xterm*on5Clicks entire page selection.

xterm.5.clicks

Selection Buffers and Keyboard Shortcuts

The last interesting option is xterm.VT100.translations which is used for keyboard shortcuts.

The first one select-end(PRIMARY, CLIPBOARD, CUT_BUFFER0) is better described in the https://davidsimmons.com/soft/xtermhacks/ page from 2005. I will try to short the meritum here. The X11 applications have two different selection buffers:

CLIPBOARD – selection buffer used for cut/paste functions – you select/highlight text and then select Copy from context menu or use [CTRL]+[C] shortcut. Then you use Paste or [CTRL]+[C] shortcut.

PRIMARY – this one receives data when user selects/highlights text with mouse. None other operations such as Copy or Paste are needed. You end selecting the text and its already in PRIMARY buffer. You then paste it with MIDDLE mouse button.

Using the option above selecting/highlighting the text in xterm(1) copies the text into both selection buffers simultaneously. You can now either Paste it info Firefox or hit MIDDLE mouse button to paste it in other xterm(1) terminal. Best of both worlds.

If that setting does not suit you then use the xterm*selectToClipboard instead. When set to true it copies selected text to CLIPBOARD buffer and when set to false it copies selection to the PRIMARY one.

Increase/Decrease Font Size on the Fly

I always missed the shortcuts to decrease or increase font size on the fly in xterm(1) and for many years I believed that its just not possible and then I found some blog post (do not remember which one now of course) in which I found these settings and started to use them.

They are smaller-vt-font() and larger-vt-font() for decrease and increase respectively with [CTRL]+[-] and [CTRL]+[+] shortcuts – keep in mind that [SHIFT] is not used here.

Copy/Paste with [CTRL]+[SHIFT]+[C/V]

If by some reason you prefer to copy and paste by using [CTRL]+[SHIFT]+[C] and [CTRL]+[SHIFT]+[V] shortcuts then copy-selection(CLIPBOARD) and insert-selection(CLIPBOARD) will do the needed job here. As you probably guessed you can use PRIMARY instead of CLIPBOARD here if that is what you desire.

Interactive Menus

The xterm(1) comes with three different interactive menus. I will now show all three of them here with screenshots.

Menu displayed with clicking [CTRL]+[LEFT-MOUSE-BUTTON] in the terminal area.

xterm.menu.mouse.LEFT

Menu displayed with clicking [CTRL]+[MIDDLE-MOUSE-BUTTON] in the terminal area.

xterm.menu.mouse.MIDDLE

Menu displayed with clicking [CTRL]+[RIGHT-MOUSE-BUTTON] in the terminal area.

xterm.menu.mouse.RIGHT

Random xterm(1) Theme

To have random xterm(1) theme on every startup you need four things:

I gathered all these themes all over the Internet, only the VERMADEN and VERMADEN-OLD themes are created by me.

Little preview of some of the included xterm(1) themes.

xterm.random

From now on to have random xterm(1) theme at each start always start it with ~/scripts/xterm.sh script. The script itself is not very complicated. It just draws random theme from the ~/.config/Xdefaults/themes dir – then loads the ~/.Xdefaults config – then merges the colors from chosen random theme – and finally starts new xterm(1) instance.

xterm.sh

urxvt(1)

I use urxvt(1) less often but still sometimes I want to use bitmap fonts instead.

urxvt.single

For a start here is the urxvt(1) configuration in the ~/.Xdefaults file.

! URXVT
! -----------------------------------------------------------------------------
  urxvt.letterSpace:    0.0
! urxvt.font:           xft:monaco:pixelsize=9,style=regular,minspace=True
  urxvt.font:           -*-clean-*-*-*-*-*-*-*-*-*-*-iso8859-2
  urxvt.boldFont:       -*-clean-*-*-*-*-*-*-*-*-*-*-iso8859-2
  urxvt.iconFile:       /home/vermaden/.icons/vermaden/xterm.xpm
  urxvt.geometry:       150x40
  urxvt.cutchars:       ,;
  urxvt.scrollBar:      false
  urxvt.imLocale:       en_US.UTF-8
  urxvt.loginShell:     true
  urxvt.saveLines:      1024000
  urxvt.inheritPixmap:  false
  urxvt.shading:        20
  urxvt.xftAntialias:   true
  urxvt.jumpScroll:     true
  urxvt.tintColor:      black
  urxvt.internalBorder: 2
  urxvt.cursorBlink:    false
  urxvt.cursorColor:    #dd9900
  urxvt.cursorColor2:   #000000
  urxvt.colorBD:        #dddddd
  urxvt.colorIT:        #bbbbbb
  urxvt.colorUL:        #999999
  urxvt.underlineColor: #999999

Tabbed Interface

To get tabs in urxvt(1) add the following option to the ~/.Xdefaults configuration file.

  urxvt.perl-ext-common:   default,tabbed

With this option you will open new tab with [SHIFT]+[DOWN] shortcut.

To switch between the tabs left and right use [CTRL]+[SHIFT]+[LEFT] and [CTRL]+[SHIFT]+[RIGHT] shortcut respectively.

You can also use [CTRL]+[LEFT] and [CTRL]+[RIGHT] to move current tab left and right.

The timeless [CTRL]+[D] – which of course is not a strictly urxvt(1) shortcut but a general shortcut for closing all terminals.

urxvt.tabbed

Daemon and Client Mode

The urxvt(1) can be run in special daemon mode where you start one urxvtd(1) server and many urxvtc(1) clients.

% urxvtd
rxvt-unicode daemon listening on /home/vermaden/.urxvt/urxvtd-w520.local.

Now you will start each new urxvt(1) terminal with urxvtc(1) command.

The drawback of that approach is that when urxvtd(1) dies or crashes then also all your urxvtc(1) client terminals disappear πŸ™‚

Random urxvt(1) Theme and Background

To have random urxvt(1) theme and background on every startup you need four things:

Little preview of some of the included urxvt(1) themes and backgrounds.

urxvt.random

From now on to have random urxvt(1) theme at each start always start it with ~/scripts/urxvt.sh script. The script for urxvt(1) is little more advanced. First it draws random theme from the ~/.config/Xdefaults/themes dir – then checks if its DARK or LIGHT theme – then draws either random LIGHT or DARK background from the ~/.config/Xdefaults/urxvt dir – finally loads the ~/.Xdefaults config and then merges the colors from chosen LIGHT or DARK theme. Of course then it finally starts new urxvt(1) instance.

urxvt.sh

sakura(1)

The more modern and GTK based sakura(1) also supports tabs. To open new tab use [CTRL]+[SHIFT]+[T] shortcut. To switch between the tabs use [CTRL]+[ALT]+[LEFT] and [CTRL]+[ALT]+[RIGHT] shortcuts. You can also move tab between left and right with [CTRL]+[SHIFT]+[LEFT] and [CTRL]+[SHIFT]+[RIGHT].

sakura.single

One things that sakura(1) impresses me is that you can scale down its window and then scale that window up and the contents that did not fit in the window after downscalling are back again in the terminal. Doing the same operation in xterm(1) or urxvt(1) terminals will result in these characters being lost. The output is also dynamically ‘fit’ into the new larger window while maintaining the new lines etc. Besides that nice feature it is small and fast and uses relatively small amount of RAM.

sakura.tabbed

Different sakura(1) Themes

If you would also like to start sakura(1) with different theme everytime the options are quite limited here. The palettes and color sets are hardcoded into the sakura(1) source code.

I will not show you how to modify them using the FreeBSD Ports system.

The sakura(1) port is located at /usr/ports/x11/sakura directory. For the record – I use WRKDIRPREFIX option in the /etc/make.conf file. This means that when I type make extract in the /usr/ports/x11/sakura dir the work directory will not be created at /usr/ports/x11/sakura/work directory but at /usr/ports/obj/usr/ports/x11/sakura/work instead. That way I can clean my Ports tree fast by removing the /usr/ports/obj directory.

We will now extract and patch the sakura(1) port on FreeBSD.

% grep WRKDIRPREFIX /etc/make.conf
WRKDIRPREFIX=${PORTSDIR}/obj

# cd /usr/ports/x11/sakura

# make patch

# cd /usr/ports/obj/$( pwd )/work/*/src || cd work/*/src

# pwd
/usr/ports/obj/usr/ports/x11/sakura/work/sakura-3.7.1/src

# grep -m 1 DEFAULT_PALETTE sakura.c
#define DEFAULT_PALETTE "tango"

# grep -o -E '[a-z]+_palette\[PALETTE_SIZE\]' sakura.c
gruvbox_palette[PALETTE_SIZE]
tango_palette[PALETTE_SIZE]
linux_palette[PALETTE_SIZE]
dark_palette[PALETTE_SIZE]
xterm_palette[PALETTE_SIZE]
rxvt_palette[PALETTE_SIZE]

As you can see the default sakura(1) palette is Tango. Fortunately you can use palette=solarized_dark option in the ~/.config/sakura/sakura.conf config file to change it into Solarized Dark for example.

Besides hardcoded palettes sakura(1) also has several Color Sets.

If you would like to make random theme (from the hardcoded ones) for each start you would have to use this syntax with prepared dedicated config files for each palette.

% sakura --config-file ~/.config/sakura/sakura.solarized_dark.conf --colorset 1 
% sakura --config-file ~/.config/sakura/sakura.tango.conf          --colorset 3

I do not use sakura(1) that much so I was too lazy to write random startup theme script also for it πŸ™‚

Changing sakura(1) palette or color set manually is shown below.

sakura.colors.menu

sakura.colors.window

sakura.palette

RAM Usage Comparison

Just started xterm(1) terminal takes about 16 MB or RAM as you have seen on the urxvt(1) screenshots. The urxvt(1) started without tabs uses more then 2 TIMES of xterm(1) terminal RAM usage. The urxvt(1) started in tabbed mode uses more then 3 TIMES of xterm(1) terminal RAM usage. The sakura(1) also uses more then 3 TIMES of xterm(1) terminal RAM usage.

Table below shows RAM usage comparison. I have added more feature packed mate-terminal(1) to the list and also added the st(1) minimalistic terminal from Suckless project for compassion.

RAM/MB  TERMINAL  
    64  mate-terminal
    53  sakura
    52  urxvt (tabbed)
    38  urxvt
    16  xterm
    12  st

CPU Time Usage Comparison

I also made simple benchmark of the CPU used. The ‘benchmark’ was to check how much time each terminal would take to print output ofΒ  dmesg | lolcat -b -r command. Here xterm(1) does not shine that much.

   TIME  TERMINAL
0:00.93  xterm
0:00.82  mate-terminal
0:00.52  sakura
0:00.43  urxvt
0:00.23  st

The above ‘benchmark’ was was quite ‘hard’ because of all the colors generated by lolcat(1) command. Lets try something more practical now. We will measure CPU time used to display out of the find find /usr/local/share/doc command.

   TIME  TERMINAL
0:01.34  xterm
0:01.18  mate-terminal
0:00.85  sakura
0:00.32  urxvt
0:00.28  st

Seems that lolcat(1) was not that ‘hard’. The st(1) minimalistic terminal really seems to suck less here πŸ™‚

EOF

Manage Contacts the UNIX Way

About two years ago my neighbor asked me a question – “How do you manage contacts on your devices?” – and that was my ‘a-ha’ moment in that topic – I do not. I do not at all. He had a problem of having an iPhone with iTunes and Android phone and wanted to manage contacts between them in one single sensible place. Finally he settled on some closed source freeware software which run on Windows. But that was not the answer – that was just the beginning – how to manage contacts the UNIX open source way?

I have tried to search for some open source software that is capable of doing that efficiently and without too much effort and PITA … and I failed miserably.

So as usual I came with my set of scripts that will do the job and after several years of using this ‘system’ I am quite satisfied with the results and PITA reduced to minimum.

Export from Phone

The VCF file (also called VCARD) exported from a mobile Android based phone looks like below.

% cat export.vcf
BEGIN:VCARD
VERSION:2.1
N:pierre;herbert;hugues;;
FN:herbert pierre hugues
TEL;PREF:555123456
NOTE:executive
EMAIL:pierre@gmail.com
END:VCARD
BEGIN:VCARD
VERSION:2.1
N:(local);butcher;;
FN:butcher (local)
TEL;PREF:555123457
TEL;HOME:225553457
TEL;HOME:451232421
NOTE:cheap
END:VCARD
BEGIN:VCARD
VERSION:2.1
N:(f1);martin;brundle;;
FN:martin brundle (f1)
TEL;PREF:555987654
TEL;HOME:451232421
X-QQ;HOME:gg:32847916
NOTE:fast
END:VCARD


I have used colors to distinguish different contacts.

The most annoying field seems to be 'N' which tries to be smarter then needed – trying really hard to first put surname, then name, and then other names. The 'FN' field is a lot more useful here. The remaining fields as 'TEL' or 'EMAIL' does not try to outsmart us and work as desired. The VCARD of course starts with 'BEGIN:VCARD' and ends with 'END:VCARD', that is obvious. In 2015 when I initially wrote those scripts the Instant Messaging was still used by me. Now several years fast forward I use it very rarely, but its still in use. I keep this Instant Messaging number/account information in the VCARD 'X-QQ' field in which I use protocol:number notation and use it for all different Instant Messaging solutions. The 'gg:' is for example for the Polish solution called Gadu-Gadu.

I do not find this VCARD format readable, nor grepable/searchable, thus I convert it into the plain text file which looks like follows and is grep(1) and awk(1) friendly (columns separated by spaces).

NAME  PHONE  IM  MAIL  NOTES
====  =====  ==  ====  =====

Here is how the above VCARD information looks after converting it with my script to the plain text columns.

% contacts-convert-vcf-from.sh -t export.vcf | column -t > contacts
% cat contacts
NAME                                    PHONE                                                IM                MAIL                                                    NOTES
======================================  ===================================================  ================  ======================================================  =====
butcher-(local)                         555123457;225553457;451232421                        -                 -                                                       cheap
herbert-pierre-hugues                   555123456                                            -                 pierre@gmail.com                                        executive
martin-brundle-(f1)                     555987654;451232421                                  gg:32847916       -                                                       fast

The length of ‘=====’ underscores is defined/hardcoded in the scripts itself. Why hardcode this? For comparison purposes – more on that later. The entries are also sorted by name. I could embed/rework the script to contain also the column -t command but I did not saw the need to – but its of course possible.

Now – lets suppose you want to generate new VCARD with some of your contacts, then you could use grep(1) to filter out the unneeded entries, like that.

% grep -v butcher contacts > contacts.NOBUTCHER
% contacts-convert-vcf-to.sh contacts.NOBUTCHER > import.vcf
% cat import.vcf

BEGIN:VCARD
VERSION:2.1
FN:herbert pierre hugues
TEL;PREF:555123456
EMAIL:pierre@gmail.com
NOTE:executive
END:VCARD

BEGIN:VCARD
VERSION:2.1
FN:martin brundle (f1)
TEL;PREF:555987654
TEL:451232421
X-QQ:gg:32847916
NOTE:fast
END:VCARD

Its obvious but the generated VCARD does not contain the 'butcher (local)' contact. You can now send this import.vcf file to your phone using email and then import these contacts as you would from any other VCARD shared with you.

Scripts

I use three scripts to convert/export/import/check that data in VCARD form.

The contacts-convert-vcf-from.sh script as the name suggests converts VCARD data (VCF file) into the plain text information. but I also implemented the CSV method which may be useful for some people – to put that data into the spreadsheet.

% contacts-convert-vcf-from.sh
usage: contacts-convert-vcf-from.sh TYPE FILE
  TYPE: -c | --csv
        -p | --plain
        -t | --text

Here is example CSV output from the script.

% contacts-convert-vcf-from.sh -c export.vcf
NAME,PHONE,IM,MAIL,NOTES
butcher-(local),555123457;225553457;451232421,-,-,cheap
herbert-pierre-hugues,555123456,-,pierre@gmail.com,executive
martin-brundle-(f1),555987654;451232421,gg:32847916,-,fast

The contacts-convert-vcf-to.sh script converts the plain text data into the VCARD format.

% contacts-convert-vcf-to.sh
usage: contacts-convert-vcf-to.sh FILE

The last contacts-check.sh script is used to find duplicated phone information within the plain text file. Many time I have found duplicated contacts with different names but with the same phone number.

% contacts-check.sh contacts | column -t
butcher-(local)      555123457;225553457;451232421  -            -  cheap
martin-brundle-(f1)  555987654;451232421            gg:32847916  -  fast

All of the three are available in my GitHub scripts page – https://github.com/vermaden/scripts/ – available here.

You can of course download them using command line like that.

% wget https://raw.githubusercontent.com/vermaden/scripts/master/contacts-check.sh
% wget https://raw.githubusercontent.com/vermaden/scripts/master/contacts-convert-vcf-from.sh
% wget https://raw.githubusercontent.com/vermaden/scripts/master/contacts-convert-vcf-to.sh
% chmod +x contacts-*

Updating Contacts

Its easy to maintain several contacts – no matter in which format – but when you grow to have about a 1000 of contacts (and I do) then you need to deal with it intelligently.

Not to mention that you can add a new contact on your phone (more often) but You can also update your local plain text contacts file.

This is where UNIX comes handy. You may use diff(1) to compare these ‘updates’ with following command.

% diff -u contacts contacts.NEW | egrep '^\-|^\+'
--- contacts            2019-12-13 15:29:23.541256000 +0100
+++ contacts.NEW        2019-12-13 15:29:36.087084000 +0100
-john-doe-the-third                      -                                                    -                 jogh.doe@gmail.com                                      -
+jan-kowalski                            555192384                                            gg:11844916       -                                                       slow


This way you know that there are two new contacts, one '-' from the local contacts file and one '+' from the plain text version generated from phone exported VCF file called contacts.NEW here.

You can also use vim(1) with its diff mode enabled by starting with -d flag as shown below.

% vim -d contacts contacts.NEW

Here is how it looks like.

vim.png

… and we now get back to the amount of '====' used in the columns in the plain text file. If you keep the same amount of these each time, then diff is possible. If I would not put them there the column -t command would generate larger NAME column for example because of longer contact name – and because of additional space in the remaining contacts both diff(1) and vim(1) tools will show that all contacts are new.

This is how I manage the contacts the UNIX way, if you have more fun way of dealing with the contacts then please let me know πŸ™‚

EOF

Manage Photography the UNIX Way

After using UNIX for so many years you start to think the UNIX way. This article aims to automate and accelerate the flow of importing photos from camera and storing it for future use.

When I had a lot of time I shoot both RAW and JPEG images at the same time (RAW and JPEG file were written for every picture). Then I used one of the DxO Optics Pro/Raw Theraphee/Darktable applications to make these RAW files shine even more with mass conversion. Then I compared these to out of camera JPEG files and left only the one that suited me best. Its was probably the best way of having ‘the best version’ of each photo but it also took whole a lot of time. Now as I do not have that much time I needed to find a way to make this process fast and almost seamless.

Hardware

I use SONY cameras because they are superior to other brands when it comes to price/performance ratio and also have some important features that are absent in other brands. For example SONY A-mount based cameras – SONY a68 camera offers just so much more for very small amount of money then any near Nikon or Canon competitor. If you want to get grip on these differences take a look at my SONY a68 review at DPReview site – https://www.dpreview.com/forums/thread/4152155 – available here.

a68-lcd.jpg

Besides the price/performance ratio SONY cameras are just too fun/too comfortable to use something different – while providing similar or better results then Nikon/Canon competition. Take the viewfinder for example. Nikon/Canon cameras are ‘by default’ using the optical viewfinder and to switch to LCD panel you need to manually push a button and switch into the PAINFULLY SLOW (autofocus is actually unusable) mode called Live View … but if you want to use viewfinder again then you again need to switch that mode off with a button. How its implemented in SONY? SONY camera just automatically switches to EVF when you attach your eye to the viewfinder and switches back to LCD automatically when you take your eye off of it … and autofocus is same fast on both viewfinder and LCD. This is just one of the examples of course. For example Nikon cameras can not record movie when you are using viewfinder – you can only do it with LCD.

a68-flash.jpg

There is also SONY E-mount system which utilizes newer/different ideas – its generally much more expensive then older A-mount system but has even more features then Canon/Nikon cameras. One of the selling points of SONY E-mount cameras is also their small size – for which feature I recently switched from SONY a68 (A-mount) to SONY a5100 (E-mount) camera.

Approach

I basically use two SONY cameras.

The small and ultra portable SONY RX100 III which is probably the best pocket/compact camera in the world when it comes to price/performance ratio. As it has quite large 1 INCH sensor (2.7 crop factor) it allows to use high ISO values without that much noise which allows to shoot indoors in low light without much loss of quality. It also has tiltable flash which you can point to ceiling to get extra bounced light in low light situations indoors. This small gem generally has all the features that all SONY APS-C/Full Frame cameras have. Same menu interface with same features. Its not some small handicapped cripple like a lot of compact cameras. And its fast too. It even features EVF! It also features XAVC S 50 Mbit video codec which helps greatly in low light situations. Of course in good light conditions this camera shines even more. As it has 24-70mm f/1.8-2.8 light/fast lens it its very universal. The Full Frame depth of field equivalent is even better then most APS-C cameras because its f/4.9-7.6 Full Frame depth of field equivalent is better – for example – then SONY a6400 with its f/3.5-5.6 kit lens – which only has f/5.3-8.4 (because of 1.5 crop ratio for APS-C).

rx100-evf-lcd-on.jpg

You can read more about depth of field equivalence here – https://www.dpreview.com/articles/2666934640/what-is-equivalence-and-why-should-i-care – a good article on DPReview explaining this.

The other SONY camera I used was SONY a68 with following lenses:

  • TAMRON 18-270mm f/3.5-5.6 – all-rounder
  • SONY 35mm f/1.8 – small bokeh low light friend
  • SIGMA 50-150mm f/2.8 – large bokeh friend
  • SAMYANG 85mm f/1.4 – manual focus bokeh master

… but as I checked my ‘habits’ it was that way most of the time:
– use/take small/portable SONY RX100 III because its convenient
– grab SONY a68 with 35mm f/1.8 at house for some bokeh pictures

If you are not sure what ‘bokeh’ means then please check Wikipedia article about it – https://en.wikipedia.org/wiki/Bokeh – available here.

I very rarely used other lenses. Which made me to think how to ‘optimize’ the SONY a68 A-mount camera. Also because SONY a68 built-in flash is not able to point up (to get extra light from ceiling indoors) I also needed dedicated external SONY HVL F20M flash on ISO hot shoe which made this large camera even bigger.

I checked the SONY portfolio and got older SONY a5100 E-mount camera instead. It has nice and fast autofocus from SONY a6000 camera along with XAVC S video codec and useful tiling LCD screen. It even has a touch screen which allows you to take a photo on the place when you touched the screen! It works similar in movies – just touch when you want it to focus. Its probably smallest SONY APS-C body – very close in size to SONY RX100 III … and I got SONY E-mount 35mm f/1.8 lens to it. I also missed 85mm f/1.4 lens so I take different route now. As E-mount system allows one to adapt older lenses with Lens Turbo adapters (about 0.7 ratio) I get an old used Minolta MD 56mm f/1.4 lens and E-mount to MD Lens Turbo adapter from ALIEXPRESS. This way I got small ultimate bokeh machine – with only one downside – manual autofocus – but SONY a5100 provides very nice implementation of Focus Peaking so its still a pleasure to use.

a5100-lcd.jpg

Of course SONY a5100 has its limitations – no viewfinder for example – but I VERY rarely used it anyway – of course intensive outdoor light can be problematic sometimes without EVF – but if someone wants to have EVF then one should get one of the SONY a6000/a6300/a6400/a6500 cameras – they are not much more larger and provide both EVF and hot shoe.

a5100-flash.jpg

Generally SONY RX100 III when powered on its comparable in size with SONY a5100 with SONY 35mm f/1.8 lens. Its the powered off state and lens range (24-70mm on SONY RX100 III) that make a difference – the SONY RX100 III even fits in the pocket – SONY a5100 does not – maybe with SONY 20mm f/2.8 lens.

If you have quite more budget to spend I also recommend the SONY RX100 V/VA which also incorporates very fast phase detection autofocus and 4k video. The SONY RX100 IV only offers 4k video but still has slower contrast autofocus – thus its IMHO pointless to get it. For the record – the SONY RX100 III also uses slower contrast based autofocus and has video up to FullHD (1080p).

top-a5100-a68.jpg

These cameras also share nice feat – they can be charged directly by attaching USB micro cable to them – very convenient – no need to provide dedicated external chargers for batteries. I really liked SONY a68 grip and lots of direct controls but I really like the size/compactness of SONY a5100. While SONY a5100 body weights 283 grams the SONY a68 is 690 grams – for the body alone. Add flash and larger lens to it and you get the idea.

top-rx100-a5100-with-lens-size.jpg

Comparing to the other side the SONY RX100 III weights 290 grams while SONY a5100 wights 437 grams with SONY 35mm f/1.8 lens attached, not bad.

Gear Summary

I have settled on these two cameras for now.

  • SONY RX100 III – gives 24-70mm f/4.9-7.6 depth of field Full Frame equivalent
  • SONY a5100 with these lenses:
    • Sony 35mm f/1.8 OSS – gives 53mm f/2.7 depth of field Full Frame equivalent
    • Minolta MD 56mm f/1.4 with Lens Turbo 0.7x adapter – gives 59mm f/1.5 depth of field Full Frame equivalent

Scripts

I switched off shooting RAW+JPEG images and now I only shoot EXTRA FINE JPEG images with Vivid profile and -0.7 EV (to not have over-burned images).

The 1st part is copying the images to new directory. That means pictures from DCIM directory and movies from PRIVATE directory.

Now the first two scripts come to play – to rename images to something useful. Each Picture and Video will have YYYY.MM.DD.HHMM(x) name.

These are made by these two scripts:

  • photo-rename-images.sh
  • photo-rename-movies.sh

Links to the scripts will be posted later in the article.

The photo-rename-images.sh uses jhead as dependency.

Now as we have everything named as it should be the size needs to be addressed. The videos will be converted using ffmpeg and images will be compressed to 92% JPEG quality with convert utility from ImageMagick suite.

  • photo-requality.sh
  • photo-movie-audio-ac3.sh

One may ask why convert JPEG from 99% to 92% and lose more quality even more? Well, you should check the differences – and one have to try really hard with very large zoom to find any. For most purposes these differences are negligible. You can also use larger value to have quite better quality and less storage savings -take photo-requality.sh 95 for example as consensus.

This is the comparison between original out of camera JPEG file and the same file compressed to 92% quality using convert utility. I was not able to stop any differences – maybe you will.

diff-crop.jpg

One may be also worried about quality loss in the videos as the size savings are that big. I also tried to find these differences and if its really hard to find them then storage savings are justified – at least for me.

I also recently added photo-flow.sh which takes two arguments. First is the device under which the camera SD card is mounted – its mmcsd0s1 on FreeBSD for most of the times. The second is directory ~/photo.NEW in which the pictures and videos will be dumped, renamed and (re)compressed.

I have put these scripts to my external (from WordPress) account on GitHub – https://github.com/vermaden/scripts – here they are:

Flow

As I attached the SD card from one of my cameras to my laptop it was automounted by my automount solution – described here – Automount Removable Media – as /media/mmcsd0s1 directory – that will be first argument for the import scripts. As I import new pictures to ~/photo.NEW directory – that will be the second argument for the import scripts.

Below you will find example output of such import/convertion process. It took about half an hour on 2011 dual-core laptop (ThinkPad T420s). I omitted/cut large parts of the same output with (…) chars in the output.

% photo-flow.sh /media/mmcsd0s1 ~/photo.NEW
/media/mmcsd0s1/DCIM/100MSDCF/DSC00390.JPG -> /home/vermaden/photo.NEW/2019.06.10.DUMP/DSC00390.JPG
/media/mmcsd0s1/DCIM/100MSDCF/DSC00391.JPG -> /home/vermaden/photo.NEW/2019.06.10.DUMP/DSC00391.JPG
/media/mmcsd0s1/DCIM/100MSDCF/DSC00393.JPG -> /home/vermaden/photo.NEW/2019.06.10.DUMP/DSC00393.JPG
(...)
/media/mmcsd0s1/DCIM/100MSDCF/DSC00462.JPG -> /home/vermaden/photo.NEW/2019.06.10.DUMP/DSC00462.JPG
/media/mmcsd0s1/DCIM/100MSDCF/DSC00463.JPG -> /home/vermaden/photo.NEW/2019.06.10.DUMP/DSC00463.JPG
/media/mmcsd0s1/DCIM/100MSDCF/DSC00464.JPG -> /home/vermaden/photo.NEW/2019.06.10.DUMP/DSC00464.JPG
/media/mmcsd0s1/PRIVATE/M4ROOT/CLIP/C0015.MP4 -> /home/vermaden/photo.NEW/2019.06.10.DUMP/C0015.MP4
/media/mmcsd0s1/PRIVATE/M4ROOT/CLIP/C0015M01.XML -> /home/vermaden/photo.NEW/2019.06.10.DUMP/C0015M01.XML

DSC00390.JPG --> 2019.05.08.0732.jpg
DSC00391.JPG --> 2019.05.08.0732a.jpg
DSC00393.JPG --> 2019.05.08.0732b.jpg
(...)
DSC00462.JPG --> 2019.06.07.2110c.jpg
DSC00463.JPG --> 2019.06.07.2110d.jpg
DSC00464.JPG --> 2019.06.07.2110e.jpg
C0015.MP4 -> 2019.06.01.2140.MP4
C0015M01.XML -> 2019.06.01.2140.XML
File './2019.05.22.0543.jpg' converted to '92' quality.
File './2019.06.07.0508a.jpg' converted to '92' quality.
File './2019.06.01.2141.jpg' converted to '92' quality.
(...)
File './2019.05.23.0124c.jpg' converted to '92' quality.
File './2019.06.01.2140e.jpg' converted to '92' quality.
File './2019.05.22.0548a.jpg' converted to '92' quality.
ffmpeg version 4.1.3 Copyright (c) 2000-2019 the FFmpeg developers
(...)
Guessed Channel Layout for Input Stream #0.1 : stereo
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from '2019.06.01.2140.MP4':
  Metadata:
    major_brand     : XAVC
    minor_version   : 16785407
    compatible_brands: XAVCmp42iso2
    creation_time   : 2019-06-01T19:40:52.000000Z
  Duration: 00:00:21.60, start: 0.000000, bitrate: 52049 kb/s
    Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p(tv, bt709/bt709/iec61966-2-4), 1920x1080 [SAR 1:1 DAR 16:9], 50101 kb/s, 50 fps, 50 tbr, 50k tbn, 100 tbc (default)
    Metadata:
      creation_time   : 2019-06-01T19:40:52.000000Z
      handler_name    : Video Media Handler
      encoder         : AVC Coding
    Stream #0:1(und): Audio: pcm_s16be (twos / 0x736F7774), 48000 Hz, stereo, s16, 1536 kb/s (default)
    Metadata:
      creation_time   : 2019-06-01T19:40:52.000000Z
      handler_name    : Sound Media Handler
    Stream #0:2(und): Data: none (rtmd / 0x646D7472), 409 kb/s (default)
    Metadata:
      creation_time   : 2019-06-01T19:40:52.000000Z
      handler_name    : Timed Metadata Media Handler
      timecode        : 83:01:01;02
Stream mapping:
  Stream #0:0 -> #0:0 (h264 (native) -> h264 (libx264))
  Stream #0:1 -> #0:1 (pcm_s16be (native) -> ac3 (native))
Press [q] to stop, [?] for help
[libx264 @ 0x80ddfb400] using SAR=1/1
[libx264 @ 0x80ddfb400] using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX
[libx264 @ 0x80ddfb400] profile High, level 4.2, 4:2:0, 8-bit
[libx264 @ 0x80ddfb400] 264 - core 157 - H.264/MPEG-4 AVC codec - Copyleft 2003-2018 - http://www.videolan.org/x264.html - options: cabac=1 ref=3 deblock=1:0:0 analyse=0x3:0x113 me=hex subme=7 psy=1 psy_rd=1.00:0.00 mixed_ref=1 me_range=16 chroma_me=1 trellis=1 8x8dct=1 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=-2 threads=1 lookahead_threads=1 sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=3 b_pyramid=2 b_adapt=1 b_bias=0 direct=1 weightb=1 open_gop=0 weightp=2 keyint=250 keyint_min=25 scenecut=40 intra_refresh=0 rc_lookahead=40 rc=crf mbtree=1 crf=23.0 qcomp=0.60 qpmin=0 qpmax=69 qpstep=4 vbv_maxrate=25000 vbv_bufsize=25000 crf_max=0.0 nal_hrd=none filler=0 ip_ratio=1.40 aq=1:1.00
Output #0, matroska, to '2019.06.01.2140.MP4.mkv':
  Metadata:
    major_brand     : XAVC
    minor_version   : 16785407
    compatible_brands: XAVCmp42iso2
    encoder         : Lavf58.20.100
    Stream #0:0(und): Video: h264 (libx264) (H264 / 0x34363248), yuv420p(progressive), 1920x1080 [SAR 1:1 DAR 16:9], q=-1--1, 50 fps, 1k tbn, 50 tbc (default)
    Metadata:
      creation_time   : 2019-06-01T19:40:52.000000Z
      handler_name    : Video Media Handler
      encoder         : Lavc58.35.100 libx264
    Side data:
      cpb: bitrate max/min/avg: 25000000/0/0 buffer size: 25000000 vbv_delay: -1
    Stream #0:1(und): Audio: ac3 ([0] [0][0] / 0x2000), 48000 Hz, stereo, fltp, 160 kb/s (default)
    Metadata:
      creation_time   : 2019-06-01T19:40:52.000000Z
      handler_name    : Sound Media Handler
      encoder         : Lavc58.35.100 ac3
frame= 1080 fps=4.1 q=31.0 Lsize=   30522kB time=00:00:21.59 bitrate=11578.4kbits/s speed=0.0815x    
video:30086kB audio:422kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.046764%
(...)

This is how the pictures look like imported and converted after running the import flow. We still have original 2019.06.01.2140.MP4 movie but we can delete it of course.

% exa ~/photo.NEW/2019.06.10.DUMP
2019.05.08.0732.jpg   2019.05.22.0548.jpg   2019.05.25.2111.jpg   2019.06.01.0914.jpg   2019.06.01.2140.jpg      2019.06.07.0509.jpg
2019.05.08.0732a.jpg  2019.05.22.0548a.jpg  2019.05.25.2111a.jpg  2019.06.01.0915.jpg   2019.06.01.2140.MP4      2019.06.07.0509a.jpg
2019.05.08.0732b.jpg  2019.05.22.0548b.jpg  2019.05.25.2111b.jpg  2019.06.01.2043.jpg   2019.06.01.2140.MP4.mkv  2019.06.07.0509b.jpg
2019.05.08.0733.jpg   2019.05.22.0549.jpg   2019.05.25.2111c.jpg  2019.06.01.2043a.jpg  2019.06.01.2140.XML      2019.06.07.2110.jpg
2019.05.22.0541.jpg   2019.05.22.0550.jpg   2019.05.27.0712.jpg   2019.06.01.2043b.jpg  2019.06.01.2140a.jpg     2019.06.07.2110a.jpg
2019.05.22.0541a.jpg  2019.05.22.0551.jpg   2019.05.27.0712a.jpg  2019.06.01.2043c.jpg  2019.06.01.2140b.jpg     2019.06.07.2110b.jpg
2019.05.22.0542.jpg   2019.05.23.0124.jpg   2019.05.27.0712b.jpg  2019.06.01.2043d.jpg  2019.06.01.2140c.jpg     2019.06.07.2110c.jpg
2019.05.22.0542a.jpg  2019.05.23.0124a.jpg  2019.05.27.0712c.jpg  2019.06.01.2043e.jpg  2019.06.01.2140d.jpg     2019.06.07.2110d.jpg
2019.05.22.0542b.jpg  2019.05.23.0124b.jpg  2019.05.27.0712d.jpg  2019.06.01.2043f.jpg  2019.06.01.2140e.jpg     2019.06.07.2110e.jpg
2019.05.22.0542c.jpg  2019.05.23.0124c.jpg  2019.05.27.0712e.jpg  2019.06.01.2043g.jpg  2019.06.01.2141.jpg
2019.05.22.0543.jpg   2019.05.23.1831.jpg   2019.05.27.0712f.jpg  2019.06.01.2043h.jpg  2019.06.01.2141a.jpg
2019.05.22.0543a.jpg  2019.05.25.2110.jpg   2019.05.27.0713.jpg   2019.06.01.2043i.jpg  2019.06.07.0508.jpg
2019.05.22.0543b.jpg  2019.05.25.2110a.jpg  2019.05.27.0713a.jpg  2019.06.01.2044.jpg   2019.06.07.0508a.jpg

These are differences in size before and after conversion – both for example picture and video.

% ls -lh ~/photo.NEW/2019.06.10.DUMP/2019.06.01.2140.MP4*
-rw-r--r--  1 vermaden  vermaden   134M 2019.06.01 21:41 /home/vermaden/photo.NEW/2019.06.10.DUMP/2019.06.01.2140.MP4
-rw-r--r--  1 vermaden  vermaden    30M 2019.06.10 22:57 /home/vermaden/photo.NEW/2019.06.10.DUMP/2019.06.01.2140.MP4.mkv

% ls -lh /media/mmcsd0s1/DCIM/100MSDCF/DSC00430.JPG ~/photo.NEW/2019.06.10.DUMP/2019.05.27.0712f.jpg
-rw-r--r--  1 vermaden  vermaden   4.4M 2019.06.10 22:53 /home/vermaden/photo.NEW/2019.06.10.DUMP/2019.05.27.0712f.jpg
-rw-r--r--  1 vermaden  vermaden   6.4M 2019.05.27 07:12 /media/mmcsd0s1/DCIM/100MSDCF/DSC00430.JPG

The best savings are in the video – more then 4 times smaller file. The pictures are about 30% smaller.

Totals of the size differences for the whole import are below. First the original dump from camera SD card.

% du -scm /media/mmcsd0s1/DCIM /media/mmcsd0s1/PRIVATE/M4ROOT/CLIP
400     /media/mmcsd0s1/DCIM
135     /media/mmcsd0s1/PRIVATE/M4ROOT/CLIP
534     total

… and converted/imported size.

% rm ~/photo.NEW/2019.06.10.DUMP/2019.06.01.2140.MP4

% du -scm /home/vermaden/photo.NEW/2019.06.10.DUMP/*jpg | tail -1
265     total

% du -scm /home/vermaden/photo.NEW/2019.06.10.DUMP/*mkv | tail -1
30      total

% du -scm ~/photo.NEW/2019.06.10.DUMP
295     /home/vermaden/photo.NEW/2019.06.10.DUMP
295     total

So after import and conversion the pictures went from 400 to 265 MB and movies (actually one movie) went from 135 to 30 MB. The most important thing is that I can import and convert this convent without any interactive and lengthy process.

These scripts (definitely the video renamer one) may be SONY related but nothing stops you from modifying them to the files provided by your camera manufacturer.

Feel free to share your photography flow πŸ™‚

EOF

Ghost in the Shell – Part 4

Long time no see. Its been a while since last post in the Ghost in the Shell series. Its also exactly one full year since I started this blog – from the first Ghost in the Shell series article – the Part 1 – that was published on 2018/03/15 day.

Today I would like to show you new pack of useful tricks and features for productive terminal/shell use. Lets start with something simple yet useful.

You may want to check other articles in the Ghost in the Shell series on the Ghost in the Shell – Global Page where you will find links to all episodes of the series along with table of contents for each episode’s contents.

Named Pipes

We all (or at least most :>) know and love pipes in UNIX. For the record – ls | grep match | awk '{print $3}' | sed 's/.jpg//g' – command ‘chains’ like that one πŸ™‚

What is a named pipe then? A manually defined pipe for special purposes. For example some applications – especially the so called Enterprise ones – often do not support UNIX pipes mechanisms – they only can dump something to a file. A great example of such Enterprise software is Oracle database whose dump command can only make dump to a file. With tool that supports UNIX pipes you would probably want to pipe that data to gzip(1)/xz(1) to compress it on the fly or even pipe it directly to ssh(1) to the Backup server for example, but not with Oracle.

This is where named pipes feature helps. We will create named pipe called /tmp/PIPE so Oracle’s dump command will be able to use it and on the other side of this pipe we will attach a pipe to gzip -9 command to compress that data on the fly.

Below example is from Linux system so mknod(1) command will be used. For example on FreeBSD you would use mkfifo(1) command for named pipe. Complete example of such named pipe is presented below.

root # cd /tmp
root # mknod /tmp/PIPE p
root # chown oracle:oinstall /tmp/PIPE
root # dd if=/tmp/PIPE bs=1M | gzip -9 > /mnt/oracle/oracle-database-backup.dmp.gz &

Now the /tmp/PIPE named pipe is ready to be used. When any process will start to write something to the /tmp/PIPE named pipe it will be automatically grabbed by dd(8) command and piped to the gzip(1) command that will compress that input and write it into the /mnt/oracle/oracle-database-backup.dmp.gz file.

Now we can start the Oracle dumping process with dump command.

root # su - oracle
oracle % dump file=/tmp/PIPE

When the dump command finishes its work you will find all your dumped data compressed in the /mnt/oracle/oracle-database-backup.dmp.gz file.

Other example of named pipes usage is my desktop dzen2 setup with unusual update schedule – described in detail in the FreeBSD Desktop – Part 13 – Configuration – Dzen2 article.

Modify Command Environment on the Fly

For most of the time we use export(1) builtin to export needed environment values that our command needs. You can then check what environment exported values are with the env(1) command of course … but you can use the same env(1) command to run any command with modified environment without exporting variables using export(1).

Here is brief example of this feature.

For the record – the gls(1) command is a GNU/Linux ls(1) command from sysutils/coreutils package/port but to make it work without name conflicts on FreeBSD where BSD ls(1) is also present it had to be renamed to gls(1).

% gls -l | head -1
total 8609K

% env LC_ALL=pl_PL.UTF-8 gls -l | head -1
razem 8609K

In the example above we run gls(1) command with default environment – I use en_US.UTF-8 locale daily. The second invocation with LC_ALL=pl_PL.UTF-8 modified environment made gls(1) command display its output in Polish (pl_PL.UTF-8) language. The word ‘razem‘ means ‘total‘ in Polish.

Other useful example may be using make(1) to build FreeBSD port with known vulnerabilities. By default FreeBSD’s build(7) system will not allow us to build such port (and that is good defaults) but if we know what we are doing we will use following spell.

# env DISABLE_VULNERABILITIES=yes make -C /usr/ports/security/bdes/ build install clean

Its also useful with commands that do not play well with UTF-8 input like tr(1) for example. When LC_ALL is set to en_US.UTF-8 it will throw an error upon as.

% tr -cd '0-9' < /dev/random | head -c 16
tr: Illegal byte sequence
%

We just wanted to generate random 16 numbers.

To make it work we will modify the LC_ALL environment for this invocation.

% env LC_ALL=C tr -cd '0-9' < /dev/random | head -c 16
9571949869123855
%

Much better πŸ™‚

Other example with timezones using date(1) command and TZ variable as shown in the example below.

% date
Fri Mar 15 14:03:38 CET 2019

% env TZ=Australia/Darwin date 
Fri Mar 15 22:35:26 ACST 2019

The Real Path

The symlinks with ln(1) are very useful for many ways, to organize stuff, for quick fixes, for versioning … you will find tons of other use cases.

There is just one problem, if you make to many levels or symlinks or its just too much nested you do not know where you are anymore … this is where the realpath(1) comes handy. No matter how many levels of links you have made, it will tell you the truth – what is the current real path. The pwd(1) command will not help you here thou.

Here is a short example how it works.

% pwd
/home/vermaden
% ln -s /home/vermaden ASD
% cd ASD
% pwd
/home/vermaden/ASD
% realpath
/home/vermaden

Browsing the PATH

Many times I wanted to ‘browse’ through the PATH to search for something. As you possibly know the PATH variable stores paths that are colon (:) separated.

You can redefine the IFS variable which by default contains space ‘ ‘ which will work as field delimited for the for loop.

Here is the example.

% export IFS=":"

% for I in $( echo ${PATH} ); do echo ${I}; done
/sbin
/bin
/usr/sbin
/usr/bin
/usr/local/sbin
/usr/local/bin 

% for I in $( echo ${PATH} ); do find "${I}" -name ifconfig; done
/sbin/ifconfig

The other way to do this is to use plain old tr tool to translate colons (:) into newlines (\n) so we will be able to use the while loop here.

Here is the tr(1) example.

% echo ${PATH} | tr ':' '\n' | while read I; do echo ${I}; done
/sbin
/bin
/usr/sbin
/usr/bin
/usr/local/sbin
/usr/local/bin

% echo ${PATH} | tr ':' '\n' | while read I; do find ${I} -name dd; done
/bin/dd

You can also achieve same thing using the Parameter Expansion in which we will change the colons (:) into newlines (\n) as shown in the example below.

% echo "${PATH//:/\n}"
/sbin
/bin
/usr/sbin
/usr/bin
/usr/local/sbin
/usr/local/bin

# echo "${PATH//:/\n}" | while read I; do find ${I} -name camcontrol; done
/sbin/camcontrol

Parameter Expansion

I will not show all possible Parameter Expansion methods – just the most useful ones.

The typical use is to get the extension of a file or to ’emulate’ basename(1) or dirname(1) commands – it will be faster to use Parameter Expansion instead of invoking these commands each time. Below are two tables showing what you will get from which Parameter Expansion method.

PARAMETER    RESULT                       DESC 
-----------  ---------------------------  --------------
${name}      kubica.polish.racing.legend  content
${name#*.}          polish.racing.legend  -
${name##*.}                       legend  extension
${name%%.*}  kubica                       -
${name%.*}   kubica.polish.racing         -

… and with slash (/) character.

PARAMETER    RESULT                       DESC 
-----------  ---------------------------  --------------
${name}      kubica/polish/racing/legend  content
${name#*/}          polish/racing/legend  -
${name##*/}                       legend  basename(1)
${name%%.*}  kubica                       root directory
${name%/*}   kubica/polish/racing         dirname(1)

You can also use Parameter Expansion methods to grab the protocol from an URL like shown below.

% URL="https://vermaden.wordpress.com"

% echo "${URL%%/*}"
https:

Sort Human Readable Values

Its simple and easy to sort just numerical values, we use sort -n for that – but values sometimes comes in human readable form like 4G, 350M and 120K. To sort these properly you will have to use sort -h flag as shown in the example below.

% du -sh /usr/*
102M    /usr/bin
228G    /usr/home
9.0M    /usr/include
 53M    /usr/lib
 43M    /usr/lib32
116K    /usr/libdata
1.9M    /usr/libexec
365M    /usr/local
512B    /usr/obj
9.5M    /usr/sbin
 39M    /usr/share
251K    /usr/tests

% du -sh /usr/* | sort -h
512B    /usr/obj
116K    /usr/libdata
251K    /usr/tests
1.9M    /usr/libexec
9.0M    /usr/include
9.5M    /usr/sbin
 39M    /usr/share
 43M    /usr/lib32
 53M    /usr/lib
102M    /usr/bin
365M    /usr/local
228G    /usr/home

If the values are in the first column then its simple but what to do when the values are not in the first column? You will use -k parameter of sort(1) which takes which column to sort as argument. Needed example below sorted bu human readable values and on the second USED column.

% zfs list | sort -h -k 2
NAME                         USED  AVAIL  REFER  MOUNTPOINT
local/usr/obj                 88K   130G    88K  /usr/obj
local/var/cache/pkg          128K   130G   128K  /var/cache/pkg
local/var/cache              216K   130G    88K  none
local/var                    304K   130G    88K  none
sys/ROOT/11.1-RELEASE        482M  2.39G  6.04G  /
local/usr/ports              729M   130G   729M  /usr/ports
local/jail/nextcloud         927M   130G   897M  /jail/nextcloud
local/jail                  1.00G   130G   100M  /jail
local/usr/src               1.28G   130G  1.28G  /usr/src
local/usr                   1.99G   130G    88K  none
sys/ROOT/11.2-RELEASE       8.69G  2.39G  7.10G  /
sys/ROOT                    9.16G  2.39G    88K  none
sys                         9.17G  2.39G    88K  none
local/home                   281G   130G   281G  /home
local                        288G   130G    88K  none

Write a File from vi(1) with Different Rights

How many times you have opened a system configuration file like /etc/sysctl.conf or /etc/fstab in your favorite vi(1) editor, made some changes and then when you wanted to save it – no luck – you are trying to write to file owned by root with regular user … the Read-only file, not written; use ! to override. message will be displayed. Of course you can save that file somewhere else like your home directory and them move it with doas(1)/sudo(8)/su(8) help to original location and fix its rights … or you may do that in one step instead.

After opening a file with vi(1) and some changes to write a file with doas(1)/sudo(8) rights you just need to type this.

:w !doas tee %

Then exit the vi(1) editor with force.

:q!

Here is how it looks in the editor.

:w !doas tee %

+=+=+=+=+=+=+=+
File contents are displayed here.

Press any key to continue [: to enter more ex commands]: [ENTER]

Here is the ‘legend’ for that spell.

:      vi(1) prompt
w      write a file
!doas  invoke doas(1) command
tee    command that will be started using doas(1) command
%      tells vi(1) to use current filename

In this process the current vi(1) contents will be redirected using tee(1) with doas(1) rights to the current (open that you opened) filename.

Of course it also works in vim(1) or neovim(1) and if sudo(8) is your poison then just use sudo instead doas(1) there.

Search Contents of PDF Files

We all love plain text files then they can be searched using grep(1) for data that is interesting for us … but grep(1) does not work with PDF files … or should I say its pointless/useless to use grep(1) to search PDF files. Fortunately pdfgrep(1) command exists and works beautifully with PDF files – including colored output.

Recently FreeBSD Journal has been made free and you will like to search for bhyve articles in FreeBSD Journal issues then this is the command for you.

% cd books/unix-bsd-journal
% exa
FreeBSD Journal - 2014-01-02.pdf FreeBSD Journal - 2016-09-10.pdf
FreeBSD Journal - 2014-03-04.pdf FreeBSD Journal - 2016-11-12.pdf
FreeBSD Journal - 2014-05-06.pdf FreeBSD Journal - 2017-01-02.pdf
FreeBSD Journal - 2014-07-08.pdf FreeBSD Journal - 2017-03-04.pdf
FreeBSD Journal - 2014-09-10.pdf FreeBSD Journal - 2017-05-06.pdf
FreeBSD Journal - 2014-11-12.pdf FreeBSD Journal - 2017-07-08.pdf
FreeBSD Journal - 2015-01-02.pdf FreeBSD Journal - 2017-09-10.pdf
FreeBSD Journal - 2015-03-04.pdf FreeBSD Journal - 2017-11-12.pdf
FreeBSD Journal - 2015-05-06.pdf FreeBSD Journal - 2018-01-02.pdf
FreeBSD Journal - 2015-07-08.pdf FreeBSD Journal - 2018-03-04.pdf
FreeBSD Journal - 2015-09-10.pdf FreeBSD Journal - 2018-05-06.pdf
FreeBSD Journal - 2015-11-12.pdf FreeBSD Journal - 2018-07-08.pdf
FreeBSD Journal - 2016-01-02.pdf FreeBSD Journal - 2018-09-10.pdf
FreeBSD Journal - 2016-03-04.pdf FreeBSD Journal - 2018-11-12.pdf
FreeBSD Journal - 2016-05-06.pdf FreeBSD Journal - 2019-01-02.pdf
FreeBSD Journal - 2016-07-08.pdf

% pdfgrep -i -n bhyve *.pdf
FreeBSD Journal - 2014-01-02 - Old Release.pdf:6: machine hypervisors, such as BHy
FreeBSD Journal - 2014-01-02 - Old Release.pdf:6: BHyVe
FreeBSD Journal - 2014-01-02 - Old Release.pdf:6: BHyVe IS THE BSD Hypervisor, de
FreeBSD Journal - 2014-01-02 - Old Release.pdf:6: Grehan and Neel Natu. The desig
FreeBSD Journal - 2014-01-02 - Old Release.pdf:6: BHyVe requires Intel CPUs w
FreeBSD Journal - 2014-01-02 - Old Release.pdf:6: BHyVe appeared in FreeBSD 1
FreeBSD Journal - 2014-01-02.pdf:42: machine hypervisors, such as BHyVe, Virtual
FreeBSD Journal - 2014-01-02.pdf:42: BHyVe e d
FreeBSD Journal - 2014-01-02.pdf:42: BHyVe IS THE BSD Hypervisor, developed by P
FreeBSD Journal - 2014-01-02.pdf:42: Grehan and Neel Natu. The design goal of BH
FreeBSD Journal - 2014-01-02.pdf:42: BHyVe requires Intel CPUs with VT-x and
FreeBSD Journal - 2014-01-02.pdf:42: BHyVe appeared in FreeBSD 10-CURRENT in
(...)

Here is how it looks in the xterm(1) terminal.

xterm-pdfgrep.png

Hope that today’s pack of spells will end up useful for you.

EOF

Wallpapers from Tech Pron

The Tech Pron at Twitter – @techno_pron – is a bot that posts aesthetic tech pics. Most of these computers pictures were made on a solid (or close to it) background color so I though it may be a cool idea to create wallpapers from them.

I have picked up 20 most interesting ones and made high resolution backgrounds of them. Here is their montage.

wallpapers-oldschool-machines.jpg

To download them all just follow this wallpapers-oldschool-machines.tar.gz file.

When needed use my random_wallpaper.sh handler from the https://github.com/vermaden/scripts repository to setup random wallpaper from directory.

EOF