HOME  |  GIT Overview  |  Script-Archive: (docs) : (wiki) : (git)  |  ...

Peter's Script-Archive - Shell Functions and Setup

This section is home of shell support functions and shell setup.

ksh93: Note that the korn shell ksh 93s+ suffers from various memory management issues on 64bit incl. eating return codes (possibly restricted to x86_64, Linux and the January08 AST release) for Ubuntu intrepid, jaunty and Centos 5.3. Karmic will be the May09 AST release and has a ksh93t+ that plays nice on 64bit (bzr branch lp:ubuntu/karmic/ksh, there doesn't seem to be a PPA). If you're affected, just place the official standalone ksh binary early in your PATH (if you're evil, you might also consider /bin/ksh93 for replacement). It doesn't require the remainder of the AST stuff: http://www.research.att.com/~gsf/cgi-bin/download.cgi?action=list&name=ksh)

set -o pipefail - Workaround: Some of the scripts in other sections use set -o pipefail and thus require a proper ksh (ksh93!) or bash (e.g. emv, sfre). If you don't mind changing the scripts and efficiency, you can change the invocations to instead source pipestatus, which works for all POSIX shells.

Check that you're using at least the version 0.6, as this release added support for pathological and unsafe filenames containing blanks, quotes and worse (including embedded newlines) as part of filenames. cat | nl thus becomes source PATH-TO-PIPESTATUS; runpipe_base cat '|' nl. I tried breaking Aleksey's changes, only to report to him my utter lack of success - so it's looking good now :). A note on nesting pipes: Define complex commands (like nested runpipe-invocations, pipes, lists) as functions and invoke the functions as simple commands in the pipeline stages of the outer runpipe statement.

Some of the more interesting functions

Git View of this section / Download

• cd: portable version of nico_cd for bash and ksh (NOT pd-ksh and derivatives; try looking for e.g. ksh93 instead) - a proper directory stack with the ability to grep and edit the stack (cd.nico_cd_for_bash_and_ksh.func)
• cd: an extension of Halchenko's dyndirstack - a dynamic dirstack building upon the bash builtins with completion support (cd.dyndirstack_bash.func)
• cd: see also mt_dir elsewhere in this archive (key-word based cd aliases)
• _echo, _warn (bash actually manages to have no echo/print honoring -- as end-of-options-option; the only reasonably safe function happens to be printf; 00_lib.func)
• f2a is a shell function reading a file into an array, allowing e.g. safe argument passing (00_lib.func):
  ksh93$ find . -print | f2a -a ARGS; less "$ARGS[@]"
  bash$  f2a -a ARGS < <(find . -print); less "$ARGS[@]"
• uniqpath deals with :-separated strings and removes duplicates and optionally '.' and '' (both of which are a security risk e.g. in $PATH; 00_lib.func)
• grep.func / gr_*: simple line-based filters in grep.func, e.g. uniq (gr_UNIQ; w/o sort), a duplicate line suppression filter with a persistant cache (gr_NEW), and some support for easy use of Schwartzian transforms within pipes (gr_MAP | ... | gr_UNMAP: remember and retrieve lines a persistant cache, regardless of changes, provided the line ID embedded by gr_MAP survives until gr_UNMAP. gr_MAPORDER orders the lines as per the occurance of their IDs in the cache, thus allowing transient reordering in addition to the transient line modifications permitted by gr_MAP/gr_UNMAP).
  Shell, with some perl (mostly one-liners); the filters lack both efficiency and support for \0-lines.
• gr_MIXED2UTF: line-based hack to convert a file with a mix of lines in latin1 and utf8 file to utf8 (heuristic based on the "invalid char" \x{FFFD}). Helpful when grepping e.g. over a mix of latin1/utf8 files.
• rss.skim.example - a fairly raw and basic example pipeline using shell functions/Perl one-liners gr_UNIQ and gr_NEW (see the grep.func file) to throw together an rss "reader" for interactive item selection and forwarding of links to firefox.
  The main difference to ordinary rss readers is the implementation of a proper regex-based kill file / select file, combined with caching/suppression of previously seen items/"threads", reminiscent of the kill/select/threading(/thread-supression) features found in the nn news-reader. Which are an absolute necessity for huge feeds (say those with more than 100 items/day, grouped in a large number of "threads" of related items). However the "full-featured" rss-reader applications always seem to lack powerful kill/select/threading. This includes apps like newsfox or even newsbeuter (oh, please do give me at least filtering options to automatically set flags, instead of just merely offering to filter item display based on manually set flags :( ...).
  The file can be sourced to use the function rss_skim for feeds of your choice. It depends on other stuff from the script-archive, but those can be replaced with just egrep and a few simple perl/ruby/python one-liners for basic text mangling.
  This example pipe is probably also a good starting point for a small pipe implementing rss-reader plugins that filter remote rss-feeds. This would also fix the usual lack of powerful select/kill-files of rss-reader applications. AFAIR at least newsbeuter should support such program-generated feeds.
  All in all, this is a nice example of the power of the concepts behind UNIX such as pipes and orthogonality: throw a few rather basic commands plus a pipe to get a customized time-saver of vastly different functionality.

HOME  |  GIT Overview  |  Script-Archive: (docs) : (wiki) : (git)  |  ...

jakobi(at)acm.org, 2009-07 - 2012-03