HOME | GIT Overview | Script-Archive: (docs) : (wiki) : (git) | ...

Peter's Script-Archive - List Processing incl. Mass-Editing

This toolbox compartment contains the heavy lifting tools for generic pipeline 'line'-based text processing.

About Stream-Editing aka Mass-Editing

I think extending the range of grep usability to allow mass-editing files makes for a nice and powerful concept. This is demonstrated in Grep.xchange below, provided the editing is more or less contained in the grepped lines. But there are also other very useful mass-editing (=non-interactive aka stream-editing) methods to add to your toolbox:

From perl -i.bak -lne ... and sed -i.bak -e ... to a more mature implementation of stream-editing such as Lee Eakin's ped, which is somewhat like a sed-done-right rewrite in perl, with flock and everything. Then there are more generic apps, that in a specific domain are very suitable for mass-edits. Let me give just some examples from the field of system-administration, more specific system and network configuration management: cfengine, puppet, or the 'tool cluster' of augeas/pad/boomerang (PEGs and lenses!).

From patching to vimming: Don't forget about patch (and check out the way it is used by Grep.xchange) and a true classic interactive/non-interactive line editor that provided the first base format for patch: ed; its ex successor embedded within e.g. vim is still quite similar. With the vimscript script from this very archive you can extend the usable stream-editing command-set to most of vim's commands including normal mode! Just be a bit careful and turn off unnecessary niceties like syntax-highlighting or unlimited undo while editing log files in the 100MB region and beyond. Combined with an external language - vim's embedded Python interface is probably the most stable - things get truly interesting. That's not to say that vim's builtin language is a slouch, but still, vimscript is not quite like Python, Perl or Lisp for elegance, expressiveness, speed of writing or community support and range of available modules.

Let's conclude this tiny and very subjective overview with a little gem of an OR-article about perl mass-editing techniques.

Mass-Editing and Regular Expressions

Non-interactive editing requires some way to describe locations and ranges in files. Usually this is done with (sets of) both line-numbers and regular expressions. In this context you can think of the boolean regular expressions of Grep.pm below as just a more convenient way to combine sub-expressions.

Newer developments to watch: The more recent PEGs (e.g. in perl6/Rakudo), tree parsers+transformations, and lenses are also very fun stuff and do extend the Reach Of The GREP and its expressiveness, while reducing the -ahem- line-noise aspect.

A Challenge: Any one up to implementing a prolog interpreter and using an extended regex/peg engine itself as the implementation of the backtracking code (mail me if you did!) :)? The current feature set of e.g. full perl5.10 regexes goes way beyond classic DFA-limits...

If you want to go to the real basics of regular expressions and think about the way PCREs (and worse, Perl5.10's own regex implementation) defy and bend the classics, do check out the computer science terms of EBNF, recursive descent parsing, deterministic finite automatons, the Chomsky hierarchy of grammars (esp. Chomsky-3), the Turing-Machine, and regular expressions. Probably best start from back to front and choose wikipedia as the starting point, but don't forget to look at the seminal papers listed as references.

Commands

Git View of this section / Download

cat0 converts from and to \0-terminated lines. Depending on invocation it's either a glorified but faster to type perl -ne 'chomp; print "$_\0"' or a perl -0ne 's/\0\z//; print "$_\n"'. The GNU tools grep z|-Z], sort -z, xargs -0 already provide support. nl0 reimplements a primitive nl with support, and xargs0 merely exists to allow faster typing of ...| cat0|xargs0: This allows me to forget about IFS, quoting and whitespace in filenames and still have a well-behaved xargs doing the 'right thing' (provided you don't insist on using file names with embedded '\n').
cat0par converts paragraphs into \0 lines and also allows for arbitrary regex-based splitting. An overgrown perl --0ne/perl -0777ne.
Grep.pm is a fork of tcgrep, which nowadays has learned about context, matching from start or end of file (line count or byte count), size limits, and highlighting. It extends matching to boolean expressions structuring regular expressions or even arbitrary perl scraps. It can perform basic stemming and synonym-expansion in regular expressions (using expansyn). It also handles \0-lines, paragraphs and file slurping, as well as directory recursion and compressed files (both as in tcgrep). Dependencies: expansyn. It can masquerade as a perl module. Published on freshmeat (together with Grep.xchange).
How about a kind of emv not for mass-renaming filenames, but for mass-editing file-contents?
Grep.xchange bridges the gap between the grep and patch commands: it takes grep output and a perl expression and uses both to change an arbitrary number of files by applying the expression at each of the grep matches, while generating a patch file as its UNDO / REDO log. It's a bit paranoid, so combined with edited grep output (or Grep.pm -C -LINES marked grep context output), the 3-Step Grep.pm|vim.pipe|Grep.xchange is a rather safe way to mass-edit files. Dependencies: For flock-locking edited files, you also need to copy Compact_pm::Flock.pm; otherwise you can disable it by commenting the use statement.
expansyn is the command implementing basic English stemming and synonym expansion for both Grep.pm / tagls. It uses a configuration file expansyn.cfg containing the synonymlists in perl syntax. I wrote it for automatic expansion of some computer science terms I always seem to use in parallel.