TITLE

Replace format built-in with format function


VERSION

  Maintainer: Damian Conway <damian@conway.org>
  Date: 15 Sep 2000
  Last Modified: 20 Sep 2000
  Mailing List: perl6-language@perl.org
  Number: 230
  Version: 3
  Status: Frozen


ABSTRACT

This RFC proposes that Perl's existing format mechanism be replaced with a standard module based on parts of the Text::Autoformat module.


DESCRIPTION

I can never remember how formats work. The specification syntax is confusing to me. And I usually don't want the formatted text going straight down some magical output stream.

It all came to a head when I was building Text::Autoformat. The smart text recognition was easy -- trying to do the final formatting with formline and $^A was just too painful.

So I created the Text::Autoformat::form subroutine. It uses a template specification than fits my brain better, it's deeply DWIMical, it's highly configurable, and it's re-entrant (so you can use a form to format another form's headers and footers, for example).

I propose that the existing format mechanism be removed from Perl 6 and be replaced with an add-in function from a standard module (based on the semantics of Text::Autoformat::form) as described in the following sections.

The Format::format function

The function itself would be called format and would be imported with a use Format call.

It takes a format (or "picture" or "template") string followed by one or more replacement values. It then interpolates those values into each picture string, and either:

In a void context, format would emit a warning: "Useless use of format in void context".

A picture string consists of sequences of the following characters:

<<<<<<<<

Left-justified field indicator. A series of two or more sequential <'s specify a left-justified field to be filled by a subsequent value.

>>>>>>>>

Right-justified field indicator. A series of two or more sequential >'s specify a right-justified field to be filled by a subsequent value.

^^^^^^^^

Centre-justified field indicator. A series of two or more sequential ^'s specify a centred field to be filled by a subsequent value.

>>>.<<<<

A numerically formatted field with the specified number of digits to either side of the decimal place. See Numerical formatting below.

[[[[[[[[

Left-justified block field indicator. Just like a <<<<<<<<< field, except it repeats as required on subsequent lines. See below.

]]]]]]]]

Right-justified block field indicator. Just like a >>>>>>>> field, except it repeats as required on subsequent lines. See below.

||||||||

Centre-justified block field indicator. Just like a ^^^^^^^^ field, except it repeats as required on subsequent lines. See below.

]]].[[[[

A numerically formatted block field with the specified number of digits to either side of the decimal place. Just like a >>>.<<<< field, except it repeats as required on subsequent lines. See below.

~

A single character field. Interpolates a single character from a subsequent data value. Repeats on subsequent lines as required.

\0

A zero-width field separator. A null character in the template field is ignored (i.e. not included in the formatted text). This allows two fields to be adjacent. For example: "[[[[[[[[[[[\0[[[[[[[[[[[". To put a literal "\0" in a formatted text, use a '~' field and pass it the data string "\0".

\

Literal escape of next character (e.g. \~ is formatted as a literal '~', not a one character wide field).

Any other character

That literal character.

Hence a typical use of format might look like this:

        $formatted = format "<[[[[[[[[[[[[[[[[>    >>>>>>>>>>>>>>>>",
                               $aphorism,          "page $page_num";

and might produce an output like this:

        <Like a camel    >            page 123
        <through the eye >
        <of a needle, so >
        <are the days of >
        <our lives       >

Note that, because every field (except a ~ field) must be at least two characters wide, the single < and > brackets in the format string are correctly interpreted as literals.

Multi-line filling

As the previous example indicates, any line with a block field continues interpolation of that field on subsequent lines until all block fields in the format have consumed all their data. Non-block fields on these lines are replaced by the appropriate number of spaces.

For example:

        $title = "On The Evil That Is Spam";
        $text1 = "How many times have you longed to smash...";
        $text2 = "...the bedevilment that is spam?";
        print format "<<<<<<<<<<<   [[[[[[[[[[[[[[[   [[[[[[[[[[",
                      $title,       $text1,           $text2;
        # prints:
        #
        #             On The Evil   How many times    ...the be- 
        #                           have you longed   devilment
        #                           to smash...       that is
        #                                             spam

Hyphenation and line-breaking

As the previous example indicates, within a block field, words are wrapped whole, unless they will not fit into the field at all, in which case they are broken and hyphenated. Simple hyphenation is used (i.e. break at the N-1th character and insert a '-'), unless a suitable alternative subroutine is specified instead.

Words will not be broken if the break would leave less than 2 characters of the word on the current line. This minimum can be varied by setting the 'minbreak' option to some positive numeric value. This value indicates the minumum total broken characters (including hyphens) that must be left on the current line.

Alternative word breaking subroutines can be specified using the "break" option in a configuration hash. For example:

        print format { break => \&my_line_breaker },
                     $template,
                     @data;

format expects any user-defined line-breaking subroutine to take three arguments (the string to be broken, the maximum permissible length of the initial section, and the total width of the field being filled). The subroutine must return a list of two strings: the initial (broken) section of the word, and the remainder of the string respectively).

For example:

        sub tilde_break = sub($$$)
        {
                (substr($_[0],0,$_[1]-1).'~', substr($_[0],$_[1]-1));
        }
        print format { break => \&tilde_break },
                     $template,
                     @data;

This causes '~' to be used as the hyphenation character.

The standard Format.pm module provides two exportable functions to simplify the use of variant hyphenation schemes.

The exportable function Format::break_with takes a single string argument and returns a reference to a sub which hyphenates with that string. Hence the previous example could be rewritten:

        use Format qw( break_wrap );
        print format { break => break_with('~') },
                     $template,
                     @data;

The exportable function Format::break_TeX takes an optional argument specifying a TeX hyphenation file and returns a reference to a sub which hyphenates using Jan Pazdziora's TeX::Hyphen module (assuming it is installed). For example:

        use Format qw( break_TeX );
        print format { break => break_TeX("lithuanian.hy") }
                     $template,
                     @data;

Note that in the previous examples there is no leading '\&' before break_with or break_TeX, since each is being directly called, arther than referred to. They each return a reference to some other suitable word-breaking subroutine.

Specifying {break = undef}> reverts format to its default line-breaking behaviour (see A note about format options and Lexically permanent options below).

The Format.pm module might also provide other predefined breaking subroutines; for example, to cope with localization and two-byte character issues such as those addressed by existing modifications to the Perl 5 format mechanism in JPerl.

Controlling line filling

format preserves the original whitespace (including newlines) from an interpolated string, unless called with certain options.

The "squeeze" option (when specified with a true value) causes any interpolated sequence of spaces and/or tabs (but not newlines) to be replaced with a single space.

The "fill" option independently causes newlines to be "squeezed".

Hence:

        $frmt = "# [[[[[[[[[[[[[[[[[[[[[";
        $data = "h  e\t \tl lo\nworld\t\t\t\t\t";
        print format $frmt, $data;
        # h  e            l lo
        # world
        print format {squeeze=>1}, $frmt, $data;
        # h e l lo
        # world
        print format {fill=>1}, $frmt, $data;
        # h  e            l lo world
        print format {squeeze=>1, fill=>1}, $frmt, $data;
        # h e l lo world

Whether or not filling or squeezing is in effect, format can also be directed to trim any extra whitespace from the end of each line it formats, using the "trim" option. If this option is specified with a true value, every line returned by format will automatically have the substitution s/[ \t]+$//gm applied to it.

Hence:

        print length format "[[[[[[[[[[", "short";
        # 11
        print length format {trim=>1}, "[[[[[[[[[[", "short";
        # 6

A note about format options

Like templates and data, option sets (passed via hash references) can be interleaved within a single format call.

For example, to leave the text in $quote unsqueezed, unfilled, and unhyphenated, whilst squeezing, filling, and hyphenating the text in $intro and $commentary:

        print format
                { squeeze=>1, fill=>1, break=>Format::break_TeX },
                '[[[[[[[[[[[[[[[[[[[[[[[[[]]]]]]]]]]]]]]]]]]]]]]]]]]',
                $intro,
                { squeeze=>0, fill=>0, break=>undef },
                '     > [[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[',
                $quote,
                { squeeze=>1, fill=>1, break=>Format::break_TeX },
                '[[[[[[[[[[[[[[[[[[[[[[[[[]]]]]]]]]]]]]]]]]]]]]]]]]]',
                $commentary;

or, more cleanly:

        $TIDY = { squeeze=>1, fill=>1, break=>Format::break_TeX };
        $RAW  = { squeeze=>0, fill=>0, break=>undef };
        print format
                $TIDY, '[[[[[[[[[[[[[[[[[[[[[[[[[]]]]]]]]]]]]]]]]]]]]]]]]]]',
                        $intro,
                $RAW,  '     > [[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[',
                               $quote;
                $TIDY, '[[[[[[[[[[[[[[[[[[[[[[[[[]]]]]]]]]]]]]]]]]]]]]]]]]]',
                        $commentary;

Note that -- in Text::Autoformat::form -- changes in options cannot currently be made between interpolated data arguments (i.e. they apply to the next complete interpolation template and all the data interpolated into it).

However, the ability to apply options to a single interpolated field might be a useful feature in the proposed Format::format. For example, to create two columns where the left column is (only) filled and the right is (only) squeezed:

        print format
                '[[[[[[[[[[[[[[[[[[[    [[[[[[[[[[[[[[[[[[[',
                { squeeze=>0, fill=>1 }, $column1_text,
                { squeeze=>1, fill=>0 }, $column2_text;

How format consumes strings

Within a single call to format fields consume data text as they format it, so the following:

        $text = "a line of text to be formatted over three lines";
        print format "<<<<<<<<<<\n  <<<<<<<<\n    <<<<<<\n",
                      $text,        $text,        $text;

produces:

        a line of
          text to
            be fo-

not:

        a line of
          a line 
            a line

To achieve the latter effect, the variable arguments must be converted to independent literals (by double-quoted interpolation):

        print format "<<<<<<<<<<\n  <<<<<<<<\n    <<<<<<\n",
                     "$text",      "$text",      "$text";

Although values passed from variable arguments are progressively consumed within format, the values of the original variables passed to format are not altered. In other words, internally format operates on a copy of each variable's string. Hence:

        print format "<<<<<<<<<<\n  <<<<<<<<\n    <<<<<<\n",
                      $text,        $text,        $text;
        print $text, "\n";

will print:

        a line of
          text to
            be fo-
        a line of text to be formatted over three lines

To cause format to visibly consume the values of the original variables passed to it, they must be passed as references. Thus:

        print format "<<<<<<<<<<\n  <<<<<<<<\n    <<<<<<\n",
                      \$text,       \$text,       \$text;
        print $text, "\n";

will print:

        a line of
          text to
            be fo-
        rmatted over three lines

Note that, for safety, the "non-consuming" behaviour takes precedence, so if a variable is passed to format both by reference and by value, its final value will be unchanged.

Numerical formatting

The ">>>.<<<" and "]]].[[[" field specifiers may be used to format numeric values about a fixed decimal place marker. For example:

        print format '(>>>>>.<<)', <<EONUMS;
           1
           1.0
           1.001
           1.009
           123.456
           1234567
           one two
        EONUMS

would print:


        (    1.0 )
        (    1.0 )
        (    1.00)
        (    1.01)
        (  123.46)
        (#####.##)
        (?????.??)
        (?????.??)

Fractions are rounded (a la sprintf) to the specified number of places after the decimal, but only the minimal number of significant digits are shown. That's why, in the above example, 1 and 1.0 are formatted as "1.0", whilst 1.001 is formatted as "1.00".

It is possible to specify that the maximal number of decimal places always be used, by giving the configuration option 'numeric' the value 'AllPlaces'. For example:

        print format { numeric => 'AllPlaces' },
                     '(>>>>>.<<)', <<'EONUMS';
           1
           1.0
        EONUMS

would print:


        (    1.00)
        (    1.00)

Note that although decimal digits are rounded to fit the specified width, the integral part of a number is never modified. If there are not enough places before the decimal place to represent the number, the entire number is replaced with hashes.

If a non-numeric sequence is passed as data for a numeric field, it is formatted as a series of question marks. This querulous behaviour can be changed by giving the configuration option 'numeric' the value 'SkipNaN', in which case, any invalid numeric data is simply ignored. For example:

        print format { numeric => 'SkipNaN' },
                     '(>>>>>.<<)',
                     <<EONUMS;
           1
           two three
           4
        EONUMS

would print:


        (    1.0 )
        (    4.0 )

Other values for the "numeric" option might also be provided, such as:

'Commas'

Locale-sensitive commification of numbers. For example

        print format {numeric=>'Commas'},
                     ']]]]]]]]].[',
                     [100,1000,10000,1000000,10000000];
        #       100.0
        #     1,000.0
        #    10,000.0
        # 1,000,000.0
        # #########.#
'Currency'

Cause the locale's currency sign to be properly inserted as part of each interpolation. This option would also imply 'AllPlaces'. For example:

        print format {numeric=>'Currency'},
                     ']]]]].[[',
                     [1,100,1000,10000];
        #    $1.00
        #  $100.00
        # $1000.00
        # #####.##
'Fill(char)'

Use the specified character as the fill character, rather than spaces. For example:

        print format {numeric=>'Fill(*)'},
                     ']]]]].[[[[',
                     [1,10.01,100.001,1000.0001,10000.00001],
                     {numeric=>'Fill(+)'},
                     ']]]]].[[[[',
                     [10,100,10000];
        # ****1.0***
        # ***10.01**
        # **100.001*
        # *1000.0001
        # 10000.0000
        # +++10.0+++
        # ++100.0+++
        # 10000.0+++

Filling block fields with lists of values

If the data provided for a particular field is an array reference, then format automatically joins the elements of the array into a single string, separating each element with a newline character. As a result, a call like this:

        @values = qw( 1 10 100 1000 );
        print format "(]]]].[[)", \@values;

will print out

         (   1.00)
         (  10.00)
         ( 100.00)
         (1000.00)

as might be expected.

Note that, because arrays must be passed using a reference, their original contents are consumed by format, just like the contents of scalars passed by reference.

To avoid having an array consumed by format, pass it as an anonymous array:

        print format "(]]]].[[)", [@values];

Headers, footers, and pages

The format subroutine can also insert headers, footers, and page-feeds as it formats. These features are controlled by the "pagenum", "pagelen", "header", "footer", and "pagefeed" options.

The "pagenum" option takes a scalar value or a reference to a scalar variable and starts page numbering at that value. If a reference to a scalar variable is specified, the value of that variable is updated as the formatting proceeds, so that the next page number is available in the variable after formatting. This can be useful for multi-part reports.

The "pagelen" option specifies the total number of lines in a page (including headers and footers, but not page-feeds).

If the "header" option is specified with a string value, that string is used as the header of every page generated. If it is specified as a reference to a subroutine, that subroutine is called at the start of every page and its return value used as the header string. When called, the subroutine is passed the current page number.

Likewise, if the "footer" option is specified with a string value, that string is used as the footer of every page generated. If it is specified as a reference to a subroutine, that subroutine is called at the start of every page and its return value used as the footer string. When called, the footer subroutine is passed the current page number.

Both the header and footer options can -- alternatively -- be specified as hash references. In this case the hash entries for the keys "left", "centre" (or "center"), and "right" specify what is to appear on the left, centre, and right of the header/footer. The entry for the key "width" specifies how wide the footer is to be. The "left", "centre", and "right" values may be literal strings or subroutines (just as a normal header/footer specification may be.) See the second example below.

The "pagefeed" option acts in exactly the same way as "header" and "footer", using a literal string or a subroutine to produce a pagefeed string (but not necessarily "\014"), which is appended after the footer. Note however that the pagefeed is not counted as part of the page length, nor can it be specified as a left-centre-right hash.

The header, footer, and pagefeed page components are recomputed at the start of each new page, before the page contents are formatted (recomputing the header and footer makes it possible to determine how many lines of data to format so as to adhere to the specified page length).

When the call to format is complete and the data has been fully formatted, the footer subroutine is called one last time, with an extra argument of 1. The string returned by this final call is used as the final footer.

So for example, a series of 60-line pages with appropriate headers and footers might be formatted like so:

        print format
                { header => sub { "Page $_[0]\n\n" },
                  footer => sub { return "" if $_[1];
                                  "-"x50."\n".format ">"x50", "...".($_[0]+1);
                                },
                  pagefeed => "\n\n",
                  pagelen  => 60
                },
                $template,
                @data;

Note the recursive use of format within the "footer" option.

As a second example, to set up headers and footers such that the running head is right justified in the header and the page number (starting at 7) is centred in the footer:

        print format
                { header => { right => "Running head" },
                  footer => { centre => sub { "Page $_[0]" } },
                  pagelen  => 60,
                  pagenum  => 7,
                },
                $template,
                @data;

Lexically permanent options

If use Format is called with a hash reference as an argument, the entries of that hash specify options that are to be made the defaults until the end of the current lexical scope. For example, to cause format to always squeeze and trim whitespace but respect newlines, the following line would be placed at the start of the source file:

        use Format { squeeze=>1, fill=>0, trim=>1 };

With these defaults in effect, filling could be turned on and trimming disabled within a particular subroutine like so:

        sub particular {
                use Format { fill => 1, trim => 0 };
                # do filled, untrimmed formatting here
        } # format's defaults revert to previous values at end of scope

format examples

As an example of the use of format, the following:

        $count = 1;
        $text = "A big long piece of text to be formatted exquisitely";
        print format q{
                ||||  <<<<<<<<<<
                ----------------
                ^^^^  ]]]]]]]]]]\|
                                =
                ]]].[[[
                
        }, $count, $text, $count+11, $text, "123 123.4\n123.456789";

produces the following output:

                 1    A big long
                ----------------
                 12     piece of|
                      text to be|
                       formatted|
                      exquisite-|
                              ly|
                                =
                123.0
                123.4
                123.456

Alternatively, picture strings and replacement values can be interleaved. For example:

        $report = format
                'Name           Rank    Serial Number',
                '====           ====    =============',
                '<<<<<<<<<<<<<  ^^^^    <<<<<<<<<<<<<',
                 $name,         $rank,  $serial_number,
                ''
                'Age    Sex     Description',
                '===    ===     =====================',
                '^^^    ^^^     [[[[[[[[[[[]]]]]]]]]]',
                 $age,  $sex,   $description;


MIGRATION ISSUES

Non-trivial, but do-able.

I also intend to extract Text::Autoformat::form from Text::Autoformat and create a new Perl 5 module: Text::Reformat.pm. This would provide the same functionality (and a migration path) for Perl 5 users.


IMPLEMENTATION

See Text::Autoformat module.

The proposed built-in/add-in ought to be reimplemented in C for speed, with Text::Autoformat::form (or Text::Reformat::format) as a reference.


REFERENCES

None.