This is recode.info, produced by makeinfo version 4.0 from recode.texi.

INFO-DIR-SECTION Internationalization and character sets
START-INFO-DIR-ENTRY
* recode: (recode).     Conversion between character sets and surfaces.
END-INFO-DIR-ENTRY

   This file documents the `recode' command, which has the purpose of
converting files between various character sets and surfaces.

   Copyright (C) 1990, 93, 94, 96, 97, 98, 99, 00 Free Software
Foundation, Inc.

   Permission is granted to make and distribute verbatim copies of this
manual provided the copyright notice and this permission notice are
preserved on all copies.

   Permission is granted to copy and distribute modified versions of
this manual under the conditions for verbatim copying, provided that
the entire resulting derived work is distributed under the terms of a
permission notice identical to this one.

   Permission is granted to copy and distribute translations of this
manual into another language, under the above conditions for modified
versions, except that this permission notice may be stated in a
translation approved by the Foundation.


File: recode.info,  Node: HTML,  Next: LaTeX,  Prev: Miscellaneous,  Up: Miscellaneous

World Wide Web representations
==============================

   Character entities have been introduced by SGML and made widely
popular through HTML, the markup language in use for the World Wide
Web, or Web or WWW for short.  For representing _unusual_ characters,
HTML texts use special sequences, beginning with an ampersand `&' and
ending with a semicolon `;'.  The sequence may itself start with a
number sigh `#' and be followed by digits, so forming a "numeric
character reference", or else be an alphabetic identifier, so forming a
"character entity reference".

   The HTML standards have been revised into different HTML levels over
time, and the list of allowable character entities differ in them.  The
later XML, meant to simplify many things, has an option
(`standalone=yes') which much restricts that list.  The `recode'
library is able to convert character references between their mnemonic
form and their numeric form, depending on aimed HTML standard level.
It also can, of course, convert between HTML and various other charsets.

   Here is a list of those HTML variants which `recode' supports.  Some
notes have been provided by François Yergeau <yergeau@alis.com>.

`XML-standalone'
     This charset is available in `recode' under the name
     `XML-standalone', with `h0' as an acceptable alias.  It is
     documented in section 4.1 of `http://www.w3.org/TR/REC-xml'.  It
     only knows `&amp;', `&gt;', `&lt;', `&quot;' and `&apos;'.

`HTML_1.1'
     This charset is available in `recode' under the name `HTML_1.1',
     with `h1' as an acceptable alias.  HTML 1.0 was never really
     documented.

`HTML_2.0'
     This charset is available in `recode' under the name `HTML_2.0',
     and has `RFC1866', `1866' and `h2' for aliases.  HTML 2.0 entities
     are listed in RFC 1866.  Basically, there is an entity for each
     _alphabetical_ character in the right part of ISO 8859-1.  In
     addition, there are four entities for syntax-significant ASCII
     characters: `&amp;', `&gt;', `&lt;' and `&quot;'.

`HTML-i18n'
     This charset is available in `recode' under the name `HTML-i18n',
     and has `RFC2070' and `2070' for aliases.  RFC 2070 added entities
     to cover the whole right part of ISO 8859-1.  The list is
     conveniently accessible at
     `http://www.alis.com:8085/ietf/html/html-latin1.sgml'.  In
     addition, four i18n-related entities were added: `&zwnj;'
     (`&#8204;'), `&zwj;' (`&#8205;'), `&lrm;' (`&#8206') and `&rlm;'
     (`&#8207;').

`HTML_3.2'
     This charset is available in `recode' under the name `HTML_3.2',
     with `h3' as an acceptable alias.  HTML 3.2
     (http://www.w3.org/TR/REC-html32.html) took up the full Latin-1
     list but not the i18n-related entities from RFC 2070.

`HTML_4.0'
     This charset is available in `recode' under the name `HTML_4.0',
     and has `h4' and `h' for aliases.  Beware that the particular
     alias `h' is not _tied_ to HTML 4.0, but to the highest HTML level
     supported by `recode'; so it might later represent HTML level 5 if
     this is ever created.  HTML 4.0 (http://www.w3.org/TR/REC-html40/)
     has the whole Latin-1 list, a set of entities for symbols,
     mathematical symbols, and Greek letters, and another set for
     markup-significant and internationalization characters comprising
     the 4 ASCII entities, the 4 i18n-related from RFC 2070 plus some
     more.  See `http://www.w3.org/TR/REC-html40/sgml/entities.html'.

   Printable characters from Latin-1 may be used directly in an HTML
text.  However, partly because people have deficient keyboards, partly
because people want to transmit HTML texts over non 8-bit clean
channels while not using MIME, it is common (yet debatable) to use
character entity references even for Latin-1 characters, when they fall
outside ASCII (that is, when they have the 8th bit set).

   When you recode from another charset to `HTML', beware that all
occurrences of double quotes, ampersands, and left or right angle
brackets are translated into special sequences.  However, in practice,
people often use ampersands and angle brackets in the other charset for
introducing HTML commands, compromising it: it is not pure HTML, not it
is pure other charset.  These particular translations can be rather
inconvenient, they may be specifically inhibited through the command
option `-d' (*note Mixed::).

   Codes not having a mnemonic entity are output by `recode' using the
`&#NNN;' notation, where NNN is a decimal representation of the UCS
code value.  When there is an entity name for a character, it is always
preferred over a numeric character reference.  ASCII printable
characters are always generated directly.  So is the newline.  While
reading HTML, `recode' supports numeric character reference as alternate
writings, even when written as hexadecimal numbers, as in `&#xfffd'.
This is documented in:

     http://www.w3.org/TR/REC-html40/intro/sgmltut.html#h-3.2.3

   When `recode' translates to HTML, the translation occurs according to
the HTML level as selected by the goal charset.  When translating _from_
HTML, `recode' not only accepts the character entity references known at
that level, but also those of all other levels, as well as a few
alternative special sequences, to be forgiving to files using other
HTML standards.

   The `recode' program can be used to _normalise_ an HTML file using
oldish conventions.  For example, it accepts `&AE;', as this once was a
valid writing, somewhere.  However, it should always produce `&AElig;'
instead of `&AE;'.  Yet, this is not completely true.  If one does:

     recode h3..h3 < INPUT

the operation will be optimised into a mere copy, and you can get `&AE;'
this way, if you had some in your input file.  But if you explicitly
defeat the optimisation, like this maybe:

     recode h3..u2,u2..h3 < INPUT

then `&AE;' should be normalised into `&AElig;' by the operation.


File: recode.info,  Node: LaTeX,  Next: Texinfo,  Prev: HTML,  Up: Miscellaneous

LaTeX macro calls
=================

   This charset is available in `recode' under the name `LaTeX' and has
`ltex' as an alias.  It is used for ASCII files coded to be read by
LaTeX or, in certain cases, by TeX.

   Whenever you recode from another charset to `LaTeX', beware that all
occurrences of backslashes `\' are translated into the string
`\backslash{}'.  However, in practice, people often use backslashes in
the other charset for introducing TeX commands, compromising it: it is
not pure TeX, nor it is pure other charset.  This translation of
backslashes into `\backslash{}' can be rather inconvenient, it may be
inhibited through the command option `-d' (*note Mixed::).


File: recode.info,  Node: Texinfo,  Next: Vietnamese,  Prev: LaTeX,  Up: Miscellaneous

GNU project documentation files
===============================

   This charset is available in `recode' under the name `Texinfo' and
has `texi' and `ti' for aliases.  It is used by the GNU project for its
documentation.  Texinfo files may be converted into Info files by the
`makeinfo' program and into nice printed manuals by the TeX system.

   Even if `recode' may transform other charsets to Texinfo, it may not
read Texinfo files yet.  In these times, usages are also changing
between versions of Texinfo, and `recode' only partially succeeds in
correctly following these changes.  So, for now, Texinfo support in
`recode' should be considered as work still in progress (!).


File: recode.info,  Node: Vietnamese,  Next: African,  Prev: Texinfo,  Up: Miscellaneous

Vietnamese charsets
===================

   We are currently experimenting the implementation, in `recode', of a
few character sets and transliterated forms to handle the Vietnamese
language.  They are quite briefly summarised, here.

`TCVN'
     The TCVN charset has an incomplete name.  It might be one of the
     three charset `VN1', `VN2' or `VN3'.  Yes `VN2' might be a second
     version of `VISCII'.  To be clarified.

`VISCII'
     This is an 8-bit character set which seems to be rather popular for
     writing Vietnamese.

`VPS'
     This is an 8-bit character set for Vietnamese.  No much reference.

`VIQR'
     The VIQR convention is a 7-bit, `ASCII' transliteration for
     Vietnamese.

`VNI'
     The VNI convention is a 8-bit, `Latin-1' transliteration for
     Vietnamese.

   Still lacking for Vietnamese in `recode', are the charsets `CP1129'
and `CP1258'.


File: recode.info,  Node: African,  Next: Others,  Prev: Vietnamese,  Up: Miscellaneous

African charsets
================

   Some African character sets are available for a few languages, when
these are heavily used in countries where French is also currently
spoken.

   One African charset is usable for Bambara, Ewondo and Fulfude, as
well as for French.  This charset is available in `recode' under the
name `AFRFUL-102-BPI_OCIL'.  Accepted aliases are `bambara', `bra',
`ewondo' and `fulfude'.  Transliterated forms of the same are available
under the name `AFRFUL-103-BPI_OCIL'.  Accepted aliases are
`t-bambara', `t-bra', `t-ewondo' and `t-fulfude'.

   Another African charset is usable for Lingala, Sango and Wolof, as
well as for French.  This charset is available in `recode' under the
name `AFRLIN-104-BPI_OCIL'.  Accepted aliases are `lingala', `lin',
`sango' and `wolof'.  Transliterated forms of the same are available
under the name `AFRLIN-105-BPI_OCIL'.  Accepted aliases are
`t-lingala', `t-lin', `t-sango' and `t-wolof'.

   To ease exchange with `ISO-8859-1', there is a charset conveying
transliterated forms for Latin-1 in a way which is compatible with the
other African charsets in this series.  This charset is available in
`recode' under the name `AFRL1-101-BPI_OCIL'.  Accepted aliases are
`t-fra' and `t-francais'.


File: recode.info,  Node: Others,  Next: Texte,  Prev: African,  Up: Miscellaneous

Cyrillic and other charsets
===========================

   The following Cyrillic charsets are already available in `recode'
through RFC 1345 tables: `CP1251' with aliases `1251', ` ms-cyrl' and
`windows-1251'; `CSN_369103' with aliases `ISO-IR-139' and `KOI8_L2';
`ECMA-cyrillic' with aliases `ECMA-113', `ECMA-113:1986' and
`iso-ir-111', `IBM880' with aliases `880', `CP880' and
`EBCDIC-Cyrillic'; `INIS-cyrillic' with alias `iso-ir-51'; `ISO-8859-5'
with aliases `cyrillic', ` ISO-8859-5:1988' and `iso-ir-144'; `KOI-7';
`KOI-8' with alias `GOST_19768-74'; `KOI8-R'; `KOI8-RU' and finally
`KOI8-U'.

   There seems to remain some confusion in Roman charsets for Cyrillic
languages, and because a few users requested it repeatedly, `recode'
now offers special services in that area.  Consider these charsets as
experimental and debatable, as the extraneous tables describing them are
still a bit fuzzy or non-standard.  Hopefully, in the long run, these
charsets will be covered in Keld Simonsen's works to the satisfaction of
everybody, and this section will merely disappear.

`KEYBCS2'
     This charset is available under the name `KEYBCS2', with
     `Kamenicky' as an accepted alias.

`CORK'
     This charset is available under the name `CORK', with `T1' as an
     accepted alias.

`KOI-8_CS2'
     This charset is available under the name `KOI-8_CS2'.


File: recode.info,  Node: Texte,  Next: Mule,  Prev: Others,  Up: Miscellaneous

Easy French conventions
=======================

   This charset is available in `recode' under the name `Texte' and has
`txte' for an alias.  It is a seven bits code, identical to `ASCII-BS',
save for French diacritics which are noted using a slightly different
convention.

   At text entry time, these conventions provide a little speed up.  At
read time, they slightly improve the readability over a few alternate
ways of coding diacritics.  Of course, it would better to have a
specialised keyboard to make direct eight bits entries and fonts for
immediately displaying eight bit ISO Latin-1 characters.  But not
everybody is so fortunate.  In a few mailing environments, and sadly
enough, it still happens that the eight bit is often willing-fully
destroyed.

   Easy French has been in use in France for a while.  I only slightly
adapted it (the diaeresis option) to make it more comfortable to several
usages in Que'bec originating from Universite' de Montre'al.  In fact,
the main problem for me was not to necessarily to invent Easy French,
but to recognise the "best" convention to use, (best is not being
defined, here) and to try to solve the main pitfalls associated with
the selected convention.  Shortly said, we have:

`e''
     for `e' (and some other vowels) with an acute accent,

`e`'
     for `e' (and some other vowels) with a grave accent,

`e^'
     for `e' (and some other vowels) with a circumflex accent,

`e"'
     for `e' (and some other vowels) with a diaeresis,

`c,'
     for `c' with a cedilla.

There is no attempt at expressing the `ae' and `oe' diphthongs.  French
also uses tildes over `n' and `a', but seldomly, and this is not
represented either.  In some countries, `:' is used instead of `"' to
mark diaeresis.  `recode' supports only one convention per call,
depending on the `-c' option of the `recode' command.  French quotes
(sometimes called "angle quotes") are noted the same way English quotes
are noted in TeX, _id est_ by ```' and `'''.  No effort has been put to
preserve Latin ligatures (`ae', `oe') which are representable in
several other charsets.  So, these ligatures may be lost through Easy
French conventions.

   The convention is prone to losing information, because the diacritic
meaning overloads some characters that already have other uses.  To
alleviate this, some knowledge of the French language is boosted into
the recognition routines.  So, the following subtleties are
systematically obeyed by the various recognisers.

  1. A comma which follows a `c' is interpreted as a cedilla only if it
     is followed by one of the vowels `a', `o' or `u'.

  2. A single quote which follows a `e' does not necessarily means an
     acute accent if it is followed by a single other one.  For example:

    `e''
          will give an `e' with an acute accent.

    `e'''
          will give a simple `e', with a closing quotation mark.

    `e''''
          will give an `e' with an acute accent, followed by a closing
          quotation mark.

     There is a problem induced by this convention if there are English
     quotations with a French text.  In sentences like:

          There's a meeting at Archie's restaurant.

     the single quotes will be mistaken twice for acute accents.  So
     English contractions and suffix possessives could be mangled.

  3. A double quote or colon, depending on `-c' option, which follows a
     vowel is interpreted as diaeresis only if it is followed by
     another letter.  But there are in French several words that _end_
     with a diaeresis, and the `recode' library is aware of them.
     There are words ending in "igue", either feminine words without a
     relative masculine (besaigue" and cigue"), or feminine words with
     a relative masculine(1) (aigue", ambigue", contigue", exigue",
     subaigue" and suraigue").  There are also words not ending in
     "igue", but instead, either ending by "i"(2) (ai", congai", goi",
     hai"kai", inoui", sai", samurai", thai" and tokai"), ending by "e"
     (canoe") or ending by "u"(3) (Esau").

     Just to complete this topic, note that it would be wrong to make a
     rule for all words ending in "igue" as needing a diaerisis, as
     there are counter-examples (becfigue, be`sigue, bigue, bordigue,
     bourdigue, brigue, contre-digue, digue, d'intrigue, fatigue,
     figue, garrigue, gigue, igue, intrigue, ligue, prodigue, sarigue
     and zigue).

   ---------- Footnotes ----------

   (1) There are supposed to be seven words in this case.  So, one is
missing.

   (2) Look at one of the following sentences (the second has to be
interpreted with the `-c' option):

     "Ai"e!  Voici le proble`me que j'ai"
     Ai:e!  Voici le proble`me que j'ai:

   There is an ambiguity between an ai", the small animal, and the
indicative future of _avoir_ (first person singular), when followed by
what could be a diaeresis mark.  Hopefully, the case is solved by the
fact that an apostrophe always precedes the verb and almost never the
animal.

   (3) I did not pay attention to proper nouns, but this one showed up
as being fairly evident.


File: recode.info,  Node: Mule,  Prev: Texte,  Up: Miscellaneous

Mule as a multiplexed charset
=============================

   This version of `recode' barely starts supporting multiplexed or
super-charsets, that is, those encoding methods by which a single text
stream may contain a combination of more than one constituent charset.
The only multiplexed charset in `recode' is `Mule', and even then, it
is only very partially implemented: the only correspondence available
is with `Latin-1'.  The author fastly implemented this only because he
needed this for himself.  However, it is intended that Mule support to
become more real in subsequent releases of `recode'.

   Multiplexed charsets are not to be confused with mixed charset texts
(*note Mixed::).  For mixed charset input, the rules allowing to
distinguish which charset is current, at any given place, are kind of
informal, and driven from the semantics of what the file contains.  On
the other side, multiplexed charsets are _designed_ to be interpreted
fairly precisely, and quite independently of any informational context.

   The spelling `Mule' originally stands for `_mul_tilingual
_e_nhancement to GNU Emacs', it is the result of a collective effort
orchestrated by Handa Ken'ichi since 1993.  When `Mule' got rewritten
in the main development stream of GNU Emacs 20, the FSF renamed it
`MULE', meaning `_mul_tilingual _e_nvironment in GNU Emacs'.  Even if
the charset `Mule' is meant to stay internal to GNU Emacs, it sometimes
breaks loose in external files, and as a consequence, a recoding tool
is sometimes needed.  Within Emacs, `Mule' comes with `Leim', which
stands for `_l_ibraries of _e_macs _i_nput _m_ethods'.  One of these
libraries is named `quail'(1).

   ---------- Footnotes ----------

   (1) Usually, quail means quail egg in Japanese, while egg alone is
usually chicken egg.  Both quail egg and chicken egg are popular food
in Japan.  The `quail' input system has been named because it is
smaller that the previous `EGG' system.  As for `EGG', it is the
translation of `TAMAGO'.  This word comes from the Japanese sentence
`_ta_kusan _ma_tasete _go_mennasai', meaning `sorry to have let you
wait so long'.  Of course, the publication of `EGG' has been delayed
many times...  (Story by Takahashi Naoto)


File: recode.info,  Node: Surfaces,  Next: Internals,  Prev: Miscellaneous,  Up: Top

All about surfaces
******************

   The "trivial surface" consists of using a fixed number of bits
(often eight) for each character, the bits together hold the integer
value of the index for the character in its charset table.  There are
many kinds of surfaces, beyond the trivial one, all having the purpose
of increasing selected qualities for the storage or transmission.  For
example, surfaces might increase the resistance to channel limits
(`Base64'), the transmission speed (`gzip'), the information privacy
(`DES'), the conformance to operating system conventions (`CR-LF'), the
blocking into records (`VB'), and surely other things as well(1).  Many
surfaces may be applied to a stream of characters from a charset, the
order of application of surfaces is important, and surfaces should be
removed in the reverse order of their application.

   Even if surfaces may generally be applied to various charsets, some
surfaces were specifically designed for a particular charset, and would
not make much sense if applied to other charsets.  In such cases, these
conceptual surfaces have been implemented as `recode' charsets, instead
of as surfaces.  This choice yields to cleaner syntax and usage.  *Note
Universal::.

   Surfaces are implemented within `recode' as special charsets which
may only transform to or from the `data' or `tree' special charsets.
Clever users may use this knowledge for writing surface names in
requests exactly as if they were pure charsets, when the only need is
to change surfaces without any kind of recoding between real charsets.
In such contexts, either `data' or `tree' may also be used as if it
were some kind of generic, anonymous charset: the request
`data..SURFACE' merely adds the given SURFACE, while the request
`SURFACE..data' removes it.

   The `recode' library distinguishes between mere data surfaces, and
structural surfaces, also called tree surfaces for short.  Structural
surfaces might allow, in the long run, transformations between a few
specialised representations of structural information like MIME parts,
Perl or Python initialisers, LISP S-expressions, XML, Emacs outlines,
etc.

   We are still experimenting with surfaces in `recode'.  The concept
opens the doors to many avenues; it is not clear yet which ones are
worth pursuing, and which should be abandoned.  In particular,
implementation of structural surfaces is barely starting, there is not
even a commitment that tree surfaces will stay in `recode', if they do
prove to be more cumbersome than useful.  This chapter presents all
surfaces currently available.

* Menu:

* Permutations::        Permuting groups of bytes
* End lines::           Representation for end of lines
* MIME::                MIME contents encodings
* Dump::                Interpreted character dumps
* Test::                Artificial data for testing

   ---------- Footnotes ----------

   (1) These are mere examples to explain the concept, `recode' only
has `Base64' and `CR-LF', actually.


File: recode.info,  Node: Permutations,  Next: End lines,  Prev: Surfaces,  Up: Surfaces

Permuting groups of bytes
=========================

   A permutation is a surface transformation which reorders groups of
eight-bit bytes.  A _21_ permutation exchanges pairs of successive
bytes.  If the text contains an odd number of bytes, the last byte is
merely copied.  An _4321_ permutation inverts the order of quadruples
of bytes.  If the text does not contains a multiple of four bytes, the
remaining bytes are nevertheless permuted as _321_ if there are three
bytes, _21_ if there are two bytes, or merely copied otherwise.

`21'
     This surface is available in `recode' under the name
     `21-Permutation' and has `swabytes' for an alias.

`4321'
     This surface is available in `recode' under the name
     `4321-Permutation'.


File: recode.info,  Node: End lines,  Next: MIME,  Prev: Permutations,  Up: Surfaces

Representation for end of lines
===============================

   The same charset might slightly differ, from one system to another,
for the single fact that end of lines are not represented identically
on all systems.  The representation for an end of line within `recode'
is the `ASCII' or `UCS' code with value 10, or `LF'.  Other conventions
for representing end of lines are available through surfaces.

`CR'
     This convention is popular on Apple's Macintosh machines.  When
     this surface is applied, each line is terminated by `CR', which has
     `ASCII' value 13.  Unless the library is operating in strict mode,
     adding or removing the surface will in fact _exchange_ `CR' and
     `LF', for better reversibility.  However, in strict mode, the
     exchange does not happen, any `CR' will be copied verbatim while
     applying the surface, and any `LF' will be copied verbatim while
     removing it.

     This surface is available in `recode' under the name `CR', it does
     not have any aliases.  This is the implied surface for the Apple
     Macintosh related charsets.

`CR-LF'
     This convention is popular on Microsoft systems running on IBM PCs
     and compatible.  When this surface is applied, each line is
     terminated by a sequence of two characters: one `CR' followed by
     one `LF', in that order.

     For compatibility with oldish MS-DOS systems, removing a `CR-LF'
     surface will discard the first encountered `C-z', which has
     `ASCII' value 26, and everything following it in the text.  Adding
     this surface will not, however, append a `C-z' to the result.

     This surface is available in `recode' under the name `CR-LF' and
     has `cl' for an alias.  This is the implied surface for the IBM or
     Microsoft related charsets or code pages.

   Some other charsets might have their own representation for an end of
line, which is different from `LF'.  For example, this is the case of
various `EBCDIC' charsets, or `Icon-QNX'.  The recoding of end of lines
is intimately tied into such charsets, it is not available separately
as surfaces.


File: recode.info,  Node: MIME,  Next: Dump,  Prev: End lines,  Up: Surfaces

MIME contents encodings
=======================

   RFC 2045 defines two 7-bit surfaces, meant to prepare 8-bit messages
for transmission.  Base64 is especially usable for binary entities,
while Quoted-Printable is especially usable for text entities, in those
case the lower 128 characters of the underlying charset coincide with
ASCII.

`Base64'
     This surface is available in `recode' under the name `Base64',
     with `b64' and `64' as acceptable aliases.

`Quoted-Printable'
     This surface is available in `recode' under the name
     `Quoted-Printable', with `quote-printable' and `QP' as acceptable
     aliases.

   Note that `UTF-7', which may be also considered as a MIME surface,
is provided as a genuine charset instead, as it necessary relates to
`UCS-2' and nothing else.  *Note UTF-7::.

   A little historical note, also showing the three levels of
acceptance of Internet standards.  MIME changed from a "Proposed
Standard" (RFC 1341-1344, 1992) to a "Draft Standard" (RFC 1521-1523)
in 1993, and was _recycled_ as a "Draft Standard" in 1996-11.  It is
not yet a "Full Standard".


File: recode.info,  Node: Dump,  Next: Test,  Prev: MIME,  Up: Surfaces

Interpreted character dumps
===========================

   Dumps are surfaces meant to express, in ways which are a bit more
readable, the bit patterns used to represent characters.  They allow
the inspection or debugging of character streams, but also, they may
assist a bit the production of C source code which, once compiled,
would hold in memory a copy of the original coding.  However, `recode'
does not attempt, in any way, to produce complete C source files in
dumps.  User hand editing or `Makefile' trickery is still needed for
adding missing lines.  Dumps may be given in decimal, hexadecimal and
octal, and be based over chunks of either one, two or four eight-bit
bytes.  Formatting has been chosen to respect the C language syntax for
number constants, with commas and newlines inserted appropriately.

   However, when dumping two or four byte chunks, the last chunk may be
incomplete.  This is observable through the usage of narrower expression
for that last chunk only.  Such a shorter chunk would not be compiled
properly within a C initialiser, as all members of an array share a
single type, and so, have identical sizes.

`Octal-1'
     This surface corresponds to an octal expression of each input byte.

     It is available in `recode' under the name `Octal-1', with `o1'
     and `o' as acceptable aliases.

`Octal-2'
     This surface corresponds to an octal expression of each pair of
     input bytes, except for the last pair, which may be short.

     It is available in `recode' under the name `Octal-2' and has `o2'
     for an alias.

`Octal-4'
     This surface corresponds to an octal expression of each quadruple
     of input bytes, except for the last quadruple, which may be short.

     It is available in `recode' under the name `Octal-4' and has `o4'
     for an alias.

`Decimal-1'
     This surface corresponds to an decimal expression of each input
     byte.

     It is available in `recode' under the name `Decimal-1', with `d1'
     and `d' as acceptable aliases.

`Decimal-2'
     This surface corresponds to an decimal expression of each pair of
     input bytes, except for the last pair, which may be short.

     It is available in `recode' under the name `Decimal-2' and has
     `d2' for an alias.

`Decimal-4'
     This surface corresponds to an decimal expression of each
     quadruple of input bytes, except for the last quadruple, which may
     be short.

     It is available in `recode' under the name `Decimal-4' and has
     `d4' for an alias.

`Hexadecimal-1'
     This surface corresponds to an hexadecimal expression of each
     input byte.

     It is available in `recode' under the name `Hexadecimal-1', with
     `x1' and `x' as acceptable aliases.

`Hexadecimal-2'
     This surface corresponds to an hexadecimal expression of each pair
     of input bytes, except for the last pair, which may be short.

     It is available in `recode' under the name `Hexadecimal-2', with
     `x2' for an alias.

`Hexadecimal-4'
     This surface corresponds to an hexadecimal expression of each
     quadruple of input bytes, except for the last quadruple, which may
     be short.

     It is available in `recode' under the name `Hexadecimal-4', with
     `x4' for an alias.

   When removing a dump surface, that is, when reading a dump results
back into a sequence of bytes, the narrower expression for a short last
chunk is recognised, so dumping is a fully reversible operation.
However, in case you want to produce dumps by other means than through
`recode', beware that for decimal dumps, the library has to rely on the
number of spaces to establish the original byte size of the chunk.

   Although the library might report reversibility errors, removing a
dump surface is a rather forgiving process: one may mix bases, group a
variable number of data per source line, or use shorter chunks in
places other than at the far end.  Also, source lines not beginning
with a number are skipped.  So, `recode' should often be able to read a
whole C header file, wrapping the results of a previous dump, and
regenerate the original byte string.


File: recode.info,  Node: Test,  Prev: Dump,  Up: Surfaces

Artificial data for testing
===========================

   A few pseudo-surfaces exist to generate debugging data out of thin
air.  These surfaces are only meant for the expert `recode' user, and
are only useful in a few contexts, like for generating binary
permutations from the recoding or acting on them.

   Debugging surfaces, _when removed_, insert their generated data at
the beginning of the output stream, and copy all the input stream after
the generated data, unchanged.  This strange removal constraint comes
from the fact that debugging surfaces are usually specified in the
_before_ position instead of the _after_ position within a request.
With debugging surfaces, one often recodes file `/dev/null' in filter
mode.  Specifying many debugging surfaces at once has an accumulation
effect on the output, and since surfaces are removed from right to left,
each generating its data at the beginning of previous output, the net
effect is an _impression_ that debugging surfaces are generated from
left to right, each appending to the result of the previous.  In any
case, any real input data gets appended after what was generated.

`test7'
     When removed, this surface produces 128 single bytes, the first
     having value 0, the second having value 1, and so forth until all
     128 values have been generated.

`test8'
     When removed, this surface produces 256 single bytes, the first
     having value 0, the second having value 1, and so forth until all
     256 values have been generated.

`test15'
     When removed, this surface produces 64509 double bytes, the first
     having value 0, the second having value 1, and so forth until all
     values have been generated, but excluding risky `UCS-2' values,
     like all codes from the surrogate `UCS-2' area (for `UTF-16'), the
     byte order mark, and values known as invalid `UCS-2'.

`test16'
     When removed, this surface produces 65536 double bytes, the first
     having value 0, the second having value 1, and so forth until all
     65536 values have been generated.

   As an example, the command `recode l5/test8..dump < /dev/null' is a
convoluted way to produce an output similar to `recode -lf l5'.  It says
to generate all possible 256 bytes and interpret them as `ISO-8859-9'
codes, while converting them to `UCS-2'.  Resulting `UCS-2' characters
are dumped one per line, accompanied with their explicative name.


File: recode.info,  Node: Internals,  Next: Concept Index,  Prev: Surfaces,  Up: Top

Internal aspects
****************

   The incoming explanations of the internals of `recode' should help
people who want to dive into `recode' sources for adding new charsets.
Adding new charsets does not require much knowledge about the overall
organisation of `recode'.  You can rather concentrate of your new
charset, letting the remainder of the `recode' mechanics take care of
interconnecting it with all others charsets.

   If you intend to play seriously at modifying `recode', beware that
you may need some other GNU tools which were not required when you first
installing `recode'.  If you modify or create any `.l' file, then you
need Flex, and some better `awk' like `mawk', GNU `awk', or `nawk'.  If
you modify the documentation (and you should!), you need `makeinfo'.
If you are really audacious, you may also want Perl for modifying
tabular processing, then `m4', Autoconf, Automake and `libtool' for
adjusting configuration matters.

* Menu:

* Main flow::           Overall organisation
* New charsets::        Adding new charsets
* New surfaces::        Adding new surfaces
* Design::              Comments on the library design


File: recode.info,  Node: Main flow,  Next: New charsets,  Prev: Internals,  Up: Internals

Overall organisation
====================

   The `recode' mechanics slowly evolved for many years, and it would
be tedious to explain all problems I met and mistakes I did all along,
yielding the current behaviour.  Surely, one of the key choices was to
stop trying to do all conversions in memory, one line or one buffer at
a time.  It has been fruitful to use the character stream paradigm, and
the elementary recoding steps now convert a whole stream to another.
Most of the control complexity in `recode' exists so that each
elementary recoding step stays simple, making easier to add new ones.
The whole point of `recode', as I see it, is providing a comfortable
nest for growing new charset conversions.

   The main `recode' driver constructs, while initialising all
conversion modules, a table giving all the conversion routines
available ("single step"s) and for each, the starting charset and the
ending charset.  If we consider these charsets as being the nodes of a
directed graph, each single step may be considered as oriented arc from
one node to the other.  A cost is attributed to each arc: for example,
a high penalty is given to single steps which are prone to losing
characters, a lower penalty is given to those which need studying more
than one input character for producing an output character, etc.

   Given a starting code and a goal code, `recode' computes the most
economical route through the elementary recodings, that is, the best
sequence of conversions that will transform the input charset into the
final charset.  To speed up execution, `recode' looks for subsequences
of conversions which are simple enough to be merged, and then
dynamically creates new single steps to represent these mergings.

   A "double step" in `recode' is a special concept representing a
sequence of two single steps, the output of the first single step being
the special charset `UCS-2', the input of the second single step being
also `UCS-2'.  Special `recode' machinery dynamically produces
efficient, reversible, merge-able single steps out of these double
steps.

   I made some statistics about how many internal recoding steps are
required between any two charsets chosen at random.  The initial
recoding layout, before optimisation, always uses between 1 and 5
steps.  Optimisation could sometimes produce mere copies, which are
counted as no steps at all.  In other cases, optimisation is unable to
save any step.  The number of steps after optimisation is currently
between 0 and 5 steps.  Of course, the _expected_ number of steps is
affected by optimisation: it drops from 2.8 to 1.8.  This means that
`recode' uses a theoretical average of a bit less than one step per
recoding job.  This looks good.  This was computed using reversible
recodings.  In strict mode, optimisation might be defeated somewhat.
Number of steps run between 1 and 6, both before and after
optimisation, and the expected number of steps decreases by a lesser
amount, going from 2.2 to 1.3.  This is still manageable.


File: recode.info,  Node: New charsets,  Next: New surfaces,  Prev: Main flow,  Up: Internals

Adding new charsets
===================

   The main part of `recode' is written in C, as are most single steps.
A few single steps need to recognise sequences of multiple characters,
they are often better written in Flex.  It is easy for a programmer to
add a new charset to `recode'.  All it requires is making a few
functions kept in a single `.c' file, adjusting `Makefile.am' and
remaking `recode'.

   One of the function should convert from any previous charset to the
new one.  Any previous charset will do, but try to select it so you will
not lose too much information while converting.  The other function
should convert from the new charset to any older one.  You do not have
to select the same old charset than what you selected for the previous
routine.  Once again, select any charset for which you will not lose
too much information while converting.

   If, for any of these two functions, you have to read multiple bytes
of the old charset before recognising the character to produce, you
might prefer programming it in Flex in a separate `.l' file.  Prototype
your C or Flex files after one of those which exist already, so to keep
the sources uniform.  Besides, at `make' time, all `.l' files are
automatically merged into a single big one by the script `mergelex.awk'.

   There are a few hidden rules about how to write new `recode'
modules, for allowing the automatic creation of `decsteps.h' and
`initsteps.h' at `make' time, or the proper merging of all Flex files.
Mimetism is a simple approach which relieves me of explaining all these
rules!  Start with a module closely resembling what you intend to do.
Here is some advice for picking up a model.  First decide if your new
charset module is to be be driven by algorithms rather than by tables.
For algorithmic recodings, see `iconqnx.c' for C code, or `txtelat1.l'
for Flex code.  For table driven recodings, see `ebcdic.c' for
one-to-one style recodings, `lat1html.c' for one-to-many style
recodings, or `atarist.c' for double-step style recodings.  Just select
an example from the style that better fits your application.

   Each of your source files should have its own initialisation
function, named `module_CHARSET', which is meant to be executed
_quickly_ once, prior to any recoding.  It should declare the name of
your charsets and the single steps (or elementary recodings) you
provide, by calling `declare_step' one or more times.  Besides the
charset names, `declare_step' expects a description of the recoding
quality (see `recodext.h') and two functions you also provide.

   The first such function has the purpose of allocating structures,
pre-conditioning conversion tables, etc.  It is also the way of further
modifying the `STEP' structure.  This function is executed if and only
if the single step is retained in an actual recoding sequence.  If you
do not need such delayed initialisation, merely use `NULL' for the
function argument.

   The second function executes the elementary recoding on a whole file.
There are a few cases when you can spare writing this function:

   * Some single steps do nothing else than a pure copy of the input
     onto the output, in this case, you can use the predefined function
     `file_one_to_one', while having a delayed initialisation for
     presetting the `STEP' field `one_to_one' to the predefined value
     `one_to_same'.

   * Some single steps are driven by a table which recodes one
     character into another; if the recoding does nothing else, you can
     use the predefined function `file_one_to_one', while having a
     delayed initialisation for presetting the `STEP' field
     `one_to_one' with your table.

   * Some single steps are driven by a table which recodes one
     character into a string; if the recoding does nothing else, you
     can use the predefined function `file_one_to_many', while having a
     delayed initialisation for presetting the `STEP' field
     `one_to_many' with your table.

   If you have a recoding table handy in a suitable format but do not
use one of the predefined recoding functions, it is still a good idea
to use a delayed initialisation to save it anyway, because `recode'
option `-h' will take advantage of this information when available.

   Finally, edit `Makefile.am' to add the source file name of your
routines to the `C_STEPS' or `L_STEPS' macro definition, depending on
the fact your routines is written in C or in Flex.


File: recode.info,  Node: New surfaces,  Next: Design,  Prev: New charsets,  Up: Internals

Adding new surfaces
===================

   Adding a new surface is technically quite similar to adding a new
charset.  *Note New charsets::.  A surface is provided as a set of two
transformations: one from the predefined special charset `data' or
`tree' to the new surface, meant to apply the surface, the other from
the new surface to the predefined special charset `data' or `tree',
meant to remove the surface.

   Internally in `recode', function `declare_step' especially
recognises when a charset is so related to `data' or `tree', and then
takes appropriate actions so that charset gets indeed installed as a
surface.

