PERLEBCDIC(1) Perl Programmers Reference Guide PERLEBCDIC(1)
NAME
perlebcdic - Considerations for running Perl on EBCDIC platforms
DESCRIPTION
An exploration of some of the issues facing Perl programmers on EBCDIC
based computers. We do not cover localization, internationalization,
or multi byte character set issues other than some discussion of UTF-8
and UTF-EBCDIC.
Portions that are still incomplete are marked with XXX.
COMMON CHARACTER CODE SETS
ASCII
The American Standard Code for Information Interchange is a set of
integers running from 0 to 127 (decimal) that imply character interpre
tation by the display and other system(s) of computers. The range
0..127 can be covered by setting the bits in a 7-bit binary digit,
hence the set is sometimes referred to as a "7-bit ASCII". ASCII was
described by the American National Standards Institute document ANSI
X3.4-1986. It was also described by ISO 646:1991 (with localization
for currency symbols). The full ASCII set is given in the table below
as the first 128 elements. Languages that can be written adequately
with the characters in ASCII include English, Hawaiian, Indonesian,
Swahili and some Native American languages.
There are many character sets that extend the range of integers from
0..2**7-1 up to 2**8-1, or 8 bit bytes (octets if you prefer). One
common one is the ISO 8859-1 character set.
ISO 8859
The ISO 8859-$n are a collection of character code sets from the Inter
national Organization for Standardization (ISO) each of which adds
characters to the ASCII set that are typically found in European lan
guages many of which are based on the Roman, or Latin, alphabet.
Latin 1 (ISO 8859-1)
A particular 8-bit extension to ASCII that includes grave and acute
accented Latin characters. Languages that can employ ISO 8859-1
include all the languages covered by ASCII as well as Afrikaans, Alba
nian, Basque, Catalan, Danish, Faroese, Finnish, Norwegian, Portuguese,
Spanish, and Swedish. Dutch is covered albeit without the ij ligature.
French is covered too but without the oe ligature. German can use ISO
8859-1 but must do so without German-style quotation marks. This set
is based on Western European extensions to ASCII and is commonly
encountered in world wide web work. In IBM character code set identi
fication terminology ISO 8859-1 is also known as CCSID 819 (or some
times 0819 or even 00819).
EBCDIC
The Extended Binary Coded Decimal Interchange Code refers to a large
collection of slightly different single and multi byte coded character
sets that are different from ASCII or ISO 8859-1 and typically run on
host computers. The EBCDIC encodings derive from 8 bit byte extensions
of Hollerith punched card encodings. The layout on the cards was such
that high bits were set for the upper and lower case alphabet charac
ters [a-z] and [A-Z], but there were gaps within each latin alphabet
range.
Some IBM EBCDIC character sets may be known by character code set iden
tification numbers (CCSID numbers) or code page numbers. Leading zero
digits in CCSID numbers within this document are insignificant. E.g.
CCSID 0037 may be referred to as 37 in places.
13 variant characters
Among IBM EBCDIC character code sets there are 13 characters that are
often mapped to different integer values. Those characters are known
as the 13 "variant" characters and are:
\ [ ] { } ^ ~ ! # | $ @
0037
Character code set ID 0037 is a mapping of the ASCII plus Latin-1 char
acters (i.e. ISO 8859-1) to an EBCDIC set. 0037 is used in North Amer
ican English locales on the OS/400 operating system that runs on AS/400
computers. CCSID 37 differs from ISO 8859-1 in 237 places, in other
words they agree on only 19 code point values.
1047
Character code set ID 1047 is also a mapping of the ASCII plus Latin-1
characters (i.e. ISO 8859-1) to an EBCDIC set. 1047 is used under Unix
System Services for OS/390 or z/OS, and OpenEdition for VM/ESA. CCSID
1047 differs from CCSID 0037 in eight places.
POSIX-BC
The EBCDIC code page in use on Siemens BS2000 system is distinct from
1047 and 0037. It is identified below as the POSIX-BC set.
Unicode code points versus EBCDIC code points
In Unicode terminology a code point is the number assigned to a charac
ter: for example, in EBCDIC the character "A" is usually assigned the
number 193. In Unicode the character "A" is assigned the number 65.
This causes a problem with the semantics of the pack/unpack "U", which
are supposed to pack Unicode code points to characters and back to num
bers. The problem is: which code points to use for code points less
than 256? (for 256 and over theres no problem: Unicode code points
are used) In EBCDIC, for the low 256 the EBCDIC code points are used.
This means that the equivalences
pack("U", ord($character)) eq $character
unpack("U", $character) == ord $character
will hold. (If Unicode code points were applied consistently over all
the possible code points, pack("U",ord("A")) would in EBCDIC equal A
with acute or chr(101), and unpack("U", "A") would equal 65, or non-
breaking space, not 193, or ord "A".)
Remaining Perl Unicode problems in EBCDIC
Many of the remaining seem to be related to case-insensitive match
ing: for example, "/[\x{131}]/" (LATIN SMALL LETTER DOTLESS I) does
not match "I" case-insensitively, as it should under Unicode. (The
match succeeds in ASCII-derived platforms.)
The extensions Unicode::Collate and Unicode::Normalized are not
supported under EBCDIC, likewise for the encoding pragma.
Unicode and UTF
UTF is a Unicode Transformation Format. UTF-8 is a Unicode conforming
representation of the Unicode standard that looks very much like ASCII.
UTF-EBCDIC is an attempt to represent Unicode characters in an EBCDIC
transparent manner.
Using Encode
Starting from Perl 5.8 you can use the standard new module Encode to
translate from EBCDIC to Latin-1 code points
use Encode from_to;
my %ebcdic = ( 176 => cp37, 95 => cp1047, 106 => posix-bc );
# $a is in EBCDIC code points
from_to($a, $ebcdic{ord ^}, latin1);
# $a is ISO 8859-1 code points
and from Latin-1 code points to EBCDIC code points
use Encode from_to;
my %ebcdic = ( 176 => cp37, 95 => cp1047, 106 => posix-bc );
# $a is ISO 8859-1 code points
from_to($a, latin1, $ebcdic{ord ^});
# $a is in EBCDIC code points
For doing I/O it is suggested that you use the autotranslating features
of PerlIO, see perluniintro.
Since version 5.8 Perl uses the new PerlIO I/O library. This enables
you to use different encodings per IO channel. For example you may use
use Encode;
open($f, ">:encoding(ascii)", "test.ascii");
print $f "Hello World!\n";
open($f, ">:encoding(cp37)", "test.ebcdic");
print $f "Hello World!\n";
open($f, ">:encoding(latin1)", "test.latin1");
print $f "Hello World!\n";
open($f, ">:encoding(utf8)", "test.utf8");
print $f "Hello World!\n";
to get two files containing "Hello World!\n" in ASCII, CP 37 EBCDIC,
ISO 8859-1 (Latin-1) (in this example identical to ASCII) respective
UTF-EBCDIC (in this example identical to normal EBCDIC). See the docu
mentation of Encode::PerlIO for details.
As the PerlIO layer uses raw IO (bytes) internally, all this totally
ignores things like the type of your filesystem (ASCII or EBCDIC).
SINGLE OCTET TABLES
The following tables list the ASCII and Latin 1 ordered sets including
the subsets: C0 controls (0..31), ASCII graphics (32..7e), delete (7f),
C1 controls (80..9f), and Latin-1 (a.k.a. ISO 8859-1) (a0..ff). In the
table non-printing control character names as well as the Latin 1
extensions to ASCII have been labelled with character names roughly
corresponding to The Unicode Standard, Version 3.0 albeit with substi
tutions such as s/LATIN// and s/VULGAR// in all cases, s/CAPITAL LET
TER// in some cases, and s/SMALL LETTER ([A-Z])/\l$1/ in some other
cases (the "charnames" pragma names unfortunately do not list explicit
names for the C0 or C1 control characters). The "names" of the C1 con
trol set (128..159 in ISO 8859-1) listed here are somewhat arbitrary.
The differences between the 0037 and 1047 sets are flagged with ***.
The differences between the 1047 and POSIX-BC sets are flagged with
###. All ord() numbers listed are decimal. If you would rather see
this table listing octal values then run the table (that is, the pod
version of this document since this recipe may not work with a
pod2_other_format translation) through:
recipe 0
perl -ne if(/(.{33})(\d+)\s+(\d+)\s+(\d+)\s+(\d+)/) \
-e {printf("%s%-9o%-9o%-9o%o\n",$1,$2,$3,$4,$5)} perlebcdic.pod
If you want to retain the UTF-x code points then in script form you
might want to write:
recipe 1
open(FH,") {
if (/(.{33})(\d+)\s+(\d+)\s+(\d+)\s+(\d+)\s+(\d+)\.?(\d*)\s+(\d+)\.?(\d*)/) {
if ($7 ne && $9 ne ) {
printf("%s%-9o%-9o%-9o%-9o%-3o.%-5o%-3o.%o\n",$1,$2,$3,$4,$5,$6,$7,$8,$9);
}
elsif ($7 ne ) {
printf("%s%-9o%-9o%-9o%-9o%-3o.%-5o%o\n",$1,$2,$3,$4,$5,$6,$7,$8);
}
else {
printf("%s%-9o%-9o%-9o%-9o%-9o%o\n",$1,$2,$3,$4,$5,$6,$8);
}
}
}
If you would rather see this table listing hexadecimal values then run
the table through:
recipe 2
perl -ne if(/(.{33})(\d+)\s+(\d+)\s+(\d+)\s+(\d+)/) \
-e {printf("%s%-9X%-9X%-9X%X\n",$1,$2,$3,$4,$5)} perlebcdic.pod
Or, in order to retain the UTF-x code points in hexadecimal:
recipe 3
open(FH,") {
if (/(.{33})(\d+)\s+(\d+)\s+(\d+)\s+(\d+)\s+(\d+)\.?(\d*)\s+(\d+)\.?(\d*)/) {
if ($7 ne && $9 ne ) {
printf("%s%-9X%-9X%-9X%-9X%-2X.%-6X%-2X.%X\n",$1,$2,$3,$4,$5,$6,$7,$8,$9);
}
elsif ($7 ne ) {
printf("%s%-9X%-9X%-9X%-9X%-2X.%-6X%X\n",$1,$2,$3,$4,$5,$6,$7,$8);
}
else {
printf("%s%-9X%-9X%-9X%-9X%-9X%X\n",$1,$2,$3,$4,$5,$6,$8);
}
}
}
incomp- incomp-
8859-1 lete lete
chr 0819 0037 1047 POSIX-BC UTF-8 UTF-EBCDIC
------------------------------------------------------------------------------------
0 0 0 0 0 0
1 1 1 1 1 1
2 2 2 2 2 2
3 3 3 3 3 3
4 55 55 55 4 55
5 45 45 45 5 45
6 46 46 46 6 46
7 47 47 47 7 47
8 22 22 22 8 22
9 5 5 5 9 5
10 37 21 21 10 21 ***
11 11 11 11 11 11
|
