Sending Japanese emails by a Perl script

WARNING: Thanks to a generous feedback from J, I have realized that the following method using ISO-2022-JP for Japanese emails is obsolete. Accordingly, I would like to mark the main part of this article as NOT-RECOMMENDED. Instead, in the comment column below, I have posted a fixed script that uses UTF-8. This UTF-8 based emailing script should work not only for Japanese but also for any other non-English languages as well.

Sending Japanese emails is not as easily as sending English emails.

Simply sending your Japanese email from the Unix command line to your recipient across the Internet often does not work. Without the important practice explained in this article, your Japanese email is always susceptible to Mojibake (文字化け) where the Japanese characters in your email become broken at the recipient's side.

In this article, I introduce a simple and practical way to do the crucial pre-processing of a Japanese email (i.e. Unicode to ISO-2022-JP conversion and Japanese MIME-header generation) using Perl. Finally, I demonstrate how to send out our "well-formed" Japanese email using sendEmail program. (For an introduction to sending an email from the command line by Perl, see this previous article.)

Here is an index to this article:

• Background
• How to run the program
1. Getting resources
2. Installing nkf
3. Running the script
• Details
• References

Background

To prevent Mojibake of your Japanese email, stick to the following de facto standard rules commonly practiced by Japanese IT professionals:

1. Convert all Japanese characters into ISO-2022-JP (7-bit) character code.
2. Apply MIME encoding to Japanese characters that appear in header fields such as Subject, To, From, etc.

In modern Linux distributions, Unicode support is getting well-matured. Many text editors (e.g. Emacs, gedit, etc.) support Unicode by default. So you may compose a Japanese email and save it easily in Unicode (UTF-8) using those editors. For that reason, I assume that your Japanese email is originally written in Unicode.

Mail servers have been traditionally handling only 7-bit character codes (most notably US-ASCII code), and many of mail servers in the world still support only 7-bit character transmission mode (thus, not 8-bit clean). On the other hand, many of Japanese character codes (e.g. EUC-JP, SHIFT-JIS, and Japanese within UTF-8) are 8-bit character codes. The only 7-bit Japanese code is ISO-2022-JP. Hence using ISO-2022-JP is the only safe option to avoid Mojibake caused by the loss of the top bit of 8-bit Japanese characters while being transmitted across emails servers.

The message header of an email is re-arranged during the process of transmission. For that reason, only ASCII characters are allowed to appear in message headers. RFC1522 defines rules to embed non-ASCII characters into message headers by using the standard MIME (Multipurpose Internet Mail Extensions) encoding method. For example, to embed a Japanese string "日本語のタイトル" into the subject field, the string is MIME-encoded as

=?ISO-2022-JP?B?GyRCRnxLXDhsJE4lPyUkJUglaw==?=

following the form of

=?<charset>?<method>?<encoded string>?=

where ? is a delimiter and B indicates the BASE64 encoding method.

How to run the program

1. Getting resources

Please get the following files and save them into the same directory.

• sendEmail: A Perl script software that handles all the email-spec-related stuffs to construct and send emails. If you are using Ubuntu, you may install it with
  

  $ sudo apt-get install sendemail
  

  If sendEmail is not included in the repository of your Linux distribution (or using Windows), you may still download a copy from sendEmail's website.

• send-email-jp.pl: The main Perl script developed in this article.
• mybase64encode.pl: A Perl script that takes an ISO-2022-JP string from standard input and outputs the MIME message header of the string to standard output.
• body-utf8.txt: The Japanese message body file written in Unicode (UTF-8).

2. Installing nkf

We use nkf, a command-line Japanese character code conversion program to do conversion from Unicode (UTF-8) to ISO-2022-JP. In Ubuntu, install the program by

$ sudo apt-get install nkf

3. Running the script

Make sure that the Perl scripts have executable permission:

$ chmod 775 send-email-jp.pl mybase64encode.pl

Finally,

$ ./send-email-jp.pl

In your own applications, please modify the script (e.g. To, From, Message body) accordingly. Enjoy sending Japanese emails!

Details

send-email-jp.pl

#!/usr/bin/perl

$to_jp_part_utf8 = '担当者';
$to_eng_part_utf8 = '<info@example.com>';
$to_jp_part_7bit_mime = `echo -n "$to_jp_part_utf8" | nkf | ./mybase64encode.pl`;
$to_7bit_mime = $to_jp_part_7bit_mime . " " . $to_eng_part_utf8;

$from_jp_part_utf8 = '山田太郎';
$from_eng_part_utf8 = '<taro@example.net>';
$from_jp_part_7bit_mime = `echo -n "$from_jp_part_utf8" | nkf | ./mybase64encode.pl`;
$from_7bit_mime = $from_jp_part_7bit_mime . " " . $from_eng_part_utf8;

$sub_utf8 = '日本語のタイトル';
$sub_7bit_mime = `echo -n "$sub_utf8" | nkf | ./mybase64encode.pl`;

$cmd = "cat ./body-utf8.txt | nkf > body-7bit.txt";
system("$cmd");


$cmd = "sendEmail " .
       "-t \"$to_7bit_mime\" " .
       "-f \"$from_7bit_mime\" " .
       "-u \"$sub_7bit_mime\" " .
       "-o message-file=body-7bit.txt " .
       "-o message-charset=ISO-2022-JP " .
       "-s localhost:25" .
       "\n";

print $cmd;
system("$cmd");


system("rm -f body-7bit.txt");

This is a complete Perl script that does the proper pre-processing of a Japanese email and send it out in the command line using sendEmail program.

$to_jp_part_utf8 = '担当者';
$to_eng_part_utf8 = '<info@example.com>';
$to_jp_part_7bit_mime = `echo -n "$to_jp_part_utf8" | nkf | ./mybase64encode.pl`;
$to_7bit_mime = $to_jp_part_7bit_mime . " " . $to_eng_part_utf8;

This block constructs a MIME-encoded header of the To-field. In the original Unicode form, the field looks like

担当者 <info@example.com>

We convert it into a MIME-encoded To-field as

=?ISO-2022-JP?B?GyRCQzRFdjxU?= <info@example.com>

where the Japanese recipient name's part in Unicode is first converted into ISO-2022-JP code by nkf, then MIME-encoded by script mybase64encode.pl. The last line concatenates the Japanese part and the ASCII part (email address).

$sub_utf8 = '日本語のタイトル';
$sub_7bit_mime = `echo -n "$sub_utf8" | nkf | ./mybase64encode.pl`;

This block creates a MIME-encoded Subject header. Since this field can be entirely in Japanese, we are MIME-encoding the whole part of the field.

$cmd = "cat ./body-utf8.txt | nkf > body-7bit.txt";
system("$cmd");

This block converts a Japanese body message file body-utf8.txt in Unicode into a file body-7bit.txt in ISO-2022-JP.

$cmd = "sendEmail " .
       "-t \"$to_7bit_mime\" " .
       "-f \"$from_7bit_mime\" " .
       "-u \"$sub_7bit_mime\" " .
       "-o message-file=body-7bit.txt " .
       "-o message-charset=ISO-2022-JP " .
       "-s localhost:25" .
       "\n";

print $cmd;
system("$cmd");
system("rm -f body-7bit.txt");

Finally, we set To-, From-, Subject-fields containing MIME-encoded ISO-2022-JP strings to the options of sendEmail program. We also set the ISO-2022-JP message body file. We explicitly specify that the email message is using ISO-2022-JP character set. The command is executed by system() function. After sending out the email, we clean up the temporarily generated body file.

mybase64encode.pl

#!/usr/bin/perl

use MIME::Base64;

undef $/;
$body = <>
$body_encoded = encode_base64($body);
$body_encoded =~ s/\s+$//;               # Remove newlines
print '=?ISO-2022-JP?B?' . $body_encoded . '?=';

This script constructs an ISO-2022-JP string given in standard input into a complete MIME header. To do BASE64 encoding, we use Perl's standard encode_base64() function. The last line constructs a complete MIME header by string concatenation.

References

"Mojibake", http://en.wikipedia.org/wiki/Mojibake
"Japanese Emails on the Internet", http://www.kanzaki.com/docs/jis-mail.html (In Japanese)