The other side of the moon

This document contains information on rewriting your mail header and mail envelope through sendmail for mailing through your ISP. You should still read the sendmail docs and Sendmail-Offline howto.

0. ChangeLog

• Sat Oct 25 2003 Philip S Tellis v1.3
  • Updated the patch for sendmail v 8.12.9
  • Moved the ChangeLog to the start of the file
  • Added URLs to References and Further Reading
  • Fixed wording a little
• Tue Jan 1 2003 Philip S Tellis v1.2
  • Updates for sendmail 8.11
• Sat Dec 8 2001 Philip S Tellis v1.1
  • Added patch file
  • genericsdomain file is same as sendmail.cw
  • changed from dgenerics to egenerics
• Sun Aug 26 2001 Philip S Tellis v1.0
  • Initial release

1. About this document

1.1. Copyright

Copyright © Philip S Tellis.

Permission is granted to make and distribute copies of this manual provided the copyright notice and this permission notice are preserved on all copies.

Permission is granted to copy and distribute modified versions of this manual under the conditions for verbatim copying, provided that the derived work is distributed under the terms of a permission notice identical to this one. Translations fall under the catagory of "modified versions".

1.2. Disclaimer

The information in this document is based on my own experience. It works on my system, but may not work on yours. No guarantees are made on its accuracy. The users of this information take full responsibility for any damage that may occur through its use.

1.3. Feedback

If you have questions or comments about this document, please feel free to contact me by email at <philip DOT tellis AT gmail DOT com>.

Please, do not ask me questions about sendmail, m4 or sendmail.cf. Do not ask me questions about receiving mail.

2. Introduction

2.1. The problem

You connect to your ISP to send and receive mail. You have an email account with another email service provider (not your ISP). You want outgoing mails to appear to come from your other account, but you want to use your ISP's mail gateway to send the mail.

You must use your ISP's mail gateway, because you are not permanently connected to the Internet. Your ISP does not allow relay - i.e. only mail coming from or going to an email address in your ISP's domain will be allowed through their gateway.

You also want that mails sent out from the system do not have any local addresses in the header (To:, Cc:, From: lines), while mails between two local users always have the local addresses - i.e. you want rewrite to work only on mails leaving the system.

Finally, you want that local mails are delivered immediately, while remote mails are stored in a queue until you are connected to the Internet.

2.2. Software Environment

I use Sendmail v 8.9.3, 8.11 and 8.12.11 with the corresponding versions of sendmail-cf. I also use GNU m4 1.4 for macro processing of sendmail.mc. This HOWTO will assume that all configuration files, including sendmail.cf are stored in /etc/mail. It also assume that the sendmail binary is /usr/sbin/sendmail, that the m4 binary is in your path, and that the cf directories are correctly installed.

NOTE: Some systems keep the sendmail.cf file at /etc/sendmail.cf, in that case, you should do the same. An easy way would be to either symlink /etc/sendmail.cf to /etc/mail/sendmail.cf or copy /etc/mail/sendmail.cf to /etc/sendmail.cf after you've generated it. The latter method can be more reliable as it sort of ensures that you don't have a half generated cf file when sendmail runs. YMMV.

If you have installed from rpm, you will need the sendmail-cf rpm, if you installed from source, then these will be in your source tree. For debian, the sendmail/cf directory is installed as part of the sendmail package.

I use pine as my mail client, in particular, pine uses `sendmail -bs' to send mail. There shouldn't be any difference with a client that connects to your smtp port.

3. Configuring Sendmail

3.1. Creating the mc file

Sendmail's configuration file - /etc/sendmail.cf is not necessarily human readable. In an earlier version of this Howto, I listed changes that were needed in sendmail.cf. While it isn't important to understand what those changes were, I've decided to get rid of that, and just write some m4 files that you can use. For the main part, we will edit a sendmail.mc file, and use m4 along with these additional files to generate the sendmail.cf file. This has the added advantage of being easier to maintain when you update your sendmail.mc file and need to regenerate sendmail.cf.

The patch for sendmail-cf's macro files is attached at the end of this document.

At this point, it may be be wise to read the cf/README (or README.cf) file that came with your sendmail/sendmail-cf package. It explains the mc file well.

You should already have a file called /etc/mail/sendmail.mc, but if you don't, now is a good time to create it. You can use mine as a template:

divert(-1)
dnl This is the macro configuration file used to generate the /etc/sendmail.cf
dnl file. If you modify this file you will have to regenerate the
dnl /etc/sendmail.cf by running this macro configuration through the m4
dnl preprocessor:
dnl
dnl        m4 /etc/mail/sendmail.mc > /etc/mail/sendmail.cf
dnl
dnl You will need to have the sendmail-cf package installed for this to
dnl work.
include(`/usr/lib/sendmail-cf/m4/cf.m4')
define(`confDEF_USER_ID',``8:12'')dnl
OSTYPE(`linux')dnl
undefine(`UUCP_RELAY')dnl
undefine(`BITNET_RELAY')dnl
define(`confAUTO_REBUILD')dnl
define(`confPRIVACY_FLAGS', `goaway authwarnings')dnl
dnl
define(`SMART_HOST', `esmtp:isp.com')dnl
define(`confCON_EXPENSIVE', `True')dnl
define(`SMTP_MAILER_FLAGS', `e')dnl
define(`confTO_QUEUEWARN', `')dnl
define(`confTO_INITIAL', `10s')dnl
define(`confTO_CONNECT', `10s')dnl
define(`confTO_ICONNECT', `10s')dnl
define(`confCW_FILE', `/etc/mail/local_host_names')dnl
dnl
FEATURE(`smrsh',`/usr/sbin/smrsh')dnl
FEATURE(`always_add_domain')dnl
FEATURE(`use_cw_file')dnl
FEATURE(`nocanonify')dnl
dnl
FEATURE(`masquerade_envelope')dnl
FEATURE(`allmasquerade')dnl
FEATURE(`genericstable', `hash -o /etc/mail/genericstable')dnl
GENERICS_DOMAIN_FILE(`/etc/mail/local_host_names')dnl
dnl
dnl The egenericstable is a feature that will be added through
dnl my patch for the macro files
dnl
FEATURE(`egenericstable', `hash -o /etc/mail/egenericstable')dnl
dnl
FEATURE(`access_db')dnl
FEATURE(`blacklist_recipients')dnl
FEATURE(`accept_unresolvable_domains')dnl
dnl
MAILER(`local')dnl
MAILER(`smtp')dnl

First of all, change the path in the include line to whatever it is on your system. If you've installed from source or using apt on a debian system, this would be /usr/share/sendmail/cf. Also, replace isp.com with the host-name of your isp's smtp server.

I will now attempt to explain the important parts of this file. Most of these things are documented in the README file, so I'll skip those. I will only explain the masquerading options.

Before that, if you're wondering what all the dnl commands are. It simply means delete till new line. Since m4 is a stream processor, it does not care about newlines, and will read and write all white space. These commands just tell it to skip the extra white space. You can also place comments after a dnl.

define(`confCON_EXPENSIVE', `True')dnl
define(`SMTP_MAILER_FLAGS', `e')dnl

These two lines tell sendmail that SMTP mailers are expensive (the e flag), and not to try expensive mailers on the first try. This will cause sendmail to put any mail that requires a connect to a remote smtp server to stay in a queue until you flush the queue with `sendmail -q'.

Local mails will be delivered immediately, because the local mailer is not marked as expensive.

Furthermore, since we are not always connected to the Internet, we don't want warnings of mail being stuck in the queue too long (because that's a perfectly valid situation).

define(`confTO_QUEUEWARN', `')dnl

tells sendmail to unset the QUEUE WARN TimeOut. Set it if you want notifications of mail not being sent for more than a day. You can set it to 1d for 1 day or 1m for 1 minute, etc.

This solves the third part of our problem - mail delivery.

FEATURE(`always_add_domain')dnl
FEATURE(`use_cw_file')dnl
define(`confCW_FILE', `/etc/mail/local_host_names')dnl

These lines tell sendmail to add a domain name to all mails, even if they are sent locally. The cw file is a file containing a list of aliases for your system. It is /etc/local_host_names by default, but we tell sendmail to use /etc/mail/local_host_names through confCW_FILE. This is so that all mail related sub-configuration files are in a common subdirectory.

FEATURE(`nocanonify')dnl

This line tells sendmail not to do DNS (Domain Name System) look ups on host names. Since we are not permanently connected to the Internet, this is required to prevent sendmail from locking up until it gets a DNS time-out.

FEATURE(`masquerade_envelope')dnl
FEATURE(`allmasquerade')dnl

By default, sendmail will only masquerade addresses in the mail header, i.e. the From:, To:, Cc: lines. masquerade_envelope tells sendmail to also masquerade the address used to make the smtp connection with the remote host. In our case, this would be the smart host.

We need this address to be the one that our isp has given us.

`allmasquerade' tells sendmail to also rewrite recipient addresses. Normally, local recipients would receive mail will the local domain name attached to their address. This will cause their masqueraded name to appear.

The advantage of this is that if you send a mail to both, a remote and a local user, then the remote user will see all valid addresses in the To: line, because all local addresses have been masqueraded. He can reply to all, and the mail will reach correctly.

Unfortunately, local users receive mail with all foreign domains, so their replies will take longer to reach (as they will go to the masquerade host instead).

There is a workaround for this, as we shall see later.

FEATURE(`genericstable', `hash -o /etc/mail/genericstable')dnl
GENERICS_DOMAIN_FILE(`/etc/mail/local_host_names')dnl

The generics table gives us a very high level of flexibility in setting aliases for outgoing mails. The genericsdomain file tells sendmail which hosts the genericstable applies to. Since we already have a local_host_names file, which contains a list of domains we accept mail for, it makes sense to use the same for outgoing domains.

Finally, we add our own rule, which tells sendmail how to rewrite the envelope (hence the `e' prefix).

FEATURE(`egenericstable', `hash -o /etc/mail/egenericstable')dnl

This rule will add the code to sendmail.cf that v1 of this document required you to add manually.

3.2. Generics tables

My /etc/mail/genericstable looks like this:

philip		philip.tellis@mydomain.com

While my /etc/mail/egenericstable looks like this:

philip		tellis@isp.com

My local_host_names file is:

tae.tellis.home

which is the output of hostname -f

Once you've made your tables, you have to convert them to hash databases using the make map command:

makemap -r hash genericstable.db < genericstable
makemap -r hash egenericstable.db > egenericstable

These are the files that are actually read by sendmail.

3.3. Making sendmail.cf

Finally, convert your sendmail.mc file to sendmail.cf using m4:

m4 /etc/mail/sendmail.mc > /etc/mail/sendmail.cf

It's actually a good idea to write this to a different file, and then move that to sendmail.cf. You could first check the file for errors, and also, you won't overwrite a file that is currently being processed. However, since you're on a home machine with presumably only one person using the machine at a time, this isn't such an issue. You decide what you need to do.

3.4. Restart sendmail

Now, restart sendmail using whatever method you normally use. On systems using SysV startup scripts, this would be /etc/init.d/sendmail restart

4. Further configuration

4.1. Flushing the queue

Since I connect to the net using a ppp link, I have this in /etc/ppp/ip-up.local:

/usr/sbin/sendmail -q3m

which tells sendmail to process the queue every three minutes.

I also have this in /etc/ppp/ip-down.local:

/usr/sbin/sendmail -q1d

which tells sendmail to process the queue once a day if not connected. Make it more if you want.

4.2. A Makefile

Some distributions have a makefile to regenerate all databases and the sendmail.cf file. You'll have to add an entry for your egenericstable there. You're on your own for this.

5. Acknowledgements

• Thanks to my ISP for forcing me to do this
• Thanks to some of the members of Ilug-Bom for ideas

6. References

• cf/README Sendmail distribution
• Sendmail - friend or foe? by Matthew Palmer, talk for Manchester Lug http://ieee.uow.edu.au/~mjp16/sendmail/
• The Sendmail+Dialup mini-HOWTO v 1.00 by Leif Erlingsson <leif AT lege.com>, Jan P Tietze <jptietze AT mail.hh.provi.de>, Vincent Zweije <zweije AT xs4all.nl> http://www.lege.com/Sendmail+Dialup.v1.00.html
• Configuring Sendmail for Offline Use mini-HOWTO by Simone Govoni <simgov AT tin.it> http://www.thinkunix.net/unix/email/sendmail/sendmail-offline.html
• A quick HOWTO for genericstable (on comp.mail.sendmail) by Chris Johnston

7. Further Reading

Apart from the references above, you may want to check all the documentation that comes with sendmail.

• Eric Allman, Sendmail: Installation and Operation Guide, file doc/op/op.ps in sendmail source archive.
• Thomas Roessler, Sendmail Address Rewriting mini-HOWTO. http://www.guug.de/~roessler/genericstable.html
• sendmail.org: Sendmail FAQ, especially the section on virtual users and generics table.
• Sendmail on a dialup by Suresh Ramasubramanian http://www.hserus.net/sendmail.html

8. The patchfile

IMPORTANT: This patch is for sendmail v 8.12, if your version is different, this may not work. Have a look at older versions of this document for older versions of sendmail.

Most versions of the patch utility can patch your directories straight from this howto itself.

First change to the directory where your sendmail-cf m4 macros and documentation is. On RedHat this is /usr/share/sendmail-cf while on debian or source installs, it is /usr/share/sendmail/cf. Some older systems used /usr/lib/sendmail-cf - the patch would most likely not work with these systems.

To verify that you are in the correct directory, check that the feature, m4 and mailer directories are listed in there.

Then, run the patch command on this file:

patch -p0 < Hdr-Env-Masq-miniHOWTO.txt

That's it. Don't worry about all the stuff before the actual patch. The patch util knows to ignore everything until it finds the actual patch.

If the patch doesn't work for you, then you might have to revert to the steps outlined in version 1 of this document. If you have sendmail v 8.9, you can use the patch in version 1.1 of this document, for sendmail v 8.11, use version 1.2 of this Howto.

diff -Pru feature/egenericstable.m4 feature/egenericstable.m4
--- feature/egenericstable.m4	1970-01-01 01:00:00.000000000 +0100
+++ feature/egenericstable.m4	2003-10-25 14:30:34.000000000 +0200
@@ -0,0 +1,20 @@
+divert(-1)
+#
+# Copyright (c) 2001-2003 Philip S Tellis.  All rights reserved.
+#
+# This file added by Philip S Tellis for envelope rewriting.
+# Licence is the same as that of Sendmail.
+#
+#
+
+divert(0)
+VERSIONID(`@(#)egenericstable.m4, 1.2 2003/01/07 23:00:00 philip')
+divert(-1)
+
+define(`_EGENERICS_TABLE_', `')
+
+LOCAL_CONFIG
+# EGenerics table (mapping outgoing addresses in envelopes)
+Kegenerics ifelse(defn(`_ARG_'), `',
+                 DATABASE_MAP_TYPE MAIL_SETTINGS_DIR`egenericstable',
+                 `_ARG_')
diff -Pru m4/proto.m4 m4/proto.m4
--- m4/proto.m4	2003-10-25 13:41:48.000000000 +0200
+++ m4/proto.m4	2003-10-25 14:30:34.000000000 +0200
@@ -1333,6 +1333,25 @@
 R<lt; $+ > $* 			$#_LOCAL_ $@ $2    $: $1
 
 ###################################################################
+###  Ruleset EnvRewrite -- convert envelope names to masq form  ###
+###################################################################
+SEnvRewrite
+
+ifdef(`_EGENERICS_TABLE_', `dnl
+# handle generics database for envelope
+ifdef(`_GENERICS_ENTIRE_DOMAIN_',
+`R$+ <lt; @ $* $=G . >	$: <lt; $1@$2$3 > $1 <lt; @ $2$3 . > @	mark',
+`R$+ <lt; @ $=G . >	$: <lt; $1@$2 > $1 <lt; @ $2 . > @	mark')
+R$+ <lt; @ *LOCAL* >	$: <lt; $1@$j > $1 <lt; @ *LOCAL* > @ mark
+R<lt; $+ > $+ <lt; $* > @	$: <lt; $( egenerics $1 $: $) > $2 <lt; $3 >
+R<lt; > $+ <lt; @ $+ >	$: <lt; $( egenerics $1 $: $) > $1 <lt; @ $2 >
+R<lt; $* @ $* > $* <lt; $* >	$@ $>3 $1 @ $2			found qualified
+R<lt; $+ > $* <lt; $* >	$: $>3 $1 @ *LOCAL*		found unqualified
+R<lt; > $*		$: $1				not found',
+`dnl')
+
+
+###################################################################
 ###  Ruleset 93 -- convert header names to masqueraded form	###
 ###################################################################
 
@@ -1401,7 +1420,9 @@
 
 SMasqEnv=94
 ifdef(`_MASQUERADE_ENVELOPE_',
-`R$+			$@ $>MasqHdr $1',
+ifdef(`_EGENERICS_TABLE_',
+`R$+			$@ $>EnvRewrite $1',
+`R$+			$@ $>MasqHdr $1'),
 `R$* <lt; @ *LOCAL* > $*	$: $1 <lt; @ $j . > $2')
 
 ###################################################################
diff -Pru mailer/local.m4 mailer/local.m4
--- mailer/local.m4	2003-10-25 13:41:48.000000000 +0200
+++ mailer/local.m4	2003-10-25 14:32:28.000000000 +0200
@@ -42,7 +42,8 @@
 R@ <lt;@ $*>		$n			temporarily bypass Sun bogosity
 R$+			$: $>AddDomain $1	add local domain if needed
 ifdef(`_LOCAL_NO_MASQUERADE_', `dnl', `dnl
-R$*			$: $>MasqEnv $1		do masquerading')
+ifdef(`_EGENERICS_TABLE_', `dnl', `dnl
+R$*			$: $>MasqEnv $1		do masquerading')')
 
 #
 #  Envelope recipient rewriting