Mail-based Comment System

Motivation

To get feedback, Comment System are essential to web pages.

Most of modern web pages use 3rd-party services like Disqus. They are criticized by ads or trivial steps for sign-up/validation. For my Point of View, even using Github issue is not a good idea. I mean… You did make contents by yourself with markdown or some similar things, but why let others hold comments for you?

CGI is another solution if you can access a machine with public IP. To show comments in your pages, you just need to transform HTTP Request into HTML. For example, this one-line comment button would be easily implemented! But consider spam or offensive content, a way to identify visitor still matters.

Of course I am talking about Mail!

For each comment, all you need to know are inside Mail header: sender title, mail address, datetime and SMTP server. By SPF/DKIM/DMARC, receiver can nearly 100% confirm Mail is from a specific sender.

Mailing List

Of course there is already a Comment System based on Mail: Mailing List.

Actually I am not a mailing-list user (though I did subscribe some lists of FOSS projects). But thanks to public web page for archive1, I can realize how powerful it is.

User just need to finish subscription, then everyone can start a new thread by mail, and get replied by another mail.

But not everyone can build such this Comment System, you need to:

  1. Be system admin to build related services
  2. Tune HTTP web server to show archive
  3. Afford traffic of SMTP for each new message in thread

This might be a little overkill for basic Comment System on simple web pages.

MDA and .forward

If you can login as a normal user in Unix System (like tilde.club), ~/.forward is what you need.

From decades ago, Unix use program called MDA(Message Delivery Agent) to deliver mail/message2 to local user (Most of time it is Sendmail). User can write their own ~/.forward file to tell MDA where this mail should goes. For each entry in this file, it could be:

  1. Another local user, for example: alice, bob
  2. User in another host, for example: alice@anotherhost.com
  3. Absolute pathname of a file, the mail would be appended to it. For example:
    /home/alice/mails
    Of course you can use /dev/null to drop any mail!
  4. A command with leading char |, it would be executed with mail as STDIN. For example:
    "|/home/alice/mail-handler.sh arg1 arg2"3

With this approach, you can easily create a Common System based on mail:

  1. Put a mailto link in your web page
  2. Create a command to store incoming mail as comment in HTML format
  3. Write an entry for this command in .forward
  4. In your web page, display comments made by step.2

Let’s do it!

comment.sh

Let’s build a bash script called comment.sh for all of these:

1. Check mail is for comment

I use pattern Comment on page: <PATH_OF_WEB_PAGE> in mail subject, to determine a mail should be stored as a comment. If it is not, script just return 0 safely:

# RFC 2822 use Carriage Return, remove it for safety
MAIL="$(tr -d '\r')"

# join multi-line header field value into one line
header="$(<<<"$MAIL" sed '/^$/ q; :a; N; s/\n\s\+//; ta')"

# just mail body
body="$(<<<"$MAIL" sed -n '/^$/,$ p' | sed '1d')"

# determine mail is for comment by pattern
pattern='^Subject: .*[cC]omment on page:? +(https?://[^/]+)?(/?[^ ]+).*$'
<<<"$header" grep -E "$pattern" >/dev/null || exit 0

2. Determine output and text process tool

For your HTTP routing, set output_dir to where HTML for comments locates.

Also, I use markdown_bin to specify command to process markdown text in mail body

output_dir=...
markdown_bin=...

3. Save each header field as variable

# enable execute last command in pipe under current shell
shopt -s lastpipe; set +m;

# save each field of header into variables
echo "$header" | \
while read field value; do
  echo "$field" "$value" >>/tmp/header
  declare field=$(<<<$field tr [:lower:] [:upper:] | tr '-' '_' | tr -d ':')
  declare $field="${value}"
done

# formated datetime
DATE=${DATE:+$(date --rfc-3339 seconds --date "$DATE")}

4. Get path of output file

Based on mail subject, determine path of output. For example:
if mail subject contains Comment on page: /posts/comment-by-mail.html, then set output to $output_dir/posts/comment-by-mail.comment.html

path=...
output=$output_dir/${path}.comment.html
umask 022; mkdir -p $(dirname $output)
exec 1>$output

5. Get comment from mail body

By RFC 1341, header with Content-Type: multipart/*; is separated by value of boundary. Some mail client automatically sends mail with both text part and html part. Here we extract text part if necessary:

printTextPart() {
  boundary=$1
  beginPat="\\|^--${boundary}\$|"
  endPat="\\|^--${boundary}(--)?\$|"
  sed -En "${beginPat},${endPat} { \@^Content-Type: text/plain@,$ {1,/^$/d; ${beginPat}d; p} }"
}

# check mail includes multiple part
boundary="$(<<<"$CONTENT_TYPE" sed -En 's/^.*boundary="?([^"]+)"?.*$/\1/p')"
if [ -n "${boundary}" ]; then
  # print mail part in MIME: text/plain
  body="$(<<<"$body" printTextPart ${boundary})"
fi

6. Write comment to output file

Time to write mail body to output file. For Unix System like tilde.club, set it readable for everyone:

umask 133
Then add basic <style> and <ul> for output file if

  1. This mail is the first comment on page
  2. Output file is illegal HTML

if [ ! -f $output ] || ! xmllint --html --nofixup-base-uris $output &>/dev/null; then
  <<-LAYOUT cat >$output
    <style>
       ul {
         ...
       }
      .comment-body {
        ...
      }
      .replies {
        ...
      }
    </style>
    <ul>
    </ul>
    LAYOUT
fi
Finally, add a list item for comment:

# Append STDIN after line number with variable, or after first <ul> tag
<<-COMMENT sed -i "${line:-/<ul>/}r /dev/stdin" $output
    <li>

    <time datetime="${DATE}">${DATE}</time>
    <a href="mailto:${RECIPIENT}?subject=Comment on page: ${path}&in-reply-to=${MESSAGE_ID}">[reply]</a>

    <div class='comment-body'>
    $(<<<"${body}" ${markdown_bin})
    </div>

    <details class="replies" open="true">
    <summary>replies</summary>
    <ul>
    <!-- ${MESSAGE_ID} -->
    </ul>
    </details>

    </li>
COMMENT
Here we use:

  1. Variable $line to insert this comment into output file
  2. Variables comes from mail header, like $DATE or $RECIPIENT
  3. An anchor element to reply to this comment (with Message-ID in mail header)
  4. Command $markdown_bin to render HTML from mail body
  5. Use <details> to create a foldable part for replies
  6. In block of replies, add value of Message-ID as marker. It could be used to locate where to put other comments as replies

If visitor click the anchor element to reply, then mail client would prepare a new mail with header contains In-Reply-To with $Message-ID

With step.5, we can get $line number for new mail (which reply to this mail) by marker:

# get $line by field "In-Reply-To"
if [ -n "${IN_REPLY_TO}" ]; then
  line=$(grep -n "^<!-- ${IN_REPLY_TO} -->$" $output | cut -d':' -f1)
fi

7. ~/.forward

To co-work with MDA, just put this script into ~/.forward:

# Inside ~/.forward

alice
|"/<PATH>/<TO>/comment.sh --output /srv/http"

Set web page

To include HTML file generated by comment.sh, we can use <object> or <iframe>:

<object type="text/html" data="/mypage.comment.html"></object>
Since height of <object> grows by number of comments, comment block would be fixed sized with scroll bar. Here we can use ResizeObserver to set height by document inside: 
<object ... onload="observeResize(this)"></object>
<script>
  function observeResize(commentBlock) {
    const doc = commentBlock.contentDocument.documentElement
    new ResizeObserver(() => {
      commentBlock.style.height = doc.clientHeight + 'px';
    }).observe(doc);
  }
</script>
Then a comment system is finished! User can simply use anchor element with mailto URI to send comment:

comment system

For tilde.club

This script could be used by non-admin Unix user, like member of tilde.club.

For tilde.club user, you can easily add it into .forward by one-line command:

echo '"|/home/pham/helper/bin/mail/comment.sh --output_dir /home/<USER>/public_html"' >>~/.forward
And then add comment block into web page (index.html in this case), please replace recipient of mailto with your username:

...
<div style="border-radius: 6px; background: lightyellow">
  <a style="display: inline-block; margin: 0.5em 0.5em 0 0; float: right" href="mailto:<USER>@tilde.club?subject=Comment on page: /index.html">[Comment on this page]</a>
  <object type="text/html" data="/index.comment.html" onload="observeResize(this)" style="width: 100%;"></object>
  <script>
    function observeResize(commentBlock) {
      const doc = commentBlock.contentDocument.documentElement
      new ResizeObserver(() => {
        commentBlock.style.height = doc.clientHeight + 'px';
      }).observe(doc);
    }
  </script>
</div>
...
That’s all you need to do!

Improvement

Still, this script only implements very basic features. It could be better with:

  1. Build new HTML file from IMAP or Maildir, because they are mail! (In current stage, you will lose all comments if output file is removed)
  2. Works as CGI to get comment from a web form, I don’t think every user can/want/expect launch mail client from browser. Send a confirmation mail for their comments might be a valid solution.
  3. Prevent texts in <pre> element fails on checking pattern

I am using this script on my own site or tilde.club page. Will make changes by my need. You can always get latest version by the following URL:

Or by git archive from my git server:

git archive --remote git://git.topo.tw/helper HEAD bin/mail/comment.sh | \
tar --strip=2 -xf -
Since this page is working with it, you are welcome to leave comment with any feedback or suggestion! (Click Comment on this page at bottom-right corner)

  1. For example, OSM Mailing List: https://lists.openstreetmap.org/listinfo↩
  2. Beware I use term “Message” here. For my point of view, a mail is a well-organized message. Mail uses header to store metadata like information of sender or subject. If a MDA is delivering data to a specific destination, then it doesn’t matter data is a mail or just a plain-text message.↩
  3. For Sendmail(1) in OpenSMTPD, you should not surround the whole command with double quotes. But for Postfix, double quote is necessary.↩
[Comment on this page]