Transforming HTML clipboard contents with Emacs to smooth out Mailchimp annoyances: dates, images, comments, colours
| emacsI've recently started handling the Bike Brigade newsletter, so now I'm itching to solve the little bits of friction that get in my way when I work with the rich-text Mailchimp block editor.
I'm not quite ready to generate everything with Org Mode. Sometimes other people go in and edit the newsletter through the web interface, so I shouldn't just dump a bunch of HTML in. (We don't have the more expensive plan that would allow me to make editable templates.) I draft the newsletter as a Slack canvas so more people can weigh in with their suggestions:
And then I redo it in Mailchimp:
My process is roughly:
- Duplicate blocks.
- Copy the text for each item and paste it in. Adjust formatting.
- Update the dates and links. Flip back and forth between the dispatch webpage and Mailchimp, getting the links and the dates just right.
- Download images one by one.
- Replace the images by uploading the saved images. Hunt through lots of files named image (3).png, image (4).png, and so on. Update their attributes and links.
- Change text and link colours as needed by manually selecting the text, clicking on the colour button in the toolbar, and selecting the correct colour.
- Change the text on each button. Switch to Slack, copy the link, switch back to Mailchimp, and update the link.
I think I can get Emacs to make things easier.
Automating buttons
The newsletter includes a button to make it easier to volunteer for deliveries. In case people want to plan ahead, I also include a link to the following week's signups.
Dates are fiddly and error-prone, so I want to
automate them. I can use a Mailchimp code block to
paste in some HTML directly, since I don't think
other people will need to edit this button. Here I
take advantage of org-read-date's clever date
parsing so that I can specify dates like +2Sun
to mean two Sundays from now. That way, I don't
have to do any date calculations myself.
This code generates something like this:
Text from the screenshot
SIGN UP NOW TO DELIVER JUN 23-29
You can also sign up early to deliver Jun 30-Jul 6
Here's the code. It calculates the dates, formats
the HTML code. I use format-time-string to
format just the month part of the dates and
compare them to tell if I can skip the month part
of the end date. After the HTML is formatted, the
code uses xdotool (a Linux command-line tool) to
switch to Google Chrome so that I can paste it in.
(defun my-brigade-copy-signup-block ()
(interactive)
(let* ((newsletter-date (org-read-date nil nil "+Sun"))
(current-week (org-read-date nil t "+Mon"))
(current-week-end (org-read-date nil t "+2Sun"))
(next-week (org-read-date nil t "+2Mon"))
(next-week-end (org-read-date nil t "+3Sun")))
(kill-new
(format
"<div style=\"background-color: #223f4d; text-align: center; max-width: 384px; margin: auto; margin-bottom: 12px;\"><a href=\"https://dispatch.bikebrigade.ca/campaigns/signup?current_week=%s\" target=\"_blank\" class=\"mceButtonLink\" style=\"background-color:#223f4d;border-radius:0;border:2px solid #223f4d;color:#ffffff;display:block;font-family:'Helvetica Neue', Helvetica, Arial, Verdana, sans-serif;font-size:16px;font-weight:normal;font-style:normal;padding:16px 28px;text-decoration:none;text-align:center;direction:ltr;letter-spacing:0px\" rel=\"noreferrer\">SIGN UP NOW TO DELIVER %s-%s</a>
</div>
<p style=\"text-align: center; font-family: 'Helvetica Neue', Helvetica, Arial, Verdana\"><a href=\"https://dispatch.bikebrigade.ca/campaigns/signup?current_week=%s\" style=\"color: #476584; margin-top: 12px; margin-bottom: 12px;\" target=\"_blank\">You can also sign up early to deliver %s-%s</a></p>"
(format-time-string "%Y-%m-%d" current-week)
(upcase (format-time-string "%b %e" current-week))
(format-time-string
(if (string= (format-time-string "%m" current-week)
(format-time-string "%m" current-week-end))
"%-e"
"%b %-e")
current-week-end)
(format-time-string "%Y-%m-%d" next-week)
(format-time-string "%b %e" next-week)
(format-time-string
(if (string= (format-time-string "%m" next-week)
(format-time-string "%m" next-week-end))
"%-e"
"%b %-e")
next-week-end)))
(shell-command "xdotool search --onlyvisible --all Chrome windowactivate windowfocus")))
Now I can use an Org Mode link like
elisp:my-brigade-copy-signup-block to generate
the HTML code that I can paste into a Mailchimp
code block. The button link is underlined even
though the inline style says
text-decoration:none, but it's easy enough to
remove that with Ctrl+u.
Transforming HTML
The rest of the newsletter is less straightforward. I copy parts of the newsletter draft from the canvas in Slack to the block editor in Mailchimp. When I paste it in, I need to do a lot to format the results neatly.
I think I'll want to use this technique of transforming HTML data on the clipboard again in the future, so let's start with a general way to do it. This uses the xclip tool for command-line copying and pasting in X11 environments. It parses the HTML into a document object model (DOM), runs it through various functions sequentially, and copies the transformed results. Using DOMs instead of regular expressions means that it's easier to handle nested elements.
(defvar my-transform-html-clipboard-functions nil "List of functions to call with the clipboard contents.
Each function should take a DOM node and return the resulting DOM node.")
(defun my-transform-html-clipboard (&optional activate-app-afterwards functions text)
"Parse clipboard contents and transform it.
This calls FUNCTIONS, defaulting to `my-transform-html-clipboard-functions'.
If ACTIVATE-APP-AFTERWARDS is non-nil, use xdotool to try to activate that app's window."
(with-temp-buffer
(let ((text (or text (shell-command-to-string "unbuffer -p xclip -o -selection clipboard -t text/html 2>& /dev/null"))))
(if (string= text "")
(error "Clipboard does not contain HTML.")
(insert (concat "<div>"
text
"</div>"))))
(let ((dom (libxml-parse-html-region (point-min) (point-max))))
(erase-buffer)
(dom-print (seq-reduce
(lambda (prev val)
(funcall val prev))
(or functions my-transform-html-clipboard-functions)
dom)))
(shell-command-on-region
(point-min) (point-max)
"xclip -i -selection clipboard -t text/html -filter 2>& /dev/null"))
(when activate-app-afterwards
(call-process "xdotool" nil nil nil "search" "--onlyvisible" "--all" activate-app-afterwards "windowactivate" "windowfocus")))
Saving images
Images from Slack don't transfer cleanly to
Mailchimp. I can download images from Slack one at
a time, but Slack saves them with generic
filenames like image (2).png. Each main
newsletter item has one image, so I'd like to
automatically save the image using the item title.
When I copy HTML from the Slack canvas, images are
included as data URIs. The markup looks like this:
<img src='data:image/png;base64,iVBORw0KGgo...
With the way I do the draft in Slack, images are
always followed by the item title as an h2
heading. If there isn't a heading, the image just
doesn't get saved. If there's no image in a
section, the code clears the variable, so that's
fine too. I can parse and save the images like
this:
(defun my-transform-html-save-images (dom dir &optional file-prefix transform-fn)
(let (last-image)
(dom-search dom
(lambda (node)
(pcase (dom-tag node)
('img
(let ((data (dom-attr node 'src)))
(with-temp-buffer
(insert data)
(goto-char (point-min))
(when (looking-at "data:image/\\([^;]+?\\);base64,")
(setq last-image (cons (match-string 1)
(buffer-substring (match-end 0) (point-max))))))))
('h2
(when last-image
(with-temp-file
(expand-file-name
(format "%s%s.%s"
(or file-prefix "")
(if transform-fn
(funcall transform-fn (dom-texts node))
(dom-texts node))
(car last-image))
dir)
(set-buffer-file-coding-system 'binary)
(insert (base64-decode-string (cdr last-image)))))
(setq last-image nil)))))
dom))
I wrapped this in a small function for newsletter-specific processing:
(defvar my-brigade-newsletter-images-directory "~/proj/bike-brigade/newsletter/images")
(defun my-brigade-save-newsletter-images (dom)
(my-transform-html-save-images
dom
my-brigade-newsletter-images-directory
(concat (org-read-date nil nil "+Sun")
"-news-")
(lambda (heading)
(replace-regexp-in-string
"[^-a-z0-9]" ""
(replace-regexp-in-string
" +"
"-"
(string-trim (downcase heading)))))))
For easier testing, I used xclip -o -selection
clipboard -t text/html > ~/Downloads/test.html to
save the clipboard. To run the code with the saved
clipboard, I can call it like this:
(my-transform-html-clipboard
nil
'(my-brigade-save-newsletter-images)
(with-temp-buffer (insert-file-contents "~/Downloads/test.html") (buffer-string)))
Cleaning up
Now that I've saved the images, I can remove them:
(defun my-transform-html-remove-images (dom)
(dolist (img (dom-by-tag dom 'img))
(dom-remove-node dom img))
dom)
I can also remove the italics that I use for comments.
(defun my-transform-html-remove-italics (dom)
(dolist (node (dom-by-tag dom 'i))
(dom-remove-node dom node))
dom)
Here's how I can test it:
(my-transform-html-clipboard
nil
'(my-brigade-save-newsletter-images
my-transform-html-remove-images
my-transform-html-remove-italics)
(with-temp-buffer (insert-file-contents "~/Downloads/test.html") (buffer-string)))
Removing sections
I put longer comments and instructions under "Meta" headings, which I can automatically remove.
(defvar my-brigade-section nil)
(defun my-brigade-remove-meta-recursively (node &optional recursing)
"Remove <h1>Meta</h1> headings in NODE and the elements that follow them.
Resume at the next h1 heading."
(unless recursing (setq my-brigade-section nil))
(cond
((eq (dom-tag node) 'h1)
(setq my-brigade-section (string-trim (dom-texts node)))
(if (string= my-brigade-section "Meta")
nil
node))
((string= my-brigade-section "Meta")
nil)
(t
(let ((processed
(seq-keep
(lambda (child)
(if (stringp child)
(unless (string= my-brigade-section "Meta")
child)
(my-brigade-remove-meta-recursively child t)))
(dom-children node))))
`(,(dom-tag node) ,(dom-attributes node) ,@processed)))))
Let's try it out:
(my-transform-html-clipboard
nil
'(my-brigade-save-newsletter-images
my-transform-html-remove-images
my-transform-html-remove-italics
my-brigade-remove-meta-recursively)
(with-temp-buffer (insert-file-contents "~/Downloads/test.html") (buffer-string)))
Formatting calls to action
Mailchimp recommends using buttons for calls to action so that they're larger and easier to click than links. In my Slack canvas draft, I use [ link text ] to indicate those calls to action. Wouldn't it be nice if my code automatically transformed those into centered buttons?
(defun my-brigade-format-buttons (dom)
(dolist (node (dom-by-tag dom 'a))
(let ((text (dom-texts node)))
(if (string-match "\\[ *\\(.+?\\) *\\]" text)
;; button, wrap in a table
(with-temp-buffer
(insert
(format "<table align=\"center\" border=\"0\" cellpadding=\"0\" cellspacing=\"0\" role=\"presentation\" data-block-id=\"627\" class=\"mceButtonContainer\" style=\"margin: auto; text-align: center\"><tbody><tr class=\"mceStandardButton\"><td style=\"background-color:#000000;border-radius:0;text-align:center\" valign=\"top\" class=\"mceButton\"><a href=\"%s\" target=\"_blank\" class=\"mceButtonLink\" style=\"background-color:#000000;border-radius:0;border:2px solid #000000;color:#ffffff;display:block;font-family:'Helvetica Neue', Helvetica, Arial, Verdana, sans-serif;font-size:16px;font-weight:normal;font-style:normal;padding:16px 28px;text-decoration:none;text-align:center;direction:ltr;letter-spacing:0px\" rel=\"noreferrer\">%s</a></td></tr></tbody></table>"
(dom-attr node 'href)
(match-string 1 text)))
(dom-add-child-before
(dom-parent dom node)
(car (dom-by-tag (libxml-parse-html-region (point-min) (point-max)) 'table)) node)
(dom-remove-node dom node)))))
dom)
Now I can test those functions in combination:
(my-transform-html-clipboard
nil
'(my-brigade-save-newsletter-images
my-transform-html-remove-images
my-transform-html-remove-italics
my-brigade-remove-meta-recursively
my-brigade-format-buttons)
(with-temp-buffer (insert-file-contents "~/Downloads/test.html") (buffer-string)))
Changing link colours
I also want to change the link colours to match the colour scheme. The newsletter has two parts distinguished by background colours. Bike Brigade updates use black text on a white background, and community updates use white text on a dark blue background so that they're visually distinct. For contrast, I like to use light blue links in the community section, which doesn't match the colour of the links when I paste them in from Slack. This meant manually recolouring the text and each of the links in Mailchimp, which was tedious.
This code changes the colours of the links. It also changes the colours of text by wrapping spans around them. It skips the links we turned into buttons.
(defvar my-brigade-community-text-style "color: #ffffff")
(defvar my-brigade-community-link-style "color: #aed9ef")
(defun my-brigade-recolor-recursively (node)
"Change the colors of links and text in NODE.
Ignore links with the class mceButtonLink.
Uses `my-brigade-community-text-style' and `my-brigade-community-link-style'."
(pcase (dom-tag node)
('table node) ; pass through, don't recurse further
('a ; change the colour
(unless (string= (or (dom-attr node 'class) "") "mceButtonLink")
(dom-set-attribute node 'style my-brigade-community-link-style))
node)
(_
(let ((processed
(seq-map
(lambda (child)
(if (stringp child)
(dom-node 'span `((style . ,my-brigade-community-text-style)) child)
(my-brigade-recolor-recursively child)))
(dom-children node))))
`(,(dom-tag node) ,(dom-attributes node) ,@processed)))))
I can add that to the sequence:
(my-transform-html-clipboard
nil
'(my-brigade-save-newsletter-images
my-transform-html-remove-images
my-transform-html-remove-italics
my-brigade-remove-meta-recursively
my-brigade-format-buttons
my-brigade-recolor-recursively)
(with-temp-buffer (insert-file-contents "~/Downloads/test.html") (buffer-string)))
Wrapping it up
Now that I've made all those little pieces, I can put them together in two interactive functions. The first function will be for the regular colour scheme, and the second function will be for the light-on-dark colour scheme. For convenience, I'll have it activate Google Chrome afterwards so that I can paste the results into the right block.
(defun my-brigade-transform-html (&optional recolor file)
(interactive (list (when current-prefix-arg (read-file-name "File: "))))
(my-transform-html-clipboard
"Chrome"
(append
'(my-brigade-save-newsletter-images
my-transform-html-remove-images
my-transform-html-remove-italics
my-brigade-remove-meta-recursively
my-brigade-format-buttons)
(if recolor '(my-brigade-recolor-recursively)))
(when file
(with-temp-buffer (insert-file-contents file) (buffer-string)))))
(defun my-brigade-transform-community-html (&optional file)
(interactive (list (when current-prefix-arg (read-file-name "File: "))))
(my-brigade-transform-html t file))
And then I can use links like this for quick shortcuts:
[[elisp:(my-brigade-transform-html nil "~/Downloads/test.html")]][[elisp:(my-brigade-transform-community-html "~/Downloads/test.html")]][[elisp:(my-brigade-transform-html)]]
Since this pastes the results as formatted text, it's editable using the usual Mailchimp workflow. That way, other people can make last-minute updates.
With embedded images, the saved HTML is about 8 MB. The code makes quick work of it. This saves about 10-15 minutes per newsletter, so the time investment probably won't directly pay off. But it also reduces annoyance, which is even more important than raw time savings. I enjoyed figuring all this out. I think this technique of transforming HTML in the clipboard will come in handy. By writing the functions as small, composable parts, I can change how I want to transform the clipboard.
Next steps
It would be interesting to someday automate the campaign blocks while still making them mostly editable, as in the following examples:
- tan-yong-sheng/mailchimp_auto: A command line application to automate the email campaign creation process on MailChimp based on the google spreadsheet input
- Automating My Newsletter Generation with MailChimp, Google Sheets, and AWS Lambda - DEV Community
Maybe someday!
(Also, hat tip to this Reddit post that helped me
get xclip to work more reliably from within Emacs
by adding -filter 2>& /dev/null to the end of my
xclip call so it didn't hang.)
