Embedding some data in Lisp

Embedding data is very good because when shiping some software you only need to give singgle executable, and the client just run that executable, in C i usually using xxd command to generate header file, now that i program in Common Lisp i need way to embed some data into lisp image, the first thing that came up in my mind is generate some lisp file then i can load that file.

Generate lisp file

(defun blob->lisp (input output &key (var '*blob*))
  (with-open-file (in input :element-type '(unsigned-byte 8))
    (let* ((size (file-length in))
           (data (make-array size :element-type '(unsigned-byte 8))))
      (read-sequence data in)
      (with-open-file (out output
                           :direction :output
                           :if-exists :supersede)
        (format out "(defparameter ~A~%  ~
                       (make-array ~D~%    ~
                         :element-type '(unsigned-byte 8)~%    ~
                         :initial-contents '(" var size)
        (dotimes (i (length data))
          (format out "~D " (aref data i)))
        (format out ")))~%~%")
        (format out "(defmacro with-~A ((var ~A) &body body)~%  ~
  `(uiop:with-temporary-file (:pathname ,var)~%    ~
     (with-open-file (out ,var :direction :output~%                             ~
                               :if-exists :supersede~%                             ~
                               :element-type '(unsigned-byte 8))~%      ~
       (write-sequence ,',~A out))~%    ~
     ,@body))~%"
                (string-downcase (subseq (symbol-name var) 1))
                var
                var))))
  (uiop:file-exists-p output))

that lisp code above generate lisp file, but the problem is i just apply my C mind to Common Lisp, i need to think like Lisp Hacker.

The way of lisp hacker do is using macro system, just like previous code, but it generate at compile time.

Using Macro system

(defmacro define-blob (name path)
  (with-open-file (in path :element-type '(unsigned-byte 8))
    (let* ((size (file-length in))
           (data (make-array size :element-type '(unsigned-byte 8)))
           (macro-name (intern (format nil "WITH-BLOB-~A" (string-upcase (symbol-name name)))
                               (symbol-package name))))
      (read-sequence data in)
      `(progn
         (defparameter ,name
           (make-array ,size :element-type '(unsigned-byte 8)
                 :initial-contents ',(coerce data 'list)))
         (defmacro ,macro-name ((var) &body body)
           `(uiop:with-temporary-file (:pathname ,var)
              (with-open-file (out ,var :direction :output
                    :if-exists :supersede
                    :element-type '(unsigned-byte 8))
                (write-sequence ,,name out))
              ,@body))))))

Why Embedding data ?

embedding some data can be very usefull for example, you create some web app, but you only want ship shinggle executable and works offline, you can embed the assets (eg. bootstrap, fontawesome)

(define-blob *bootstrap-min-css-blob*
  "blob/bootstrap-5.3.8-dist/css/bootstrap.min.css")

(tbnl:define-easy-handler (bootstrap-css :uri "/bootstrap.min.css") ()
  (setf (tbnl:content-type*) "text/css"
        (tbnl:header-out "Cache-Control") "public, max-age=31536000"
        (tbnl:header-out "ETag") "\"bootstrap-5.3.8-css\"")
  (if (string= (tbnl:header-in* "if-none-match") "\"bootstrap-5.3.8-css\"")
      (setf (tbnl:return-code*) 304)
      (write-sequence *bootstrap-min-css-blob* (tbnl:send-headers))))

(define-blob *bootstrap-bundle-min-js-blob*
  "blob/bootstrap-5.3.8-dist/js/bootstrap.bundle.min.js")

(tbnl:define-easy-handler (bootstrap-js :uri "/bootstrap.bundle.min.js") ()
  (setf (tbnl:content-type*) "application/javascript"
        (tbnl:header-out "Cache-Control") "public, max-age=31536000"
        (tbnl:header-out "ETag") "\"bootstrap-5.3.8-js\"")
  (if (string= (tbnl:header-in* "if-none-match") "\"bootstrap-5.3.8-js\"")
    (setf (tbnl:return-code*) 304)
    (write-sequence *bootstrap-bundle-min-js-blob* (tbnl:send-headers))))

now you only need to give single executable to your client.