Journey of a thousand miles starts with jlox

Recently, I've been following along Robert Nystrom's book "Crafting Interpreters" about writing a Lox interpreter. I hate to admit that I've gone deeper into this rabbit hole of "making your own programming language". So, how did I get myself into this mess? Well, um... it all started when I poked around the InterWebs  about Common Lisp.  One thing leads to another and I unknowingly landed myself on an online version of "Build your own Lisp". Interestingly enough, the base code is written in C! What intrigued me the most is the author's MPC (Micro Parser Combinators) library. I realised my C skillset is virtually non-existent when I looked at the MPC repo, so I finally made a brave attempt to overcome my fear of parsers, specifically "Recursive Descent Parser".

This explains why I'm here today.

And why Clojure, you ask? Firstly, I find Clojure and any Lisp-like derivatives enigmatic. For some unexplainable reason, I keep coming back to them. Secondly, I intend to explore more about the Clojure language because of reason number one.

Alright, this is my single step out of a thousand miles, so wish me luck.

My Emacs Initialisation File

I've been writing Clojure codes and using Frederick Giasson's Emacs init file for almost a month. I find his init file gravitates towards simplicity and pragmatism. I've used other people's init file but end up disliking how my Emacs behaves. I would almost certainly blame the unnecessary packages for this.

I've copied Frederick's init file, excluded monokai-theme and turned off a few other settings. Below is the GitHub link: https://github.com/lyeung/dot-emacs

SSH X11 Forwarding and Mac OS X Sierra

I want to run remote Firefox locally on my computer. I have an SSH server running on Ubuntu 15 and an SSH client on Mac OS X Sierra. I've done this before, I would've thought it's as easy as enabling "-X", right? No so, this time. It looks like luck isn't on my side.

I have to enable debugging by using "-vvv" from SSH client to understand the problem:

ssh -vvv -X user@remote.com

The log contained:

debug1: No xauth program.

Warning: untrusted X11 forwarding setup failed: xauth key data not generated

Obviously, I need the extra "-Y" to enable trusted X11 forwarding:

ssh -vvv -XY user@remote.com

This time, it contained:

debug1: No xauth program.

Warning: No xauth data; using fake authentication data for X11 forwarding.

The problem must be related to xauth and I can't find this anywhere on Sierra. Googling the web, the link below came to my attention:
https://origin-discussions-us.apple.com/thread/7685786?start=0&tstart=0

So, I the solution here is to install XQuartz as this comes with the xauth binary.

After the installation, Sierra SSH configuration files /etc/ssh/sshd_config and etc/ssh/ssh_config both contains:

XAuthLocation /opt/X11/bin/xauth

A new directory X11 exists under "/opt" and xauth exists under "/opt/X11/bin".

Another attempt with the following command shows Firefox running remote on Ubuntu and displayed locally:

ssh -XY user@remote.com firefox

To improve local connection speed, you could request compression by adding "-C" when executing SSH:

ssh -XYC user@remote.com firefox

My First Macro

I'm writing my first test code to cover my HugSQL.  I'm still on my training wheels, so I can't be more fancier here:

-- :name get-user-type :? :*
-- :doc get user by type
select id, name from users where type = :type order by id


Now comes the mythical test code. Let's suppose "db/add-user!" exists, performs database inserts into the users table and returns 1 as return value.

(deftest test-user
  (jdbc/with-db-transaction [t-conn *db*]
    (jdbc/db-set-rollback-only! t-conn)
    (is (= 1 (db/add-user!
              t-conn
              {:username      "u123"
               :type          "admin"})))
    (is (= {:username      "u123"
            :type          "admin"}
           (db/get-user-type t-conn {:type "admin"}))))))

But alas! This is an epic failure!

(expected: {:username "u123",
            :type "admin"}
  actual: ({:id 1,
            :username "u123",
            :type "admin"})

It appears that ":id" value is an auto-generated primary key and this equality test is not going to work.

Perhaps, I could define a function to remove the id before comparing? This might work as the id is a surrogated key automatically populated by the database.

(defn match-record

  [expected record]

  (= expected (dissoc record :id)))

Or perhaps I could raise the bar higher by writing a macro? After all, I've heard Clojure macro is a powerful language extension.

(defmacro match-record

  [expected record]

  `(= ~expected (dissoc ~record :id)))

This is really exciting - my first ever Clojure macro!

So, rewriting my test code gives me,

(deftest test-user
  (jdbc/with-db-transaction [t-conn *db*]
    (jdbc/db-set-rollback-only! t-conn)
    (is (= 1 (db/add-user!
              t-conn
              {:username      "u123"
               :type          "admin"})))

    (let [result (db/get-user-type t-conn {:type "admin"})]
        (is (match-record {:username   "u123"
                           :type       "admin"}
                          (first result))))))



I know my macro is sub-optimal here and there are better ways of doing this. But hey! I'm certain my dear mother will be proud of me!

Clojure Atom, Swap!, Reset! and Concurrency

I've encourage myself to keep writing simple Clojure codes. Obviously, this encouragement should extend beyond the traditional "Hello World!" program, shouldn't they?

Recently, I've attempted to write a in-memory database program that keeps track of person record. This in-memory database is nothing but a list (in an abstract sense) and each element in the list contains a map of properties that describes a person.

So, we'll have a variable called "db-ref" bounded to an atom of Clojure vector:

(def db-ref (atom []))

We also need a function to create a person containing id, first name, last name and email :

(defn person [pid lname fname email]

  {:id pid :lname lname :fname fname :email email})

(Does this remind you of a factory pattern?)



To add a person to the database

(defn add-person [p]

  (swap! db-ref conj p))

As you can see, we're adding the parameter "p" (for person) into db-ref using "swap!"



We also need a way to remove a person by their "id". All we have to do is use the same pattern as "add-person", but as oppose to conjoining, we use the remove function (how difficult could this be?):

(defn find-id-p [id element]

  (let [k (:id element)]

    (if (= id k)

      element)))

(defn delete-person [id]

  (swap! db-ref (into [] (remove (partial find-id-p id) @db-ref))))

Surely, our test code below should quickly tell us how awesome we are after the first attempt...

(deftest delete-person-test

  (testing

  (is (= 0 (count-db)))

  (add-person (person "100" "fred" "flintstone" "fred@flintstone"))

  (add-person (person "101" "barney" "rubble" "barney@rubble"))

  (is (= 2 (count-db)))

  (delete-person "100")

  (is (= 1 (count-db)))

  (delete-person "101")

  (is (= 0 (count-db)))))

Running "lein test":

ERROR in (delete-person-test) (APersistentVector.java:292)

Uncaught exception, not in assertion.

expected: nil

  actual: java.lang.IllegalArgumentException: Key must be integer

 at clojure.lang.APersistentVector.invoke (APersistentVector.java:292)

  ...

  ...

  ...

Oopsie! A big fat error telling us something is terribly wrong. Maybe we could sidestep the problem by using reset! function.

(defn delete-person [id]

  (reset! db-ref (into [] (remove (partial find-id-p id) @db-ref))))

Re-running the test should tell us we could totally sidestep the issue.

0 failures, 0 errors.

But before we conclude, Clojure documentation says,

(reset! atom newval)

Sets the value of atom to newval without regard for the
current value. Returns newval.

Our in-memory database may not work correctly in concurrent environments.

Alright, let's try to revisit the problem. Afterall, we're awesome aren't we?

The swap! function documentation says:

(swap! atom f) (swap! atom f x) (swap! atom f x y) 

    (swap! atom f x y & args)

Atomically swaps the value of atom to be:
(apply f current-value-of-atom args). Note that f may be called
multiple times, and thus should be free of side effects.  Returns
the value that was swapped in.

It looks like we shouldn't dereference "db-ref" when calling on swap.

Below is another take to delete a person using the swap! function:

(defn remove-person [id db-ref]

  (remove #(find-id-p id %) db-ref))

(defn delete-person [id]

  (swap! db-ref (partial remove-person id)))

We added an extra helper function to call on the actual remove function. 

Running the test displays:

0 failures, 0 errors.

Destructuring Clojure Vector and Map

Clojure documentation describes destructuring as:

Destructuring is a way to concisely bind names to the values inside a data structure. Destructuring allows us to write more concise and readable code.

So, what does this mean?

Suppose we have a vector that contains the bridge officers found in Star Trek original series:

(def star-trek-bridge-officers [ :kirk :spock :sulu :uhura ])

To retrieve the values, we could destructure the elements this way:

(let [[a b c d ] star-trek-bridge-officers] 

    println (str a " " b " " c " " d))

;; ":kirk :spock :sulu :uhura"

What happened here was the keyword :kirk is bound to the variable a, :spock to b and so on and so forth.


Suppose we have a slightly different vector, where :sulu and :uhura belongs to another subvector:

(def star-trek-bridge-officers-2 [ :kirk :spock [:sulu :uhura]])

To retrieve the values:

(let [[a b [c d]] star-trek-bridge-officers-2] println (str a " " b " " c " " d))

;; ":kirk :spock :sulu :uhura"

If we're not interested with Spock and Uhura, we replace the variables with underscore character "_":

(let [[a _ [c _]] star-trek-bridge-officers-2] 

    println (str a " " c))

;; ":kirk :sulu"

Jim, you can't risk your life on theory!

Now, let's talk about destructuring a map.

Suppose we have a map that contains the rank and names of the crew:

(def star-trek-crew { :captain "kirk" :lt-cmd "spock" :lt-1 "sulu" :lt-2 "uhura" })

To retrieve the values by the keys:

(let [{a :captain b :lt-cmd c :lt-1 d :lt-2 } star-trek-bridge-officers] 

    println (str a " " b " " c " " d))

;; "kirk spock sulu uhura"

The variable a is bound to the value of the key :captain and b is bound to the value of the key :lt-cmd.

To complicate the map structure a bit, I'm going to add a few more members as an embedded map:

(def star-trek-officers { :captain "kirk" :lt-cmd "spock" :lt-1 "sulu" :lt-2 "uhura" 

    :non-bridge-officers {:lt-cmd-1 "scotty" :lt-cmd-2 "mccoy" :nurse "chapel"}})

To retrieve the values including the embedded map:

(let [{a :captain b :lt-cmd c :lt-1 d :lt-2 {x :lt-cmd-1 y :lt-cmd-2 z :nurse} 

    :non-bridge-officers} star-trek-officers] 

        println (str a " " b " " c " " d " " x " " y " " z))

;; "kirk spock sulu uhura scotty mccoy chapel"

As you can see, we need to use :non-bridge-officers as the key to refer to the embedded map.

We all have our darker side. We need it; it's half of what we are.
        It's not really ugly, it's human.

Clojure Plan, Map and Courage

This is one of those moments when the computer says NO!

Let's look at the interesting nature of Clojure's map function.

(defn foo[filenames]
    (map #(touchFile (str "/tmp/" %)) filenames))

Suppose we have a pre-existing function called touchFile that creates an empty file.

This foo function never created any files when called. I must have made a mistake with the directory path or perhaps my filesystem ran out of space or perhaps a bug somewhere in my touchFile function or perhaps...

Hang on, my test code for touchFile passes without any problem!

An attempt to prove something is wrong with the map call....

(defn foo[filenames]
    (map #(println (str "/tmp/" %)) filenames))

And nothing was printed out on my screen!

Something really sinister is going on here...

All you need is the plan, the road map, and the courage to press on to your destination.

After going back to the docs, it mentioned that "map returns a lazy sequence of...". Did you just hear the penny dropped?

So, the fix here is to invoke doall to force the map to evaluate each items immediately.

(defn foo[filenames]
    (doall (map #(touchFile (str "/tmp/" %)) filenames)))

Yay it worked!

Looking at a simple problem at 2 in the morning doesn't help a lot. It's probably time for me to hit the Zs.