Daniel O'Shea Blog Posts

Determining Changes in Data Across Machines Efficiently with a Merkle Tree

2011-12-07T00:00:00-08:00

Determining Changes in Data Across Machines Efficiently with a Merkle Tree

Introduction

When you have data distributed on multiple machines, you often want to be able to ensure that the data on one computer is the same as data on another. There are several scenarios where knowing the set of data across machines is in sync. In this article we will build a straight forward implementation of a Merkle tree that can be used to efficiently compute the differences in a set of data between multiple computers.

A What Tree?

A Merkle Tree is a data structure that allows the efficient comparison of a series of elements to find differences. They are used in distributed systems when you need to be able to efficiently find differences between machines without having to transmit the entire data set. They can also be used to efficiently check that a set of files or one very large file has not been modified in transit. For example in a peer to peer scenario, you can query a trusted host to ensure that the data you are receiving from a peer has not been modified.

How Do They Work

Let’s start with a simple example. Suppose you have two computers each running the same program. You want modifications made to the data on one computer to be reflected on the other. You are happily transmitting changes from one machine to the other when one of the machines crashes, or goes offline due to networking issues. When the second machine is reachable again, you want to only transmit the data that is missing or different. A Merkle Tree will help you do just that.

Before we dive into the actual Merkle tree implementation, let’s do the simple naive implementation. You could compute a hash function for every element in the set and transmit those hashes between machines. This will result in transmitting the hash for every element in the set and for a large number of elements we would like to transmit the minimum amount of data.

To do that, we will build a Merkle Tree. A Merkle tree is a special binary tree where only the leaf nodes hold values, the other nodes in the tree are for searching, but we’ll get to that later.

To construct the tree we will start with the root and then build a tree based on the number of leaf nodes we will have. For now, let’s assume that eight leaves are sufficient, then we need a tree of height 3 (remember 2^3 = 8). We construct our balanced binary tree recursively at each level until we reach a leaf nodes. This results in a tree that looks like the one in the diagram below:

For each node in the tree we can compute a hash value for it’s children. Using a hash function (crc if low security, or SHA1 or another cryptographically secure hash when required) we can compute the hash value for each node by recursively computing the hash for the left node and concatenating that hash with the hash of the right node. In the case of a leaf node (no children) then the hash is computed based on the stored value.

Using the example tree in diagram1 we can compute the hash value of the root node according to the following rules.

Let h be the hash function you have decided for your implementation. In our example we will use SHA1. The hash for any node in the tree is h(N) = h(h(N.left) + h(N.right)) where N.left and N.right are left and right sub trees of that node. Therefore the hash for the root h(A) = h(h(B) + h(C))

Where:

    h(B) = h(h(D) + h(E))
    h(C) = h(h(F) + h(G))
    h(D) = h(h(1) + h(2))
    h(E) = h(h(3) + h(4))
    h(F) = h(h(5) + h(6))
    h(G) = h(h(7) + h(8))

In our scala implementation the hashes are computed lazily when requested, but they could be computed once and cached, that optimization is left as an exercise for the reader.

With our tree constructed we can determine if two trees are identical by comparing the hash of the root: h(A) If the root matches congratulations your trees are identical. The identical case is not that interesting, what happens if the third and fifth leaf nodes are different. How do you determine that using the Merkle tree.

In that scenario h(A) != h(A’) where A’ is the root of a tree on another machine. We would then look at the hash for the children of A and see that D and D’ matched, but E and E’ did not, we then compare h(3) and h3’) see they do not match but h(4 and h(4’) do. We would use a similar set of comparisons on the sub tree rooted at C to discover that h(5) and h(5’) do not match.

In our implementation we don’t name nodes with letters and numbers, instead we address sub trees based on the range of leaf nodes they encompass. For example the hash for h(A) would represent the entire range one through eight (1,8), while B represents one through four (1,4) and so on down the tree. A leaf node’s range is it’s position from the left so one would be (1,1).

CODE, or it didn’t happen

I have published an implementation an implementation of a sha1 hashing merkle tree in scala on Github. I chose scala for fun to learn the language. The implementation consists of two classes. Node.scala and Tree.scala

Node Class

The node is not very interesting, it uses the SHA1 message digest algorithm, and recursively determines the hash for any of it’s child nodes. Empty leaf nodes have a hash value equal to empty string, and that value will bubble up the tree if the neighboring children are also empty.

I will call attention to the SHA1 method is, because converting the digest from bytes to a string was somewhat annoying, it’s worth copying it from here should you need to do something similar.

Tree Class

The tree class is where the magic happens :-)

Tree exposes three methods

insert(value : String) - Insert a String into the tree.
hash_for_range(search_start : Int, search_end : Int) - Determine the hash for a range of child nodes.
leaf_node(number : Int) - Get the actual Node object representing a leaf node.

The tree class does a couple “clever” things. On the first insert it creates a balanced tree based on an initial size. The tree will resize itself like a hash table when it is full and another node is inserted. The resize doubles the size of the tree and copies the leaves from the smaller tree to the larger one. Originally I created a new tree on resize but when refactoring the tree, I know only add one more level of the tree, and move the values from the old leaves to the new ones.

Only leaf nodes hold values. The other nodes contain a hash representing their child nodes. Insert automatically assigns the value to the next available leaf node in the tree.

The hash for range method is interesting you specify a range of leaf nodes you would like the hash for. A tree with 8 elements hash_for_range(1, 8) would return the hash of the root, hash_for_range(1,4) would return the hash for the node left of the root, and hash_for_range(8,8) would return just the hash for the 8th leaf node. Hash for range is used to compare the hashes between two trees. This way you can navigate to just the child nodes that are different. Again as we discussed above If the root hash of the two trees is the same, then every node is identical, otherwise you get more and more specific until you find the leaf nodes that are not identical. Finally you use the leaf_node method to retrieve the specific node that is different between the two trees.

The other methods are for utility purposes. This code is still very new and not yet ripe. Please don’t use it in a production system. I will include a license file once the code has some tests.

Until next time. Happy Coding!

Crossfit Update

2011-08-14T00:00:00-07:00

Crossfit Update

It’s been over five months since my first post about Crossfit. It’s been a terrific journey so far. I’ve lost almost sixty pounds, gotten decently strong, and went from being a couch potato to running a couple 5 KM runs per week plus doing 2 Crossfit workouts.

This past Sunday I attended my first group class. Up to this point I’ve been working out one on one with a trainer, focusing on technique and working up the amount of weight I can move safely. The group class was great. I feel like I should have started attending sooner I had such a good time. A few short months ago I would not say running, burpees, and carrying a kettle bell 200m was a good time on a weekend. Now I’m hooked (you might even say addicted) and really look forward to the workouts, even though I hurt like the dickens afterwards.

I’ve changed around my work schedule so I can workout in the middle of the day and still go running in the morning, or at the end of the day. The great thing about Sparkart is we really do have flexible schedules, and everyone at work has been very supportive even when I blow off their meetings because I’m “in the gym”.

Also ending the day with a 5k run along the bay trail in Emeryville with the sun setting behind the mountains is beautiful and relaxing. The hot shower or a trip to the hot tub afterward is like heaven.

The experience overall has been fantastic. The trainers at Crossfit East Bay are excellent and I can not recommend Maximus enough. If you want to get seriously fit, some might even say “Elite” fitness, and have a great time doing it, you owe it to yourself to check them out.

Presentation at the Eastbay HTML5 Meetup

2011-03-17T00:00:00-07:00

Presentation at the Eastbay HTML5 Meetup

Getting out there

Niraj and I presented the work we did for “The Killers” at the Eastbay HTML5 Meetup tonight. Overall I think the the presentation went pretty well. We tried to strike a good balance between talking about the creative process that went into creating the site, and actionable advice regarding SVG and Canvas.

I also gave a simple canvas animation demo:

The code above adds three circles to a canvas element, and moves them from (xCoord, yCoord) to (xDest, yDest) and then back. We apply some easing to give them some acceleration, and to make the animation slightly more interesting.

This is very similar to what we did with on the actual Killers site, just with far fewer elements :-)

Crossfit Part 1

2011-02-07T00:00:00-08:00

Crossfit Part 1

The Beginning

Another year has rolled around, and another birthday on top of that, the big Three Ohh; and I find myself a bit rounder than I’d like. Every year I say I’ll do something about it, sometimes I’ll even make it so far as to join a gym, which lasts for about two months. Then I get bored, stop going and have to wait for the terms of the contract to expire so I can cancel it. This year I wanted to do something different, something that I wouldn’t get bored doing, and something where I could get help along the way. I thought about hiring a trainer in traditional gym, it’s a pretty big investment, so I immediately start googling trainers, reading reviews, asking friends, doing due diligence like I would before any big purchase. A couple friends thought the trainer idea was pretty bogus. “Why hire someone when you are pretty much guaranteed results the first couple months, if you just do any kind of workout”, they would say. Perhaps they are right, maybe anyone can get some short term results, blubbing around in the gym; but I’m looking for a program, a movement, a freaking way to keep this going for the long term. So I was about ninety percent confident that the trainer route was the right path.

Now, there are many trainer options out there. The people available at the local 24hr fitness, boutique gyms, boot camps, etc, etc. It’s quite a field to navigate with all the options. Perhaps drawn by the videos on the internet of what appears to be crazy buff beasts doing amazing thinks like 50 inch box jumps (50” are you kidding me, that’s crazy shit). Anyway, I jumped on yelp.com and browsed around some of the crossfit gyms in my area.

Crossfit - Holy Shit Man, What Are You Thinking?

Yeah, the workouts are intense, some people even told me there is no way I will be able to do a crossfit workout, that I’m crazy for trying. Maybe they are right, but you have to try to find out for sure what you can do.

Crossfit - Session 1

Today I had my first Crossfit’ish workout. It was just an intro, and I didn’t make it all the way through. We managed to get through a rowing warmup, streaching, air squats, deadlifts, overhead press, and some pushups. After the deadlift, I “almost” puked, but was able to walk it off, not the greatest start to looking like one of the guys from 300 :-)

Wednesday will mark day two, tune back then for an update.