Adventures in Clojure and Datomic2018-12-07T22:02:57+00:00http://pelle.github.comPelle Braendgaardpelle@stakeventures.comDatomic transactions as entities2012-07-09T00:00:00+00:00http://pelle.github.com/datomic/2012/07/09/transactions-as-entities
<p>In the previous post you saw <a href="http://pelle.github.com/Datomic/2012/07/08/thinking-in-datomic/">how datomic data is made up of facts</a>. In this post I will discuss how you add facts to Datomic using transactions.</p>
<p>Just like Clojure code is just data, datomic transactions are also just data. They consist of collections of facts. This is very useful as there is not much to learn. You don’t have to learn any particular language just a data format.</p>
<p>I won’t go in to detail about how Datomic works, but for running transactions there is a single transactor responsible for running the transactions. For local dev use this is done in memory in the same process. See <a href="http://docs.datomic.com/architecture.html">Datomic’s Architecture page</a> for more.</p>
<h2 id="the-basics">The basics</h2>
<p>Here is the example I gave in the last article of facts about me:</p>
<figure class="highlight"><pre><code class="language-clojure" data-lang="clojure"><span class="p">{</span><span class="no">:db/id</span><span class="w"> </span><span class="mi">123</span><span class="w"> </span><span class="no">:person/name</span><span class="w"> </span><span class="s">"Pelle Braendgaard"</span><span class="w"> </span><span class="no">:location/city</span><span class="w"> </span><span class="s">"Miami, FL"</span><span class="w"> </span><span class="no">:contact/phone</span><span class="w"> </span><span class="s">"+1 305-555-5555"</span><span class="p">}</span></code></pre></figure>
<p>I add this to the database like this:</p>
<figure class="highlight"><pre><code class="language-clojure" data-lang="clojure"><span class="p">(</span><span class="nf">datomic.api/transact</span><span class="w"> </span><span class="n">conn</span><span class="w">
</span><span class="p">[{</span><span class="no">:db/id</span><span class="w"> </span><span class="o">#</span><span class="n">db/id</span><span class="p">[</span><span class="no">:db.part/user</span><span class="w"> </span><span class="mi">-1000001</span><span class="p">]</span><span class="w"> </span><span class="no">:person/name</span><span class="w"> </span><span class="s">"Pelle Braendgaard"</span><span class="w"> </span><span class="no">:location/city</span><span class="w"> </span><span class="s">"Miami, FL"</span><span class="w"> </span><span class="no">:contact/phone</span><span class="w"> </span><span class="s">"+1 305-555-5555"</span><span class="p">}])</span></code></pre></figure>
<p>Note as I’m creating a new entity I don’t have an entity id. I need to create a temporary entity id which is what <code>#db/id[:db.part/user -1000001]</code> does. Don’t worry to much about it, there are various ways of doing it, but none are important right now. I spent too much time being confused by this in the beginning.</p>
<p>I can modify an entity by adding new facts</p>
<figure class="highlight"><pre><code class="language-clojure" data-lang="clojure"><span class="p">(</span><span class="nf">datomic.api/transact</span><span class="w"> </span><span class="n">conn</span><span class="w">
</span><span class="p">[{</span><span class="no">:db/id</span><span class="w"> </span><span class="mi">123123</span><span class="w"> </span><span class="no">:location/city</span><span class="w"> </span><span class="s">"Santiago, Chile"</span><span class="w"> </span><span class="no">:contact/phone</span><span class="w"> </span><span class="s">"+56 9999 9999"</span><span class="p">}])</span></code></pre></figure>
<p>I could also have done this by adding individual attributes:</p>
<figure class="highlight"><pre><code class="language-clojure" data-lang="clojure"><span class="p">(</span><span class="nf">datomic.api/transact</span><span class="w"> </span><span class="n">conn</span><span class="w">
</span><span class="p">[[</span><span class="no">:db/add</span><span class="w"> </span><span class="mi">123123</span><span class="w"> </span><span class="no">:location/city</span><span class="w"> </span><span class="s">"Santiago, Chile"</span><span class="p">]</span><span class="w">
</span><span class="p">[</span><span class="no">:db/add</span><span class="w"> </span><span class="mi">123123</span><span class="w"> </span><span class="no">:contact/phone</span><span class="w"> </span><span class="s">"+56 9999 9999"</span><span class="p">]])</span></code></pre></figure>
<p>You can retract a datom. This doesn’t actually remove it, it just marks that fact as no longer valid.</p>
<figure class="highlight"><pre><code class="language-clojure" data-lang="clojure"><span class="p">(</span><span class="nf">datomic.api/transact</span><span class="w"> </span><span class="n">conn</span><span class="w">
</span><span class="p">[[</span><span class="no">:db/retract</span><span class="w"> </span><span class="mi">123123</span><span class="w"> </span><span class="no">:contact/phone</span><span class="p">]])</span></code></pre></figure>
<p>You can also retract a whole entity, which retracts all facts about an entity:</p>
<figure class="highlight"><pre><code class="language-clojure" data-lang="clojure"><span class="p">(</span><span class="nf">datomic.api/transact</span><span class="w"> </span><span class="n">conn</span><span class="w">
</span><span class="p">[[</span><span class="no">:db/retractEntity</span><span class="w"> </span><span class="mi">123123</span><span class="p">]])</span></code></pre></figure>
<p>You can still query past database values and retrieve it.</p>
<h2 id="creating-multiple-related-entities">Creating multiple related entities</h2>
<p>Since the transactions just consist of a vector of facts you can easily create multiple entities at once:</p>
<figure class="highlight"><pre><code class="language-clojure" data-lang="clojure"><span class="p">(</span><span class="nf">datomic.api/transact</span><span class="w"> </span><span class="n">conn</span><span class="w">
</span><span class="p">[{</span><span class="no">:db/id</span><span class="w"> </span><span class="o">#</span><span class="n">db/id</span><span class="p">[</span><span class="no">:db.part/user</span><span class="w"> </span><span class="mi">-1000001</span><span class="p">]</span><span class="w"> </span><span class="no">:person/name</span><span class="w"> </span><span class="s">"Pelle Braendgaard"</span><span class="w"> </span><span class="no">:location/city</span><span class="w"> </span><span class="s">"Miami, FL"</span><span class="w"> </span><span class="no">:contact/phone</span><span class="w"> </span><span class="s">"+1 305-555-5555"</span><span class="p">}</span><span class="w">
</span><span class="p">{</span><span class="no">:db/id</span><span class="w"> </span><span class="o">#</span><span class="n">db/id</span><span class="p">[</span><span class="no">:db.part/user</span><span class="w"> </span><span class="mi">-1000002</span><span class="p">]</span><span class="w"> </span><span class="no">:person/name</span><span class="w"> </span><span class="s">"Bob Smith"</span><span class="w"> </span><span class="no">:location/city</span><span class="w"> </span><span class="s">"Coral Gables, FL"</span><span class="w"> </span><span class="no">:contact/phone</span><span class="w"> </span><span class="s">"+1 305-555-9999"</span><span class="p">}])</span></code></pre></figure>
<p>If I wanted to relate them to each other I can use a reference type. I won’t go into details on schema yet. But see <a href="http://docs.datomic.com/schema.html">Datomic’s Schema documentation</a> for more.</p>
<p>I start out by creating a temporary id for each entity outside the transaction so I can use them to reference each other. You create this temporary id specifying the partition your data is in. While your just playing use <code>:db.part/user</code>. I’m still exploring the benefits of creating multiple partitions and will write that up later.</p>
<figure class="highlight"><pre><code class="language-clojure" data-lang="clojure"><span class="p">(</span><span class="k">let</span><span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="n">pelle</span><span class="w"> </span><span class="p">(</span><span class="nf">datomic.api/tempid</span><span class="w"> </span><span class="no">:db.part/user</span><span class="p">)</span><span class="w">
</span><span class="n">bob</span><span class="w"> </span><span class="p">(</span><span class="nf">datomic.api/tempid</span><span class="w"> </span><span class="no">:db.part/user</span><span class="p">)</span><span class="w"> </span><span class="p">]</span><span class="w">
</span><span class="p">(</span><span class="nf">datomic.api/transact</span><span class="w"> </span><span class="n">conn</span><span class="w">
</span><span class="p">[{</span><span class="no">:db/id</span><span class="w"> </span><span class="n">pelle</span><span class="w"> </span><span class="no">:person/name</span><span class="w"> </span><span class="s">"Pelle Braendgaard"</span><span class="w"> </span><span class="no">:location/city</span><span class="w"> </span><span class="s">"Miami, FL"</span><span class="w"> </span><span class="no">:contact/phone</span><span class="w"> </span><span class="s">"+1 305-555-5555"</span><span class="w"> </span><span class="no">:role/friends</span><span class="w"> </span><span class="n">bob</span><span class="p">}</span><span class="w">
</span><span class="p">{</span><span class="no">:db/id</span><span class="w"> </span><span class="n">bob</span><span class="w"> </span><span class="no">:person/name</span><span class="w"> </span><span class="s">"Bob Smith"</span><span class="w"> </span><span class="no">:location/city</span><span class="w"> </span><span class="s">"Coral Gables, FL"</span><span class="w"> </span><span class="no">:contact/phone</span><span class="w"> </span><span class="s">"+1 305-555-9999"</span><span class="w"> </span><span class="no">:role/friends</span><span class="w"> </span><span class="n">pelle</span><span class="p">}])</span></code></pre></figure>
<h2 id="database-functions">Database functions</h2>
<p>Database functions are clojure or java functions you add to the schema of the database. I haven’t touched on the schema yet but here is a very simple example of a function that increments an attribute:</p>
<figure class="highlight"><pre><code class="language-clojure" data-lang="clojure"><span class="p">{</span><span class="w"> </span><span class="no">:db/id</span><span class="w"> </span><span class="o">#</span><span class="n">db/id</span><span class="w"> </span><span class="p">[</span><span class="no">:db.part/user</span><span class="p">]</span><span class="w">
</span><span class="no">:db/ident</span><span class="w"> </span><span class="no">:inc</span><span class="w">
</span><span class="no">:db/fn</span><span class="w"> </span><span class="o">#</span><span class="n">db/fn</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="no">:lang</span><span class="w"> </span><span class="s">"clojure"</span><span class="w">
</span><span class="no">:params</span><span class="w"> </span><span class="p">[</span><span class="n">db</span><span class="w"> </span><span class="n">id</span><span class="w"> </span><span class="n">attr</span><span class="w"> </span><span class="n">amount</span><span class="p">]</span><span class="w">
</span><span class="no">:code</span><span class="w"> </span><span class="s">"(let [ e (datomic.api/entity db id)
orig (attr e 0) ]
[[:db/add id attr (+ orig amount) ]])"</span><span class="p">}}</span></code></pre></figure>
<p>The clojure function it installs is basically:</p>
<figure class="highlight"><pre><code class="language-clojure" data-lang="clojure"><span class="p">(</span><span class="k">defn</span><span class="w"> </span><span class="nb">inc</span><span class="w"> </span><span class="p">[</span><span class="n">db</span><span class="w"> </span><span class="n">id</span><span class="w"> </span><span class="n">attr</span><span class="w"> </span><span class="n">amount</span><span class="p">]</span><span class="w">
</span><span class="p">(</span><span class="k">let</span><span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="n">e</span><span class="w"> </span><span class="p">(</span><span class="nf">datomic.api/entity</span><span class="w"> </span><span class="n">db</span><span class="w"> </span><span class="n">id</span><span class="p">)</span><span class="w">
</span><span class="n">orig</span><span class="w"> </span><span class="p">(</span><span class="nf">attr</span><span class="w"> </span><span class="n">e</span><span class="w"> </span><span class="mi">0</span><span class="p">)</span><span class="w"> </span><span class="p">]</span><span class="w">
</span><span class="p">[[</span><span class="no">:db/add</span><span class="w"> </span><span class="n">id</span><span class="w"> </span><span class="n">attr</span><span class="w"> </span><span class="p">(</span><span class="nb">+</span><span class="w"> </span><span class="n">orig</span><span class="w"> </span><span class="n">amount</span><span class="p">)</span><span class="w"> </span><span class="p">]]))</span></code></pre></figure>
<p>It is based the value of the database as it is at the moment of the transaction and then any other parameters you want to pass it. It should return a vector of data elements just like you would when creating a transaction manually. They can also call other functions.</p>
<p>This is can be called by adding it to your transactional data like this:</p>
<figure class="highlight"><pre><code class="language-clojure" data-lang="clojure"><span class="p">(</span><span class="nf">datomic.api/transact</span><span class="w"> </span><span class="n">conn</span><span class="w">
</span><span class="p">[[</span><span class="no">:inc</span><span class="w"> </span><span class="mi">123123</span><span class="w"> </span><span class="no">:person/age</span><span class="w"> </span><span class="mi">1</span><span class="p">]])</span></code></pre></figure>
<p>Database functions are especially needed when you want to create facts based on existing facts. For example increasing a value in the database.</p>
<p>You could also use it for validation. In this case you throw an exception in your function if something is invalid. The whole transaction will rollback.</p>
<p>It may also be a neat way of abstracting out the creation of common elements within a transaction.</p>
<h2 id="transactions-as-entities">Transactions as entities</h2>
<p>Each transaction has an entity created for it. This by default just has a :db/txInstant timestamp attribute value. But you can add as much information about your transaction that you wish.</p>
<p>This can be useful for auditing by adding user ids, ip addresses etc. But taking it to the extreme lets look at this simple bank application where you transfer funds from one account to the other.</p>
<figure class="highlight"><pre><code class="language-clojure" data-lang="clojure"><span class="p">(</span><span class="k">let</span><span class="w"> </span><span class="p">[</span><span class="n">txid</span><span class="w"> </span><span class="p">(</span><span class="nf">datomic.api/tempid</span><span class="w"> </span><span class="no">:db.part/tx</span><span class="p">)]</span><span class="w">
</span><span class="p">(</span><span class="nf">datomic.api/transact</span><span class="w"> </span><span class="n">conn</span><span class="w"> </span><span class="p">[[</span><span class="no">:transfer</span><span class="w"> </span><span class="n">from</span><span class="w"> </span><span class="n">to</span><span class="w"> </span><span class="n">amount</span><span class="p">]</span><span class="w">
</span><span class="p">{</span><span class="no">:db/id</span><span class="w"> </span><span class="n">txid,</span><span class="w"> </span><span class="no">:db/doc</span><span class="w"> </span><span class="n">note</span><span class="w"> </span><span class="no">:ot/from</span><span class="w"> </span><span class="n">from</span><span class="w"> </span><span class="no">:ot/to</span><span class="w"> </span><span class="n">to</span><span class="w"> </span><span class="no">:ot/amount</span><span class="w"> </span><span class="n">amount</span><span class="p">}]))</span></code></pre></figure>
<p>The <code>[:transfer ...]</code> section is a database function performing a transfer between accounts.</p>
<p>The important thing here is that we create a new tempid in the <code>:db.part/tx partition</code>. We can add as many facts as we want to this id which will represent the transaction.</p>
<p>This can be queried as if it was just a regular entity:</p>
<figure class="highlight"><pre><code class="language-clojure" data-lang="clojure"><span class="p">(</span><span class="nf">datomic.api/q</span><span class="w"> </span><span class="o">'</span><span class="p">[</span><span class="no">:find</span><span class="w"> </span><span class="n">?tx</span><span class="w"> </span><span class="no">:in</span><span class="w"> </span><span class="n">$</span><span class="w"> </span><span class="no">:where</span><span class="w"> </span><span class="p">[</span><span class="n">?tx</span><span class="w"> </span><span class="no">:ot/amount</span><span class="w"> </span><span class="n">_</span><span class="p">]]</span><span class="w"> </span><span class="p">(</span><span class="nf">datomic.api/db</span><span class="w"> </span><span class="n">conn</span><span class="p">))</span></code></pre></figure>
<p>The above returns any entity with a <code>:ot/amount</code> attribute.</p>
<p>This transaction anotation is extremely powerful. Instead of creating your own activity tables you can just annotate the transaction instead.</p>
<h2 id="complete-example">Complete example</h2>
<p>I’ve written a <a href="https://gist.github.com/2635666">complete example showing most uses of Datomic transactions in a bank like application</a> that you can look at.</p>
<p>The <a href="https://gist.github.com/2635666#file_schema.dtm">schema</a> contains 2 database functions <code>:transfer</code> and <code>:credit</code>. <code>:transfer</code> calls <code>:credit</code> on two accounts with oposite amounts. <code>:credit</code> throws an exception to ensure sufficient funds in an account.</p>
<p>The <a href="https://gist.github.com/2635666#L52">transfer function</a> calls the above transfer function and annotates the transaction with information about the transaction.</p>
Thinking in Datomic2012-07-08T00:00:00+00:00http://pelle.github.com/datomic/2012/07/08/thinking-in-datomic
<p><a href="http://datomic.com">Datomic</a> is so different than regular databases that your average developer will probably choose to ignore it. But for the developer and startup who takes the time to understand it properly I think it can be a real unfair advantage as a choice for a data layer in your application.</p>
<p>In this article I will deal with the core fundamental definition of how data is stored in Datomic. This is very different from all other databases so before we even deal with querying and transactions I think it’s a good idea to look at it.</p>
<h2 id="datoms-are-facts-about-entities-not-tables">Datoms are facts about entities not tables</h2>
<p>When you initially look at Datomic you can be fooled into thinking that it is just a better relational database as the query language makes it look that way.</p>
<p>In fact it is much more similar to the RDF concept of triples, which consist of entities, attribute keys and values.</p>
<p>Datomic’s core data element is the <em>datom</em> which is like a triple but adds time to it, so I guess you could say it uses Quadruples.</p>
<p>A datom could look like this:</p>
<table class="table table-striped table-bordered"><tr><th>Entity</th><th>Attribute</th><th>Value</th><th>Time</th></tr>
<tr><td><code>123</code></td><td><code>:person/name</code></td><td><code>Pelle</code></td><td><code>2012-07-08T20:19:03.176-00:00</code></td></tr></table>
<h2 id="entities-are-not-rows">Entities are not rows</h2>
<p>An entity is a collection of related facts. An entity is created by creating an entity id and mapping facts to it. These facts can change through time and datomic indexes and remembers this.</p>
<p>Stuart Halloway from Datomic likes to point out that in real life data is not square. A person table in a SQL database is a large square containing the same shaped facts about everyone.</p>
<p>So while loosely speaking you can think of Datomic’s entity id as a primary key in a relational table. Instead of just mapping the columns in that table to the id you can match any attribute in your schema to it.</p>
<p>I’ve been doing a lot of travelling the last half year so I could create an entity representing me at the beginning of the year:</p>
<figure class="highlight"><pre><code class="language-clojure" data-lang="clojure"><span class="p">{</span><span class="no">:db/id</span><span class="w"> </span><span class="mi">123</span><span class="w"> </span><span class="no">:person/name</span><span class="w"> </span><span class="s">"Pelle Braendgaard"</span><span class="w"> </span><span class="no">:location/city</span><span class="w"> </span><span class="s">"Miami, FL"</span><span class="w"> </span><span class="no">:contact/phone</span><span class="w"> </span><span class="s">"+1 305-555-5555"</span><span class="p">}</span></code></pre></figure>
<p>This adds 3 facts about my entity. My name, location and phone number.</p>
<table class="table table-striped table-bordered"><tr><th>Entity</th><th>Attribute</th><th>Value</th><th>Time</th></tr>
<tr><td><code>123</code></td><td><code>:person/name</code></td><td><code>Pelle</code></td><td><code>1970-08-11T20:19:03.176-00:00</code></td></tr>
<tr><td><code>123</code></td><td><code>:location/city</code></td><td><code>Miami, FL</code></td><td><code>2012-01-01T00:00:00.000-00:00</code></td></tr>
<tr><td><code>123</code></td><td><code>:contact/phone</code></td><td><code>+1 305-555-5555</code></td><td><code>2012-01-01T00:00:00.000-00:00</code></td></tr>
</table>
<p>When I moved to Santiago, Chile in late January I added new facts:</p>
<div lang="clojure" class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>{:db/id 123 :location/city "Santiago, Chile" :contact/phone "+56 9999 9999"}
</code></pre></div></div>
<p>This doesn’t change the fact that I lived in Miami before then. It just adds new facts to me. That I now lived in Santiago, Chile and had a new phone number.</p>
<p>So Datomic would now have the following datoms about me:</p>
<table class="table table-striped table-bordered"><tr><th>Entity</th><th>Attribute</th><th>Value</th><th>Time</th></tr>
<tr><td><code>123</code></td><td><code>:person/name</code></td><td><code>Pelle</code></td><td><code>1970-08-11T20:19:03.176-00:00</code></td></tr>
<tr><td><code>123</code></td><td><code>:location/city</code></td><td><code>Miami, FL</code></td><td><code>2012-01-01T00:00:00.000-00:00</code></td></tr>
<tr><td><code>123</code></td><td><code>:contact/phone</code></td><td><code>+1 305-555-5555</code></td><td><code>2012-01-01T00:00:00.000-00:00</code></td></tr>
<tr><td><code>123</code></td><td><code>:location/city</code></td><td><code>Santiago, Chile</code></td><td><code>2012-01-24T00:00:00.000-00:00</code></td></tr>
<tr><td><code>123</code></td><td><code>:contact/phone</code></td><td><code>+56 9999 9999</code></td><td><code>2012-01-24T00:00:00.000-00:00</code></td></tr>
</table>
<p>At any point I can check where did I live at the beginning of the year. This is always available.</p>
<p>I won’t get into how to querying this yet, but have a look at <a href="http://docs.datomic.com/query.html">Datomic’s Querying documentation</a> to wet your beak.</p>
<h2 id="thinking-outside-the-square">Thinking outside the square</h2>
<p>We are so used to putting related data in these large squares that it can take a bit of time to understand that you don’t need to do this anymore.</p>
<p>Datomic namespaces the attributes as a convenience to your application. Instead of creating a schema for a table organize related attributes within the same name space.</p>
<h2 id="duck-typed-data">Duck typed data</h2>
<p>For example you could create a <code>:person/name</code> attribute. But you could also save yourself some repetition and create a general name attribute lets say <code>:general/name</code> that could be used to name any kind of entity.</p>
<p>In my application I have people, organizations, apps and many other kind of entities. By having a generic name attribute I can share code. It is irrelevant that the name is for an organization, app or a person.</p>
<p>In some respects you can think of this as duck typing data. If an entity has a name it is a named entity.</p>
<p>Datomic already has one such attribute that it uses in it’s schema amongst other things called <code>:db/doc</code>. I’m using this as a generic “description” attribute for my entities. So instead of adding a description column to 15 different tables as I might do in a SQL schema I just add a <code>:db/doc</code> value to any entities that need a human readable and changeable description.</p>
<p>In general when modelling your data and you see various patterns repeat try to name space the attributes and reuse them like you would a Mixin in Ruby code.</p>
<h3 id="future-posts">Future posts</h3>
<p>In future posts I probably wont try to duplicate any of <a href="http://docs.datomic.com/getting-started.html">Datomic’s own documentation</a> but focus on small areas that I’ve found useful in getting my mind around Datomic.</p>