Jekyll2019-07-28T11:43:47+05:30https://blog.waffles.space/feed.xmlArrow of CodeBlogRavi ShankarExperience Report: Swift from a Rustacean’s perspective2018-12-31T23:59:59+05:302018-12-31T23:59:59+05:30https://blog.waffles.space/2018/12/31/swift-experience-report<p>I’ve been using Swift for over a year now, mostly for the <a href="https://naamio.cloud/projects/kauppa">decentralized e-commerce platform</a> I’ve been working on (and a bunch of side projects like OpenSSL bindings, WebID-TLS prototype, etc.). So, this is something I’ve been wanting to write for a long time, but never found time until now.<sup id="fnref:1"><a href="#fn:1" class="footnote">1</a></sup></p>
<p>A few things about this write-up. Firstly, I love the language!<sup id="fnref:2"><a href="#fn:2" class="footnote">2</a></sup> And second, even though I’ll be comparing bits of both the languages, I’ll never say that one is better than the other. Both are amazing languages and they aim to solve different things. Also, I began using Rust from around its 1.0 release (about 3.5 years back), and Swift only <em>after</em> its 4.0 release (last year), so I’ll be talking about the features they currently have, not what they had / lacked ages ago!</p>
<blockquote>
<p>As a side-note, please don’t expect this post to be a tutorial or have in-depth discussions about language internals or have some sort of order in comparisons.</p>
</blockquote>
<!-- more -->
<p>Before we begin, here are a few basic differences between both the languages, just to show you why a direct comparison isn’t fair.</p>
<table>
<thead>
<tr>
<th>Rust</th>
<th>Swift</th>
</tr>
</thead>
<tbody>
<tr>
<td>Low-level</td>
<td>Not that low-level</td>
</tr>
<tr>
<td>Embedded, kernels, browser/game engines, etc.</td>
<td>Web services, apps for iOS, macOS, etc.</td>
</tr>
<tr>
<td>No garbage collection</td>
<td>Reference-counted garbage collector</td>
</tr>
<tr>
<td>Supports pretty much all platforms</td>
<td>Only XCode and Ubuntu (as of now)</td>
</tr>
<tr>
<td>Supports static builds</td>
<td>Foundation (Swift’s stdlib) in Linux <a href="https://bugs.swift.org/browse/SR-2205">can’t be statically linked yet</a></td>
</tr>
</tbody>
</table>
<p>With this in place, let’s proceed.</p>
<h2 id="basic-syntax">Basic syntax</h2>
<p>In Rust (being expression-based), semicolons have meaning, whereas Swift doesn’t need semicolons to separate statements (unless they’re on the same line), like Python.</p>
<p>Here’s an example demonstrating a function that simply fetches the value of the given env variable. It’s a common practice - instead of dealing with importing <a href="https://developer.apple.com/documentation/foundation/"><code class="highlighter-rouge">Foundation</code></a> (stdlib) and calling <code class="highlighter-rouge">ProcessInfo</code> everywhere, we have this nice abstraction.</p>
<div class="language-swift highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">import</span> <span class="kt">Foundation</span>
<span class="kd">func</span> <span class="nf">getEnvVariable</span><span class="p">(</span><span class="n">name</span> <span class="nv">variable</span><span class="p">:</span> <span class="kt">String</span><span class="p">)</span> <span class="o">-></span> <span class="kt">String</span><span class="p">?</span> <span class="p">{</span>
<span class="k">return</span> <span class="kt">ProcessInfo</span><span class="o">.</span><span class="n">processInfo</span><span class="o">.</span><span class="n">environment</span><span class="p">[</span><span class="n">variable</span><span class="p">]</span> <span class="c1">// explicit return</span>
<span class="p">}</span>
<span class="k">let</span> <span class="nv">port</span> <span class="o">=</span> <span class="nf">getEnvVariable</span><span class="p">(</span><span class="nv">name</span><span class="p">:</span> <span class="s">"LISTEN_PORT"</span><span class="p">)</span><span class="o">!</span>
</code></pre></div></div>
<p>If we were to write the same thing in Rust, then we’d do something like:</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">use</span> <span class="nn">std</span><span class="p">::</span><span class="n">env</span><span class="p">;</span>
<span class="k">fn</span> <span class="nf">get_env_variable</span><span class="p">(</span><span class="n">name</span><span class="p">:</span> <span class="o">&</span><span class="nb">str</span><span class="p">)</span> <span class="k">-></span> <span class="nb">Option</span><span class="o"><</span><span class="nb">String</span><span class="o">></span> <span class="p">{</span>
<span class="nn">env</span><span class="p">::</span><span class="nf">var</span><span class="p">(</span><span class="n">name</span><span class="p">)</span><span class="nf">.ok</span><span class="p">()</span>
<span class="p">}</span>
<span class="k">let</span> <span class="n">port</span> <span class="o">=</span> <span class="nf">get_env_variable</span><span class="p">(</span><span class="s">"LISTEN_PORT"</span><span class="p">)</span><span class="nf">.unwrap</span><span class="p">();</span>
</code></pre></div></div>
<p>Similar, right?</p>
<h3 id="imports-optionals-and-argument-labels">Imports, optionals and argument labels</h3>
<p><code class="highlighter-rouge">import</code> syntax was probably the first thing I felt weird about. It’s like a “recursive glob import”. In Rust, you need to be very specific about what you want in your module, but here you import a package at the top of the file, and <strong>you get to use all</strong> (public) stuff from that package! This means, if we’d imported multiple libs, then it’s hard to know (without an IDE) where some item (type / function / whatever) is from - not to mention that we also need to <code class="highlighter-rouge">grep</code> the item in that lib to find wherever it’s located.</p>
<p>In order to reduce the burden, Swift devs arrange their modules in such a way that they’re self-explanatory. An example would be Vapor’s <a href="https://github.com/vapor/postgresql">PostgreSQL driver lib</a>. There, we have <a href="https://github.com/vapor/postgresql/blob/92881d8b29b7fef572b0d3d56b71527e8a4baeca/Sources/PostgreSQL/Connection/PostgreSQLConnection.swift"><code class="highlighter-rouge">PostgreSQLConnection</code> type in its own module</a>, but then we also have <a href="https://github.com/vapor/postgresql/tree/92881d8b29b7fef572b0d3d56b71527e8a4baeca/Sources/PostgreSQL/Connection">a number of <code class="highlighter-rouge">PostgreSQLConnection+Foo.swift</code> files</a> that contain additional implementations for that type related to some behavior “Foo” (in different modules).</p>
<p>Then, there are the optionals. In Rust, <code class="highlighter-rouge">Option</code> is <a href="(https://doc.rust-lang.org/std/option/enum.Option.html)">just like any other enum</a>, which means <code class="highlighter-rouge">None</code> is just another value. In Swift, even though <a href="https://github.com/apple/swift/blob/0d4a5853bf665eb860ad19a16048664899c6cce3/stdlib/public/core/Optional.swift#L122"><code class="highlighter-rouge">Optional</code> is an enum</a>, it’s baked into the compiler such that <code class="highlighter-rouge">?</code> operator (in suffix) represents an optional type, which means <code class="highlighter-rouge">nil</code> is a special value to indicate nothing. As a result of this, unwrapping an optional can be as simple as using another (exclamation <code class="highlighter-rouge">!</code>) operator.</p>
<p>In the above example, <code class="highlighter-rouge">name</code> is the label and <code class="highlighter-rouge">variable</code> is the actual argument to be used in the function body. Argument labeling was weird at first, but nowadays, I wanna label everything! Here’s a snippet from our platform:</p>
<div class="language-swift highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">let</span> <span class="nv">product</span> <span class="o">=</span> <span class="k">try</span> <span class="n">products</span><span class="o">.</span><span class="nf">getProductVariant</span><span class="p">(</span><span class="nv">for</span><span class="p">:</span> <span class="n">unit</span><span class="o">.</span><span class="n">product</span><span class="p">,</span>
<span class="nv">from</span><span class="p">:</span> <span class="n">order</span><span class="o">.</span><span class="n">shippingAddress</span><span class="p">)</span>
<span class="k">try</span> <span class="nf">checkCurrency</span><span class="p">(</span><span class="nv">for</span><span class="p">:</span> <span class="n">product</span><span class="p">)</span>
<span class="k">let</span> <span class="nv">inventoryItem</span> <span class="o">=</span> <span class="k">try</span> <span class="n">inventory</span><span class="o">.</span><span class="nf">getInventoryItem</span><span class="p">(</span><span class="nv">for</span><span class="p">:</span> <span class="n">product</span><span class="o">.</span><span class="n">inventoryId</span><span class="p">)</span>
<span class="k">try</span> <span class="nf">updateConsumedInventory</span><span class="p">(</span><span class="nv">for</span><span class="p">:</span> <span class="n">inventoryItem</span><span class="p">,</span> <span class="nv">with</span><span class="p">:</span> <span class="n">product</span><span class="p">,</span> <span class="nv">in</span><span class="p">:</span> <span class="n">unit</span><span class="p">)</span>
</code></pre></div></div>
<p>With labeling, it’s possible to write some cool expressive code.</p>
<h3 id="try--catch">try … catch</h3>
<p>In Swift, <code class="highlighter-rouge">Error</code> is a <em>protocol</em> (interface, if you want) which can be implemented for any type, just like Rust, where <a href="https://doc.rust-lang.org/std/error/trait.Error.html">it’s a trait</a> (another buzzword!). Only difference is that here, an error value can be <em>thrown</em> by some operation and can be <em>caught</em> elsewhere when it bubbles up.</p>
<p>To see this in action, let’s write a function which throws 95% of the time:</p>
<div class="language-swift highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">enum</span> <span class="kt">Luck</span><span class="p">:</span> <span class="kt">Error</span> <span class="p">{</span>
<span class="k">case</span> <span class="n">worse</span>
<span class="k">case</span> <span class="n">bad</span>
<span class="p">}</span>
<span class="kd">func</span> <span class="nf">iFeelLucky</span><span class="p">()</span> <span class="k">throws</span> <span class="p">{</span> <span class="c1">// mark explicitly</span>
<span class="k">let</span> <span class="nv">i</span> <span class="o">=</span> <span class="kt">Int</span><span class="o">.</span><span class="nf">random</span><span class="p">(</span><span class="nv">in</span><span class="p">:</span> <span class="mi">0</span> <span class="o">..<</span> <span class="mi">100</span><span class="p">)</span> <span class="c1">// generate Int in [0, 100)</span>
<span class="c1">// switch: not just ranges, but patterns (tuples and enums too!)</span>
<span class="k">switch</span> <span class="n">i</span> <span class="p">{</span>
<span class="k">case</span> <span class="mi">0</span> <span class="o">..<</span> <span class="mi">50</span><span class="p">:</span>
<span class="k">throw</span> <span class="kt">Luck</span><span class="o">.</span><span class="n">worse</span>
<span class="k">case</span> <span class="mi">51</span> <span class="o">..<</span> <span class="mi">95</span><span class="p">:</span> <span class="c1">// biased rejection</span>
<span class="k">throw</span> <span class="kt">Luck</span><span class="o">.</span><span class="n">bad</span>
<span class="k">default</span><span class="p">:</span>
<span class="k">return</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="k">do</span> <span class="p">{</span>
<span class="k">try</span> <span class="nf">iFeelLucky</span><span class="p">()</span>
<span class="p">}</span> <span class="k">catch</span> <span class="kt">Luck</span><span class="o">.</span><span class="n">worse</span> <span class="p">{</span>
<span class="nf">print</span><span class="p">(</span><span class="s">"Definitely not!"</span><span class="p">)</span>
<span class="p">}</span> <span class="k">catch</span> <span class="kt">Luck</span><span class="o">.</span><span class="n">bad</span> <span class="p">{</span>
<span class="nf">print</span><span class="p">(</span><span class="s">"Try again?"</span><span class="p">)</span>
<span class="p">}</span> <span class="k">catch</span> <span class="p">{</span>
<span class="c1">// some other sorcery?</span>
<span class="p">}</span>
</code></pre></div></div>
<p>The <code class="highlighter-rouge">do { }</code> block represents your <em>trial</em> area, and you <em>catch</em> the error next to that block. It’s worth mentioning that when you <code class="highlighter-rouge">throw</code> your error value, the actual error type is erased (since it gets casted to the <code class="highlighter-rouge">Error</code> interface), so you can throw any kind of error from the same block (if you’ve got good reason to do that). Later, when you <code class="highlighter-rouge">catch</code>, you can pattern match by casting it back to the actual error type.<sup id="fnref:3"><a href="#fn:3" class="footnote">3</a></sup></p>
<h2 id="types">Types</h2>
<p>All types (and protocols) in Swift can be <em>extended</em> - regardless of whether they’re from a foreign package, like Foundation. For example, we could extend strings with an alphanumeric check like so:</p>
<div class="language-swift highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">import</span> <span class="kt">Foundation</span>
<span class="kd">extension</span> <span class="kt">String</span> <span class="p">{</span>
<span class="c1">/// Checks if the given string is alphanumeric.</span>
<span class="kd">public</span> <span class="kd">func</span> <span class="nf">isAlphaNumeric</span><span class="p">()</span> <span class="o">-></span> <span class="kt">Bool</span> <span class="p">{</span>
<span class="k">return</span> <span class="k">self</span><span class="o">.</span><span class="nf">range</span><span class="p">(</span><span class="nv">of</span><span class="p">:</span> <span class="s">"[^a-zA-Z0-9]"</span><span class="p">,</span> <span class="nv">options</span><span class="p">:</span> <span class="o">.</span><span class="n">regularExpression</span><span class="p">)</span> <span class="o">==</span> <span class="kc">nil</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<p>We’ve just added some functionality to a type that doesn’t belong to us! It’s an useful abstraction, yes, but Rust doesn’t allow you to do this,<sup id="fnref:4"><a href="#fn:4" class="footnote">4</a></sup> and I think there’s a good reason for it. When you start extending stuff you don’t own, users will have trouble finding the implementation - whether it’s from your package, or it’s from a dependency, or whether this has existed in a core package all this time!</p>
<p>That said, I’m not against it either (I’m doing it myself!). I’m simply unsure about the downsides (if any) to not using / having this feature.</p>
<h3 id="structs-and-classes">Structs and Classes</h3>
<p>In addition to enums and tuples, Swift supports structs and classes - both have static and stored properties (readable, writable or computed). Also, access to types, properties and methods can be <a href="https://docs.swift.org/swift-book/LanguageGuide/AccessControl.html">controlled with modifiers</a>.<sup id="fnref:5"><a href="#fn:5" class="footnote">5</a></sup></p>
<p>Let’s take a dumb struct:</p>
<div class="language-swift highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">import</span> <span class="kt">Foundation</span>
<span class="kd">public</span> <span class="kd">struct</span> <span class="kt">Customer</span> <span class="p">{</span>
<span class="kd">public</span> <span class="k">let</span> <span class="nv">id</span><span class="p">:</span> <span class="kt">UUID</span>
<span class="kd">public</span> <span class="k">let</span> <span class="nv">createdAt</span><span class="p">:</span> <span class="kt">Date</span>
<span class="kd">public</span> <span class="k">var</span> <span class="nv">updatedAt</span><span class="p">:</span> <span class="kt">Date</span>
<span class="kd">public</span> <span class="k">var</span> <span class="nv">firstName</span> <span class="o">=</span> <span class="s">""</span>
<span class="kd">public</span> <span class="k">var</span> <span class="nv">lastName</span> <span class="o">=</span> <span class="s">""</span>
<span class="kd">public</span> <span class="k">var</span> <span class="nv">primaryEmail</span> <span class="o">=</span> <span class="s">""</span>
<span class="kd">public</span> <span class="k">var</span> <span class="nv">isValid</span><span class="p">:</span> <span class="kt">Bool</span> <span class="p">{</span>
<span class="k">return</span> <span class="o">!</span><span class="n">firstName</span><span class="o">.</span><span class="n">isEmpty</span> <span class="o">&&</span> <span class="o">!</span><span class="n">lastName</span><span class="o">.</span><span class="n">isEmpty</span> <span class="c1">// && validate email</span>
<span class="p">}</span>
<span class="kd">public</span> <span class="nf">init</span><span class="p">(</span><span class="nv">id</span><span class="p">:</span> <span class="kt">UUID</span><span class="p">,</span> <span class="n">createdAt</span> <span class="nv">creation</span><span class="p">:</span> <span class="kt">Date</span><span class="p">,</span> <span class="n">updatedAt</span> <span class="nv">updated</span><span class="p">:</span> <span class="kt">Date</span><span class="p">)</span> <span class="p">{</span>
<span class="k">self</span><span class="o">.</span><span class="n">id</span> <span class="o">=</span> <span class="n">id</span>
<span class="n">createdAt</span> <span class="o">=</span> <span class="n">creation</span> <span class="c1">// no name collision - can ignore `self`</span>
<span class="n">updatedAt</span> <span class="o">=</span> <span class="n">updated</span>
<span class="p">}</span>
<span class="kd">public</span> <span class="nf">init</span><span class="p">()</span> <span class="p">{</span>
<span class="k">let</span> <span class="nv">date</span> <span class="o">=</span> <span class="kt">Date</span><span class="p">()</span>
<span class="k">self</span><span class="o">.</span><span class="nf">init</span><span class="p">(</span><span class="nv">id</span><span class="p">:</span> <span class="kt">UUID</span><span class="p">(),</span> <span class="nv">createdAt</span><span class="p">:</span> <span class="n">date</span><span class="p">,</span> <span class="nv">updatedAt</span><span class="p">:</span> <span class="n">date</span><span class="p">)</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<p>I’m ambivalent about having methods as part of the type itself, but other than that, I like a number of things here:<sup id="fnref:6"><a href="#fn:6" class="footnote">6</a></sup></p>
<ol>
<li><strong>Foundation has a lot of stuff!</strong> So far, we’ve seen random number generation, regex, UUID and datetime. It’s nice to have all these things in stdlib, so it’s one less worry for us.</li>
<li>Mutation is field-specific. In the above example, <code class="highlighter-rouge">id</code> cannot be changed for an instance (even if the instance itself is mutable).</li>
<li>Functions could have the same names, as long as they have different signatures. <code class="highlighter-rouge">init</code> is special (in that it’s the constructor), but it’s no different from any other function.</li>
</ol>
<h3 id="values-and-references">Values and References</h3>
<p>All classes in Swift are “reference types” and all other types are “value types”. The difference is that instances of <strong>value types are copied</strong>. Coming from Rust, this felt like <em>infidelity</em>, but well, that’s what you pay for using languages with automatic memory management. Arrays and dictionaries are structs, so every time you assign them to some variable or pass them to another function, they get copied!</p>
<div class="language-swift highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">var</span> <span class="nv">a</span> <span class="o">=</span> <span class="p">[</span><span class="mi">0</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">5</span><span class="p">,</span> <span class="mi">10</span><span class="p">]</span> <span class="c1">// array is a struct</span>
<span class="k">var</span> <span class="nv">b</span> <span class="o">=</span> <span class="n">a</span> <span class="c1">// copied</span>
<span class="n">b</span><span class="o">.</span><span class="nf">append</span><span class="p">(</span><span class="mi">11</span><span class="p">)</span> <span class="c1">// "a" still has 4 elements</span>
<span class="kd">class</span> <span class="kt">Foo</span> <span class="p">{</span>
<span class="k">var</span> <span class="nv">inner</span> <span class="o">=</span> <span class="p">[</span><span class="mi">0</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">5</span><span class="p">,</span> <span class="mi">10</span><span class="p">]</span>
<span class="p">}</span>
<span class="k">let</span> <span class="nv">f</span> <span class="o">=</span> <span class="kt">Foo</span><span class="p">()</span>
<span class="k">let</span> <span class="nv">g</span> <span class="o">=</span> <span class="n">f</span> <span class="c1">// "f" and "g" hold reference to same class</span>
<span class="n">g</span><span class="o">.</span><span class="n">inner</span><span class="o">.</span><span class="nf">append</span><span class="p">(</span><span class="mi">15</span><span class="p">)</span> <span class="c1">// "f.inner" and "g.inner" are same (5 elements).</span>
</code></pre></div></div>
<h3 id="protocols">Protocols</h3>
<p>Traits are one of those lovely things in Rust. With generics, they’re simply <em>beautiful</em>. Having used to them, it wasn’t hard for me to get into protocols (interfaces of Swift). All types in Swift can implement protocols.<sup id="fnref:7"><a href="#fn:7" class="footnote">7</a></sup></p>
<p>That said, Swift has its limitations when it comes to protocols. Generics and traits in Rust are rather robust. For example, we can do this in Rust but not in Swift:</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">impl</span> <span class="n">MyTrait</span> <span class="k">for</span> <span class="n">T</span> <span class="n">where</span> <span class="n">T</span><span class="p">:</span> <span class="n">MyOtherTrait</span> <span class="p">{</span>
<span class="c">// MyTrait impl</span>
<span class="p">}</span>
</code></pre></div></div>
<p>This translates to, “Implement <code class="highlighter-rouge">MyTrait</code> for <strong>all types</strong> that implement <code class="highlighter-rouge">MyOtherTrait</code>”. This has some wonderful effects. <a href="https://doc.rust-lang.org/std/convert/trait.From.html"><code class="highlighter-rouge">From</code></a> and <a href="https://doc.rust-lang.org/std/convert/trait.Into.html"><code class="highlighter-rouge">Into</code></a> traits are my favorites. If your type implements <code class="highlighter-rouge">From</code>, then (because of this feature) it gets the <code class="highlighter-rouge">Into</code> implementation for free!</p>
<p>In Swift, you can add a protocol extension with such a constraint.</p>
<div class="language-swift highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">extension</span> <span class="kt">MyProtocol</span> <span class="k">where</span> <span class="k">Self</span><span class="p">:</span> <span class="kt">MyOtherProtocol</span> <span class="p">{</span>
<span class="c1">// MyProtocol impl</span>
<span class="p">}</span>
</code></pre></div></div>
<p>But, this doesn’t automatically apply the implementation for all <code class="highlighter-rouge">MyOtherProtocol</code> implementors. You still need to extend your types <em>specifically</em> and mark them like:</p>
<div class="language-swift highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">extension</span> <span class="kt">MyStruct</span><span class="p">:</span> <span class="kt">MyProtocol</span> <span class="p">{}</span>
</code></pre></div></div>
<p>This hasn’t become a big deal for me yet, just saying.</p>
<p>Otherwise, protocols are quite cool. There’s a protocol for <a href="https://developer.apple.com/documentation/swift/hashable">hashing</a>, <a href="https://developer.apple.com/documentation/swift/equatable">equality</a> and <a href="https://developer.apple.com/documentation/swift/sequence">iteration</a> (just like Rust), and there are others like one <a href="https://developer.apple.com/documentation/swift/rawrepresentable">for types that could be raw values</a>, <a href="https://developer.apple.com/documentation/swift/encodable">encoding</a> and <a href="https://developer.apple.com/documentation/swift/decodable">decoding</a>.</p>
<p>Then, there’s <a href="https://developer.apple.com/documentation/swift/codable"><code class="highlighter-rouge">Codable</code></a> which unifies serialization and deserialization (again, built into Foundation). The problem is that it’s not even close to <a href="https://serde.rs/">serde</a>, which is the commonly used encoding / decoding lib in Rust.</p>
<p>In serde, you can do almost anything with a bunch of attributes (you rarely need to write custom code), which is a great perk for using a statically typed language, whereas in Swift, let it be skipping a property, managing a particular property on your own or performing additional validation, <strong>anything</strong> that deviates even a little <strong>requires custom code</strong>. It’s not hard to write, but it’s difficult to maintain - whenever you alter the structs, you need to modify that custom implementation. It’d be nice if it could be done with less effort.</p>
<p>If there’s one thing I like about Swift protocols, it’s automatic box’ing (another perk of managed languages). In Rust, you need to specify the pointer which holds a particular <a href="https://doc.rust-lang.org/book/ch17-02-trait-objects.html">trait object</a>. I don’t want this to change. It’s always been and should always be that way in Rust (I need to know whether I’m using <code class="highlighter-rouge">Box</code>, <code class="highlighter-rouge">Arc</code> or a simple reference), it’s just that it’s sometimes annoying (depending on the use case) to box stuff on our own when you’re dealing with trait objects.</p>
<h2 id="packaging">Packaging</h2>
<p><a href="https://github.com/apple/swift-package-manager/blob/ad40fc69276d5dafd213af9b3aafca1cccd6fe3c/Documentation/PackageDescriptionV4.md">Swift Package Manager</a> reminded me of <a href="https://doc.rust-lang.org/cargo/reference/build-scripts.html">build scripts</a> in Rust, because you write in Swift to build your Swift package, and I like it. But, there’s no central registry upon which SPM relies on (like <a href="https://crates.io/">crates.io</a> for cargo). Instead, it needs Git. In order to specify a package as a dependency, you have to specify the URL of a git repo, and versions are based on tags. That said, you can specify a branch / revision in that repo, or simply use your local path - everything works, so I haven’t had any trouble with it.</p>
<p>I also liked SPM’s model - a package has a name, a number of products (libraries and executables), dependencies and targets. Products depend on targets. Tests are part of targets (called test targets). This means, a package can output any number of executables and libraries, and tests can be located anywhere (typically they’re inside <code class="highlighter-rouge">Tests/</code> in project root). In Rust, we can use <a href="https://doc.rust-lang.org/cargo/reference/manifest.html#the-workspace-section">workspaces</a> to output multiple products, but unit tests cannot exist elsewhere.<sup id="fnref:8"><a href="#fn:8" class="footnote">8</a></sup></p>
<h2 id="the-future">The Future</h2>
<p>I think Swift and Rust have a number of similarities in their features (other than the obvious differences).<sup id="fnref:9"><a href="#fn:9" class="footnote">9</a></sup> I have a blind wish that it gets procedural macros from Rust at some point!</p>
<p>Anyway, it didn’t require much effort for someone coming from Rust to get into Swift (but I guess that’s the case for jumping from Rust into any other language, because well… we’ve learned from the master!). I like the way things are in Swift right now, and I’m looking forward to where it’s headed.</p>
<p>If I were to write a web service today, then Swift will be my choice without any second thoughts.</p>
<hr />
<p><small>I may have left out some things along the way, but I’ll update the post whenever something comes to mind.</small></p>
<div class="footnotes">
<ol>
<li id="fn:1">
<p>I don’t think I’ll be able to write about my WebID-TLS work just yet. I’ve been diagnosed with <a href="https://en.wikipedia.org/wiki/Carpal_tunnel_syndrome">CTS</a> lately, and it’s ascending, so I’ll probably be taking a break from my computer starting this February or something and I need to wrap up some work before that. <a href="#fnref:1" class="reversefootnote">↩</a></p>
</li>
<li id="fn:2">
<p>Not as much as I love Rust though! <a href="#fnref:2" class="reversefootnote">↩</a></p>
</li>
<li id="fn:3">
<p>I uhh… personally hate try-catching - I like Rust’s way of dealing with fallible types using the <a href="https://doc.rust-lang.org/std/result/enum.Result.html"><code class="highlighter-rouge">Result</code></a> enum, but again that’s just my preference. <a href="#fnref:3" class="reversefootnote">↩</a></p>
</li>
<li id="fn:4">
<p>In Rust, you can only extend your own type or implement your own trait to other types. <a href="#fnref:4" class="reversefootnote">↩</a></p>
</li>
<li id="fn:5">
<p>Rust 1.18 introduced <a href="https://blog.rust-lang.org/2017/06/08/Rust-1.18.html">support for even fine-grained access control</a> like enabling access in a particular crate, in a module or even a specified path, etc. (besides public and private) <a href="#fnref:5" class="reversefootnote">↩</a></p>
</li>
<li id="fn:6">
<p>We don’t have any of this in Rust, although sometimes I wish we had argument labeling, differentiating functions based on signatures (not names), and some core crates getting stabilized. <a href="#fnref:6" class="reversefootnote">↩</a></p>
</li>
<li id="fn:7">
<p>Although, protocols marked with <code class="highlighter-rouge">class</code> can only be implemented by classes. <a href="#fnref:7" class="reversefootnote">↩</a></p>
</li>
<li id="fn:8">
<p>They need to exist in the same module. They can stay outside, but they won’t have access to any internally used types, whereas in Swift, you can mark packages as <code class="highlighter-rouge">@testable</code> in imports just for testing. <a href="#fnref:8" class="reversefootnote">↩</a></p>
</li>
<li id="fn:9">
<p>Speaking of the future, Swift has <a href="https://github.com/apple/swift-nio">NIO</a> for event loops and futures (again, somewhat similar to Rust). <a href="#fnref:9" class="reversefootnote">↩</a></p>
</li>
</ol>
</div>Ravi ShankarI’ve been using Swift for over a year now, mostly for the decentralized e-commerce platform I’ve been working on (and a bunch of side projects like OpenSSL bindings, WebID-TLS prototype, etc.). So, this is something I’ve been wanting to write for a long time, but never found time until now.1 I don’t think I’ll be able to write about my WebID-TLS work just yet. I’ve been diagnosed with CTS lately, and it’s ascending, so I’ll probably be taking a break from my computer starting this February or something and I need to wrap up some work before that. ↩The Swiss Army Knife of Hashmaps2018-12-07T22:28:46+05:302018-12-07T22:28:46+05:30https://blog.waffles.space/2018/12/07/deep-dive-into-hashbrown<p>A while back, there was <a href="https://github.com/rust-lang/rust/issues/55514">a discussion</a> comparing the performance of using the <a href="https://github.com/Amanieu/hashbrown">hashbrown crate</a> (based on <a href="https://www.youtube.com/watch?v=ncHmEUmJZf4">Google’s SwissTable</a> implementation<sup id="fnref:1"><a href="#fn:1" class="footnote">1</a></sup>) in the Rust compiler. In the <a href="https://rome.rustfest.eu/">last RustFest</a>, Amanieu was <a href="https://github.com/rust-lang/rust/pull/56241">experimenting on integrating</a> his crate into stdlib, which turned out to have <a href="https://twitter.com/Gankro/status/1067816516652015616">some really promising results</a>. As a result, it’s being planned to move the crate into stdlib.</p>
<p>While the integration is still ongoing, there’s currently no blog post out there explaining SwissTable at the moment. So, I thought I’d dig deeper into the Rust implementation to try and explain how its (almost) identical twin <code class="highlighter-rouge">hashbrown::HashMap</code> works.</p>
<!-- more -->
<h2 id="hashing-and-hashmaps">Hashing and Hashmaps</h2>
<p>In order to establish some terminology, I’m gonna start from scratch. If you know about hashing, hashmaps, open-addressing and cache performance in general, feel free to <a href="#robin-hood-hashing">skip this section</a>.</p>
<p>While arrays and linked lists hold a sequence of items, hashmaps (or tables) hold key/value pairs i.e., they bind <em>values</em> to <em>keys</em>. So, you insert a value for a key and you can later address those values (fetch/remove/whatever) <strong>using the same key</strong>.</p>
<p><em>Hashing</em> is basically computing a special number (the <em>hash</em>) for some <em>hashable</em> object, provided that if two objects are equal, their hashes <em>must</em> be equal. When a <em>hash function</em> produces the same hash for two different objects, we call it a <em>hash collision</em>. A perfect hash function doesn’t result in collisions, but since we’re not in an ideal world, collisions happen (the <em>rate</em> depends on the algorithm).</p>
<p>Hashmaps use arrays as their backend. In the context of hashmaps, the array elements are called <em>buckets</em> or <em>slots</em> (I’ll be using this interchangeably). Let’s take the following array:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code> index | 0 | 1 | 2 | 3 |
-------|-----|-----|-----|-----|
value | | | | |
</code></pre></div></div>
<blockquote>
<p>The <em>value</em> here is the array’s element in that index (usually, it’s the key/value pair as a whole!).</p>
</blockquote>
<p>This array has 4 slots (it could be just one!). We wish to insert a key/value pair <code class="highlighter-rouge">(4, 8)</code>, for which we use a hash function <code class="highlighter-rouge">H(x)</code>. In order to find the index at which the key/value pair should be inserted, the <strong>key is hashed</strong> using the hash function <code class="highlighter-rouge">H(K)</code>, and the index is obtained by modulo’ing the hash using the length of the array.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>H(4) = 12638164110811449565
i = H(4) % 4 = 3
</code></pre></div></div>
<p>For the sake of keeping this post simple, I’ve used an <em>unspecified</em> hash function<sup id="fnref:2"><a href="#fn:2" class="footnote">2</a></sup> (so don’t worry about it) - only the actual hashes matter as far as we’re concerned. And, note that the hash is 64 bits - we’ll be sticking with this size throughout the post. Again for simplicity, we’re only focusing on 64-bit machines.</p>
<p>Anyway, now that we’ve found the index of the bucket, let’s insert the key/value pair. Our array now looks like this:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code> index | 0 | 1 | 2 | 3 |
-------|-----|-----|-----|-----|
value | | | | 4,8 |
</code></pre></div></div>
<p>But, why did we have to add both the key and value <code class="highlighter-rouge">(4, 8)</code>, instead of just the value <code class="highlighter-rouge">8</code>?</p>
<h3 id="dealing-with-collisions">Dealing with collisions</h3>
<p>Right now, for performing an operation on a key/value pair in the map, we <em>find</em> it by following the same steps - hash the key, modulo <code class="highlighter-rouge">N</code>, land on the index and perform the desired action.</p>
<p>Hashing functions, despite that they have a whole 8 bytes, encounter hash collisions. By doing the <em>modulo</em> operation, we’ve greatly reduced their range. Let’s try inserting <code class="highlighter-rouge">(8, 2)</code> into our map;</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>H(8) = 12638161911788193143
i = H(8) % 4 = 3
</code></pre></div></div>
<p>But, we already have an element at index 3!</p>
<p>In order to deal with these collisions, we group similar data together. One way is to assign a linked list to <strong>each bucket</strong>. Now, our map will look like:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code> index | 0 | 1 | 2 | 3 |
-------|-----|-----|-----|-----|
value | | | | |o| |
__|__
| 4,8 |
|-----|
| 8,2 |
-------
</code></pre></div></div>
<p>… and when we wish to find an element, we stop at the bucket, traverse through the linked list and <strong>compare the keys</strong> for locating the value. This method of using another data structure for storing the values in each bucket is called <em>separate chaining</em>. So, if we keep getting collisions for subsequent insertions, our linked list will get bigger and that will impact our performance, right?</p>
<p>Not exactly. This is where we talk about the <em>load factor</em>. Just like all other dynamic data structures, hashmaps should be able to resize at will! Load factor is the ratio of the number of elements in the hashmap to the number of buckets. Once we reach a certain load factor (say, 50%, 70% or 90% depending on your configuration), hashmaps will resize and <em>rehash</em> all the key/value pairs.</p>
<p>Let’s say that our hashmap <em>doubles</em> in size at a load factor of 60%. This means, once we add a third element <code class="highlighter-rouge">(5, 7)</code>, our new hashmap will resize (regardless of whether it’s colliding). It will now look something like:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code> index | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 |
-------|-----|-----|-----|-----|-----|-----|-----|-----|
value | |o| | | | |o| | | | | |o| |
__|__ __|__ __|__
| 5,7 | | 4,8 | | 8,2 |
------- ------- -------
</code></pre></div></div>
<p>The choice of the load factor depends entirely on the internals of your map i.e., how it hashes and determines buckets. For example, you have a weaker hash function which results in adding a number of elements to the same bucket, and in order to reduce traversal, you’d probably go for a resize / rehash when you hit a <em>relatively lesser</em> load factor (the choice is based on a gazillion performance tests!).</p>
<p>However, we have a major performance bottleneck. Firstly, the usage of external data structures require additional allocations / pointers which themselves consume some memory <strong>per element</strong>. And second, when it comes to linked lists, they have worse <em>processor</em> cache performance.</p>
<p>Okay, what does that mean?</p>
<h3 id="cpu-cache">CPU Cache</h3>
<p>Our RAM is faster than say, our SSD (like, a hundred times!), but it’s also a hundred times slower than the CPU<sup id="fnref:3"><a href="#fn:3" class="footnote">3</a></sup>. That’s why CPUs have their own caches (L1, L2, L3, etc.). Data flows from memory to the CPU cache in fixed sized blocks called <em>cache lines</em>. This fetch can take up to a 100 ns (~200-300 clock cycles). In contrast, L1 cache reference is ~1-2 ns, whereas for L2, it’s ~8-12ns (CPU caches are hierarchical, think of L1 as close to the CPU and smaller in size compared to L2).</p>
<p>What this means is that, whenever the CPU needs to read/write to a memory location, it checks the cache(s), and if it’s present, it’s a <em>cache hit</em>, otherwise it’s a <em>cache miss</em>. Whenever a cache miss occurs, we pay the cost of fetching the data from main memory (thereby losing a few hundred CPU cycles by <strong>waiting</strong>).</p>
<p>Coming back to linked lists, the pointers of subsequent nodes could be anywhere, which results in fetching cache lines randomly. This indirection leads to the poor cache performance of linked lists.</p>
<h3 id="the-other-way">The other way</h3>
<p>The second way (for our hashmap) is to get rid of using external data structures completely, and use the same array for storing values alongside buckets. With another element <code class="highlighter-rouge">(12,9)</code>, our map will look like this:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code> index | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 |
-------|-----|-----|-----|-----|------|-----|-----|-----|
value | 5,7 | | | 4,8 | 12,9 | | | 8,2 |
-------|-----|-----|-----|-----|------|-----|-----|-----|
slot | 0 | | | 3 | 3 | | | 7 |
</code></pre></div></div>
<blockquote>
<p>I’ve added another row to indicate the buckets/slots computed from the hashes.</p>
</blockquote>
<p>Note that the new key/value pair is on index 4, even though its slot index is 3. This way, we sequentially fill the empty slots. Also, the ordering is unnecessary.</p>
<p>Let’s try and add a few more elements…</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code> index | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 |
-------|-----|------|-----|-----|------|-----|-----|-----|
value | 5,7 | 29,7 | 6,6 | 4,8 | 12,9 | 3,4 | | 8,2 |
-------|-----|------|-----|-----|------|-----|-----|-----|
slot | 0 | 0 | 1 | 3 | 3 | 2 | | 7 |
</code></pre></div></div>
<p>As you can see, <code class="highlighter-rouge">(3, 4)</code> has a slot index of 2, but since that index already has an element, we’re inserting it into the first empty element we find (which is at 5). During fetching, we land on a slot (based on the hash), compare the keys one by one, and traverse all the way until we either <strong>find a matching pair or an empty slot</strong> (<em>probing</em>).</p>
<p>Here’s the catch. When you remove an element, you can’t simply remove the key/value pair from a slot and be done with it. If you do that, then you’ve created an empty slot, which could result in the map complaining existing values to be non-existent (because the search has encountered an empty slot). So, you have two choices:</p>
<ol>
<li>In addition to removing a pair, we also move the next pair to that slot (shift backwards).<sup id="fnref:4"><a href="#fn:4" class="footnote">4</a></sup> For example, if we removed <code class="highlighter-rouge">(4, 8)</code> in the above map, then we move <code class="highlighter-rouge">(3, 4)</code> to its position (which has the lowest slot index), and the table will now look like:</li>
</ol>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code> index | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 |
-------|-----|------|-----|-----|------|-----|-----|-----|
value | 5,7 | 29,7 | 6,6 | 3,4 | 12,9 | | | 8,2 |
-------|-----|------|-----|-----|------|-----|-----|-----|
slot | 0 | 0 | 1 | 2 | 3 | | | 7 |
</code></pre></div></div>
<ol>
<li>Or, you add a special flag (<em>tombstone</em>) to the removed slots, and when you probe, you skip the slots containing that flag. But, this will also affect the load factor of the hashmap - removing a number of elements in a map followed by inserting a few could trigger a <em>rehash</em>.</li>
</ol>
<p>There are other ways to resolve hash collisions in open addressing (<em>quadratic probing</em>, where we quadratically address the elements instead of sequentially, and <em>double hashing</em> where we use another hash function to land on the slot), but they’re out of scope of this post.</p>
<p>The advantage of linear probing is that it has the best cache performance - if you think about it, linear probing sequentially visits the elements, which means, (most often, depending on the size of the object) the data is part of a cache line, which is great, because we don’t waste CPU cycles (in contrast, quadratic probing doesn’t offer such cache performance and double hashing uses another hash function, which by definition means that we’re spending more work on computing another hash).</p>
<p>Although, we do have a problem with openly addressed maps (linear probing in particular) - <em>clustering</em>. Whenever a collision occurs, we start queueing the key/value pairs. For example, in the above table, because <code class="highlighter-rouge">(29, 7)</code> has the same slot as <code class="highlighter-rouge">(5, 7)</code>, it’s at index 1, and we had to place <code class="highlighter-rouge">(6, 6)</code> at 2 (even though its slot index is 1), and other elements follow.</p>
<p>This way, as the table grows, more elements get pushed away from their actual index (which elongages the search/probe sequence, thereby increasing the cache misses). It also depends on the hash function, in that, a weaker hash function can lead to more collisions, and hence, more clusters.</p>
<h2 id="robin-hood-hashing">Robin Hood hashing</h2>
<p>I’m also going through Robin Hood hashing, because it’s a nice optimization and at the time of writing of this post, Rust used this in <code class="highlighter-rouge">std::collections::HashMap</code>, but again if you know this, feel free to <a href="#hashbrown">skip this section</a>. It has nothing to do with hashbrown itself.</p>
<p>This is one of the few methods to improve the overall efficiency of openly addressed hashmaps. It offers a way to redistribute an existing key/value pair during the insertion of a new pair. For example, let’s take this <em>openly addressed</em> map:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code> index | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 |
-------|-----|------|-----|-----|------|-----|-----|-----|
value | 5,7 | 29,7 | 6,6 | 3,4 | 12,9 | | | 8,2 |
-------|-----|------|-----|-----|------|-----|-----|-----|
slot | 0 | 0 | 1 | 2 | 3 | | | 7 |
</code></pre></div></div>
<p>We’re interested in a variable called <em>distance to the actual slot</em> - I’ll call it <code class="highlighter-rouge">D(value)</code>. For example, the actual slot of <code class="highlighter-rouge">(5, 7)</code> is 0, and it’s located at 0, so the distance is 0. For <code class="highlighter-rouge">(29, 7)</code> on the other hand, the distance is 1, because it’s at 1, even though it should’ve been at 0.</p>
<p>In robin hood hashing, you follow one rule - if the distance to the actual slot of the current element in the slot is <em>less than</em> the distance to the actual slot of the element to be inserted, then we swap both the elements and proceed.</p>
<p>Okay, that was ugly. An example will really help us.</p>
<p>We’d like to insert <code class="highlighter-rouge">(13, 3)</code> into this map. The slot index for 13 is 0. So, let’s start at 0. We already have <code class="highlighter-rouge">(5, 7)</code> located there, which also has a slot index 0.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code> index | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 |
-------|-----|------|-----|-----|------|-----|-----|-----|
value | 5,7 | 29,7 | 6,6 | 3,4 | 12,9 | | | 8,2 |
-------|-----|------|-----|-----|------|-----|-----|-----|
slot | 0 | 0 | 1 | 2 | 3 | | | 7 |
^
| D(13) == 0
(13,3)
</code></pre></div></div>
<p>Both have the same distances i.e., <code class="highlighter-rouge">D(5) == D(13) == 0</code>. We move forward.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code> index | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 |
-------|-----|------|-----|-----|------|-----|-----|-----|
value | 5,7 | 29,7 | 6,6 | 3,4 | 12,9 | | | 8,2 |
-------|-----|------|-----|-----|------|-----|-----|-----|
slot | 0 | 0 | 1 | 2 | 3 | | | 7 |
^
| D(13) == 1
(13,3)
</code></pre></div></div>
<p>Still the same. <code class="highlighter-rouge">D(29) == D(13) == 1</code> (both are 1 slot away from their actual slots). Moving on…</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code> index | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 |
-------|-----|------|-----|-----|------|-----|-----|-----|
value | 5,7 | 29,7 | 6,6 | 3,4 | 12,9 | | | 8,2 |
-------|-----|------|-----|-----|------|-----|-----|-----|
slot | 0 | 0 | 1 | 2 | 3 | | | 7 |
^
| D(13) == 2
(13,3)
</code></pre></div></div>
<p>In the next stop, we see something. <code class="highlighter-rouge">D(13) == 2</code> whereas <code class="highlighter-rouge">D(6) == 1</code> and hence <code class="highlighter-rouge">D(6) < D(13)</code>, which means we should <strong>swap the pairs</strong> and proceed.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code> index | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 |
-------|-----|------|------|-----|------|-----|-----|-----|
value | 5,7 | 29,7 | 13,3 | 3,4 | 12,9 | | | 8,2 |
-------|-----|------|------|-----|------|-----|-----|-----|
slot | 0 | 0 | 0 | 2 | 3 | | | 7 |
^
| D(6) == 1
(6, 6)
</code></pre></div></div>
<p>Now, we go looking for a place to insert <code class="highlighter-rouge">(6, 6)</code>. This way, we compare and swap the key/value pairs based on their distances. In the end, we have this nicely distributed hashmap:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code> index | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 |
-------|-----|------|------|-----|-----|------|-----|-----|
value | 5,7 | 29,7 | 13,3 | 6,6 | 3,4 | 12,9 | | 8,2 |
-------|-----|------|------|-----|-----|------|-----|-----|
slot | 0 | 0 | 0 | 1 | 2 | 3 | | 7 |
</code></pre></div></div>
<p>Since the hashmap has reached almost its capacity, you might be wondering whether this would’ve already triggered a <em>rehash</em>. But, the resize/rehash depends on the load factor, and because this method redistributes the key/value pairs regardless of when they get inserted (<strong>takes away from the rich and gives it to the poor</strong>, hence the name), the hashmaps could now have higher load factors of even 90-95%.</p>
<p>This also brings another improvement to searching. We don’t have to probe all the way until we find an empty slot. We can stop when <code class="highlighter-rouge">D(slot value) < D(query value)</code>, since our rule guarantees that this shouldn’t happen for the key we’re looking for. For example, in the above table, if we wanna query for the key <code class="highlighter-rouge">21</code> (whose slot index is 0), then we can stop at index 3, because at that point <code class="highlighter-rouge">D(6) == 2</code> which is less than <code class="highlighter-rouge">D(21) == 3</code> which wouldn’t have happened if the key/value pair were there. So, we can safely declare that the key doesn’t exist.</p>
<p>Now that we’ve grazed over a lot of things associated with openly-addressed hashmaps<sup id="fnref:5"><a href="#fn:5" class="footnote">5</a></sup>, let’s proceed to hashbrown.</p>
<h2 id="hashbrown">Hashbrown</h2>
<p>I’m not gonna call it “SwissTable” from here on because firstly, even though hashbrown was a port of SwissTable, the author has made a few changes to improve its performance, and second, I didn’t read the C++ code at all - I followed the <code class="highlighter-rouge">hashbrown</code> crate.</p>
<p>An optimization for linear probing is storing some metadata for each key/value pair. The first thing that comes to mind is the check for <em>equality</em> - once we’ve landed on a slot, we probe by comparing the keys in the slots one by one. While this is easier for integers, it gets expensive for bigger objects. We could store the hash, but, that’s another <strong>8 bytes per slot</strong>, which is a huge deal for memory-eating hashmaps!</p>
<p>Let’s reset our map and make some changes to its behavior.</p>
<ol>
<li>Let’s make the initial size of the internal array to <strong>16</strong> (I’ll get into why we’re doing that in a bit, trust me for now). We call this group of 16 elements a <em>group</em>.<sup id="fnref:6"><a href="#fn:6" class="footnote">6</a></sup> So, a map is made of a number of consecutive groups.</li>
<li>For each slot in a group, let’s assign <strong>a byte for metadata</strong> and call it <em>control byte</em>. Again, we’ll see what it is soon.</li>
</ol>
<p>It will now look something like this:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code> index | 0 | 1 | 2 | 3 | 4 | ... | 15 |
-------|--------|--------|--------|--------|--------| |--------|
value | | | | | | ... | |
-------|--------|--------|--------|--------|--------| |--------|
ctrl |00000000|00000000|00000000|00000000|00000000| ... |00000000|
</code></pre></div></div>
<p>I’m skipping a number of elements (irrelevant to us) so that it fits to the screen. I’m also representing the “control byte” in bits, because we’ll be playing with bits. As a result, there are 128 bits in this group - a nice round number (we’ll see why).</p>
<p>Our first candidate for insertion is <code class="highlighter-rouge">(5, 7)</code>.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>H(5) = 12638147618137026400
</code></pre></div></div>
<p>Taking the top (<em>most significant</em>) 7 bits of this hash<sup id="fnref:7"><a href="#fn:7" class="footnote">7</a></sup> and calling it <code class="highlighter-rouge">H2(x)</code>, we get:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>H2(5) = H(5) >> 57 = 87 = 0b1010111
</code></pre></div></div>
<p>These will be the <em>bottom</em> 7 bits of our control byte.<sup id="fnref:8"><a href="#fn:8" class="footnote">8</a></sup> Then, we use a special bit for our own purposes (to indicate whether the slot value is empty, full or deleted) and this will be the top bit. The states are now represented as follows:<sup id="fnref:9"><a href="#fn:9" class="footnote">9</a></sup></p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>0b11111111 // EMPTY (all bits are set)
0b10000000 // DELETED (top bit is set)
0b0....... // FULL (whenever the top bit isn't set)
</code></pre></div></div>
<p>In light of this information, all the slots are empty, so our map will look like:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code> index | 0 | 1 | 2 | 3 | 4 | ... | 15 |
-------|--------|--------|--------|--------|--------| |--------|
value | | | | | | ... | |
-------|--------|--------|--------|--------|--------| |--------|
ctrl |11111111|11111111|11111111|11111111|11111111| ... |11111111|
</code></pre></div></div>
<p>To recall what we’ve done so far, we’re storing the top 7 bits of our key’s hash in our control byte, and in addition to that, we use the top bit of the control byte to indicate whether the slot is full, empty or deleted.</p>
<p>Going back to our candidate <code class="highlighter-rouge">(5, 7)</code>, its slot index is 0 i.e., <code class="highlighter-rouge">H(5) % 16 == 0</code>.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code> index | 0 | 1 | 2 | 3 | 4 | ... | 15 |
-------|--------|--------|--------|--------|--------| |--------|
value | (5,7) | | | | | ... | |
-------|--------|--------|--------|--------|--------| |--------|
ctrl |01010111|11111111|11111111|11111111|11111111| ... |11111111|
</code></pre></div></div>
<p>Once the pair is inserted, we’ve also set its control byte to <code class="highlighter-rouge">H2(5)</code>, since the top bit is zero anyway (because it’s now <code class="highlighter-rouge">FULL</code>). Now, let’s try inserting <code class="highlighter-rouge">(39, 8)</code>.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>H(39) = 17050702200253021726
i = H(39) % 16 = 2
H2(39) = H(39) >> 57 = 54 = 0b110110
</code></pre></div></div>
<p>And, we do the same thing.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code> index | 0 | 1 | 2 | 3 | 4 | ... | 15 |
-------|--------|--------|--------|--------|--------| |--------|
value | (5,7) | | (39,8) | | | ... | |
-------|--------|--------|--------|--------|--------| |--------|
ctrl |01010111|11111111|00110110|11111111|11111111| ... |11111111|
</code></pre></div></div>
<p>Now, we’re all set. Let’s start addressing the reasons behind whatever we’ve done.</p>
<h3 id="why-the-round-number">Why the round number?</h3>
<p>Each group contains 16 slots summing up to 8 control bytes. The first natural question is why we’ve restricted the group size to 128 bits.</p>
<p>When we query for a key in the map, we first use the hash to land on the group corresponding to a slot, find the offset of the slot inside that group and start probing by comparing against the 7-bit hashes in the control byte.</p>
<p>The boost here is that the control bytes (being 128 bits) can fit into an L1 cache line. This means, we can probe an <em>entire</em> group really quickly, before having to fetch another group from L2 or L3 or whatever. And, we don’t have to worry about comparing the keys at all, until we encounter all 7 bits matching in a control byte.</p>
<p>There’s one other cool optimization for modern processors. Modern CPUs support SIMD instructions, which basically means that we can do some operation (add or multiple or compare, etc.) on multiple values at the same time in a processor! <a href="https://en.wikipedia.org/wiki/Streaming_SIMD_Extensions">SSE</a> is a subset of that where we can work on different types such as two 64-bit floats, four 32-bit integers or <strong>sixteen 8-bit integers</strong>.</p>
<p>Now, our workflow will simply be:</p>
<ol>
<li><a href="https://doc.rust-lang.org/nightly/core/arch/x86/fn._mm_set1_epi8.html?search=_mm_load%20si128">Load up</a> these bytes from an array.</li>
<li><a href="https://doc.rust-lang.org/nightly/core/arch/x86_64/fn._mm_set1_epi8.html">Set the byte</a> to compare.</li>
<li><a href="https://doc.rust-lang.org/nightly/core/arch/x86_64/fn._mm_cmpeq_epi8.html">Compare</a> both the values.</li>
<li><a href="https://doc.rust-lang.org/nightly/core/arch/x86_64/fn._mm_movemask_epi8.html">Mask</a> the values from comparison to true or false.</li>
</ol>
<p>And, that’s finding the results from 16 slots in <strong>four CPU instructions!</strong></p>
<p>In the above example, if we wish to find <code class="highlighter-rouge">39</code>, then all we have to do is find the position of the group using the hash, find the 7-bit value <code class="highlighter-rouge">0b110110</code> from the hash, and do this:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>1. Load group (A)
--------------------------------------------- ------------
| 01010111 | 11111111 | 00110110 | 11111111 | ... | 11111111 |
--------------------------------------------- ------------
2. Set comparable 0b110110 (B)
--------------------------------------------- ------------
| 00110110 | 00110110 | 00110110 | 00110110 | ... | 00110110 |
--------------------------------------------- ------------
3. Compare A and B
--------------------------------------------- ------------
| 00000000 | 00000000 | 11111111 | 00000000 | ... | 00000000 |
--------------------------------------------- ------------
(success!)
4. Mask values
--------------------------------------------- ------------
| 0 | 0 | 1 | 0 | ... | 0 |
--------------------------------------------- ------------
(true)
</code></pre></div></div>
<p>After masking, we actually get an integer, because the final result for each group is either <code class="highlighter-rouge">0</code> or <code class="highlighter-rouge">1</code>, which could all be accumulated into an integer. In other words, the value and position of each bit in the returned integer corresponds to a match of a slot in the group.</p>
<p>So, we have the results! Now, all we need to do is check the indices of bits that are set in the final integer, compare the key(s) in those slots against the querying key (for equality), find the corresponding value, and we’ll land on <code class="highlighter-rouge">(39, 8)</code>.</p>
<h3 id="hints-to-compiler">Hints to compiler!</h3>
<p>Futher optimizations can be done on this implementation. If we’ve used a good hash function that distributes the bits reasonably well, then we can hint the compiler that the final equality check (for the key) will almost always be true. In Rust, we have <a href="https://doc.rust-lang.org/nightly/core/intrinsics/fn.likely.html"><code class="highlighter-rouge">likely</code></a> and <a href="https://doc.rust-lang.org/nightly/core/intrinsics/fn.unlikely.html"><code class="highlighter-rouge">unlikely</code></a> to achieve this. So, we can <a href="https://github.com/Amanieu/hashbrown/blob/6b9cc4e01090553c5928ccc0ee4568319ee0ed33/src/raw/mod.rs#L666">tell the compiler</a> that the equality is <em>likely</em> to be true.</p>
<p>The next hint is on whether we should probe to the next group looking for that key. Again, if our hash function is good enough, then the odds of that happening is very <em>very</em> low.<sup id="fnref:10"><a href="#fn:10" class="footnote">10</a></sup> So, we can <a href="https://github.com/Amanieu/hashbrown/blob/6b9cc4e01090553c5928ccc0ee4568319ee0ed33/src/raw/mod.rs#L670">hint the compiler again</a> that it’s <em>likely</em> to stop probing.</p>
<p>When we remove a key/value pair, we can follow the tombstone method - we query the key as usual, find the slot (in some group), set a tombstone (by marking the control byte as <code class="highlighter-rouge">DELETED</code>) and later mark it back as <code class="highlighter-rouge">EMPTY</code> (when we resize, for example). But, we could take advantage of the previous fact and say that if the group had at least one empty element, then we don’t have to add a tombstone. We could simply set it to <code class="highlighter-rouge">EMPTY</code>, because the probing is gonna stop with this group anyway (because the probing would’ve encountered an <code class="highlighter-rouge">EMPTY</code>).</p>
<p>Amanieu has added more stuff like making the hashmap efficient for 32-bit platforms, supporting maps smaller than a group, and a ton of other features to make it compatible with <code class="highlighter-rouge">std</code>, which is pretty cool!</p>
<p>I hope you found this post interesting. As always, feel free to drop any comment if you have anything to add.</p>
<hr />
<p><small>A huge thanks to <a href="https://github.com/Hoverbear">Ana</a> and <a href="https://github.com/Amanieu">Amanieu</a> for reviewing this post!</small></p>
<div class="footnotes">
<ol>
<li id="fn:1">
<p>I insist on watching this talk when you have some free time! <a href="#fnref:1" class="reversefootnote">↩</a></p>
</li>
<li id="fn:2">
<p>Although, hashbrown uses FX hash. <a href="#fnref:2" class="reversefootnote">↩</a></p>
</li>
<li id="fn:3">
<p><a href="https://akkadia.org/drepper/cpumemory.pdf">What Every Programmer Should Know About Memory</a>. <a href="#fnref:3" class="reversefootnote">↩</a></p>
</li>
<li id="fn:4">
<p>This is what Rust used in its Robin hood hashing implementation. <a href="#fnref:4" class="reversefootnote">↩</a></p>
</li>
<li id="fn:5">
<p>I haven’t talked about a number of improvements that could be made to Robin Hood hashing - backward shifting entries during deletion (instead of tombstones), caching hashes, slot indices or “distances” to improve probing, etc. all improve the performance of the hashmap. <a href="#fnref:5" class="reversefootnote">↩</a></p>
</li>
<li id="fn:6">
<p>This doesn’t mean that a hashmap that contains, say 2 elements <em>should</em> have a 16 element array (for the values) - only the group has 16 elements, the actual array containing the key/value pairs will still be 2 elements. <a href="#fnref:6" class="reversefootnote">↩</a></p>
</li>
<li id="fn:7">
<p>In the SwissTable implementation, the bottom (<em>least significant</em>) 7 bits were used for <code class="highlighter-rouge">H2</code>, but Amanieu has claimed that his choice lead to slightly more efficient code. <a href="#fnref:7" class="reversefootnote">↩</a></p>
</li>
<li id="fn:8">
<p>Again, this is how hashbrown is implemented. SwissTable used the top 7 bits. <a href="#fnref:8" class="reversefootnote">↩</a></p>
</li>
<li id="fn:9">
<p>One more Rust-specific enhancement - SwissTable had <code class="highlighter-rouge">kSentinel</code> to deal with C++ iterators, which wasn’t required in Rust. <a href="#fnref:9" class="reversefootnote">↩</a></p>
</li>
<li id="fn:10">
<p>According to the talk, it’d take a load factor of 94% for an ideal map to reach that situation. <a href="#fnref:10" class="reversefootnote">↩</a></p>
</li>
</ol>
</div>Ravi ShankarA while back, there was a discussion comparing the performance of using the hashbrown crate (based on Google’s SwissTable implementation1) in the Rust compiler. In the last RustFest, Amanieu was experimenting on integrating his crate into stdlib, which turned out to have some really promising results. As a result, it’s being planned to move the crate into stdlib. I insist on watching this talk when you have some free time! ↩1 year…2018-11-20T15:40:26+05:302018-11-20T15:40:26+05:30https://blog.waffles.space/2018/11/20/1-year<p>It’s been a year since I’ve blogged, despite <a href="https://manishearth.github.io/blog/2018/08/26/why-i-enjoy-blogging/">people reminding me</a> that blogging <a href="https://myrrlyn.net/blog/misc/to-all-the-posts-ive-blogged-before">is a good thing</a> and that it helps in the longer run. I had at least 3 posts in mind over the past few months, but I kept procrastinating on them. I now realize that if I keep doing that, then I might just as well stop blogging.</p>
<p>So, I’ve planned to blog about two topics by the end of this year:</p>
<ol>
<li>Experience report on using Swift for a year from a Rustacean’s perspective (what I liked, what I feared about, what I hated, etc.).</li>
<li>Building a HTTP (+ TLS) server from scratch using NIO (<a href="https://github.com/apple/swift-nio">Swift NIO</a> is a low-level framework for building network applications, which has built-in support for event loops, futures, HTTP/2, etc.).</li>
</ol>
<p>Firstly, here’s the story of my 2018 projects…</p>
<!-- more -->
<h2 id="building-services-for-a-client">Building services for a client</h2>
<p>Although my job began with the promise of working on Rust (FFI-bridging Rust and Swift), we had to move to Swift in a few days, because it felt easy and productive to build web services in a managed language like Swift compared to Rust (that said, we do use Rust for other things - our infrastructure, for example).</p>
<p><a href="https://omnijar.studio/">Our studio</a> had one goal - build open-source, decentralized web and e-commerce platforms, which are free, intuitive, and easy-to-use (and are <a href="https://naamio.cloud/"><strong>now open for early access!</strong></a>). My colleague (our technical lead and director) was working on the web platform, and me on the e-commerce thingy.</p>
<p>We got a client who wanted to move away from their Wordpress stack and start using our platforms (especially for e-commerce, as they were selling coffee-related products). The problem was that they had no developers, and my colleagues only had time for the ongoing projects in our studio (so they were unavailable to commit full-time).</p>
<p>Our agreement was that I would work full-time (as a full stack developer), our designer would be contracted for a few days every month, our director would be spending 3 days every week for them (code review, meetings, sprint planning, etc.), and that they should hire developers along the way, because it’s simply too much work for one person to accomplish in the given time frame (2 months).</p>
<p>Moreover, we need to polish our platforms every now and then, because the whole point of joining them was for mutual benefit - they get our services, and we use their data to improve our platform (not to mention that we also had to add features specifically <em>for</em> them).</p>
<p>After a while, they ended up extending their milestone (3 more months), but they never hired any developer to support us. After 2 months, I was still lagging behind. I was already spending my weekends on our platform, and the last thing I wanted was to spend 12+ hours on their frontend implementing <em>features</em>. With only one month left to go, stressed by the sprint features, deadlines and a ton of broken things to fix, I knew that I was gonna hate working on frontends eventually.</p>
<p>And then, it happened.</p>
<p>They hadn’t paid us for 5 months already (we kept reminding them, they kept postponing it), and when we reminded them about it (in 4 months), we got a message saying that <strong>they weren’t gonna pay us</strong> until we finish the platform. Firstly, they breached our contract by saying this, which means we’re free to pursue other clients and work on our own projects, and second, we knew that it’s gonna take more time to finish their platform, and if we continued to spend time for them, our studio will run out of money.</p>
<h2 id="aftermath">Aftermath</h2>
<p>As for me, that’s one more failed startup (in addition to <a href="/2017/11/09/8-months/#into-a-yc-startup">what happened last year</a>).</p>
<p>We stopped working for them (waiting on them to pay their debts) and resumed our work on the platform. We <a href="https://naamio.cloud/">launched it</a> for early access users and it’s registered as a separate company (which makes me… a cofounder!).</p>
<p>We’ve got two more clients now, for whom we’ll be building (non-Naamio) applications over the next year, and fund Naamio with that money. Let’s see how that goes…</p>
<p><small>Thanks to <a href="https://twitter.com/HelloFillip">Phil</a> for reviewing this post…</small></p>Ravi ShankarIt’s been a year since I’ve blogged, despite people reminding me that blogging is a good thing and that it helps in the longer run. I had at least 3 posts in mind over the past few months, but I kept procrastinating on them. I now realize that if I keep doing that, then I might just as well stop blogging.8 months…2017-11-09T21:21:47+05:302017-11-09T21:21:47+05:30https://blog.waffles.space/2017/11/09/8-months<p>It’s been a while since I blogged. Last few months have been… well, <em>complicated</em>. Now that I’m travelling, I won’t get a better chance to tell the story.</p>
<p>As far as the world knows, I’ve been writing some Rust and Python code for some bioinformatics company (which is one of the few companies using Rust in production in South India). Job was good until the start of this year. As time flew, the folks who run the company made some fuckups - they introduced new rules, they kept postponing the product release, and eventually, I got tired of researching stuff for them.</p>
<p>So, I got back into open-source and started looking for a new job.</p>
<!-- more -->
<h2 id="job-hunt">Job hunt</h2>
<p>Job hunting was <strong>hard</strong>, especially with my experience. Back then, it was only a year since I began writing production code. Almost everyone’s looking for senior developers with 3-5 years of experience in PHP or Node or whatever.<sup id="fnref:1"><a href="#fn:1" class="footnote">1</a></sup> The rest who look for junior/mid-level devs either didn’t respond, or rejected with an automated mail - which I understand, they get a lot of traffic, they can’t hire everyone!</p>
<p>I longed for a human response, even if it was a rejection. Because, I don’t simply go around and bulk apply for all the jobs that I could find! I take my time - only if I’m convinced that I <em>might</em> be a good fit for their open position, I tailor my email and send it to them.</p>
<p>Out of the 57 companies I’d applied in 2 months, 8 rejected with an automated mail (it’s funny to think that one company rejected me after 3 months), 6 rejected because I wasn’t around their location (which they didn’t specify in the job listing), 3 rejected after technical round (I hate competitive coding, really!), and the rest <strong>never replied!</strong></p>
<p>Meanwhile, I was working on Servo stuff. <a href="https://github.com/servo/servo/pulls?utf8=%E2%9C%93&q=is%3Apr%20author%3Awafflespeanut%20grid">I wrote the parsing and serialization code</a> for CSS grid (for <a href="https://wiki.mozilla.org/Quantum/Stylo">Stylo</a>). I rewrote a <a href="https://github.com/servo-automation/highfive/">bot as a Github integration</a> and added some cool features for it, and I was one of the coaches for two girls who participated in <a href="https://teams.railsgirlssummerofcode.org/teams/511">RGSoC 2017</a>.</p>
<h2 id="into-a-yc-startup">Into a YC startup</h2>
<p>Around mid-May, @Manishearth <a href="https://twitter.com/OmnijarStudio/status/865072706550718464">linked me to a tweet</a> which claimed that some startup called “Surematics” is looking to hire Rust devs. I applied, wrote some code, had a chat with the CTO (Phil) and CEO (when I also realized that the startup is part of YCombinator’s 2017 summer batch), booked my flights, and by June, I was in Mountain View on a 3-month contract (with a possible employment after demo day). It all happened quite fast!</p>
<p>My stack was totally new! Typescript for frontend, Rust for backend services, Kubernetes (in Azure) for orchestrating the microservices, and some cool new technologies like <a href="https://www.docker.com/">docker</a>, <a href="https://www.vaultproject.io/">vault</a>, <a href="https://coreos.com/etcd/">etcd</a> and <a href="https://www.cockroachlabs.com/">cockroach</a>. The learning curve was <em>huge!</em></p>
<p>What surprised me was the fact that we’re in YC’s batch <strong>without a product!</strong> I’ve been told that YC usually funds and guides startups which already have a product that’s being used by some clients. In our case, we didn’t even have a proper layout of the product!</p>
<p>Anyway, we’d planned to deliver something usable every week, but we couldn’t, because (in short) the goals/decisions kept changing, and we went on hacking stuff just to achieve those goals (which eventually kicked us in our asses). While I’d love to talk about this, it’s a big story, which Phil has summarized in <a href="https://hackernoon.com/through-the-looking-glass-part-i-2a710a865c67">a lovely series of posts</a>.</p>
<p>In the end, the startup merged with another company, and we all had to handover our work and go home. The irony is that, by that point, we had a nice working version of the product (because we’d been coding for 2 days straight!), but it didn’t matter.</p>
<p>In my last few days at CA, I was mostly enjoying. But, aside that, I bought a new domain, <a href="https://github.com/wafflespeanut/waffles.space">designed my website</a>, and launched it along with my projects in DigitalOcean. Now, I have a customized static server in Rust, my projects, Servo’s bots - all in docker containerized environments in CoreOS machines.</p>
<h2 id="post-yc-life">Post-YC life</h2>
<p>My post-YC project was a <a href="https://github.com/OmnijarBots/beryllium">bot library</a> for the <a href="https://wire.com/en/">Wire messenger</a> in Rust,<sup id="fnref:2"><a href="#fn:2" class="footnote">2</a></sup> which actually took me into the async features of Rust for the first time. It was fun!</p>
<p>After some latency period, I began hunting for jobs again. Now, things were easy (probably because of the YC stunt). I got through some interviews and got some cool offers, but in the end, I rejected those because I got a better offer from Phil (the one who hired me for Surematics). He’d hooked me up with a Finnish startup, where I get to be a full-stack developer again! Only difference is that this time, I’ll be working on Swift along with Typescript and Rust.</p>
<p>My contract officially began this month, and I’m already enjoying it, because my work (right now) is mostly on FFI-bridging Swift and Rust (dark arts!). I’ll try to write up a technical post on this.</p>
<p>Anyway, last few months were interesting. Glad to be back now!</p>
<div class="footnotes">
<ol>
<li id="fn:1">
<p>And yeah, <strong>I hate ‘em both!</strong> There, I said it. <a href="#fnref:1" class="reversefootnote">↩</a></p>
</li>
<li id="fn:2">
<p>Well, they had a lot of libraries in Rust, but this particular SDK was in Node and Java, so I went ahead and took this project! <a href="#fnref:2" class="reversefootnote">↩</a></p>
</li>
</ol>
</div>Ravi ShankarIt’s been a while since I blogged. Last few months have been… well, complicated. Now that I’m travelling, I won’t get a better chance to tell the story.Drawing an ASCII sketch2017-03-01T00:46:53+05:302017-03-01T00:46:53+05:30https://blog.waffles.space/2017/03/01/ascii-sketch<p>Every once in a while, I get a (seemingly) nice and interesting idea (thanks to a wonderful female creature, who’s always been my <em>muse</em>), and whenever I get one, I go straight to researching more about it, allocating most of my free time, so that I finish it up ASAP and show it to her. Last time, it was a <a href="https://github.com/wafflespeanut/AISH">CSS spewing thingy</a>. This time, it was about generating an ASCII sketch of a picture.</p>
<p>I’m sure you’d have seen all those “Image to ASCII converter”, “ASCII art generator”, and all sorts of boring variants of this online. But, I’ll tell you where they all fail and how I managed to bring up a decent sketch in ASCII. It took me about 3 hours to come up with a basic sketch, and then a few more for making it more generic and <a href="https://waffles.space/ascii-gen/">deploying it in my website</a>.</p>
<!-- more -->
<h2 id="mapping-ascii-values-over-rgb">Mapping ASCII values over RGB…</h2>
<p>Let’s say I want to draw the ASCII sketch of this JPEG image…<sup id="fnref:1"><a href="#fn:1" class="footnote">1</a></sup></p>
<p><img src="/images/ascii/sample.jpg" alt="sample" /></p>
<p>I always tend to hack on stuff with Python. And, it has an <a href="https://en.wikipedia.org/wiki/Python_Imaging_Library">amazing image library</a> to play with images. With PIL, we can do something like this,</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="o">>>></span> <span class="kn">from</span> <span class="nn">PIL</span> <span class="kn">import</span> <span class="n">Image</span>
<span class="o">>>></span> <span class="n">img</span> <span class="o">=</span> <span class="n">Image</span><span class="o">.</span><span class="nb">open</span><span class="p">(</span><span class="s">'sample.jpg'</span><span class="p">)</span>
<span class="o">>>></span> <span class="n">px</span> <span class="o">=</span> <span class="n">img</span><span class="o">.</span><span class="n">load</span><span class="p">()</span>
<span class="o">>>></span> <span class="n">px</span><span class="p">[</span><span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">]</span>
<span class="p">(</span><span class="mi">72</span><span class="p">,</span> <span class="mi">94</span><span class="p">,</span> <span class="mi">91</span><span class="p">)</span></code></pre></figure>
<p>So, we now have all those 3-tuple RGB values in a 2D array. The first step would be to convert these RGB values to intensities<sup id="fnref:2"><a href="#fn:2" class="footnote">2</a></sup> (i.e., <em>grayscale</em>, since the final ASCII art will look very similar to its grayscale version). It’s <a href="https://en.wikipedia.org/wiki/Grayscale#Luma_coding_in_video_systems">very easy</a>, and PIL eases this a bit more,</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="o">>>></span> <span class="n">img</span> <span class="o">=</span> <span class="n">img</span><span class="o">.</span><span class="n">convert</span><span class="p">(</span><span class="s">'L'</span><span class="p">)</span>
<span class="o">>>></span> <span class="n">px</span> <span class="o">=</span> <span class="n">img</span><span class="o">.</span><span class="n">load</span><span class="p">()</span>
<span class="o">>>></span> <span class="n">px</span><span class="p">[</span><span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">]</span>
<span class="mi">87</span></code></pre></figure>
<p>Next stop is to have a bunch of characters sorted with respect to their pixel densities (like <code class="highlighter-rouge">' '</code> (space) for white, <code class="highlighter-rouge">'.'</code> (dot) for gray, <code class="highlighter-rouge">'#'</code> for black and so on), Once we have a character map, we simply map the characters over the grayscale image, like so…</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="o">>>></span> <span class="n">width</span><span class="p">,</span> <span class="n">height</span> <span class="o">=</span> <span class="n">img</span><span class="o">.</span><span class="n">size</span>
<span class="o">>>></span> <span class="k">for</span> <span class="n">j</span> <span class="ow">in</span> <span class="nb">xrange</span><span class="p">(</span><span class="n">height</span><span class="p">):</span>
<span class="o">...</span> <span class="k">print</span> <span class="s">''</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">CHARS</span><span class="p">[</span><span class="n">px</span><span class="p">[</span><span class="n">i</span><span class="p">,</span> <span class="n">j</span><span class="p">]</span> <span class="o">%</span> <span class="nb">len</span><span class="p">(</span><span class="n">CHARS</span><span class="p">)]</span> <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">xrange</span><span class="p">(</span><span class="n">width</span><span class="p">))</span></code></pre></figure>
<p>Looks very simple, right? All we need to do is find <code class="highlighter-rouge">CHARS</code> (the character map).</p>
<p>There’s an ASCII art generator in Python, which has an <a href="https://github.com/ajalt/pyasciigen/blob/48a5e5ffa5d2ab28637a4724e5b1ce0609b982dd/asciigen.py#L84">interesting implementation</a> for the character map. First, the printable ASCII characters are drawn in an image. Then, they’re sorted according to the pixel density of their <em>render</em>. This means, “space” has zero pixels, and so it will be the first thing you’ll find in the sorted list. Just what I wanted!</p>
<p>Let’s see how the ASCII image turns out after this mapping…</p>
<p><img src="/images/ascii/screwed.png" alt="screwed" /></p>
<p>A gradient <em>spew</em>. Almost all the necessary details are gone! <em>Bad luck, huh?</em></p>
<p>People work around this by clamping ranges of values to some character instead of using the entire ASCII table (like, all the colors ranging from light gray to white will be mapped to “space”). But, that’s still a workaround. It doesn’t help much.</p>
<p>Let’s see if we can tune this, by extracting necessary details from the image.</p>
<h2 id="getting-the-details">Getting the details…</h2>
<p>An image (just like any other signal) can be represented as a sum of periodic (2D) <em>waves</em> of colors. It’s the magic of <em>Fourier transform</em>, that any signal can be represented as a <em>sum</em> of periodic waves of certain amplitudes and frequencies.</p>
<p>Let’s take our grayscale image and see how the first wave looks like,</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="o">>>></span> <span class="kn">import</span> <span class="nn">numpy</span> <span class="k">as</span> <span class="n">np</span>
<span class="o">>>></span> <span class="kn">from</span> <span class="nn">numpy</span> <span class="kn">import</span> <span class="n">fft</span> <span class="k">as</span> <span class="n">fourier</span>
<span class="o">>>></span>
<span class="o">>>></span> <span class="n">ft</span> <span class="o">=</span> <span class="n">fourier</span><span class="o">.</span><span class="n">rfft2</span><span class="p">(</span><span class="n">img</span><span class="p">)</span> <span class="c"># get the 2D fourier transform of the grayscale image</span>
<span class="o">>>></span> <span class="n">ft_new</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">zeros_like</span><span class="p">(</span><span class="n">ft</span><span class="p">)</span>
<span class="o">>>></span> <span class="n">ft_new</span><span class="p">[</span><span class="mi">0</span><span class="p">:</span><span class="mi">1</span><span class="p">,</span> <span class="mi">0</span><span class="p">:</span><span class="mi">1</span><span class="p">]</span> <span class="o">=</span> <span class="n">ft</span><span class="p">[</span><span class="mi">0</span><span class="p">:</span><span class="mi">1</span><span class="p">,</span> <span class="mi">0</span><span class="p">:</span><span class="mi">1</span><span class="p">]</span>
<span class="o">>>></span> <span class="n">rft</span> <span class="o">=</span> <span class="n">fourier</span><span class="o">.</span><span class="n">irfft2</span><span class="p">(</span><span class="n">ft_new</span><span class="p">)</span> <span class="c"># inverse transform</span>
<span class="o">>>></span> <span class="n">img</span> <span class="o">=</span> <span class="n">Image</span><span class="o">.</span><span class="n">fromarray</span><span class="p">(</span><span class="n">rft</span><span class="p">)</span>
<span class="o">>>></span> <span class="n">img</span><span class="o">.</span><span class="n">convert</span><span class="p">(</span><span class="s">'L'</span><span class="p">)</span><span class="o">.</span><span class="n">save</span><span class="p">(</span><span class="s">'fourier-1.jpg'</span><span class="p">)</span></code></pre></figure>
<p>And, we get this - the initial component.</p>
<p><img src="/images/ascii/fourier-1.jpg" alt="fourier-1" /></p>
<p>Now, let’s get the sum of first 3 waves…</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="o">>>></span> <span class="n">ft_new</span><span class="p">[</span><span class="mi">0</span><span class="p">:</span><span class="mi">3</span><span class="p">,</span> <span class="mi">0</span><span class="p">:</span><span class="mi">3</span><span class="p">]</span> <span class="o">=</span> <span class="n">ft</span><span class="p">[</span><span class="mi">0</span><span class="p">:</span><span class="mi">3</span><span class="p">,</span> <span class="mi">0</span><span class="p">:</span><span class="mi">3</span><span class="p">]</span> <span class="c"># we only need to change this line</span>
<span class="o">>>></span> <span class="c"># everything else is the same</span></code></pre></figure>
<p>… and, we get the lowest frequencies from the image. This would be the base gradient.</p>
<p><img src="/images/ascii/fourier-3.jpg" alt="fourier-3" /></p>
<p>… for 10 waves,</p>
<p><img src="/images/ascii/fourier-10.jpg" alt="fourier-10" /></p>
<p>… and, for 50 waves,</p>
<p><img src="/images/ascii/fourier-50.jpg" alt="fourier-50" /></p>
<p>Clearly, as we go further, the details are starting to show up. So, smaller frequencies indicate gradients, and higher frequencies indicate finer things. In other words, smaller frequencies tell you that there’s a <em>face</em>, whereas the higher frequencies show the finer details like edges, curvature, hairs, etc.</p>
<p>This means, we need the higher frequencies for the details. Well, we don’t have to fiddle around Fourier transform for achieving this, but it gives you an idea. Perhaps, the easiest way to get the details is by filtering out the lower frequencies.</p>
<p>First, we <em>blur</em> the image. Let’s apply a Gaussian blur filter (I usually pick a radius of 7 or 8).</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="o">>>></span> <span class="kn">from</span> <span class="nn">PIL</span> <span class="kn">import</span> <span class="n">Image</span><span class="p">,</span> <span class="n">ImageFilter</span>
<span class="o">>>></span> <span class="n">img</span> <span class="o">=</span> <span class="n">Image</span><span class="o">.</span><span class="nb">open</span><span class="p">(</span><span class="s">'sample.jpg'</span><span class="p">)</span>
<span class="o">>>></span> <span class="n">blur_filter</span> <span class="o">=</span> <span class="n">ImageFilter</span><span class="o">.</span><span class="n">GaussianBlur</span><span class="p">(</span><span class="n">radius</span><span class="o">=</span><span class="mi">8</span><span class="p">)</span>
<span class="o">>>></span> <span class="n">blur_img</span> <span class="o">=</span> <span class="n">img</span><span class="o">.</span><span class="nb">filter</span><span class="p">(</span><span class="n">blur_filter</span><span class="p">)</span></code></pre></figure>
<p>Since it’s a low pass filter, we get the image with the higher frequencies stripped out.</p>
<p><img src="/images/ascii/blur.jpg" alt="blur" /></p>
<p>Now, we invert the image, and blend it with the original image (with 50% opacity)…</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="o">>>></span> <span class="kn">from</span> <span class="nn">PIL</span> <span class="kn">import</span> <span class="n">ImageOps</span>
<span class="o">>>></span> <span class="n">inv_img</span> <span class="o">=</span> <span class="n">ImageOps</span><span class="o">.</span><span class="n">invert</span><span class="p">(</span><span class="n">blur_img</span><span class="p">)</span>
<span class="o">>>></span> <span class="n">blend</span> <span class="o">=</span> <span class="n">Image</span><span class="o">.</span><span class="n">blend</span><span class="p">(</span><span class="n">inv_img</span><span class="p">,</span> <span class="n">img</span><span class="p">,</span> <span class="mf">0.5</span><span class="p">)</span></code></pre></figure>
<p>This leaves us with the details…</p>
<p><img src="/images/ascii/blend.jpg" alt="blend" /></p>
<p>Now, we find a lot of <em>gray</em> areas. So, we have one last (and perhaps the important) step, which is to adjust the <em>levels</em>. For this, we move the image from RGB space to HSV space, clamp the levels to a certain minimum, maximum and gamma values, and convert it back to RGB.</p>
<p>You can think of this as making darker areas <em>black</em> and lighter areas <em>white</em> altogether! It’s quite simple. <a href="https://stackoverflow.com/a/3125421/2313792">Here’s an answer</a> from Stackoverflow that provides a Python implementation of how the levels are clamped.</p>
<p>As for our picture, once I clamp the levels (min = 78, max = 125, gamma = 0.78) and convert it to grayscale, I get this…</p>
<p><img src="/images/ascii/sketch.jpg" alt="sketch" /></p>
<p>Looks like we’ve <em>narrowed down</em> more than enough details to get the ASCII art! Now, if we use the character mapping…</p>
<p><img src="/images/ascii/ascii.png" alt="ascii" /></p>
<p>And, <em>voila!</em></p>
<p>As a sidenote, you’re very welcome to play with <a href="https://waffles.space/ascii-gen/">my ASCII art generator</a>, which does whatever I’d just shown you above.</p>
<p>See you next time!</p>
<div class="footnotes">
<ol>
<li id="fn:1">
<p>Okay, I know what you’re thinking, but FYI, that’s definitely <em>not</em> the girl I talked about! <a href="#fnref:1" class="reversefootnote">↩</a></p>
</li>
<li id="fn:2">
<p>We <em>can</em> however use the color data to get weighted colors and apply them over the final ASCII values, but meh… <a href="#fnref:2" class="reversefootnote">↩</a></p>
</li>
</ol>
</div>Ravi ShankarEvery once in a while, I get a (seemingly) nice and interesting idea (thanks to a wonderful female creature, who’s always been my muse), and whenever I get one, I go straight to researching more about it, allocating most of my free time, so that I finish it up ASAP and show it to her. Last time, it was a CSS spewing thingy. This time, it was about generating an ASCII sketch of a picture.Exploring the human genome (Part 1)2017-02-12T16:54:14+05:302017-02-12T16:54:14+05:30https://blog.waffles.space/2017/02/12/exploring-the-human-genome-part-1<p>You may already know that <a href="https://blog.waffles.space/2016/07/12/new-job-new-field/">my work in bioinformatics</a> is mostly, well, <em>research</em>. All these months, I’ve been writing little tools in Rust (things that help speed up some <em>boring</em> analysis). These days, I’m involved in something new, something <em>very interesting!</em> Before we get into all that, I’ll try to give a general overview of the flow of data (without going way too much into biology), what kinds of data we deal with, how we analyze it, where I come in, etc., starting from this post.</p>
<!-- more -->
<h2 id="preamble---reading-the-dna">Preamble - Reading the DNA…</h2>
<p>I’m sure you already have an idea about the DNA - the double helix thing, a bunch of ATCGs, the genetic code, the basis of life, etc., but I’ll tell you a better story. To begin, we should go <em>deeper</em>, all the way down to the nucleus of a cell.</p>
<p>In there (for humans), we’ll find 23 pairs of wiggly thingies (chromosomes, if you want). That’s where we’ll also find the tightly packed and coiled DNA strand. Each chromosome has a specific set of nucleotides (ATCGs) and are labeled based on their size - one has ~250 million of them, and so it’s “Chr. 1”, another one has ~50 million of them, and so it’s “Chr. 22”.<sup id="fnref:1"><a href="#fn:1" class="footnote">1</a></sup></p>
<p>A genome is a collection of all the genes. A gene is just a sequence of bases used to manufacture a protein (more on this next time). When we say “human genome”, we mean the whole thing, starting from the first base of the first chromosome to the last base of the last chromosome, which contains all the genes necessary for a human being.</p>
<p>Even though the DNA is very small, it’s rather <em>long</em> in its scale. If the size of each nucleotide is around 3Å, then there are ~3 billion of them in total, which means if you stretch the DNA end-to-end, then it will span about 1 meter! So, every cell in your body has ~1 meter of DNA.<sup id="fnref:2"><a href="#fn:2" class="footnote">2</a></sup> Your body <a href="https://biology.stackexchange.com/q/3327/3446">has about 10 trillion cells</a> and if they produce a DNA in every cycle, then your body alone <a href="https://calculatedimages.blogspot.in/2015/04/light-years-of-dna.html">produces a light year of DNA</a> in your lifetime! More importantly, that’s <em>almost</em> the same copy of DNA that began in the embryo.</p>
<p><em>… which is a lot!</em></p>
<p>We know that the DNA is the basis of all the wonderful mechanisms going on inside, so the first step is to <em>read</em> the DNA. That’s what we call <em>sequencing</em>. The basic process goes something like this. A sample (say, from blood, or liver) goes through a complex preparation and gets loaded into the sequencing machine. Since there are only 4 bases and they bind only with their <em>mate</em> (A to T, or C to G, and vice versa), reading a single helix is enough (as we can always <em>deduce</em> that if one side was “ATTG…”, then the other side would be “TAAC…”).</p>
<p>Firstly, the double helix is <a href="https://en.wikipedia.org/wiki/Nucleic_acid_thermodynamics#Denaturation">unwound</a> and individual strands are separately sequenced. As we’re still in the molecular level, identifying the bases is <em>tough</em> (limited by the sensitivity of our instruments), and so, the sequence is amplified <a href="https://en.wikipedia.org/wiki/Polymerase_chain_reaction">by making huge copies</a>. All you need is a suitable environment and a bunch of <a href="https://en.wikipedia.org/wiki/Primer_%28molecular_biology%29">additional ATCGs</a> to stick to the chain.</p>
<p>The interesting catch here is that the DNA sample won’t be a single straight strand during the process. <strong>It should be split into numerous sequences</strong> (ranging from 100 to a few thousand bases) and laid into multiple container-like thingies (<em>lanes</em>), where they’re <em>individually</em> amplified and <em>collectively</em> sequenced. Now, <a href="https://en.wikipedia.org/wiki/Dideoxynucleotide">special ATCGs</a> are used to stop the amplification, which light up (by fluorescing a color for each base) as they attach themselves to a particular nucleotide. Sensitive photoelectric devices are then used to take snapshots of these lights in the lanes.</p>
<p>Each color indicates the presence of the corresponding nucleotide, and <em>voila!</em> DNA sequenced! Here’s a <a href="https://www.youtube.com/watch?v=MvuYATh7Y74">wonderful TED lesson</a> to visualize this.</p>
<h2 id="the-reference-genome">The Reference Genome</h2>
<p>Remember that we had to slice the DNA into fragments? Once the sample has been <em>read</em>, the sequencing machine generates a <a href="https://en.wikipedia.org/wiki/FASTQ_format">FASTQ</a> file<sup id="fnref:3"><a href="#fn:3" class="footnote">3</a></sup>, which looks something like this,</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ head -8 demo_blood.fastq
@ERR009127.307 IL22_2005:8:1:5:1941 length=36
GCAGACCCAGCGGGGCATGGGCGGACAGAGCCGCAC
+
<B?<@BB>>3BB>))<>43)94@=11=A?@=@B:/-
@ERR009127.308 IL22_2005:8:1:5:944 length=36
GGCGAACGCTTCGCTGGCCATTTAGGAGCTCTGCTC
+
B@AAAA=ABBB@B@BBB><;<A;<9<<;<;=<A;;3
</code></pre></div></div>
<p>Every 4 lines in this file is a “read”. People are usually interested in the second line (the sequence fragment) and the fourth line (the ASCII-encoded quality of individual bases) in each read. The <a href="https://en.wikipedia.org/wiki/Phred_quality_score">quality score</a> is much like the machine’s confidence on a particular base.</p>
<p>For example, the last base in the first sequence “C” has a value 45 (‘minus’ in ASCII) whereas in the second sequence, it’s 51 (number ‘3’ in ASCII), which means we’re relatively more confident about “C” in the second sequence. The scale is logarithmic and so, you can’t expect an ASCII value more than 100 to show up all the time in reality.</p>
<p>Even though we’ve done so much to get this FASTQ file, it won’t be of any use by itself, since it doesn’t have the necessary information like where the particular sequence <em>was</em> in the sample, which means we’ve no idea what gene we’re looking at, which isn’t really useful.</p>
<p>So, we should reconstruct the DNA!</p>
<p><a href="https://en.wikipedia.org/wiki/Sequence_assembly">Assembly of DNA</a> is a <em>big deal</em>, because you have to figure out where a sequence belongs to. It’s like shredding a novel into bits of paper (which contain nothing more than a few words) and recreating it back from the bits. This takes a <em>long</em> time! And, it’s error-prone. Years have passed since <a href="https://en.wikipedia.org/wiki/Human_Genome_Project">the first attempt</a>, and yet, we don’t know some parts of our own genome.</p>
<p>Building a DNA from the FASTQ file every time we read a sample would be rather <em>silly</em>. So, people have spent years to land on a basic template DNA. This is called the <em>reference genome</em>. It’s incomplete, but it’s accurate enough. Every species has its own reference genome. The human’s is the largest - <a href="http://hgdownload.cse.ucsc.edu/downloads.html#human">a horrible 3GB file!</a> It serves as a template because all humans share more than 99% of the genome. In other words, we differ only by a few million bases.</p>
<p>This is what the reference genome looks like…</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span>head <span class="nt">-5</span> ~/data/BWAIndex/hs37d5.fa
<span class="o">></span>chr1 dna:chromosome chromosome:GRCh37:1:1:249250621:1
NNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNN
NNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNN
NNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNN
NNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNN</code></pre></figure>
<p>It begins with “chr1” indicating the first chromosome (and its range - 1 to 249,250,621). Note that it begins with a lot of Ns. “N” indicates that we’ve no idea what base occupies that position (like I said, parts are still incomplete). Let’s seek to a span with some data…</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span>sed <span class="nt">-n</span> <span class="s1">'200,204p'</span> ~/data/BWAIndex/hs37d5.fa
TCAGCCTTTTCTTTGACCTCTTCTTTCTGTTCATGTGTATTTGCTGTCTCTTAGCCCAGA
CTTCCCGTGTCCTTTCCACCGGGCCTTTGAGAGGTCACAGGGTCTTGATGCTGTGGTCTT
CATCTGCAGGTGTCTGACTTCCAGCAACTGCTGGCCTGTGCCAGGGTGCAAGCTGAGCAC
TGGAGTGGAGTTTTCCTGTGGAGAGGAGCCATGCCTAGAGTGGGATGGGCCATTGTTCAT
CTTCTGGCCCCTGTTGTCTGCATGTAACTTAATACCACAACCAGGCATAGGGGAAAGATT</code></pre></figure>
<p>Throughout the file, this is what you find most of the time. So, it’s accurate enough.</p>
<h2 id="stitching-the-dna">Stitching the DNA…</h2>
<p>Our <em>quest</em> is always to answer questions like, “Where the changes have happened?”, “Why they happened there?”, “Does/Doesn’t it cause any disease/disorder?”, etc. Now that we have a digital version of the reference genome, the next checkpoint is to <a href="https://en.wikipedia.org/wiki/Sequence_alignment">align</a> the reads from our FASTQ file to the reference.</p>
<p>Before that, we also do a <em>quality check</em> on the FASTQ file (using a Rust-powered tool) to ensure that it’s good. For instance, you can’t have a lot of Ns in a particular sequence (we obviously don’t want a lot of unknown bases). Also, if we assume that ATCGs are randomly distributed (which is often the case), then you can expect each base to occur 25% of the time on the average (if A is less than 5% and T is more than 40%, then we clearly don’t want that). Then, we look for patterns of DNA which are <em>junk</em> (apparently, we won’t get any useful info out of those). So, after these (and a lot more similar) quality checks, we trim the sequences, we filter some sequences, or even throw away the FASTQ file as needed.</p>
<p>If the FASTQ file is good, then we proceed for alignment. This is where things get a bit more interesting, because alignment is very much a string searching problem. If we’re able to afford ~20 GB of RAM (which we <em>can</em>, given the low cost of cloud services like AWS or Google Cloud), then <a href="https://en.wikipedia.org/wiki/Suffix_array#Applications">suffix arrays</a> could be used for the exact string matching problem. We build the suffix array for the reference genome (takes around 20-30 mins) and binary search for the occurrence of the FASTQ sequence. Once we find the position, we can find the chromosome it belongs to (if it’s less than ~250 million, then it’s definitely the first chromosome and so on).</p>
<p>But, that wouldn’t be enough. All the sequencing machines make mistakes. What if we encounter a “N” along the way? What if the machine wasn’t very confident about the base it just read? What if a mutation has happened at a particular position? What if a virus infection deleted a bunch of nucleotides consecutively from the sample genome?</p>
<p>Statistically, this happens 20-30% of the time in a FASTQ file. Apparently, it’s an <a href="https://en.wikipedia.org/wiki/Approximate_string_matching">approximate string matching</a> problem. So, how do we solve it?</p>
<p>Almost all the sequence aligners out there use the <em>magical</em> <a href="https://en.wikipedia.org/wiki/Burrows%E2%80%93Wheeler_transform">Burrows-Wheeler transform</a>. It’s much like a wrapper over the suffix array (as a matter of fact, for longer strings, you need the suffix array to build the BWT - the usual sorting could take <em>days!</em>). Once the BWT is in place, we use the <a href="https://en.wikipedia.org/wiki/FM-index">FM-index</a> to find things in O(1) time. For the human genome (without any optimization), you may need ~25 GB of RAM to do this, but it works pretty well, and it’s worth every penny!</p>
<p>Moreover, since the reference genome doesn’t change (until we get the next version), its BWT will be the same. It won’t take more than 5-6 mins to build the FM-index from the BWT.</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span>wc <span class="nt">-c</span> ref_genome gram_bwt
3137454505 ref_genome
3137454506 gram_bwt <span class="c"># BWT needs one additional byte</span>
6274909011 total</code></pre></figure>
<p>I’d <a href="https://github.com/wafflespeanut/nucleic-acid">written an Rust library</a> for this. Let’s take a look at its powers! I’ll try the first sequence from the FASTQ file I just showed you above.</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span>./gram <span class="nt">-c</span> GCAGACCCAGCGGGGCATGGGCGGACAGAGCCGCAC
<span class="o">[[</span><span class="s2">"chr16"</span>,69776079,<span class="s2">"NOB1"</span><span class="o">]]</span></code></pre></figure>
<p>And, we have a wonderful exact match! The sequence matches to position 69776079 in 16th chromosome, which is the range of the <code class="highlighter-rouge">NOB1</code> gene (don’t worry about genes for now, we’ll get into it in future posts). You can <a href="https://www.wolframalpha.com/input/?i=GCAGACCCAGCGGGGCATGGGCGGACAGAGCCGCAC">check this with Wolfram Alpha</a> (you know, just in case, you don’t believe me). The time taken for the Rust backend<sup id="fnref:4"><a href="#fn:4" class="footnote">4</a></sup> is ~50μs.</p>
<p>This means, we can align ~5000 sequences at a time (if they match perfectly) using a single CPU, and that’s <em>lightning</em> fast!</p>
<h2 id="magic-behind-the-fm-index">Magic behind the FM-index</h2>
<p>Now, we come to the question of how it’s so <em>magical</em>, and how we go about approximately matching a sequence. In order to generate the BWT, we append a null byte to the string, get all its rotations and sort it. For a string, say, “AACCG”, we append the null-byte, say <code class="highlighter-rouge">$</code> to its end, and get the sorted rotations like this,</p>
<p><code class="highlighter-rouge">$ A A C C G</code> <br />
<code class="highlighter-rouge">A A C C G $</code> <br />
<code class="highlighter-rouge">A C C G $ A</code> <br />
<code class="highlighter-rouge">C C G $ A A</code> <br />
<code class="highlighter-rouge">C G $ A A C</code> <br />
<code class="highlighter-rouge">G $ A A C C</code> <br /></p>
<p>Here, the last column <code class="highlighter-rouge">G$AACC</code> is the BWT of <code class="highlighter-rouge">AACCG$</code>. The suffix array of the same string is <code class="highlighter-rouge">[5,0,1,3,2,4]</code>. If we map the suffixes to their indices in the suffix array, we get this,</p>
<p><code class="highlighter-rouge">$</code> <br />
<code class="highlighter-rouge">A A C C G $</code> <br />
<code class="highlighter-rouge">A C C G $</code> <br />
<code class="highlighter-rouge">C C G $</code> <br />
<code class="highlighter-rouge">C G $</code> <br />
<code class="highlighter-rouge">G $</code> <br /></p>
<p>And, we find a strong resemblance. They share the first column. The FM-index has the BWT and some additional information regarding the first column and the BWT itself. Using this, we can narrow down our search space. If your query starts with “C”, then you’ll know that “C” suffixes lie in the range <code class="highlighter-rouge">[3, 5]</code> in the suffix array. For the next base, you feed the previous range along with the base, and the index will return the next range.<sup id="fnref:5"><a href="#fn:5" class="footnote">5</a></sup> If it’s a valid range, then the sequence exists, or if it’s invalid, then we’re <em>unlucky</em>.</p>
<p><em>I won’t go into the details here (I’m excited to talk about it, but this post is already big!).</em></p>
<p>Let’s try a sequence…</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span>head <span class="nt">-4</span> SRR413130.fastq
@SRR413130.1 A2097DABXX:4:1:2222:2104/1
GGACNGAGTTATCGAGGCACATACTCCACCACTGTCACAGGAAGAACCT
+
AA?@#BCCCCEGGGGGGEGGGGFGGGGGGGGGGGGGGGFGGGGGFGGGG
<span class="nv">$ </span>./gram <span class="nt">-c</span> GGACNGAGTTATCGAGGCACATACTCCACCACTGTCACAGGAAGAACCT
<span class="o">[]</span></code></pre></figure>
<p>We don’t have a match, because we have a “N” there (most often, that’s the case). Let’s try query’ing the sequence after the “N”…</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span>./gram <span class="nt">-c</span> GAGTTATCGAGGCACATACTCCACCACTGTCACAGGAAGAACCT
<span class="o">[[</span><span class="s2">"chr6"</span>,-161038213,<span class="s2">"LPA"</span><span class="o">]</span>,
<span class="o">[</span><span class="s2">"chr6"</span>,-161043759,<span class="s2">"LPA"</span><span class="o">]</span>,
<span class="o">[</span><span class="s2">"chr6"</span>,-161049303,<span class="s2">"LPA"</span><span class="o">]</span>,
<span class="o">[</span><span class="s2">"chr6"</span>,-161060396,<span class="s2">"LPA"</span><span class="o">]</span>,
<span class="o">[</span><span class="s2">"chr6"</span>,-161065943,<span class="s2">"LPA"</span><span class="o">]]</span></code></pre></figure>
<p>Whoa! We have a lot of matches now. They all belong to the 6th chromosome and “LPA” gene, but the positions are negative. Why? That means we have matched its reverse complement <code class="highlighter-rouge">AGGTTCTTCCTGTGACAGTGGTGGAGTATGTGCCTCGATAACTC</code>. We’ve reversed query, complemented the bases (A for T, C for G, etc.), and then we get a match. Let’s try it.</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span>./gram <span class="nt">-c</span> AGGTTCTTCCTGTGACAGTGGTGGAGTATGTGCCTCGATAACTC
<span class="o">[[</span><span class="s2">"chr6"</span>,161038213,<span class="s2">"LPA"</span><span class="o">]</span>,
<span class="o">[</span><span class="s2">"chr6"</span>,161043759,<span class="s2">"LPA"</span><span class="o">]</span>,
<span class="o">[</span><span class="s2">"chr6"</span>,161049303,<span class="s2">"LPA"</span><span class="o">]</span>,
<span class="o">[</span><span class="s2">"chr6"</span>,161060396,<span class="s2">"LPA"</span><span class="o">]</span>,
<span class="o">[</span><span class="s2">"chr6"</span>,161065943,<span class="s2">"LPA"</span><span class="o">]]</span></code></pre></figure>
<p>There we go! The same matches (in forward direction). What this really means is that we’ve matched the other side of the double helix (which is fine too, since both are from the same DNA). Now, let’s try the whole sequence, changing the “N” to “A”…</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span>./gram <span class="nt">-c</span> GGACAGAGTTATCGAGGCACATACTCCACCACTGTCACAGGAAGAACCT
<span class="o">[[</span><span class="s2">"chr6"</span>,-161038213,<span class="s2">"LPA"</span><span class="o">]</span>,
<span class="o">[</span><span class="s2">"chr6"</span>,-161043759,<span class="s2">"LPA"</span><span class="o">]</span>,
<span class="o">[</span><span class="s2">"chr6"</span>,-161049303,<span class="s2">"LPA"</span><span class="o">]</span>,
<span class="o">[</span><span class="s2">"chr6"</span>,-161060396,<span class="s2">"LPA"</span><span class="o">]</span>,
<span class="o">[</span><span class="s2">"chr6"</span>,-161065943,<span class="s2">"LPA"</span><span class="o">]]</span></code></pre></figure>
<p>And, <a href="https://www.wolframalpha.com/input/?i=GGACAGAGTTATCGAGGCACATACTCCACCACTGTCACAGGAAGAACCT">we have the matches!</a></p>
<p>Let’s have a look at the ranges while querying FM-index…</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span>./random unmatched_SRR413130.fastq
@SRR413130.12176383 A2097DABXX:4:62:1132:102596/1
AGAGCGGAGGCAGGAGTTGGGCCCCAATTTGCTTCACGTNAAATTTATG
+
DDDDDDDCDDD:CD<span class="o">=</span><span class="p">;</span>2<<?CBDDCBBBBBBBBBB<span class="p">;</span><span class="o">></span>:7#08665BBBB
<span class="nv">$ </span>./gram <span class="nt">-c</span> AGAGCGGAGGCAGGAGTTGGGCCCCAATTTGCTTCACGTNAAATTTATG
<span class="o">[]</span></code></pre></figure>
<p>So, we have another sequence which doesn’t match. I’ve modified the wrapper to show us the range output for each base. Let’s feed the bases one by one…</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span>./gram <span class="nt">-c</span> e:::G
<span class="o">(</span>G, 1449387873, 2042847480<span class="o">)</span>
<span class="nv">$ </span>./gram <span class="nt">-c</span> e:1449387873:2042847480:T
<span class="o">(</span>T, 2642364268, 2853319690<span class="o">)</span></code></pre></figure>
<p>We begin from the last base<sup id="fnref:6"><a href="#fn:6" class="footnote">6</a></sup> “G”, and we get a range. Then, we feed the second last base “T” with G’s range, and I get a new range. It’s valid. Now, let’s try feeding the whole thing.</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span>./gram <span class="nt">-c</span> AGAGCGGAGGCAGGAGTTGGGCCCCAATTTGCTTCACGTNAAATTTATG <span class="nt">--debug</span>
<span class="o">(</span>G, 1449387873, 2042847480<span class="o">)</span>
<span class="o">(</span>GT, 2642364268, 2853319690<span class="o">)</span>
<span class="o">(</span>GTA, 730337789, 783909233<span class="o">)</span>
<span class="o">(</span>GTAT, 2436460939, 2448269684<span class="o">)</span>
<span class="o">(</span>GTATT, 2902060112, 2905545457<span class="o">)</span>
<span class="o">(</span>GTATTT, 3044849837, 3046261437<span class="o">)</span>
<span class="o">(</span>GTATTTA, 832453834, 832844405<span class="o">)</span>
<span class="o">(</span>GTATTTAA, 276956227, 277060107<span class="o">)</span>
<span class="o">(</span>GTATTTAAA, 109010853, 109046976<span class="o">)</span>
<span class="o">(</span>GTATTTAAAN, 2042847950, 2042847950<span class="o">)</span>
<span class="o">[]</span></code></pre></figure>
<p>Apparently, we’re losing wonderful matches just because of a few accidental/incidental substitutions, insertions or deletions (and like I said, this happens 20-30% of the time). In our case, it has stopped at “N”, because <code class="highlighter-rouge">GTATTTAAAN</code> has returned an invalid range. This is how we approach the fuzzy string matching. All we have to do is once (and whenever) we encounter an invalid range, we try querying a new base with the previous range, and <em>backtrack</em> from there. Since there are only 4 possible bases (and since the FM-index is <em>fast</em>), depending on our algorithm, we won’t be risking a lot of computing time unnecessarily.</p>
<p>As for this query, changing the “N” to “C” will <a href="https://www.wolframalpha.com/input/?i=AGAGCGGAGGCAGGAGTTGGGCCCCAATTTGCTTCACGTCAAATTTATG">return a match</a>. This is a <em>mismatch</em>. Insertions and deletions can happen too, and it’s up to the aligner to decide which alignment to take/leave.</p>
<p>However, limiting the depth of backtracking is up to our resources i.e., how much edit distance we’re willing to allow (the more we allow, the more we’re prone to bad sequences and end up spending more computing time, and disallowing them entirely results in leaving out the most important sequences).</p>
<p>Now that we’ve aligned the FASTQ file to the reference genome, the next checkpoint is to infer the alignments - like how many gaps/insertions have occurred, whether a particular substitution is a mutation or whether it’s a machine error, or whether the alignment itself is wrong, etc., but that’s for the next post.</p>
<p><em>Auf Wiedersehen…</em></p>
<div class="footnotes">
<ol>
<li id="fn:1">
<p>Then, there’s the X and Y chromosomes which determine sex. <a href="#fnref:1" class="reversefootnote">↩</a></p>
</li>
<li id="fn:2">
<p>Actually, it’s every cell with a nucleus. That’s because not all cells have a nucleus - even though the red blood cells, the cells in hair, skin, nail, etc. start with a nucleus (with a DNA), they destroy their nucleus as they mature. <a href="#fnref:2" class="reversefootnote">↩</a></p>
</li>
<li id="fn:3">
<p>I’m choosing the simpler case here, because nowadays, a machine generates two FASTQ files - one belonging to each strand of the DNA (one will be in the forward direction, and the other will be in the reverse direction, as if their tails are tied up). When we analyze the files, we get reads from the <em>paired</em> files at the same time. <a href="#fnref:3" class="reversefootnote">↩</a></p>
</li>
<li id="fn:4">
<p>I have a simple TCP server that listens to a particular port for sequence requests (or job requests for running an entire file!). That’s just a client making a request and printing the response. Rust’s <a href="https://doc.rust-lang.org/std/sync/">concurrency primitives</a> are rather <em>charming</em> to work with, and it’s always pushing me to write parallel code. <a href="#fnref:4" class="reversefootnote">↩</a></p>
</li>
<li id="fn:5">
<p>The range is very useful by itself for counting the occurrences of substrings. The difference between them indicates the number of occurrences. <a href="#fnref:5" class="reversefootnote">↩</a></p>
</li>
<li id="fn:6">
<p>That’s because I have a forward BWT, and it’s got to do with how the FM-index works. If I had a BWT of the reverse genome, then I’ll be querying from the start. <a href="#fnref:6" class="reversefootnote">↩</a></p>
</li>
</ol>
</div>Ravi ShankarYou may already know that my work in bioinformatics is mostly, well, research. All these months, I’ve been writing little tools in Rust (things that help speed up some boring analysis). These days, I’m involved in something new, something very interesting! Before we get into all that, I’ll try to give a general overview of the flow of data (without going way too much into biology), what kinds of data we deal with, how we analyze it, where I come in, etc., starting from this post.Three months…2017-02-11T23:24:29+05:302017-02-11T23:24:29+05:30https://blog.waffles.space/2017/02/11/3-months<p>It’s been 3 months since I’ve blogged. <em>A lot</em> can happen in 3 months. Someone who was once close to you could leave you, your dad could lose his job (trembling your financial support), you could get to the point when you’ll no longer want your bachelor’s degree, etc.</p>
<p>In the midst of this, wonderful things can happen too - you could get invited to the awesome <a href="https://wiki.mozilla.org/All_Hands/2016_Hawaii">Mozilla “All Hands!”</a>, where you get to see <a href="https://www.flickr.com/photos/mozillaallhands/sets/72157673850712094/with/31521175206/">all kinds of interesting people</a> from allover the world (including the ones you’ve been chatting in IRC), you get to hang out with them, you get to live with them for a week, FUN!!! <sup id="fnref:1"><a href="#fn:1" class="footnote">1</a></sup></p>
<p>And, once you get back, you get the promise of a <em>superior</em> distraction - a nice little project that could keep you distracted for months!</p>
<p>You see? Balance!</p>
<p>Anyway, I’ve planned to write a bunch of posts about my work in bioinformatics, especially the project I’m involved in right now. I’ll make sure that they’ll be interesting by having a mixture of bio-stuff and coding. See you until then…</p>
<div class="footnotes">
<ol>
<li id="fn:1">
<p>Well, you could also get allergic to seafood (lobster, in my case) and get “hives”, but <em>meh</em>… <a href="#fnref:1" class="reversefootnote">↩</a></p>
</li>
</ol>
</div>Ravi ShankarIt’s been 3 months since I’ve blogged. A lot can happen in 3 months. Someone who was once close to you could leave you, your dad could lose his job (trembling your financial support), you could get to the point when you’ll no longer want your bachelor’s degree, etc.An easy bug in Stylo…2016-10-05T08:30:34+05:302016-10-05T08:30:34+05:30https://blog.waffles.space/2016/10/05/and-i-thought-it-was-easy<p>While my <a href="https://blog.waffles.space/2016/07/12/new-job-new-field/">new job</a> demands writing backend tools in Rust, I get a lot of free time every once in a while, when I fiddle around Servo’s code. Lately, I got interested in <a href="https://wiki.mozilla.org/Stylo">Stylo</a>.</p>
<p>Stylo is interesting enough for it to need a whole writeup about itself, but this post is just about an easy stylo bug, which then turned slightly <em>ugly</em>. Well, it’s no big deal, since developers usually deal with this kind of thing every day, but since it’s an easy bug, I thought it might give some ideas to the newcomers (to stylo) about where to look when hacking on stylo, and to keep pushing and <strong>not give up</strong> if an easy issue becomes less easy…</p>
<!-- more -->
<h2 id="stylo-in-a-nutshell">“Stylo” in a nutshell!</h2>
<p>There’s parallel style code in Servo and sequential C++ code in Gecko. In stylo, we isolate Servo’s style libraries and hook it up to Gecko (with a sleek FFI) and make it use that instead. Now, that’s easier said than done, but once we have this integration, we can focus on pure style stuff, without having to worry about unimplemented layout/rendering stuff in Servo (since a Firefox build will provide feedback on how things are going).</p>
<p>Stylo contains <a href="https://hg.mozilla.org/incubator/stylo">both Gecko and Servo</a> code. The workflow is somewhat difficult for a newcomer, because sometimes it demands submitting patches to both Gecko (<a href="https://hg.mozilla.org/mozilla-central">hg repo</a>) and Servo (<a href="http://github.com/servo/servo/">git repo</a>), dealing with codegen (<a href="http://www.makotemplates.org/">Mako</a> for the glue code, and a version of <a href="https://github.com/servo/rust-bindgen">rust-bindgen</a> for translating numerous C++ stuff to Rust), and finally testing them (when you build stylo and <em>manually</em> check whether your changes work).</p>
<h2 id="how-it-began">How it began…</h2>
<p>A “good first bug” in stylo usually goes about changing (or adding) something in the glue code. Manish had scraped a few pages and put up a <a href="https://manishearth.github.io/css-properties-list/">list of CSS properties</a>, which comes in rather handy. As we can see, there are some stuff that are implemented in Servo, but not in Stylo. For those properties, the changes reside in the glue code (mostly)<sup id="fnref:1"><a href="#fn:1" class="footnote">1</a></sup>, where we only have to get the computed values from Servo and <strong>set</strong> it on Gecko.</p>
<p>I’d done <code class="highlighter-rouge">border-spacing</code> a few days back. It was pretty easy, as it required nothing more than <a href="https://github.com/servo/servo/pull/13450/files">copying some values</a> from Servo to Gecko. The next thing in my queue was <code class="highlighter-rouge">font-stretch</code>, which “looked” pretty similar.</p>
<p>The core principle behind style code is that each property belongs to a particular type! It’s always a <code class="highlighter-rouge">struct</code> field (or a bunch of fields) in Gecko, whereas it’d be an <code class="highlighter-rouge">enum</code> or a <code class="highlighter-rouge">struct</code> in Servo.</p>
<p><code class="highlighter-rouge">font-stretch</code> turned out to be <a href="http://doc.servo.org/style/properties/longhands/font_stretch/computed_value/enum.T.html">an enum</a> in Servo, whereas it’s a <a href="https://dxr.mozilla.org/mozilla-central/rev/ea104eeb14cc54da9a06c3766da63f73117723a0/gfx/src/nsFont.h#78">16-bit signed integer</a> in Gecko. And, it wasn’t straight-forward (like I thought it’d be).<sup id="fnref:2"><a href="#fn:2" class="footnote">2</a></sup> Whenever we encounter an enum, we can easily <em>cast it away</em> to an integer primitive. But, we can’t do that here, because some of the <a href="https://dxr.mozilla.org/mozilla-central/rev/ea104eeb14cc54da9a06c3766da63f73117723a0/gfx/thebes/gfxFontConstants.h#24-27">constants were negative</a>. In order to keep things future-proof, we need the constants in Servo before we can do anything.</p>
<p>While we already have most of the types and values, importing a few more is pretty easy. Emilio had done a great job with bindgen, that we now have a <a href="https://github.com/servo/servo/tree/7914f14caabaa557c9f88130443ab77162c7072b/components/style/binding_tools">bunch of tools</a> for generating the necessary bindings required for the glue code. So, simply including the file and adding the constants’ pattern <a href="https://dxr.mozilla.org/servo/rev/1a28907a8f3792b92cfbba9505d345c5ae796535/components/style/binding_tools/regen.py#51,77">should do it</a>.</p>
<p>… or so I’d thought.</p>
<p>There was a <a href="https://github.com/servo/servo/issues/13540">slight trouble with the bindgen</a> along the way, but once it got fixed, I could generate the bindings in no time. Surprisingly enough, bindgen then seemed to ignore constants with negative values. So, it was time to get into bindgen code.</p>
<h2 id="a-bug-within-a-bug">A bug within a bug…</h2>
<p>Parsers never cease to impress me.<sup id="fnref:3"><a href="#fn:3" class="footnote">3</a></sup> As complicated as they look, they can never be perfect, and always have bugs! Rust bindgen is something that translates C++ code units to Rust (with support from <code class="highlighter-rouge">clang</code> libraries). It also has a parser. So, with the current scenario on hand, it’s natural to assume that we’re ignoring the negative values whenever we parse <code class="highlighter-rouge">#define</code> directives in C++ code.</p>
<p>Initial digging showed that <a href="https://github.com/servo/rust-bindgen/blob/cfdf15f5d04d4fbca3e7fcb46a1dd658ade973cd/src/codegen/mod.rs#L1706">this</a> is where we filter the collected Rust constants (translated from <code class="highlighter-rouge">#define</code>) with respect to the whitelist of patterns, but throwing some <code class="highlighter-rouge">println!</code> there showed that those set of constants never even get there in the first place!</p>
<p>After more digging, <a href="https://github.com/servo/rust-bindgen/blob/cfdf15f5d04d4fbca3e7fcb46a1dd658ade973cd/src/lib.rs#L333">it turned</a> <a href="https://github.com/servo/rust-bindgen/blob/cfdf15f5d04d4fbca3e7fcb46a1dd658ade973cd/src/ir/item.rs#L406">out that</a> <a href="https://github.com/servo/rust-bindgen/blob/cfdf15f5d04d4fbca3e7fcb46a1dd658ade973cd/src/ir/var.rs#L67">we skip</a> <a href="https://github.com/servo/rust-bindgen/blob/cfdf15f5d04d4fbca3e7fcb46a1dd658ade973cd/src/ir/var.rs#L149">parsing</a> if we don’t find an integer literal in a code unit. In our case, since parentheses and unary minus don’t count as literals, they’d been neglected by the parser.</p>
<p>Finally, a patch <a href="https://github.com/servo/rust-bindgen/pull/74">to bindgen</a> followed by another patch <a href="https://github.com/servo/servo/pull/13566">for bindings regeneration</a> was enough for making <a href="https://github.com/servo/servo/pull/13570">my actual glue code patch</a> to work.</p>
<p>This is what I like about Stylo. It’s hard, interesting, new and definitely <strong>not</strong> straight-forward.<sup id="fnref:4"><a href="#fn:4" class="footnote">4</a></sup> So, I’m planning to keep fiddling around it for a while…</p>
<div class="footnotes">
<ol>
<li id="fn:1">
<p>Mostly… but not necessarily. It could also mean that we can’t write the glue for that particular property <em>easily</em>, because either the Gecko code is complicated, or transforming the values from Servo to Gecko has some complication. <a href="#fnref:1" class="reversefootnote">↩</a></p>
</li>
<li id="fn:2">
<p>There are tons of unimplemented <em>easy</em> properties in stylo (like this one). Some are pretty straight, while others require a bit of hacking, and spending time with both the codebases. <a href="#fnref:2" class="reversefootnote">↩</a></p>
</li>
<li id="fn:3">
<p>They <em>attract</em> me so much that whenever I get to work with them, I tend to spend more time admiring the existing code rather than concentrating on the particular problem on hand. <a href="#fnref:3" class="reversefootnote">↩</a></p>
</li>
<li id="fn:4">
<p>I admit. This consumed a few hours of mine. If I’d asked around, then maybe I’d have fixed this within an hour or so. But, where’s the fun in that? <a href="#fnref:4" class="reversefootnote">↩</a></p>
</li>
</ol>
</div>Ravi ShankarWhile my new job demands writing backend tools in Rust, I get a lot of free time every once in a while, when I fiddle around Servo’s code. Lately, I got interested in Stylo.New job! New field of science!2016-07-12T20:36:59+05:302016-07-12T20:36:59+05:30https://blog.waffles.space/2016/07/12/new-job-new-field<p>It’s been about a year since I’ve blogged. A lot of stuff has happened in the mean time. I became a <a href="https://twitter.com/ServoDev/status/684472921818017794">reviewer for the Servo browser engine</a> - especially the python code (which felt good), attended a flight training program at IIT, Kanpur (which was pretty fun), had a <em>war</em> with some of the professors (which has postponed my bachelors degree, <em>meh</em>), and now I’m working for a bioinformatics company, writing production code in <strong>Rust</strong> (which is cool!).</p>
<!-- more -->
<p>While I was doing my final year project, I applied for an internship at a <a href="http://www.genomels.com/">bioinformatics company</a>. For the first week or so, it was just python and shell scripts (boring stuff, really), until one day, I ported some of the python code to Rust and
gave a demo on that. That was it! From then on, until the end of the internship, and now, my job, is totally on Rust!</p>
<p>All these days, they’ve been using third-party tools for their analysis, connecting them with shell pipes, tinkering the output with a few scripts, and finally bringing it to the front-end. Now, there’s an opportunity (for them) to break their painful dependencies, research on things, write stuff from scratch, while I can get deeper into systems programming, and simultaneously get an actual experience in writing production code (in Rust!). So, it’s a “win-win” situation!</p>
<p>As an example, there’s this Java-powered tool called <a href="http://www.bioinformatics.babraham.ac.uk/projects/fastqc/">FastQC</a> which analyzes <a href="https://en.wikipedia.org/wiki/FASTQ_format">FASTQ</a> data. With the help of some bioinformatics fellas, it’s pretty easy to reconstruct what the tool does, from its output. By the end of the internship, I was asked to rewrite the tool (ASAP!). They helped me with the spec, and it took me exactly 12 days (~70 hours) to write the Rust version of that tool. <sup id="fnref:1"><a href="#fn:1" class="footnote">1</a></sup></p>
<p>The initial version doesn’t have any kind of <em>unsafe</em> code, and I didn’t optimize it very well. It only utilizes the APIs in the <a href="https://doc.rust-lang.org/std/">standard library</a> for efficient reading, data storage, and parallelization, but it was already ~20% faster than its (carefully crafted) <em>rival</em>, and there’s no limitation for this. Now that we’ve got our own tool, we can have as many features we want! <sup id="fnref:2"><a href="#fn:2" class="footnote">2</a></sup></p>
<p>Anyway, I’m pretty sure we’re the only ones using Rust for production <a href="https://en.wikipedia.org/wiki/Tamil_Nadu">in our state</a> (right now), and I’m quite happy about this, because my job allows me to play with the language I love, and I’ve got more than enough learning space here. So, maybe I’ll stay around for a while, and see how far this is gonna take me…</p>
<p>Even though there are languages specifically designed for scientific computing (like Julia, for example), I personally believe Rust has a great future in big data analysis!</p>
<div class="footnotes">
<ol>
<li id="fn:1">
<p>I’m planning to blog about it soon. <a href="#fnref:1" class="reversefootnote">↩</a></p>
</li>
<li id="fn:2">
<p>They were actually impressed by this, and it got me the “game changer” award :P <a href="#fnref:2" class="reversefootnote">↩</a></p>
</li>
</ol>
</div>Ravi ShankarIt’s been about a year since I’ve blogged. A lot of stuff has happened in the mean time. I became a reviewer for the Servo browser engine - especially the python code (which felt good), attended a flight training program at IIT, Kanpur (which was pretty fun), had a war with some of the professors (which has postponed my bachelors degree, meh), and now I’m working for a bioinformatics company, writing production code in Rust (which is cool!).100 shades of green: The journey of a coder…2015-09-07T18:33:18+05:302015-09-07T18:33:18+05:30https://blog.waffles.space/2015/09/07/100-shades-of-green-the-journey-of-a-coder<p>I’ve been coding for about a year now. I’ve danced with Python <em>(a lot!)</em> and nowadays, I’m playing with Rust, although I’ve also done some basic C & Javascript. Anyways, I get a lot of questions from my fellow undergrads about how I got into coding in the first place, and yesterday, <a href="https://manishearth.github.io/">Manish</a> gave me the idea to blog about it.</p>
<p>Also, since my commit streak has reached 100 days (with 1k commits), I think now’s a great time to share my story with y’all…</p>
<p><img src="/images/streak.png" alt="commit streak" /></p>
<p><small>On a side note, this post is intended for those who’re about to get involved in the <em>art</em> of coding, though I assure the rest of you that it will be interesting for others as well.</small><sup id="fnref:1"><a href="#fn:1" class="footnote">1</a></sup></p>
<!-- more -->
<h2 id="good-ol-days-with-the-computer">Good ol’ days with the computer…</h2>
<p>When I got my first computer, all I ever wanted to do was play games (well, I still do, but I love other things too). Apart from that, I liked to paint (you know, lines, squares, circles, colors, wheee…). Though I did encounter C in my school days, I didn’t know the purpose of it, because I simply didn’t realize the powers of coding. The only thing that attracted me was HTML (4), because it can do some pretty stuff - that way, I realized that with some text I can make colorful things on a computer!</p>
<p>I discovered many things during my high school days, but the only thing that mattered the most was that characters in computers can do more than just coloring - that was the time when I played with matrices and generated prime numbers in C (basic stuff you learn in high-school). That was also the time when I had developed this weird desire to fiddle around the things which are hidden from plain sight. I just don’t like it that way.</p>
<p>For instance, take the beautiful “Microsoft Windows XP” (the OS of my first computer) - the <code class="highlighter-rouge">C:\\</code> drive is forbidden by default, and it has a lot of files which are unknown to me. I had a friend who shared my trait of fiddling around with unknown files. Since his father owned an internet cafe, whenever we’re lost, or screw up something (or get screwed up by a virus, mostly a worm), we get a solution almost immediately. Later, we take those problems to our school, so that we can screw up the computers in our labs (let’s just say that we hated some of our teachers and we wanted to make their lives miserable!).</p>
<p>That was the time when I also got interested in the command prompt (which I terribly hate nowadays, since I got into bash). For example, we used this tiny snippet for recursively walking inside a path and hiding the contents inside. It was used often by us to drive the teachers and students crazy (for a month!).<sup id="fnref:2"><a href="#fn:2" class="footnote">2</a></sup></p>
<pre><code class="language-bat">attrib [path] +h +r +s /s /d
</code></pre>
<h2 id="as-a-newbie-to-the-internet">As a newbie to the internet…</h2>
<p>It wasn’t until college when I got my first laptop along with a <em>pathetic</em> excuse for an internet connection (which I still have). By the end of my fresher year, a great deal of things had happened.</p>
<p>I got into Stack Exchange, <a href="http://wp.me/p3OCmi-D">which had some great impacts on my personality</a> over the years (that’s also where I met Manish). At the start, I had the motivation to explore the system, interact with a lot of users, and learn to contribute, but once that became handy, I got bored. Yeah, reading stuff on the internet, participating in a great community is awesome and all, but I needed something more - something that could keep me equipped when I’m bored of reading.</p>
<p>I got excited about math and wanted to solve problems - it was right about that time when I discovered <a href="https://projecteuler.net/">Project Euler</a>. It demanded the users to code to solve the problems. I’m sure every newbie coder gets interested in those problems, because it demands only the solution for a particular problem, which means you ought to guess an algorithm first (sometimes, it’d be the awful bruteforce), try to implement it (when you’ll discover more about the language of choice - with the help of <a href="https://stackoverflow.com/">Stackoverflow</a>, of course), get the result and then you can refine the algorithm (unlike SPOJ or Code Chef, which are designed to concentrate more on algorithms and data structures). In those days, I used C.</p>
<p>My sophomore year began with <a href="https://wafflescrazypeanut.wordpress.com">blogging</a> - when I was mostly writing about physics-ish stuff (whatever I got excited about) or my favorite experiences (or ranting about something). At this point, I knew about <em>Mathematica</em> and its glories. Luckily, the Wolfram community had offered plenty of documentation for it, that I was able to learn its basics pretty quickly.</p>
<p>Now, I had no reason to use C for solving the problems, since <em>Mathematica</em> had all the built-in functions available in hand. I used it to generate plots and some simple animations, which I later embedded in my blog posts. One thing led to another and soon, I was making animations for teaching my classmates to visualize things.<sup id="fnref:3"><a href="#fn:3" class="footnote">3</a></sup></p>
<p>Along the way, I also got fond of a markup language - which affected me so much, that I got addicted to just seeing its <em>beauty</em>. It was the awesome LaTeX! I used to prepare notes for some of my courses with it, just because I felt happier to read all those silly formulas in LaTeX. Once I got to know about its clockworks, I began using <a href="http://lyx.org">LyX</a> and <a href="http://geogebra.org">Geogebra</a> to speed things up.</p>
<h2 id="hands-on-an-easy-aesthetic--powerful-language">Hands on an easy, aesthetic & powerful language…</h2>
<p>Then, I discovered Github. I didn’t understand the point of it at first, and so I assumed it as a “dropbox for coders” - I didn’t know about the concept of version-control or open-source. Anyway, the repositories served nicely as a <em>backup place</em> for all my code.</p>
<p>I’m a <em>perfectionist</em>, alright? I admire the beauty of things when they’re neat, but I often struggle to maintain their clean state as time progresses. Since I realized that LaTeX seemed to consume most of my time, note-taking also came to an end. And, <em>Mathematica</em> (the only language I was currently working with) seemed too abstract, since it hides all the details from my view. Now, I wanted to know how things work behind the cloak, but I also don’t wanna fall back to C. <em>(meh)</em></p>
<p>When I was a sophomore, I used to hang out a lot in our physics chatroom, <a href="https://chat.stackexchange.com/rooms/71/the-h-bar">the H-bar</a> and most often, I could hear about Python. When I asked about it, they told me that the scientific community is using Python most of the time (from handling the data to plotting the results), especially because it’s easy. <em>“Hmm, looks like something I should try out…“</em>, and I got curious just like that.</p>
<p>By then, I somehow got interested in cryptography. After reading about the old <a href="https://en.wikipedia.org/wiki/Caesar_cipher">Caesar</a> & <a href="https://en.wikipedia.org/wiki/Vigenère_cipher">Vigènere</a> ciphers, I got a desire to create my own cipher (that often happens if you get too <em>excited</em> about cryptography without thinking about the past few decades of research), and that little project of mine consumed my entire vacation at the end of my sophomore year.</p>
<p>On the brighter side, this was the point where an endless discovery was going on. As days passed, I dug deeper and deeper into Python. Some new idea kept popping up every now and then, and I immediately implemented it. Man, I was totally productive! As a free perk, I also <a href="https://github.com/wafflespeanut/scripts/blob/0f1b6ee61b3929eacce4fa32638ff328e6f8858e/site-old/javascripts/zombify.js">translated the code to Javascript</a> (by which I learned some HTML5 and JS along with a bit of CSS). This was also where my life as a coder began…</p>
<p>The day came soon, when I learned some stuff in public-key cryptography, when I realized about the depths of cryptography, when I’d also <em>finally</em> decided to <strong>ditch all my work</strong> because it just seemed too <em>stupid</em> to keep on developing a nonsensical <em>beast</em> which does nothing but consumes more time and memory in the name of a “<a href="https://github.com/wafflespeanut/scripts/tree/0f1b6ee61b3929eacce4fa32638ff328e6f8858e/python/AutomatonX">non-conformant pet cipher</a>”! I had to <em>move on</em>. By then, I knew some serious stuff in Python (thanks to Stackoverflow which helped me to learn all the way down to its internals), and so I used it to reduce the work in my academic stuff - like grabbing data from the lab machines, minor computations, iterations and plotting (which simply took too much time in Excel - some of my classmates realized that later).</p>
<p>Every time I encountered a problem requiring repetitive steps, I used Python. Since Python can do many things (given the vast amount of packages it has) - I used it to crawl through webpages to download stuff, switched to Python for solving those old Project Euler problems, clean my files (and sometimes, dirty code). Like I said, I’m a <em>perfectionist</em>.</p>
<p>After a few months, I got an idea (for another little project!). I often forget what happened every day (which is one of my problems which I had to solve) and so, I wanted something to keep track of my memories. That “something” was my next project - <a href="https://github.com/wafflespeanut/biographer">a diary</a>. Apart from Mozilla, that project was something that consumed a great deal of time, and every single line in it was <em>molded</em> by me over the months. Ideas popped up in the same way - <em>“Let’s have an option for searching!”</em>, <em>“Authentication per session would be a great idea!”</em>, <em>“Hmm, wouldn’t it be wonderful to have <a href="https://en.wikipedia.org/wiki/Block_cipher_mode_of_operation#Cipher_Block_Chaining_.28CBC.29">CBC</a> before shifting?”</em>, and I implemented those as and whenever they cropped up in mind.<sup id="fnref:4"><a href="#fn:4" class="footnote">4</a></sup></p>
<h2 id="into-an-awesome-community---open-source-and-beyond">Into an awesome community - open-source and beyond…</h2>
<p>Anyways, that thing had taken some of my time and since I was indulged in it, I never bothered to look into Project Euler, and so my problem-solving days were done (I simply didn’t get the mood!). Now, I needed something else - something I’ve never tried in my internet lifetime.</p>
<p>Then, I recalled Manish’s talks about Mozilla, Rust, etc. He had often told me that Mozilla’s one of the most welcoming communities to get involved. One of the most important things which Stack Exchange taught me is to <strong>“search more before asking questions!”</strong> Because, we often get silly questions at the site, where some users don’t even bother to put some effort on their questions. The point is that it was pretty hard for me to ask questions (especially when it’s a new place). Also, I had never heard of “IRC”.</p>
<p>I was afraid. The gigantic codebase and the numerous code it contained freaked me out! I’ve never seen a codebase that big in my life! <em>“Millions of lines of code? C’mon! Seriously? It’s just a browser!”</em>. I never understood the bugs, nor did I know how to patch those. Soon, my vacation began, and my sole aim was to get involved in a Mozilla codebase! <sup id="fnref:5"><a href="#fn:5" class="footnote">5</a></sup></p>
<p>I got into IRC and discovered that they offered mentors for new users. I got one for myself, who even suggested me a “good first bug”, which took me some days to finally submit a patch. That also required me to get into Ubuntu and a new version control system called <em>Mercurial</em>. My first bug explained me how things worked over there. It also came with a perk - I never used Windows from then on! I realized the awesomeness of Ubuntu! <a href="http://wp.me/p3OCmi-uA">That was it</a>, and later bugs became a piece of cake, thanks to those awesome Mozillians.</p>
<p>It was by that time Rust 1.0.0 was released. I never got to play with a low-level language. The release of Rust and its success was my motivation. I got excited about it, and immediately got indulged in it (with the help of the <a href="https://doc.rust-lang.org/">wonderful docs & book</a>). Now, an idea popped up while reading about <a href="https://doc.rust-lang.org/book/rust-inside-other-languages.html">concurrency and FFI</a> in Rust (that was my first time reading about “FFI”, though I had some idea about concurrency). Anyways, that diary I’d written was completely in Python. So, If I could somehow link it through FFI and hand over the searching to Rust (and utilize its concurrency), then I could save a lot of time. And, it did.</p>
<p>This task consumed an entire weekend of mine (but, it was worth it!). Though FFI was hard, I got help from a lot of Rustaceans (mostly at IRC for minor things, or Stackoverflow, whenever <a href="https://stackoverflow.com/q/31083223/2313792">things went out of hand</a>). I loved Rust so much (just like I loved Python), which is also why I never felt hopeless, and by the end of the day, I <a href="https://blog.waffles.space/2015/07/08/a-pythonist-getting-rusty-these-days-part-2/">got the thing to work!</a> <em>(finally)</em></p>
<h2 id="coding-everyday">Coding everyday…</h2>
<p>There’s something that I learned from my journey. Once I got into the art of coding with the help a language, jumping into other languages wasn’t a big deal. Moreover, if you get involved in a project (open-source, or something of your own), then you’ll more likely learn a great deal of stuff than you normally would (though that cipher and my diary wasn’t much useful, I did learn a hell lot of things while working in them).</p>
<p>Also, since I’d gone for a high-level language (Python), I didn’t have to worry about the types very much, but I did get into trouble when I tried to get into a low-level <em>beast</em> like Rust. Because, Rust doesn’t <em>abstract away</em> many things like Python does, I had to do quite a few tasks by myself! Moreover, Rust’s design made my coding pretty challenging, because of its static type system and its merciless borrow checker (among other things). <strong>I had to pay the iron price!</strong> (for my choice of a high-level language at the start). But, <a href="https://blog.waffles.space/2015/07/05/a-pythonist-getting-rusty-these-days-part-1/">I had quite a lot of fun</a> while playing with it.</p>
<p>Nowadays, I’m involved in the <a href="https://github.com/servo/servo">Servo browser engine</a>, to learn more about Rust, which gets pretty exciting every day. And, while I’m playing with Rust, I’ve also got back into C, because firstly, I’ve realized the power of these low-level languages (plus, C doesn’t have any abstractions at all!), and secondly, I’m about to finish my college and industries only know about established languages like C/C++/Java. So, I gotta learn one of those to get a job <em>(sigh)</em>.<sup id="fnref:6"><a href="#fn:6" class="footnote">6</a></sup></p>
<p><small>Thanks to <a href="https://twitter.com/ManishEarth">Manish</a> for <em>patiently</em> reviewing this post…</small></p>
<div class="footnotes">
<ol>
<li id="fn:1">
<p>If you’d grown in my locality (in your childhood), then all you know is that a computer (which you often see in school) is <em>“something”</em> where you can only type, draw, and play games - nothing more! I’m not embarrassed to say that we were never curious! <a href="#fnref:1" class="reversefootnote">↩</a></p>
</li>
<li id="fn:2">
<p>Well, we learned that from a virus (and that too because we never believed in the concept of an anti-virus, since everything is just the cause & effect of code). This command is often utilized by a harmless worm which infects pen drives. <a href="#fnref:2" class="reversefootnote">↩</a></p>
</li>
<li id="fn:3">
<p>It was for a course, where we draw vector diagrams for the velocities and accelerations of various parts of some machine. I’d even <a href="https://www.youtube.com/playlist?list=PLpqCCxmTKpa1NBYfoEJlKb8X4fq0Pf6N9">made some boring videos</a> for the junior undergrads using those animations. <a href="#fnref:3" class="reversefootnote">↩</a></p>
</li>
<li id="fn:4">
<p>It’s not a great <em>project</em> and all (I created that only to dig deeper into Python, and I have!). And yeah, I do write my story every day (I’ve got about a year of stories now). I don’t usually spend much time other than just recalling the events, and I just write a paragraph or two (like hints) - to just remind myself that something so nice/worse has happened on that day. <a href="#fnref:4" class="reversefootnote">↩</a></p>
</li>
<li id="fn:5">
<p>Meanwhile, I <a href="http://wp.me/p3OCmi-ws">got into a hackathon</a>, when I learned a lot of stuff - more JS and some frameworks - especially the MVCs like Django and Angular JS, and I’d also figured out how Git worked. <a href="#fnref:5" class="reversefootnote">↩</a></p>
</li>
<li id="fn:6">
<p>Well, to be honest with you, based on the things I’ve seen so far, I personally feel like C being quite easy, since I’ve learned the <em>“Rust”</em>. <a href="#fnref:6" class="reversefootnote">↩</a></p>
</li>
</ol>
</div>Ravi ShankarI’ve been coding for about a year now. I’ve danced with Python (a lot!) and nowadays, I’m playing with Rust, although I’ve also done some basic C & Javascript. Anyways, I get a lot of questions from my fellow undergrads about how I got into coding in the first place, and yesterday, Manish gave me the idea to blog about it.