Engineering Fantasyhttp://nafiulis.me/2018-02-03T21:41:00+06:00What Heroku Should Do2018-02-03T21:41:00+06:002018-02-03T21:41:00+06:00Nafiul Islamtag:nafiulis.me,2018-02-03:what-Heroku-should-do.html<p><em>DISCLAIMER: THESE OPINIONS ARE MY OWN AND DO NOT REFLECT THE OPINIONS OF MY FORMER OR PRESENT EMPLOYERS</em></p>
<div class="contents topic" id="table-of-contents">
<p class="topic-title first">Table of Contents</p>
<ul class="simple">
<li><a class="reference internal" href="#heroku-connect" id="id1">Heroku Connect</a><ul>
<li><a class="reference internal" href="#heterogeneous-syncing-types" id="id2">Heterogeneous Syncing Types</a></li>
<li><a class="reference internal" href="#pay-per-row" id="id3">Pay per Row</a></li>
</ul>
</li>
<li><a class="reference internal" href="#underpowered-dynos" id="id4">Underpowered Dynos</a></li>
<li><a class="reference internal" href="#deploy-anywhere-but" id="id5">Deploy Anywhere (but...)</a></li>
<li><a class="reference internal" href="#heroku-is-an-ecosystem" id="id6">Heroku is an Ecosystem</a></li>
<li><a class="reference internal" href="#what-heroku-is-doing-right" id="id7">What Heroku is Doing Right</a></li>
</ul>
</div>
<p>I've been using Heroku for a while not at my workplace and so have many of our developers. Heroku is actually a really
good idea, and thats why services like convox.io and platform.sh keep popping up. The notion, that you can have
your servers managed by people, and not need a full time devops team is a total godsend. Iterations are faster and
easier, and it allows our teams to focus on application logic. It is not without its set of caveats though.
The market place also makes for quick integrations with other service providers, which makes adding things like cache
and search very easy.</p>
<p>I myself was pretty skeptical at the beginning because although for my personal projects, my perception of Heroku
was that it was pretty expensive, and not something that you'd want to use at a <em>real</em> company. By the time I joined,
Heroku was already being used for several applications. I mean, at the time, I just thought it was a nice skin on top
of AWS (which is an accurate description even to present day).</p>
<p>However, the real reason that the company was using Heroku was because of a product called <em>Heroku Connect</em> (HC).</p>
<div class="section" id="heroku-connect">
<h2><a class="toc-backref" href="#id1">Heroku Connect</a></h2>
<p>Heroku Connect is basically a syncing tool. It basically syncs data from SalesForce (SF) to a Heroku operated Postgresql
database. The data in SalesForce (SF) was coming from the company's own servers. In other words, we were paying Heroku
to gain access to out own data. This initially perplexed me, but I soon found out is that the old platform had issues
scaling data access out, so we decided to use HC and Postgresql as a temporary solution to create a couple of
applications that we could use to test new products and new markets. SF does have an API, but that has API limits
for reads and writes, and is generally used for bulk updates and reads. Furthermore, it gets quite slow if you're
building an API on top of it. However, we soon ran into a couple of issues very quickly. HC in conjunction with
Postgresql was seen as a really good alternative, because it allowed you to bypass data access limit by essentially
creating a copy of your business data in Postgresql.</p>
<p>However, we soon ran into a couple of issues really quickly.</p>
<div class="section" id="heterogeneous-syncing-types">
<h3><a class="toc-backref" href="#id2">Heterogeneous Syncing Types</a></h3>
<p>In SF, data is stored within things called SalesForce Objects (SO). These SOs are basically tables, and through HC are
mapped to Postgresql tables. There are two ways in which syncs occur. One is that there is a set interval after which
the data from SF is synced to Postgresql and the changes with Postgresql are synced to SalesForce. The other way is that
data is synced in real time. What is perplexing is that you never know which SOs are eligible for real time updates
and which ones will only be updated at intervals. You can always see which objects are being updated at intervals,
and which ones are being updated in real time, but you never know <em>why</em> a particular SO gets real time updates while
another SO gets updates at intervals. Sometimes, SOs that had real time updates will start getting interval updates
instead. This is because you might hit what is called the "Streaming API limit". What this limit is, and how it is
reached remains a mystery to me as well as our SF developers.</p>
<p>If you're building a service that gives you quick and unlimited access to SF, then you better deliver on that promise
otherwise, people will become disillusioned by your service as a whole and so despite have some awesome product,
your reputation is going to suffer as a natural consequence.</p>
</div>
<div class="section" id="pay-per-row">
<h3><a class="toc-backref" href="#id3">Pay per Row</a></h3>
<p>If you want to use HC, you have to pay an amount for a particular number of rows. So basically, you are paying for
the data in SF, you are paying for the data in Postgresql, and on top of that you have to pay per row to use HC. This
is maddening because it makes so little sense. If you're building a syncing service, you should be charging for
throughput, and not for the amount of data. This also makes sense because the higher the throughput, the higher the
costs for Heroku as a business, so it'd make sense for both Heroku and the companies buying the HC service.</p>
<p>But the real consequence of this pricing scheme is that if you want to launch a geo-distributed application built on
top of HC, you need to pay for rows in <em>each</em> region because Heroku's postgres does not have multi-master replication,
only master-slave replication. So, if you had to pay for regions in the US, EU and China, then you'd need to pay
for the same number of rows <em>three</em> times.</p>
</div>
</div>
<div class="section" id="underpowered-dynos">
<h2><a class="toc-backref" href="#id4">Underpowered Dynos</a></h2>
<p>Dynos are basic units of computation for Heroku. In a generic sense, they are a virtual machine.</p>
<p>The lowest tier dyno is really quite pathetic. Its just 512 MBs of RAM and 1 GHz, and for the equivalent price,
you could get a <em>few</em> t2.mirco instances from AWS and I'm pretty sure that you could get the same from Azure and
Google Cloud Platform. Although you get auto-scaling as well as maintenance on those VMs, the price seems quite steep
for the kind of product that they're offering.</p>
</div>
<div class="section" id="deploy-anywhere-but">
<h2><a class="toc-backref" href="#id5">Deploy Anywhere (but...)</a></h2>
<p>You can <em>theoretically</em> deploy on pretty much any continent using Heroku. However, there are two Herokus. There is the
public cloud, and then there is the private cloud. If this is news to you, then what you've been using all this time
is Heroku's public cloud. The public cloud only has a few regions, whereas the private cloud has many regions from
Virginia to Tokyo.</p>
<p>However, the two clouds have a completely different payment scheme. To just be able to <em>use</em> the
private cloud you need to pay a certain amount every month, just for the privilege of being able to use it. You're
not getting any dynos or anything. Just to be able to use the private cloud, you need to pay an amount upfront, and its
not cheap either. Furthermore, the dynos in the private cloud are more expensive (yes, they have a little more
compute power), which makes geo-distribution of your applications quite expensive.</p>
</div>
<div class="section" id="heroku-is-an-ecosystem">
<h2><a class="toc-backref" href="#id6">Heroku is an Ecosystem</a></h2>
<p>In Heroku the <em>app</em> is the center of the universe. You <em>attach</em> add-ons like redis or Postgresql to your application.
You can indeed <em>share</em> add-ons between applications, but later on, it becomes difficult to monitor what add-on is owned
by what app. It is far better if you have an interface, where you have the add-ons separate to the applications, so
that you can share resources between applications more easily.</p>
</div>
<div class="section" id="what-heroku-is-doing-right">
<h2><a class="toc-backref" href="#id7">What Heroku is Doing Right</a></h2>
<p>Heroku has probably the best service you can get. If something breaks or something goes wrong with your application,
their support mechanism is truly exceptional, because they will ping up an on-call engineer to fix your production
problems. Usually, these issues are resolved very quickly.</p>
<p>Secondly, even if you consider that Heroku is expensive, having your own devops team is even more expensive, and
further still if you need them to be on-call. This is difficult for small companies.</p>
<p>Furthermore, Heroku also has some of the best account executives you can get. They are really quite helpful,
and truly fight for you with upper management to give you the best deals possible if you buy bulk plans. I've only
worked with two so far, but they really are fighting for you in the company, and what to see you succeed.</p>
</div>
GraphQL in the Python World2017-06-23T16:53:00+06:002017-06-23T16:53:00+06:00Nafiul Islamtag:nafiulis.me,2017-06-23:graphql-in-the-python-world.html<div class="contents topic" id="table-of-contents">
<p class="topic-title first">Table of Contents</p>
<ul class="simple">
<li><a class="reference internal" href="#getting-started" id="id2">Getting Started</a><ul>
<li><a class="reference internal" href="#here-comes-python" id="id3">Here Comes Python</a></li>
<li><a class="reference internal" href="#playing-with-graphene" id="id4">Playing with Graphene</a></li>
</ul>
</li>
<li><a class="reference internal" href="#basic-types" id="id5">Basic Types</a><ul>
<li><a class="reference internal" href="#a-few-opinions" id="id6">A Few Opinions</a></li>
</ul>
</li>
<li><a class="reference internal" href="#integration-with-orms" id="id7">Integration with ORMs</a><ul>
<li><a class="reference internal" href="#django" id="id8">Django</a></li>
<li><a class="reference internal" href="#sqlalchemy-and-others" id="id9">SQLAlchemy and Others</a></li>
</ul>
</li>
<li><a class="reference internal" href="#integration-with-web-frameworks" id="id10">Integration with Web Frameworks</a></li>
<li><a class="reference internal" href="#final-thoughts" id="id11">Final Thoughts</a></li>
</ul>
</div>
<p>Not too long ago, Facebook unveiled
<a class="reference external" href="https://code.facebook.com/posts/1691455094417024/graphql-a-data-query-language/">Graphql</a>.
This was thought up to be a flexible data access layer designed to work
as a simple interface on top of heterogeneous data sources. Not long after facebook had introduced it, github announced
that the latest version of its API (version 4) would indeed be a <a class="reference external" href="https://developer.github.com/v4/">GraphQL API</a>.
Many other companies have followed including Meteor, Pinterest, Shopify to name a few.</p>
<p>At its heart, Graphql is designed to replace REST APIs. It also has a proper specification so that developers aren't
constantly fighting over the <em>proper</em> definition of Graphql. It has documentation inbuilt, with its powerful graphiql
query application that allows people to quickly check out the results of a particular query, and get code completion
on the fly, as they are writing the query.</p>
<img alt="Graphql on Github" class="align-center" src="/images/graphql_01.png" />
<div class="section" id="getting-started">
<h2><a class="toc-backref" href="#id2">Getting Started</a></h2>
<p>Python's main competitor in the ring is a nifty little library called <a class="reference external" href="http://graphene-python.org/">graphene</a>.
But before we get into graphene, we need to talk about some of the fundamental parts of Graphql.</p>
<ul class="simple">
<li>The <em>model</em> is an object defined by a Graphql schema.</li>
<li>The <em>schema</em> defines models and their attribute.</li>
<li>Each attribute of a model has its own <em>resolver</em>.
This is a function that is responsible for getting
the data for that particular model attribute.</li>
<li>A <em>query</em> is basically what you use to get or set
data inside Graphql</li>
<li><em>Mutations</em> are particular queries that allow you
to change the data for a given model or a set
of models.</li>
<li><em>Graphiql</em> is the UI that you use to interact with
a Graphql server. The IDE-like thing that you see
in the above screenshot is the Graphiql interface.</li>
</ul>
<div class="section" id="here-comes-python">
<h3><a class="toc-backref" href="#id3">Here Comes Python</a></h3>
<p>Python's attempt at Graphql can be broken down into three parts. Graphene, ORM specific graphene add-ons and
graphene libraries that add support to web frameworks such as flask or django. Graphene allows you to define your
models, define the attributes in your model, and also parses queries to get the data for those particular models.</p>
<p>The ORM specific graphene libraries basically do the job of turning your SQLAlchemy or Django models
into Graphql objects with the help of graphene. Some libraries will will also come in with views that fit in with
your web framework like <code>graphene-django</code>.</p>
</div>
<div class="section" id="playing-with-graphene">
<h3><a class="toc-backref" href="#id4">Playing with Graphene</a></h3>
<p>Installation is quite easy, and graphene supports both python versions 2 and 3.</p>
<div class="highlight"><pre><span></span>pip install graphene
</pre></div>
<p>Once you've installed it, import it, write a simple model, and the execute it like so:</p>
<div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">graphene</span> <span class="c1"># 1</span>
<span class="k">class</span> <span class="nc">Query</span><span class="p">(</span><span class="n">graphene</span><span class="o">.</span><span class="n">ObjectType</span><span class="p">):</span> <span class="c1"># 2</span>
<span class="n">hello</span> <span class="o">=</span> <span class="n">graphene</span><span class="o">.</span><span class="n">String</span><span class="p">(</span><span class="n">description</span><span class="o">=</span><span class="s1">'A typical hello world'</span><span class="p">)</span> <span class="c1"># 3</span>
<span class="k">def</span> <span class="nf">resolve_hello</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">args</span><span class="p">,</span> <span class="n">context</span><span class="p">,</span> <span class="n">info</span><span class="p">):</span> <span class="c1"># 4</span>
<span class="k">return</span> <span class="s1">'World'</span>
<span class="n">schema</span> <span class="o">=</span> <span class="n">graphene</span><span class="o">.</span><span class="n">Schema</span><span class="p">(</span><span class="n">query</span><span class="o">=</span><span class="n">Query</span><span class="p">)</span> <span class="c1"># 5</span>
<span class="n">query</span> <span class="o">=</span> <span class="s1">'''</span>
<span class="s1"> query {</span>
<span class="s1"> hello</span>
<span class="s1"> }</span>
<span class="s1">'''</span> <span class="c1"># 6</span>
<span class="n">result</span> <span class="o">=</span> <span class="n">schema</span><span class="o">.</span><span class="n">execute</span><span class="p">(</span><span class="n">query</span><span class="p">)</span> <span class="c1"># 7</span>
</pre></div>
<p>In <code class="coderef">1</code> we are importing the package. Note that in <code class="coderef">2</code> we are creating the query class. All query classes inherit
from the <code>graphene.ObjectType</code> class. You can also embed queries within queries (this is great when you want
to modularize your applications). Furthermore, even complex objects will also inherit from <code>graphene.ObjectType</code>.</p>
<p>This class basically
holds all the models. Right now, there is only one model, and that is hello, which happens to be a simple string. Yes,
I know its not terribly exciting, but bare with me.</p>
<p>In <code class="coderef">3</code> we add an object to the schema. In this case it is a simple string. In <code class="coderef">4</code> we basically write the <em>resolver</em>
for that particular model in the schema, we will get into the details of the function signature soon enough.</p>
<p>In <code class="coderef">5</code> we basically declare the schema, and we tell it that the query
class object is <code>Query</code> (a query object can be made up of multiple query objects, and we will get into that later).</p>
<p>Then, we declare the simplest query possible, in <code class="coderef">6</code> and in <code class="coderef">7</code> we finally execute it. This is what the result looks
like:</p>
<div class="highlight"><pre><span></span><span class="n">In</span> <span class="p">[</span><span class="mi">6</span><span class="p">]:</span> <span class="n">result</span> <span class="o">=</span> <span class="n">schema</span><span class="o">.</span><span class="n">execute</span><span class="p">(</span><span class="n">query</span><span class="p">)</span>
<span class="n">In</span> <span class="p">[</span><span class="mi">7</span><span class="p">]:</span> <span class="nb">type</span><span class="p">(</span><span class="n">result</span><span class="p">)</span>
<span class="n">Out</span><span class="p">[</span><span class="mi">7</span><span class="p">]:</span> <span class="n">graphql</span><span class="o">.</span><span class="n">execution</span><span class="o">.</span><span class="n">base</span><span class="o">.</span><span class="n">ExecutionResult</span>
<span class="n">In</span> <span class="p">[</span><span class="mi">8</span><span class="p">]:</span> <span class="n">result</span><span class="o">.</span><span class="n">data</span>
<span class="n">Out</span><span class="p">[</span><span class="mi">8</span><span class="p">]:</span> <span class="n">OrderedDict</span><span class="p">([(</span><span class="s1">'hello'</span><span class="p">,</span> <span class="s1">'world'</span><span class="p">)])</span>
</pre></div>
<p>The result comes with three main attributes.</p>
<div class="highlight"><pre><span></span><span class="n">In</span> <span class="p">[</span><span class="mi">12</span><span class="p">]:</span> <span class="n">result</span><span class="o">.</span><span class="n">data</span>
<span class="n">Out</span><span class="p">[</span><span class="mi">12</span><span class="p">]:</span> <span class="n">OrderedDict</span><span class="p">([(</span><span class="s1">'hello'</span><span class="p">,</span> <span class="s1">'world'</span><span class="p">)])</span>
<span class="n">In</span> <span class="p">[</span><span class="mi">13</span><span class="p">]:</span> <span class="n">result</span><span class="o">.</span><span class="n">errors</span>
<span class="n">In</span> <span class="p">[</span><span class="mi">14</span><span class="p">]:</span> <span class="n">result</span><span class="o">.</span><span class="n">invalid</span>
<span class="n">Out</span><span class="p">[</span><span class="mi">14</span><span class="p">]:</span> <span class="bp">False</span>
</pre></div>
<p><code>data</code> gives us the return data that we want. <code>errors</code> points any errors that we might <a class="footnote-reference" href="#errors" id="id1">[1]</a>
have and finally, <code>invalid</code> basically tells us that if the query itself is valid.</p>
</div>
</div>
<div class="section" id="basic-types">
<h2><a class="toc-backref" href="#id5">Basic Types</a></h2>
<p>So, now that we've gotten the hang of the fundamental parts of the Graphql system, its time that we explored the other
data types, and started playing around with more complex objects. There is already plenty of good documentation on
this stuff, so I won't do a copy of the tutorials that are already available. The first thing to take a look at is
the <a class="reference external" href="http://docs.graphene-python.org/en/latest/types/">type reference</a>.</p>
<p>There are quite a few types to go over, but in basic terms, there is a dichotomy between scalars and non-scalars.
Scalars are the base object types, they are things like strings, integers, booleans etc. Non-Scalars on the other
hand are more complex objects, often they are containers of scalars, like lists, <code>graphene.List</code>. They can also
be interfaces, that other graphene <code>ObjectType</code>s inherit from and of course, there are mutation types, that
basically make changes on top of the data.</p>
<p>In simple terms, there are a lot of types, but what it comes down to are basically types that hold information,
types that contain basic scalar types, types that other types inherit from in a concrete sense and types that
other objects inherit from in an abstract sense. Pretty much what you'd expect from a simple object oriented system.</p>
<div class="section" id="a-few-opinions">
<h3><a class="toc-backref" href="#id6">A Few Opinions</a></h3>
<p>However, what I have come to notice is that it is almost always better to avoid using abstract types, and just go
straight for interfaces. Most of the time, you just don't need abstract objects. In the case of Graphql, composition
is almost always better than inheritance. Probably the only place that it makes sense to use the <code>AbstractType</code>
class is for <code>Query</code> types that you are later on going to use in your final <code>Query</code> object.</p>
<p>Make sure to keep your names simple, and avoid using <code>graphene.NotNull</code>, because you can always set
<code>required=True</code> for scalar types anyway. This is similar to what you get in other serializer libraries, so
people looking at the codebase can see commonalities between graphene and other libraries.</p>
</div>
</div>
<div class="section" id="integration-with-orms">
<h2><a class="toc-backref" href="#id7">Integration with ORMs</a></h2>
<p>If you think about it, at its core, graphene is basically a combination of a serialization library and a Graphql
query interpreter, so it would make a lot of sense for it to work with ORMs and it does so quite well. At the time
of writing this, there is support for django, sqlalchemy as well as google app engine. Most are really quite
simple to integrate. Most of the time, you just have to set the meta model attribute of an object type and you're done.</p>
<div class="section" id="django">
<h3><a class="toc-backref" href="#id8">Django</a></h3>
<div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">django.db</span> <span class="kn">import</span> <span class="n">models</span>
<span class="kn">from</span> <span class="nn">graphene_django</span> <span class="kn">import</span> <span class="n">DjangoObjectType</span>
<span class="k">class</span> <span class="nc">Account</span><span class="p">(</span><span class="n">models</span><span class="o">.</span><span class="n">Model</span><span class="p">):</span>
<span class="n">birth_date</span> <span class="o">=</span> <span class="n">models</span><span class="o">.</span><span class="n">DateField</span><span class="p">(</span><span class="n">db_column</span><span class="o">=</span><span class="s1">'personbirthdate'</span><span class="p">,</span> <span class="n">null</span><span class="o">=</span><span class="bp">True</span><span class="p">)</span>
<span class="n">created_date</span> <span class="o">=</span> <span class="n">models</span><span class="o">.</span><span class="n">DateTimeField</span><span class="p">(</span><span class="n">blank</span><span class="o">=</span><span class="bp">True</span><span class="p">,</span> <span class="n">null</span><span class="o">=</span><span class="bp">True</span><span class="p">,</span> <span class="n">db_column</span><span class="o">=</span><span class="s1">'createddate'</span><span class="p">)</span>
<span class="n">is_paying_customer</span> <span class="o">=</span> <span class="n">models</span><span class="o">.</span><span class="n">NullBooleanField</span><span class="p">(</span><span class="n">db_column</span><span class="o">=</span><span class="s1">'iscustomer'</span><span class="p">)</span>
<span class="n">country</span> <span class="o">=</span> <span class="n">models</span><span class="o">.</span><span class="n">CharField</span><span class="p">(</span><span class="n">db_column</span><span class="o">=</span><span class="s1">'country'</span><span class="p">,</span> <span class="n">max_length</span><span class="o">=</span><span class="mi">3</span><span class="p">,</span> <span class="n">null</span><span class="o">=</span><span class="bp">True</span><span class="p">,</span> <span class="n">blank</span><span class="o">=</span><span class="bp">True</span><span class="p">)</span>
<span class="n">customer_number</span> <span class="o">=</span> <span class="n">models</span><span class="o">.</span><span class="n">CharField</span><span class="p">(</span> <span class="n">db_column</span><span class="o">=</span><span class="s1">'cnumber'</span><span class="p">,</span> <span class="n">unique</span><span class="o">=</span><span class="bp">True</span><span class="p">,</span> <span class="n">max_length</span><span class="o">=</span><span class="mi">255</span><span class="p">,</span>
<span class="n">blank</span><span class="o">=</span><span class="bp">True</span><span class="p">,</span> <span class="n">null</span><span class="o">=</span><span class="bp">True</span><span class="p">,</span> <span class="n">editable</span><span class="o">=</span><span class="bp">False</span><span class="p">)</span>
<span class="k">class</span> <span class="nc">Meta</span><span class="p">:</span>
<span class="n">managed</span> <span class="o">=</span> <span class="bp">False</span>
<span class="n">db_table</span> <span class="o">=</span><span class="s1">'accountinfo'</span>
<span class="k">class</span> <span class="nc">AccountType</span><span class="p">(</span><span class="n">DjangoObjectType</span><span class="p">):</span>
<span class="k">class</span> <span class="nc">Meta</span><span class="p">:</span>
<span class="n">model</span> <span class="o">=</span> <span class="n">Account</span>
</pre></div>
<p>You can now use <code>AccountType</code> just like you would any other object type. But most of the time, you don't even
have to manually add things to the query objects. If you have django-filter installed, you can do something pretty
simple, and add <code>graphene.Node</code> to your list of interfaces for a give type. This allows you to set some variables
in the metaclass that allows for simple filtering and also allows your type to be easily integrated into the query
using <code>DjangoConnectedFilterField</code>. Here's an example of what that would look like from the perspective of the
<code>Account</code> model:</p>
<div class="highlight"><pre><span></span><span class="k">class</span> <span class="nc">AccountNode</span><span class="p">(</span><span class="n">DjangoObjectType</span><span class="p">):</span>
<span class="k">class</span> <span class="nc">Meta</span><span class="p">:</span>
<span class="n">model</span> <span class="o">=</span> <span class="n">Account</span>
<span class="n">interfaces</span> <span class="o">=</span> <span class="p">(</span><span class="n">graphene</span><span class="o">.</span><span class="n">Node</span><span class="p">,</span> <span class="p">)</span>
<span class="n">filter_fields</span> <span class="o">=</span> <span class="p">[</span>
<span class="s1">'customer_number'</span><span class="p">,</span>
<span class="s1">'is_paying_customer'</span><span class="p">,</span>
<span class="p">]</span>
</pre></div>
<p>And this is what it would look like in the query object:</p>
<div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">graphene_django.filter</span> <span class="kn">import</span> <span class="n">DjangoConnectedFilterField</span>
<span class="k">class</span> <span class="nc">AccountQuery</span><span class="p">(</span><span class="n">graphene</span><span class="o">.</span><span class="n">AbstractType</span><span class="p">):</span>
<span class="c1"># Gives you a particular account</span>
<span class="n">account</span> <span class="o">=</span> <span class="n">graphene</span><span class="o">.</span><span class="n">Node</span><span class="o">.</span><span class="n">Field</span><span class="p">(</span><span class="n">AccountNode</span><span class="p">)</span>
<span class="c1"># Gives you all the accounts available</span>
<span class="n">all_accounts</span> <span class="o">=</span> <span class="n">DjangoFilterConnectionField</span><span class="p">(</span><span class="n">AccountNode</span><span class="p">,</span> <span class="n">order_by</span><span class="o">=</span><span class="s1">'-customer_number'</span><span class="p">)</span>
</pre></div>
<p>This really simplifies your whole query process, and focuses on fat database models and not fat graphene objects. Note,
that I use <code>graphene.AbstractType</code> for this <code>AccountQuery</code> objects because I'm basically going to use
it as a mixin for my final <code>Query</code> Object. Here is what that looks like:</p>
<div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">.queries</span> <span class="kn">import</span> <span class="n">AccountQuery</span>
<span class="k">class</span> <span class="nc">Query</span><span class="p">(</span><span class="n">AccountQuery</span><span class="p">,</span> <span class="n">graphene</span><span class="o">.</span><span class="n">ObjectType</span><span class="p">):</span>
<span class="k">pass</span>
<span class="n">schema</span> <span class="o">=</span> <span class="n">graphene</span><span class="o">.</span><span class="n">Schema</span><span class="p">(</span><span class="n">query</span><span class="o">=</span><span class="n">Query</span><span class="p">)</span>
</pre></div>
<p>This way, your main query object won't be cluttered, and you can keep adding your queries as you please. Make sure to
add <code>graphene.ObjectType</code> as the <em>last</em> argument for inheritance, otherwise your final <code>Query</code> object
will not be a concrete class.</p>
</div>
<div class="section" id="sqlalchemy-and-others">
<h3><a class="toc-backref" href="#id9">SQLAlchemy and Others</a></h3>
<p>Other ORMs are also similarly supported. With SQLAlchemy, you just use <code>SQLAlchemyObjectType</code> instead of
<code>DjangoObjectType</code>, you can also add the <code>Node</code> interface to it, and have a similar querying experience
that you do with django using <code>SQLAlchemyConnectionField</code>. The same is true for google app engine. Peewee support
is still in the works.</p>
</div>
</div>
<div class="section" id="integration-with-web-frameworks">
<h2><a class="toc-backref" href="#id10">Integration with Web Frameworks</a></h2>
<p>As you might expect, graphene supports a few popular frameworks, with more support coming. Right now there is awesome
support for <code>flask</code> and <code>django</code>. Here is what a flask Graphql application looks like:</p>
<div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">flask</span> <span class="kn">import</span> <span class="n">Flask</span>
<span class="kn">from</span> <span class="nn">flask_graphql</span> <span class="kn">import</span> <span class="n">GraphQLView</span>
<span class="kn">from</span> <span class="nn">models</span> <span class="kn">import</span> <span class="n">db_session</span>
<span class="kn">from</span> <span class="nn">schema</span> <span class="kn">import</span> <span class="n">schema</span><span class="p">,</span> <span class="n">Department</span>
<span class="n">app</span> <span class="o">=</span> <span class="n">Flask</span><span class="p">(</span><span class="vm">__name__</span><span class="p">)</span>
<span class="n">app</span><span class="o">.</span><span class="n">debug</span> <span class="o">=</span> <span class="bp">True</span>
<span class="n">app</span><span class="o">.</span><span class="n">add_url_rule</span><span class="p">(</span>
<span class="s1">'/graphql'</span><span class="p">,</span>
<span class="n">view_func</span><span class="o">=</span><span class="n">GraphQLView</span><span class="o">.</span><span class="n">as_view</span><span class="p">(</span>
<span class="s1">'graphql'</span><span class="p">,</span>
<span class="n">schema</span><span class="o">=</span><span class="n">schema</span><span class="p">,</span>
<span class="n">graphiql</span><span class="o">=</span><span class="bp">True</span>
<span class="p">)</span>
<span class="p">)</span>
<span class="nd">@app.teardown_appcontext</span>
<span class="k">def</span> <span class="nf">shutdown_session</span><span class="p">(</span><span class="n">exception</span><span class="o">=</span><span class="bp">None</span><span class="p">):</span>
<span class="n">db_session</span><span class="o">.</span><span class="n">remove</span><span class="p">()</span>
<span class="k">if</span> <span class="vm">__name__</span> <span class="o">==</span> <span class="s1">'__main__'</span><span class="p">:</span>
<span class="n">app</span><span class="o">.</span><span class="n">run</span><span class="p">()</span>
</pre></div>
<p>I would of course suggest that you have separate endpoints for graphiql and for your HTTP requests. You can also
subclass <code>GraphQLView</code> if you want to add some custom authentication.</p>
</div>
<div class="section" id="final-thoughts">
<h2><a class="toc-backref" href="#id11">Final Thoughts</a></h2>
<p>I really like Graphql, I've recently launched a production application based on top of Graphql, and
<code>graphene-django</code> has really helped me replace my current REST API with a Graphql API (or service or whatever
it is that you want to call it). The delcarative syntax makes it quite easy to use, and so there really is not much
of a learning curve, especially if you've used serializer libraries like Marshmallow or DRF's serializers.</p>
<p>Currently, graphene is basically a python port of the Graphql nodejs package at version 0.6. I'm pretty sure there will
be version bumps in the future as the library has a healthy following. It also supports all kinds of clients including
the apollo client.</p>
<hr class="docutils" />
<table class="docutils footnote" frame="void" id="errors" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#id1">[1]</a></td><td>You will always get back errors in a separate sub-object in a query if you have any. You will not get any errors in HTTP status codes.</td></tr>
</tbody>
</table>
</div>
Only Real-Time will Remain Standing2015-11-21T12:14:00+06:002015-11-21T12:14:00+06:00Nafiul Islamtag:nafiulis.me,2015-11-21:only-real-time-will-remain-standing.html<p>The Internet has changed tremendously over the past few years. What started
off as a platform to share research has become <em>the</em> hub of communication,
commerce and entertainment. However, we are at the verge of
the next stage in the evolution of web applications: the normalization
of real-time applications.</p>
<p>This transition will not only see real-time web applications become ubiquitous
but rather, and more fundamentally, become the de-facto standard for a web
application. This eventual change is a result of a myriad of things, but mostly
as a result of increased computing power, faster and cheaper storage as well
as javascript becoming ridiculously more powerful.</p>
<p>There have been dramatic changes in technology in order for change to happen.
With the advent of SSDs and multi-core processors as well
as architectural shift towards distributed systems, we suddenly have a lot
more processing power at our disposal. These changes did not
happen solely because we <em>wanted</em> to build real-time applications but rather,
they arose out of a need to scale our sites to monolithic proportions.</p>
<p>As such, consequences will be dire for languages, frameworks and implementations
that do not have first-class support for creating asynchronous or concurrent
applications. Many will claim that non-real time web applications will
survuive, but they will survuive the way that static html pages (with little
to no styling) do. In other words, they will become artifacts of a bygone
era.</p>
<p>The underlying technology has become powerful enough to support this, but
the rise of modern javascript libraries such as react, angular and others
will only catalyze the eventual rise and dominance of such applications.
RESTful web services are merely a precursor to full blown sites running
mostly using websockets. In other words, we are in for a lot more single page
applications.</p>
<p>Language-wise, golang and scala will become much more popular
than they already are.<a class="footnote-reference" href="#javascript" id="id1">[1]</a>
Languages like Python and Ruby will no longer remain
in the positions they currently hold until and unless they get their acts
together to support this new paradigm or find another use case
entirely.<a class="footnote-reference" href="#python" id="id2">[2]</a> Although both Python and Ruby have
their asynchronous frameworks and/or libraries, they are not powerful enough
to face off against go, scala and nodejs as they do not have an implementation
fast enough to compete with the likes of go, which has concurrency primitives
and Scala which has many libraries for paralell code execution.</p>
<p>But languages alone will not face the consequences of this change;
databases will need to evolve as well. One example of a
database built solely for real-time applications is rethinkdb,
which already has many adherents. Rethinkdb started off as a MySQL engine,
but has evolved to become an independent database in and of itself, touting
features like atomic changfeeds in its latest release.</p>
<p>So, when trying to master language, framework or database, make sure
to take into consideration the eventual rise of real-time applications and
do your best to avoid languages, frameworks and databases that do not support
this new paradigm. Otherwise, the skills that you're building might eventually
have very little value in the near future.</p>
<hr class="docutils" />
<table class="docutils footnote" frame="void" id="javascript" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#id1">[1]</a></td><td>If you're asking why javascript isn't on that list, that's
because its not possible for javascript to get more popular than it
already is.</td></tr>
</tbody>
</table>
<table class="docutils footnote" frame="void" id="python" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#id2">[2]</a></td><td>Python has already found a good niche in the realm of education
and data-science, so I feel that it will still remain quite relevant or
even as popular as it is now in the years ahead simply because data is
blowing up in everyone's face.</td></tr>
</tbody>
</table>
Setting up Rethinkdb with Lunchy2015-09-27T14:35:00+06:002015-09-27T14:35:00+06:00Nafiul Islamtag:nafiulis.me,2015-09-27:setting-up-rethinkdb-with-lunchy.html<div class="contents topic" id="table-of-contents">
<p class="topic-title first">Table of Contents</p>
<ul class="simple">
<li><a class="reference internal" href="#installation" id="id6">Installation</a></li>
<li><a class="reference internal" href="#setup" id="id7">Setup</a></li>
<li><a class="reference internal" href="#launching" id="id8">Launching</a></li>
</ul>
</div>
<p>Rethinkdb is a powerful new NoSQL database that comes with its one web-based admin console. However, its a bit
awkward to set up. The installation is not hard when you have a one line homebrew installation or if you prefer to
install it as an application, then you can simply use the use their disk image provided as a url <a class="reference external" href="https://www.rethinkdb.com/docs/install/osx/">download</a> on their OSX installation page. However, the Rethinkdb
command line tool is a bit awkward in the sense that it just creates a directory to store your data in the same
directory that you launch it. This is a simple tutorial on how to set rethinkdb as a service so that it stores your
data in the same place with a proper set of default configurations.</p>
<div class="section" id="installation">
<h2><a class="toc-backref" href="#id6">Installation</a></h2>
<p>I would highly recommend using homebrew to install rethinkdb. Its a painless process and gives you full control as to
how you use the <code>rethinkdb</code> CLI.</p>
<div class="highlight"><pre><span></span>brew install rethinkdb
</pre></div>
<p>Now, we are going to install <code>lunchy</code>, which is a wrapper over <code>launchctl</code>, that basically starts and stops
our services in a simple manner.<a class="footnote-reference" href="#lunchy" id="id1">[1]</a> <a class="footnote-reference" href="#agentsanddemons" id="id2">[2]</a></p>
<div class="highlight"><pre><span></span>gem install lunchy
</pre></div>
</div>
<div class="section" id="setup">
<h2><a class="toc-backref" href="#id7">Setup</a></h2>
<p>The first thing that we need to do is install the <code>rethinkdb</code> plist file in our LaunchAgents directory. We can
do this by firstly creating a file called <code>com.rethinkdb.server.plist</code> and then running <code>lunchy install</code> on
that file. My own configuration is shown below.</p>
<div class="highlight"><pre><span></span><span class="cp"><?xml version="1.0" encoding="UTF-8"?></span>
<span class="cp"><!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd"></span>
<span class="nt"><plist</span> <span class="na">version=</span><span class="s">"1.0"</span><span class="nt">></span>
<span class="nt"><dict></span>
<span class="nt"><key></span>Label<span class="nt"></key></span>
<span class="nt"><string></span>com.rethinkdb.server<span class="nt"></string></span>
<span class="nt"><key></span>ProgramArguments<span class="nt"></key></span>
<span class="nt"><array></span>
<span class="nt"><string></span>/usr/local/bin/rethinkdb<span class="nt"></string></span>
<span class="nt"><string></span>-d<span class="nt"></string></span>
<span class="nt"><string></span>/Users/quazinafiulislam/Library/RethinkDB/data<span class="nt"></string></span>
<span class="nt"></array></span>
<span class="nt"><key></span>StandardOutPath<span class="nt"></key></span>
<span class="nt"><string></span>/Users/quazinafiulislam/Library/Rethinkdb/out.log<span class="nt"></string></span>
<span class="nt"><key></span>StandardErrorPath<span class="nt"></key></span>
<span class="nt"><string></span>/Users/quazinafiulislam/Library/Rethinkdb/error.log<span class="nt"></string></span>
<span class="nt"><key></span>RunAtLoad<span class="nt"></key></span>
<span class="nt"><false/></span>
<span class="nt"><key></span>KeepAlive<span class="nt"></key></span>
<span class="nt"><true/></span>
<span class="nt"><key></span>LowPriorityIO<span class="nt"></key></span>
<span class="nt"><false/></span>
<span class="nt"></dict></span>
<span class="nt"></plist></span>
</pre></div>
<div class="section" id="steps">
<h3>Steps</h3>
<ul class="simple">
<li>Create a file using the <code>touch</code> command, i.e. <code>touch com.rethinkdb.server.plist</code>, and just paste in
the contents.</li>
<li>Replace <code>quazinafiulislam</code> with your username in the file.<a class="footnote-reference" href="#completepaths" id="id3">[3]</a></li>
<li>Create a new directory, <code>~/Library/Rethinkdb</code>. This place will house all your rethinkdb stuff i.e. data, logs etc.</li>
<li>Create a new data directory using <code>rethinkdb create -d ~/Library/Rethinkdb/data</code>.</li>
<li>The install the file into your LaunchAgents directory using <code>lunchy install com.rethinkdb.server.plist</code>.</li>
<li>Once this is done, create two log files using the <code>touch</code> command to create the output log and the error log
inside <code>~/Library/LaunchAgents</code> named <code>out.log</code> and <code>error.log</code> respectively.</li>
</ul>
<p>This is the express installation, you can of course customize the installation to your needs. For example you can add
your own <code>.conf</code> file for rethinkdb.<a class="footnote-reference" href="#configure" id="id4">[4]</a> You can then add it as a parameter to your program arguments
by adding the following to your <code>ProgramArguments</code> inside of <code>array</code>.</p>
<div class="highlight"><pre><span></span><span class="nt"><string></span>--config-file<span class="nt"><string></span>
<span class="nt"><string></span>/path/to/your/config/file<span class="nt"><string></span>
</pre></div>
<p>Feel free to change the <code>plist</code> file to your liking. One other thing that you might want to change is the
<code>RunAtLoad</code> key. I've set this to <code>false</code>, but you might want <code>rethinkdb</code> to start up when you log
into your system.</p>
</div>
</div>
<div class="section" id="launching">
<h2><a class="toc-backref" href="#id8">Launching</a></h2>
<p>Launching this is quite easy, all you need to do is tell <code>lunchy</code> that you want to start the <code>rethinkdb</code>
service.</p>
<div class="highlight"><pre><span></span>lunchy start rethinkdb
</pre></div>
<p>We can see that the standard output is printed to the <code>out.log</code> file inside of <code>~/Library/Rethinkdb</code>.<a class="footnote-reference" href="#port" id="id5">[5]</a></p>
<img alt="Rethinkdb log" class="align-center" src="/images/rethink_01.png" />
<p>Stopping <code>rethinkdb</code> is also a breeze.</p>
<div class="highlight"><pre><span></span>lunchy stop rethinkdb
</pre></div>
<hr class="docutils" />
<table class="docutils footnote" frame="void" id="lunchy" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#id1">[1]</a></td><td>The <a class="reference external" href="https://en.wikipedia.org/wiki/Launchd">Wikipedia Page</a> will give you a good idea of what both
<code>launchd</code> and <code>launchctrl</code> are.</td></tr>
</tbody>
</table>
<table class="docutils footnote" frame="void" id="agentsanddemons" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#id2">[2]</a></td><td>If you want to know the difference between a launch agent and a demon,
take a look at <a class="reference external" href="http://www.grivet-tools.com/blog/2014/launchdaemons-vs-launchagents/">this article</a>.</td></tr>
</tbody>
</table>
<table class="docutils footnote" frame="void" id="completepaths" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#id3">[3]</a></td><td>Make sure that when you enter the path of a file or folder in a <code>.plist</code> file, you give the complete
path without aliases.</td></tr>
</tbody>
</table>
<table class="docutils footnote" frame="void" id="configure" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#id4">[4]</a></td><td>Learn to configure rethinkdb from their <a class="reference external" href="https://www.rethinkdb.com/docs/config-file/">configuration documentation page</a>.
I would strongly encourage you to create your own <code>.conf</code> file.</td></tr>
</tbody>
</table>
<table class="docutils footnote" frame="void" id="port" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#id5">[5]</a></td><td>You can specify your port as another <code>ProgramArgument</code>.</td></tr>
</tbody>
</table>
</div>
Python 3 May Become Relevant Now2015-08-03T03:30:00+06:002015-08-03T03:30:00+06:00Nafiul Islamtag:nafiulis.me,2015-08-03:python-3-may-become-relevant-now.html<div class="contents topic" id="table-of-contents">
<p class="topic-title first">Table of Contents</p>
<ul class="simple">
<li><a class="reference internal" href="#type-hints" id="id6">Type Hints</a></li>
<li><a class="reference internal" href="#async-await" id="id7">Async/Await</a></li>
<li><a class="reference internal" href="#summary" id="id8">Summary</a></li>
</ul>
</div>
<p>Python 3 has been mocked for a pretty long time for not having any <em>big-ticket</em> features. For the first four
minor releases for Python 3 (3.1 to 3.4), this did not change. There are two features that are worth noting
in Python 3.5. One being the type checker/hints.<a class="footnote-reference" href="#typechecker" id="id1">[1]</a> The other being the async/await key words
coming to Python 3.5.<a class="footnote-reference" href="#async" id="id2">[2]</a> Question is, how relevant?</p>
<div class="section" id="type-hints">
<h2><a class="toc-backref" href="#id6">Type Hints</a></h2>
<blockquote>
<p>This PEP aims to provide a standard syntax for type annotations, opening up Python code to easier static analysis
and refactoring, potential runtime type checking, and performance optimizations utilizing type information.</p>
<p><cite>PEP 0484</cite></p>
</blockquote>
<p>Python now has a type <em>checker</em>. This does not mean that python is going to magically become a
<em>statically typed</em> language. In other words, the type checker is merely a tool for a better development experience.
For example, if your IDE/Editor knows what type you're working with, it can provide better code completion. Some
IDEs actually provide rudimentary type checking.<a class="footnote-reference" href="#pycharm" id="id3">[3]</a> However, this will standardize the way in which
types are declared and that means this will raise the bar for all the python tools available at our disposal.</p>
<p>Think of this as the python equivalent to <a class="reference external" href="https://github.com/facebook/flow">flow</a>, which is a static type checker
for javascript. The type checking will happen through stub files, a lot like DefinitelyTyped's Typescript stubs.<a class="footnote-reference" href="#dt" id="id4">[4]</a></p>
<p>Other than better tooling, this also means that a type induced runtime errors can be thwarted before they happen, which is
something that makes developing large code bases with Python a lot easier. You will hear no end of war stories about
how a function that <em>sometimes</em> returned <code>None</code> deep in the source brought about <em>pyapocalypse</em>. Having type
problems deep within your source code, is by no means uncommon.</p>
<p>A friend of mine had to deal with this kind of problem once. The problem was with a function (buried deep in the
source code as one of many decorators) that usually returned a <code>list</code> but under a certain circumstances, it
returned <code>None</code>. This wasn't noted before because most of the time, the result would be checked in a
simple <code>if</code> statement.</p>
<p>In other words <code>bool(None)</code> and <code>bool([])</code> return the same thing, which is <code>False</code>. The problem was
happening because <em>another</em> branch still <em>thought</em> that the returned value was a <code>list</code> and so
tried to <code>append</code> to it. As you might expect, the traceback wasn't pretty.</p>
<p>I'm not saying that the person who originally wrote the code is a bad programmer. I'll leave that up to you. What
I am saying is that python allows you to make such silly mistakes. This is especially problematic when you're working
with really large code-bases.</p>
<p>However, while this is all good in terms of development tools, this does not mean a large boost in performance. PyPy's
FAQ talks about this issue and addresses some of the key concerns. <a class="footnote-reference" href="#pypyfaq" id="id5">[5]</a> In a nutshell, the performance will
not increase simply because python's types are objects and do not correspond directly to how types are represented in
binary. For example, python's <code>int</code> type isn't necessarily a 64-bit number, but has the ability to grow to whatever
size you need it to grow.</p>
<blockquote>
<p>... annotations are at the wrong level (e.g. a PEP 484 “int” corresponds to Python 3’s int type, which does not
necessarily fits inside one machine word; even worse, an “int” annotation allows arbitrary int subclasses).</p>
<p><cite>PyPy FAQ (Would type annotations help PyPy’s performance?)</cite></p>
</blockquote>
</div>
<div class="section" id="async-await">
<h2><a class="toc-backref" href="#id7">Async/Await</a></h2>
<blockquote>
<p>This proposal makes coroutines a native Python language feature, and clearly separates them from generators. This
removes generator/coroutine ambiguity, and makes it possible to reliably define coroutines without reliance on a
specific library. This also enables linters and IDEs to improve static code analysis and refactoring.</p>
<p><cite>PEP 0492</cite></p>
</blockquote>
<p><code>async</code> and <code>await</code> are very good ideas. In fact, these two keywords have been an integral part of C# for
quite some time now and make programing with tasks much easier. These two key words makes python feel a lot more
abstract, whereas turning generators into coroutines (even through the use of a decorator) and then using them like
generators feels rather clumsy. In simple terms, <code>async</code> creates a coroutine for you. For example, the following
is a coroutine:</p>
<div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">asyncio</span>
<span class="kn">import</span> <span class="nn">datetime</span>
<span class="nd">@asyncio.coroutine</span>
<span class="k">def</span> <span class="nf">display_date</span><span class="p">(</span><span class="n">loop</span><span class="p">):</span>
<span class="n">end_time</span> <span class="o">=</span> <span class="n">loop</span><span class="o">.</span><span class="n">time</span><span class="p">()</span> <span class="o">+</span> <span class="mf">5.0</span>
<span class="k">while</span> <span class="bp">True</span><span class="p">:</span>
<span class="k">print</span><span class="p">(</span><span class="n">datetime</span><span class="o">.</span><span class="n">datetime</span><span class="o">.</span><span class="n">now</span><span class="p">())</span>
<span class="k">if</span> <span class="p">(</span><span class="n">loop</span><span class="o">.</span><span class="n">time</span><span class="p">()</span> <span class="o">+</span> <span class="mf">1.0</span><span class="p">)</span> <span class="o">>=</span> <span class="n">end_time</span><span class="p">:</span>
<span class="k">break</span>
<span class="k">yield from</span> <span class="n">asyncio</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span>
<span class="n">loop</span> <span class="o">=</span> <span class="n">asyncio</span><span class="o">.</span><span class="n">get_event_loop</span><span class="p">()</span>
<span class="c1"># Blocking call which returns when the display_date() coroutine is done</span>
<span class="n">loop</span><span class="o">.</span><span class="n">run_until_complete</span><span class="p">(</span><span class="n">display_date</span><span class="p">(</span><span class="n">loop</span><span class="p">))</span>
<span class="n">loop</span><span class="o">.</span><span class="n">close</span><span class="p">()</span>
</pre></div>
<p>The above will turn into the following under PEP-492:</p>
<div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">asyncio</span>
<span class="kn">import</span> <span class="nn">datetime</span>
<span class="n">async</span> <span class="k">def</span> <span class="nf">display_date</span><span class="p">(</span><span class="n">loop</span><span class="p">):</span>
<span class="n">end_time</span> <span class="o">=</span> <span class="n">loop</span><span class="o">.</span><span class="n">time</span><span class="p">()</span> <span class="o">+</span> <span class="mf">5.0</span>
<span class="k">while</span> <span class="bp">True</span><span class="p">:</span>
<span class="k">print</span><span class="p">(</span><span class="n">datetime</span><span class="o">.</span><span class="n">datetime</span><span class="o">.</span><span class="n">now</span><span class="p">())</span>
<span class="k">if</span> <span class="p">(</span><span class="n">loop</span><span class="o">.</span><span class="n">time</span><span class="p">()</span> <span class="o">+</span> <span class="mf">1.0</span><span class="p">)</span> <span class="o">>=</span> <span class="n">end_time</span><span class="p">:</span>
<span class="k">break</span>
<span class="n">await</span> <span class="n">asyncio</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span>
<span class="n">loop</span> <span class="o">=</span> <span class="n">asyncio</span><span class="o">.</span><span class="n">get_event_loop</span><span class="p">()</span>
<span class="c1"># Blocking call which returns when the display_date() coroutine is done</span>
<span class="n">loop</span><span class="o">.</span><span class="n">run_until_complete</span><span class="p">(</span><span class="n">display_date</span><span class="p">(</span><span class="n">loop</span><span class="p">))</span>
<span class="n">loop</span><span class="o">.</span><span class="n">close</span><span class="p">()</span>
</pre></div>
<p>However, what is being introduced is syntactic sugar meaning that this does not change python on a fundamental level.
For example, introducing these native constructs will not <em>magically</em> make python be able to compete with node (V8).</p>
</div>
<div class="section" id="summary">
<h2><a class="toc-backref" href="#id8">Summary</a></h2>
<p>The new features being added are subtle but will go a long way into making Python 3 the abstract language that it needs
to be as well as provide the tooling required to make it a potential choice for huge code-bases. However, this does
not mean that Python has become a better choice over other languages such as golang for high performance applications.
So, although these new features will definitely persuade people to move from Python 2 to Python 3 (at least for new
projects), it does not necessarily mean that more and more people will leave Python seeking abstract yet high
performance alternatives.</p>
<hr class="docutils" />
<table class="docutils footnote" frame="void" id="typechecker" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#id1">[1]</a></td><td><a class="reference external" href="https://www.python.org/dev/peps/pep-0484/">PEP-0484</a>. This type checker is
based on the <a class="reference external" href="http://mypy-lang.org/">mypy</a> type checker created by one of the authors of PEP-0484,
Jukka Lehtosalo.</td></tr>
</tbody>
</table>
<table class="docutils footnote" frame="void" id="async" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#id2">[2]</a></td><td><a class="reference external" href="https://www.python.org/dev/peps/pep-0492/">PEP-0492</a>. This is a pretty radical change since it means
Python will get new key words.</td></tr>
</tbody>
</table>
<table class="docutils footnote" frame="void" id="pycharm" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#id3">[3]</a></td><td>PyCharm provides type checking through its <a class="reference external" href="https://github.com/JetBrains/python-skeletons">skeletons</a>.
These are a lot like the stubs that PEP-0484 is proposing.</td></tr>
</tbody>
</table>
<table class="docutils footnote" frame="void" id="dt" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#id4">[4]</a></td><td>DefinitelyTyped is a collection for stub files for a language called TypeScript which is basically a typed
superset of JavaScript. These stub files help IDE and editors in providing better code completion.</td></tr>
</tbody>
</table>
<table class="docutils footnote" frame="void" id="pypyfaq" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#id5">[5]</a></td><td>PyPy's FAQ talks about this issue, and its lack of effect on performance under
"<a class="reference external" href="http://pypy.readthedocs.org/en/latest/faq.html#would-type-annotations-help-pypy-s-performance">Would type annotations help PyPy’s performance?</a>".
Look under the "PEP-484 Type Hints" sub section.</td></tr>
</tbody>
</table>
</div>
Potential Pythonic Pitfalls2015-05-11T15:42:00+06:002015-05-11T15:42:00+06:00Nafiul Islamtag:nafiulis.me,2015-05-11:potential-pythonic-pitfalls.html<div class="contents topic" id="table-of-contents">
<p class="topic-title first">Table of Contents</p>
<ul class="simple">
<li><a class="reference internal" href="#not-knowing-the-python-version" id="id7">Not Knowing the Python Version</a></li>
<li><a class="reference internal" href="#obsessing-over-one-liners" id="id8">Obsessing Over One-Liners</a></li>
<li><a class="reference internal" href="#initializing-a-set-the-wrong-way" id="id9">Initializing a set the Wrong Way</a></li>
<li><a class="reference internal" href="#misunderstanding-the-gil" id="id10">Misunderstanding the GIL</a></li>
<li><a class="reference internal" href="#using-old-style-classes" id="id11">Using Old Style Classes</a></li>
<li><a class="reference internal" href="#iterating-the-wrong-way" id="id12">Iterating the Wrong Way</a></li>
<li><a class="reference internal" href="#using-mutable-default-arguments" id="id13">Using Mutable Default Arguments</a></li>
<li><a class="reference internal" href="#takeaway" id="id14">Takeaway</a></li>
<li><a class="reference internal" href="#update" id="id15">Update</a></li>
</ul>
</div>
<p>Python is a very expressive language. It provides us with a large standard library and many builtins to get the job
done quickly. However, many can get lost in the power that it provides, fail to make full use of the standard
library, value one liners over clarity and misunderstand its basic constructs. This is a non-exhaustive list of a
few of the pitfalls programmers new to Python fall into.</p>
<div class="section" id="not-knowing-the-python-version">
<h2><a class="toc-backref" href="#id7">Not Knowing the Python Version</a></h2>
<p>This is a recurring problem in StackOverflow questions. Many write perfectly working code for one version but they have
a <em>different</em> version of Python installed on their system.<a class="footnote-reference" href="#pythree" id="id1">[1]</a>
Make sure that you know the Python version you're working with. You can check via the following:</p>
<div class="highlight"><pre><span></span>$ python --version
Python <span class="m">2</span>.7.9
</pre></div>
<div class="section" id="not-using-a-version-manager">
<h3>Not Using a Version Manager</h3>
<p><a class="reference external" href="https://github.com/yyuu/pyenv">pyenv</a> is a great tool for managing different Python versions. Unfortunately, it only
works on *nix systems. On Mac OS, one can simply install it via <code>brew install pyenv</code> and on Linux, there is
an <a class="reference external" href="https://github.com/yyuu/pyenv-installer">automatic installer</a>. <a class="footnote-reference" href="#alternative" id="id2">[2]</a></p>
</div>
</div>
<div class="section" id="obsessing-over-one-liners">
<h2><a class="toc-backref" href="#id8">Obsessing Over One-Liners</a></h2>
<p>Some get a <em>real kick</em> out of one liners. Many boast about their one-liner solutions even if they are less efficient
than a multi-line solution.</p>
<p>What this essentially means in Python is convoluted comprehensions having multiple expressions. For example:</p>
<div class="highlight"><pre><span></span><span class="n">l</span> <span class="o">=</span> <span class="p">[</span><span class="n">m</span> <span class="k">for</span> <span class="n">a</span><span class="p">,</span> <span class="n">b</span> <span class="ow">in</span> <span class="nb">zip</span><span class="p">(</span><span class="n">this</span><span class="p">,</span> <span class="n">that</span><span class="p">)</span> <span class="k">if</span> <span class="n">b</span><span class="o">.</span><span class="n">method</span><span class="p">(</span><span class="n">a</span><span class="p">)</span> <span class="o">!=</span> <span class="n">b</span> <span class="k">for</span> <span class="n">m</span> <span class="ow">in</span> <span class="n">b</span> <span class="k">if</span> <span class="ow">not</span> <span class="n">m</span><span class="o">.</span><span class="n">method</span><span class="p">(</span><span class="n">a</span><span class="p">,</span> <span class="n">b</span><span class="p">)</span> <span class="ow">and</span> <span class="nb">reduce</span><span class="p">(</span><span class="k">lambda</span> <span class="n">x</span><span class="p">,</span> <span class="n">y</span><span class="p">:</span> <span class="n">a</span> <span class="o">+</span> <span class="n">y</span><span class="o">.</span><span class="n">method</span><span class="p">(),</span> <span class="p">(</span><span class="n">m</span><span class="p">,</span> <span class="n">a</span><span class="p">,</span> <span class="n">b</span><span class="p">))]</span>
</pre></div>
<p>To be perfectly honest, I made the above example up. But, I've seen plenty of people write code like it. Code like
this will make no sense in a week's time. If you're trying to do something a little more complex that simply adding
an item to a <code>list</code> or <code>set</code> with a condition then you're probably making a mistake.</p>
<p>One-Liners are not achievements, yes they can seem very clever but they are not achievements. Its like thinking that
shoving everything into your closet is an actual attempt at cleaning your room. Good code is clean, easy to read and
efficient.</p>
</div>
<div class="section" id="initializing-a-set-the-wrong-way">
<h2><a class="toc-backref" href="#id9">Initializing a set the Wrong Way</a></h2>
<p>This is a more subtle problem that can catch you off guard. <code>set</code> comprehensions are a lot like list
comprehensions.</p>
<div class="highlight"><pre><span></span><span class="o">>>></span> <span class="p">{</span> <span class="n">n</span> <span class="k">for</span> <span class="n">n</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">10</span><span class="p">)</span> <span class="k">if</span> <span class="n">n</span> <span class="o">%</span> <span class="mi">2</span> <span class="o">==</span> <span class="mi">0</span> <span class="p">}</span>
<span class="p">{</span><span class="mi">0</span><span class="p">,</span> <span class="mi">8</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">4</span><span class="p">,</span> <span class="mi">6</span><span class="p">}</span>
<span class="o">>>></span> <span class="nb">type</span><span class="p">({</span> <span class="n">n</span> <span class="k">for</span> <span class="n">n</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">10</span><span class="p">)</span> <span class="k">if</span> <span class="n">n</span> <span class="o">%</span> <span class="mi">2</span> <span class="o">==</span> <span class="mi">0</span> <span class="p">})</span>
<span class="o"><</span><span class="k">class</span> <span class="err">'</span><span class="nc">set</span><span class="s1">'></span>
</pre></div>
<p>The above is one such example of a set comprehension. Sets are like lists in that they are containers. The difference
is that a set cannot have any duplicate values and sets are unordered. Seeing set comprehensions people often make
the mistake of thinking that <code>{}</code> initializes an <em>empty</em> set. It does not, it initializes an empty <em>dict</em>.</p>
<div class="highlight"><pre><span></span><span class="o">>>></span> <span class="p">{}</span>
<span class="p">{}</span>
<span class="o">>>></span> <span class="nb">type</span><span class="p">({})</span>
<span class="o"><</span><span class="k">class</span> <span class="err">'</span><span class="nc">dict</span><span class="s1">'></span>
</pre></div>
<p>If we wish to initialize an empty set, then we simply call <code>set()</code>.</p>
<div class="highlight"><pre><span></span><span class="o">>>></span> <span class="nb">set</span><span class="p">()</span>
<span class="nb">set</span><span class="p">()</span>
<span class="o">>>></span> <span class="nb">type</span><span class="p">(</span><span class="nb">set</span><span class="p">())</span>
<span class="o"><</span><span class="k">class</span> <span class="err">'</span><span class="nc">set</span><span class="s1">'></span>
</pre></div>
<p>Note how an empty set is denoted as <code>set()</code> but a set containing something is denoted as items surrounded by
curly braces.</p>
<div class="highlight"><pre><span></span><span class="o">>>></span> <span class="n">s</span> <span class="o">=</span> <span class="nb">set</span><span class="p">()</span>
<span class="o">>>></span> <span class="n">s</span>
<span class="nb">set</span><span class="p">()</span>
<span class="o">>>></span> <span class="n">s</span><span class="o">.</span><span class="n">add</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span>
<span class="o">>>></span> <span class="n">s</span>
<span class="p">{</span><span class="mi">1</span><span class="p">}</span>
<span class="o">>>></span> <span class="n">s</span><span class="o">.</span><span class="n">add</span><span class="p">(</span><span class="mi">2</span><span class="p">)</span>
<span class="o">>>></span> <span class="n">s</span>
<span class="p">{</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">}</span>
</pre></div>
<p>This is rather counter intuitive, since you'd expect something like <code>set([1, 2])</code>.</p>
</div>
<div class="section" id="misunderstanding-the-gil">
<h2><a class="toc-backref" href="#id10">Misunderstanding the GIL</a></h2>
<p>The GIL (Global Interpreter Lock) means that only one thread in a Python program can be running at any one time. This
implies that when we create a thread and expect to run in <em>parallel</em> it doesn't. What the Python interpreter is
actually doing is quickly <em>switching</em> between different running threads. But this is an oversimplified version of
what is actually happening. There are many instances in which things <em>do</em> run in parallel, like when using libraries
that are essentially C extensions. But when running Python code, you don't get parallel execution <em>most</em> of the time. In
other words, threads in Python are not like Threads in Java or C++.</p>
<p>Many will try to defend Python by saying that these are <em>real</em> threads.<a class="footnote-reference" href="#realthreads" id="id3">[3]</a>
This is indeed true, but does not change the fact that how Python handles threads is different from what you'd generally
expect. This is the same case for a language like Ruby (which also has an interpreter lock).</p>
<p>The prescribed solution to this is using the <code>multiprocessing</code> module. The <code>multiprocessing</code> module provides
you with the <code>Process</code> class which is basically a nice cover over a fork. However, a fork is much more expensive
than a thread, so you might not always see the performance benefits since now the different processes have to do a lot
of work to co-ordinate with each other.</p>
<p>However, this problem does not exist every <em>implementation</em> of Python.
<a class="reference external" href="http://pypy.readthedocs.org/en/latest/stm.html">PyPy-stm</a> for example is an implementation of Python that tries
to get rid of the GIL (still not stable yet). Implementations built on top of other platforms like the JVM (Jython)
or CLR (IronPython) do not have GIL problems.</p>
<p>All in all, be careful when using the <code>Thread</code> class, what you get might not be what you expect.</p>
</div>
<div class="section" id="using-old-style-classes">
<h2><a class="toc-backref" href="#id11">Using Old Style Classes</a></h2>
<p>In Python 2 there are two types of classes, there's the "old style" classes, and there's the "new style" classes. If
you're using Python 3, then you're using the "new style" classes by default. In order to make sure that you're using
"new style" classes in Python 2, you need to inherit from <code>object</code> for <em>any</em> new class you create that
isn't already inheriting from a builtin like <code>int</code> or <code>list</code>. In other words, your base class, the class
that isn't inheriting from anything else, should always inherit from object.</p>
<div class="highlight"><pre><span></span><span class="k">class</span> <span class="nc">MyNewObject</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span>
<span class="c1"># stuff here</span>
</pre></div>
<p>These "new style" classes fix some very fundamental flaws in the old style classes that we really don't need to get
into. However, if anyone is interested they can find the information in the
<a class="reference external" href="https://docs.python.org/2/reference/datamodel.html#new-style-and-classic-classes">related documentation</a>.</p>
</div>
<div class="section" id="iterating-the-wrong-way">
<h2><a class="toc-backref" href="#id12">Iterating the Wrong Way</a></h2>
<p>Its very common to see the following code from users who are relatively new to the language:</p>
<div class="highlight"><pre><span></span><span class="k">for</span> <span class="n">name_index</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="nb">len</span><span class="p">(</span><span class="n">names</span><span class="p">)):</span>
<span class="k">print</span><span class="p">(</span><span class="n">names</span><span class="p">[</span><span class="n">name_index</span><span class="p">])</span>
</pre></div>
<p>There is no need to call <code>len</code> in the above example, since iterating over the list is actually much simpler:</p>
<div class="highlight"><pre><span></span><span class="k">for</span> <span class="n">name</span> <span class="ow">in</span> <span class="n">names</span><span class="p">:</span>
<span class="k">print</span><span class="p">(</span><span class="n">name</span><span class="p">)</span>
</pre></div>
<p>Furthermore, there are a whole host of other tools at your disposal to make iteration easier. For example, <code>zip</code>
can be used to iterate over two lists at once:<a class="footnote-reference" href="#zip" id="id4">[4]</a></p>
<div class="highlight"><pre><span></span><span class="k">for</span> <span class="n">cat</span><span class="p">,</span> <span class="n">dog</span> <span class="ow">in</span> <span class="nb">zip</span><span class="p">(</span><span class="n">cats</span><span class="p">,</span> <span class="n">dogs</span><span class="p">):</span>
<span class="k">print</span><span class="p">(</span><span class="n">cat</span><span class="p">,</span> <span class="n">dog</span><span class="p">)</span>
</pre></div>
<p>If we want to take into consideration both the index <em>and</em> the value list variable, we can use <code>enumerate</code>:<a class="footnote-reference" href="#enumerate" id="id5">[5]</a></p>
<div class="highlight"><pre><span></span><span class="k">for</span> <span class="n">index</span><span class="p">,</span> <span class="n">cat</span> <span class="ow">in</span> <span class="nb">enumerate</span><span class="p">(</span><span class="n">cats</span><span class="p">):</span>
<span class="k">print</span><span class="p">(</span><span class="n">cat</span><span class="p">,</span> <span class="n">index</span><span class="p">)</span>
</pre></div>
<p>There are also many useful functions to choose from in <a class="reference external" href="https://docs.python.org/3/library/itertools.html">itertools</a>.
Please note however, that using <code>itertools</code> functions is not always the right choice. If one of the functions
in <code>itertools</code> offers a very convenient solution to the problem you're trying to solve, like flattening a list or
creating a getting the permutations of the contents of a given list, then go for it. But <em>don't</em> try to fit it into
some part of your code just because you want to.</p>
<p>The problem with <code>itertools</code> abuse happens so often that one highly respected Python contributor on StackOverflow
has dedicated a significant part of their profile to it.<a class="footnote-reference" href="#abarnert" id="id6">[6]</a></p>
</div>
<div class="section" id="using-mutable-default-arguments">
<h2><a class="toc-backref" href="#id13">Using Mutable Default Arguments</a></h2>
<p>I've seen the following quite a lot:</p>
<div class="highlight"><pre><span></span><span class="k">def</span> <span class="nf">foo</span><span class="p">(</span><span class="n">a</span><span class="p">,</span> <span class="n">b</span><span class="p">,</span> <span class="n">c</span><span class="o">=</span><span class="p">[]):</span>
<span class="c1"># append to c</span>
<span class="c1"># do some more stuff</span>
</pre></div>
<p>Never use mutable default arguments, instead use the following:</p>
<div class="highlight"><pre><span></span><span class="k">def</span> <span class="nf">foo</span><span class="p">(</span><span class="n">a</span><span class="p">,</span> <span class="n">b</span><span class="p">,</span> <span class="n">c</span><span class="o">=</span><span class="bp">None</span><span class="p">):</span>
<span class="k">if</span> <span class="n">c</span> <span class="ow">is</span> <span class="bp">None</span><span class="p">:</span>
<span class="n">c</span> <span class="o">=</span> <span class="p">[]</span>
<span class="c1"># append to c</span>
<span class="c1"># do some more stuff</span>
</pre></div>
<p>Instead of explaining what the problem is, its better to show the effects of using mutable default arguments:</p>
<div class="highlight"><pre><span></span><span class="n">In</span><span class="p">[</span><span class="mi">2</span><span class="p">]:</span> <span class="k">def</span> <span class="nf">foo</span><span class="p">(</span><span class="n">a</span><span class="p">,</span> <span class="n">b</span><span class="p">,</span> <span class="n">c</span><span class="o">=</span><span class="p">[]):</span>
<span class="o">...</span> <span class="n">c</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">a</span><span class="p">)</span>
<span class="o">...</span> <span class="n">c</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">b</span><span class="p">)</span>
<span class="o">...</span> <span class="k">print</span><span class="p">(</span><span class="n">c</span><span class="p">)</span>
<span class="o">...</span>
<span class="n">In</span><span class="p">[</span><span class="mi">3</span><span class="p">]:</span> <span class="n">foo</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="mi">1</span><span class="p">)</span>
<span class="p">[</span><span class="mi">1</span><span class="p">,</span> <span class="mi">1</span><span class="p">]</span>
<span class="n">In</span><span class="p">[</span><span class="mi">4</span><span class="p">]:</span> <span class="n">foo</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="mi">1</span><span class="p">)</span>
<span class="p">[</span><span class="mi">1</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">1</span><span class="p">]</span>
<span class="n">In</span><span class="p">[</span><span class="mi">5</span><span class="p">]:</span> <span class="n">foo</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="mi">1</span><span class="p">)</span>
<span class="p">[</span><span class="mi">1</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">1</span><span class="p">]</span>
</pre></div>
<dl class="docutils">
<dt>The same <code>c</code> is being referenced again and again every time the function is called. This can have some very</dt>
<dd>unwanted consequences.</dd>
</dl>
</div>
<div class="section" id="takeaway">
<h2><a class="toc-backref" href="#id14">Takeaway</a></h2>
<p>These are just some of the problems that one might run into when relatively new at Python. Please note however, that
this is far from a comprehensive list of the problems that one might run into. The other pitfalls however are largely
to do with people using Python like Java or C++ and trying to use Python in a way that they are familiar with. So,
as a continuation of this, try diving into things like Python's <code>super</code> function. Take a look at
<code>classmethod</code>, <code>staticmethod</code> and <code>__slots__</code>.</p>
</div>
<div class="section" id="update">
<h2><a class="toc-backref" href="#id15">Update</a></h2>
<p><em>Last Updated on 12 May 2015 4:50 PM (GMT +6)</em></p>
<ul class="simple">
<li>Made the section on <a class="reference internal" href="#misunderstanding-the-gil">Misunderstanding the GIL</a> better.</li>
</ul>
<hr class="docutils" />
<table class="docutils footnote" frame="void" id="pythree" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#id1">[1]</a></td><td>Most people are taught Python using Python 2. However, when they go home and try things out
themselves, they install Python 3 (quite a natural thing to install the <em>latest</em> version).</td></tr>
</tbody>
</table>
<table class="docutils footnote" frame="void" id="alternative" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#id2">[2]</a></td><td><a class="reference external" href="http://saghul.github.io/pythonz/">pythonz</a> is an alternative to Pyenv thats worth checking out
if pyenv doesn't work for you.</td></tr>
</tbody>
</table>
<table class="docutils footnote" frame="void" id="realthreads" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#id3">[3]</a></td><td>When people talk about <em>real</em> threads what they essentially mean is that these threads are real CPU
threads, which are scheduled by the OS (Operating System).</td></tr>
</tbody>
</table>
<table class="docutils footnote" frame="void" id="zip" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#id4">[4]</a></td><td><a class="reference external" href="https://docs.python.org/3/library/functions.html#zip">https://docs.python.org/3/library/functions.html#zip</a></td></tr>
</tbody>
</table>
<table class="docutils footnote" frame="void" id="enumerate" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#id5">[5]</a></td><td><code>enumerate</code> can be further configured to produce the kind of index you want.
<a class="reference external" href="https://docs.python.org/3/library/functions.html#enumerate">https://docs.python.org/3/library/functions.html#enumerate</a></td></tr>
</tbody>
</table>
<table class="docutils footnote" frame="void" id="abarnert" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#id6">[6]</a></td><td><a class="reference external" href="http://stackoverflow.com/users/908494/abarnert">http://stackoverflow.com/users/908494/abarnert</a></td></tr>
</tbody>
</table>
</div>
An Open Letter To StackOverflow Election Winners2015-04-21T14:21:00+06:002015-04-21T14:21:00+06:00Nafiul Islamtag:nafiulis.me,2015-04-21:an-open-letter-stackoverflow-election-winners.html<p>Almost three years ago, I found a relatively new site on the internet. It kept popping up on the first few search
results whenever I would be looking for an answer to a programming problem. You had trouble with Java? StackOverflow.
You don't understand how python generators work? StackOverflow. You have a question about how C++ friend classes worked?
StackOverflow. Because it was such a great site open to free membership, I decided to join.</p>
<p>My time as a StackOverflow user started terribly.
The first question I asked was also one of the first question I deleted, because I thought it was so terrible!</p>
<p>I was transitioning
from Java to Python after taking an online course on Python. I was quite new to computer science too. One of the
very first question was about the unary operator, <code>~</code>.<a class="footnote-reference" href="#tilde" id="id1">[1]</a>
I didn't even know what it was called back then. Was it a <em>tridle</em>?
Was it a <em>twiddle</em>? What was it even called in the context of computer Science? I didn't know. That's why I asked the
question, I had never used it in writing.
I searched google, and all I got was several links pertaining to the <code>@</code> symbol. I checked again just
now and it still does.</p>
<img alt="Google search results for at symbol" class="align-center" src="/images/open_letter_01.png" />
<p>I thought to myself that its a simple question, and I'm sure that this would be a valid one.
It was down-voted six times in total, where I thought it would be amicably met.
If you're the kind of arse that gets a kick out of making other people feel
crap about themselves, then the above example has earned nothing but a condescending sneer from you. If so, then you
don't need to keep on reading.</p>
<p>This was a community that I <em>admired</em>, <em>valued</em> and <em>looked up to</em>. When <em>six</em> of this community
conveyed how <em>stupid</em> my question was, I was totally crushed.</p>
<p>Wouldn't you be? If the very community you had come to admire had told you that you had the intelligence of a goat,
how would you feel? I felt so ashamed that I deleted the question. Such a stupid question is <em>undeserving</em> of
StackOverflow. In a site that is completely transparent like StackOverflow, all your stupidity is neatly documented
in the questions that you've asked. All your deficiencies are there for the world to see. So I started deleting
my questions.</p>
<p>But then, I got banned from asking questions entirely. No one told me what I was banned either.
Did the StackOverflow bot determine that I was a terrible programmer, and that I had not belong here?
Did I not make the cut? Was there a limit to how many times you could be down-voted when starting out?</p>
<p>Fast-forward to now, and you <em>still</em> get this kind of behaviour.<a class="footnote-reference" href="#still" id="id2">[2]</a>
Trigger happy down-votes and close votes are the
<em>norm</em> now. The thing that's changed is that I've answered a few questions and asked a few more too.</p>
<p>I wanted to stand in the elections. I didn't because I don't think I'm anywhere <em>near</em> good enough. You need to have
plenty of reputation and badges to stand a chance of winning. I don't so I voted for the people who I believe
will be a <em>little</em> kinder and gentler to new users.</p>
<p>So, whoever you three are, I hope you're kinder to newbies. You might think that StackOverflow is too big and too
important to fail. Make no mistake that bars need to be set and examples made. However, that does not give us
license to be cruel to new users who value what we do here and <em>want</em> to contribute.</p>
<p>I know I'm <em>no one</em> to tell the highly esteemed winners of the elections what to do. I forfeited that right when
I chose not to stand. However, I beg of you to consider the consequences of the chronic levels of intolerance that
we show new comers.</p>
<hr class="docutils" />
<table class="docutils footnote" frame="void" id="tilde" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#id1">[1]</a></td><td><a class="reference external" href="http://stackoverflow.com/questions/18742078/what-does-the-operator-do-in-python">http://stackoverflow.com/questions/18742078/what-does-the-operator-do-in-python</a></td></tr>
</tbody>
</table>
<table class="docutils footnote" frame="void" id="still" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#id2">[2]</a></td><td>You can only see
<a class="reference external" href="http://stackoverflow.com/questions/29700432/searching-for-duplicate-values-in-a-dictionary">this</a>
if you have the ability to see deleted questions. OP asked a simple question, got down-voted 8 times.
Eventually deleted the question. This could eventually lead to OP getting question-banned.</td></tr>
</tbody>
</table>
The Deceptive Anagram Question2015-04-16T00:10:00+06:002015-04-16T00:10:00+06:00Nafiul Islamtag:nafiulis.me,2015-04-16:the-deceptive-anagram-question.html<div class="contents topic" id="table-of-contents">
<p class="topic-title first">Table of Contents</p>
<ul class="simple">
<li><a class="reference internal" href="#the-question" id="id5">The Question</a><ul>
<li><a class="reference internal" href="#an-example" id="id6">An Example</a></li>
</ul>
</li>
<li><a class="reference internal" href="#quadratic-time" id="id7">Quadratic Time</a><ul>
<li><a class="reference internal" href="#initial-solution" id="id8">Initial Solution</a></li>
</ul>
</li>
<li><a class="reference internal" href="#linear-time" id="id9">Linear Time</a><ul>
<li><a class="reference internal" href="#hashing-the-right-way" id="id10">Hashing The Right Way</a><ul>
<li><a class="reference internal" href="#the-pythonic-version" id="id11">The Pythonic Version</a></li>
</ul>
</li>
<li><a class="reference internal" href="#ordering-done-right" id="id12">Ordering Done Right</a></li>
<li><a class="reference internal" href="#profiling" id="id13">Profiling</a></li>
</ul>
</li>
<li><a class="reference internal" href="#a-vector-approach-to-hashing" id="id14">A Vector Approach to Hashing</a></li>
<li><a class="reference internal" href="#the-final-frontier" id="id15">The Final Frontier</a><ul>
<li><a class="reference internal" href="#but-there-s-an-import-for-that" id="id16">But There's An Import For That</a></li>
</ul>
</li>
<li><a class="reference internal" href="#lessons-learnt" id="id17">Lessons Learnt</a></li>
<li><a class="reference internal" href="#acknowledgments" id="id18">Acknowledgments</a></li>
</ul>
</div>
<p>Last month, I had a programming interview. It hadn't gone as I would've liked, but I did get asked a question that I
found interesting. The question is deceptively simple, but has a lot of depth to it. Since I failed to solve the
problem correctly in the interview, I decided explore the ways in which I could optimize my initially O(n<sup>2</sup>)
solution.</p>
<p>After a few attempts at solving the problem from different angles, I've come to appreciate the importance of
understanding complexity, as well as its limitations.</p>
<div class="section" id="the-question">
<h2><a class="toc-backref" href="#id5">The Question</a></h2>
<p>The question was simple:</p>
<blockquote>
Given a list of words L, find all the anagrams in L in the order in which they appear in L.</blockquote>
<div class="section" id="an-example">
<h3><a class="toc-backref" href="#id6">An Example</a></h3>
<p>Given the input</p>
<div class="highlight"><pre><span></span><span class="p">[</span><span class="s2">"pool"</span><span class="p">,</span> <span class="s2">"loco"</span><span class="p">,</span> <span class="s2">"cool"</span><span class="p">,</span> <span class="s2">"stain"</span><span class="p">,</span> <span class="s2">"satin"</span><span class="p">,</span> <span class="s2">"pretty"</span><span class="p">,</span> <span class="s2">"nice"</span><span class="p">,</span> <span class="s2">"loop"</span><span class="p">]</span>
</pre></div>
<p>The desired output would be</p>
<div class="highlight"><pre><span></span><span class="p">[</span><span class="s2">"pool"</span><span class="p">,</span> <span class="s2">"loco"</span><span class="p">,</span> <span class="s2">"cool"</span><span class="p">,</span> <span class="s2">"stain"</span><span class="p">,</span> <span class="s2">"satin"</span><span class="p">,</span> <span class="s2">"loop"</span><span class="p">]</span>
</pre></div>
<p>in that order <em>exactly</em>.</p>
</div>
</div>
<div class="section" id="quadratic-time">
<h2><a class="toc-backref" href="#id7">Quadratic Time</a></h2>
<p>The naive solution to this will give you an O(n<sup>2</sup>) algorithm. Be warned, the following code may burn your eyes.</p>
<div class="section" id="initial-solution">
<h3><a class="toc-backref" href="#id8">Initial Solution</a></h3>
<div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">collections</span> <span class="kn">import</span> <span class="n">Counter</span>
<span class="k">def</span> <span class="nf">anagram_finder</span><span class="p">(</span><span class="n">word_list</span><span class="p">):</span>
<span class="n">_ret</span> <span class="o">=</span> <span class="nb">set</span><span class="p">()</span>
<span class="k">for</span> <span class="n">word</span> <span class="ow">in</span> <span class="n">word_list</span><span class="p">:</span>
<span class="n">c</span> <span class="o">=</span> <span class="n">Counter</span><span class="p">(</span><span class="n">word</span><span class="p">)</span>
<span class="k">for</span> <span class="n">other_word</span> <span class="ow">in</span> <span class="p">[</span><span class="n">w</span> <span class="k">for</span> <span class="n">w</span> <span class="ow">in</span> <span class="n">word_list</span> <span class="k">if</span> <span class="n">w</span> <span class="o">!=</span> <span class="n">word</span><span class="p">]:</span>
<span class="k">if</span> <span class="n">Counter</span><span class="p">(</span><span class="n">other_word</span><span class="p">)</span> <span class="o">==</span> <span class="n">c</span><span class="p">:</span>
<span class="n">_ret</span><span class="o">.</span><span class="n">add</span><span class="p">(</span><span class="n">word</span><span class="p">)</span>
<span class="n">_ret</span><span class="o">.</span><span class="n">add</span><span class="p">(</span><span class="n">other_word</span><span class="p">)</span>
<span class="k">return</span> <span class="p">[</span><span class="n">w</span> <span class="k">for</span> <span class="n">w</span> <span class="ow">in</span> <span class="n">word_list</span> <span class="k">if</span> <span class="n">w</span> <span class="ow">in</span> <span class="n">_ret</span><span class="p">]</span>
<span class="k">if</span> <span class="vm">__name__</span> <span class="o">==</span> <span class="s1">'__main__'</span><span class="p">:</span>
<span class="k">print</span> <span class="n">anagram_finder</span><span class="p">([</span><span class="s2">"pool"</span><span class="p">,</span> <span class="s2">"loco"</span><span class="p">,</span> <span class="s2">"cool"</span><span class="p">,</span> <span class="s2">"stain"</span><span class="p">,</span> <span class="s2">"satin"</span><span class="p">,</span> <span class="s2">"pretty"</span><span class="p">,</span> <span class="s2">"nice"</span><span class="p">,</span> <span class="s2">"loop"</span><span class="p">])</span>
</pre></div>
<p>This is <em>wrong</em> on so many levels, I don't even know <em>where</em> to begin.</p>
<ul class="simple">
<li>The problem solves the solution in quadratic time meaning, for more computing power we throw at this
the slower it gets per computer. Imagine if we were to put an algorithm like this in the server, we would have
serious scaling issues.</li>
<li><code>collections.Counter</code> is expensive. It needs to create a dictionary, then it needs to add <em>each</em> character
to the dictionary, that means <em>hashing</em> each character.</li>
<li>It adds the original word every single time it finds an anagram, and relies on <code>set</code> not to add
duplicate values.</li>
</ul>
<p>I'm probably missing a few points but the point remains; this is seriously bad code.</p>
</div>
</div>
<div class="section" id="linear-time">
<h2><a class="toc-backref" href="#id9">Linear Time</a></h2>
<p>So, how can we turn this problem form an O(n<sup>2</sup>) solution into an O(n) solution? Using hash-maps correctly.
Unfortunately, I did not come up with the brilliant idea of using a hash-map, but rather my interviewer told me
that the way to get O(n) was to use a hash-map.</p>
<div class="section" id="hashing-the-right-way">
<h3><a class="toc-backref" href="#id10">Hashing The Right Way</a></h3>
<p>In general, whenever you hear the word "hash" you think md5 or SHA. But in reality, a hash is a way
to map data in a uniform way. Think of it like this, if you have a the word <em>pool</em> and <em>loop</em>, in the eyes of the
anagram solver, they are the same. Why? Because <em>both</em> words use the same characters. In other words, there had to be
a uniform way to converting these two words into the <em>same</em> thing. If we were to simply <em>sort</em> the characters in the
word, we'd get exactly what we're looking for. Here's a demonstration:</p>
<div class="highlight"><pre><span></span><span class="o">>>></span> <span class="nb">sorted</span><span class="p">(</span><span class="s2">"loop"</span><span class="p">)</span>
<span class="p">[</span><span class="s1">'l'</span><span class="p">,</span> <span class="s1">'o'</span><span class="p">,</span> <span class="s1">'o'</span><span class="p">,</span> <span class="s1">'p'</span><span class="p">]</span>
<span class="o">>>></span> <span class="nb">sorted</span><span class="p">(</span><span class="s2">"pool"</span><span class="p">)</span>
<span class="p">[</span><span class="s1">'l'</span><span class="p">,</span> <span class="s1">'o'</span><span class="p">,</span> <span class="s1">'o'</span><span class="p">,</span> <span class="s1">'p'</span><span class="p">]</span>
<span class="o">>>></span> <span class="s2">""</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="nb">sorted</span><span class="p">(</span><span class="s2">"loop"</span><span class="p">))</span>
<span class="s1">'loop'</span>
<span class="o">>>></span> <span class="s2">""</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="nb">sorted</span><span class="p">(</span><span class="s2">"pool"</span><span class="p">))</span>
<span class="s1">'loop'</span>
</pre></div>
<p>With that, I had my hashing function and with it, I had my linear solution.</p>
<div class="highlight"><pre><span></span><span class="k">def</span> <span class="nf">hasher</span><span class="p">(</span><span class="n">w</span><span class="p">):</span> <span class="c1"># 1</span>
<span class="k">return</span> <span class="s2">""</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="nb">sorted</span><span class="p">(</span><span class="n">w</span><span class="p">))</span>
<span class="k">def</span> <span class="nf">anagram_finder</span><span class="p">(</span><span class="n">word_list</span><span class="p">):</span>
<span class="n">hash_dict</span> <span class="o">=</span> <span class="p">{}</span> <span class="c1"># 2</span>
<span class="k">for</span> <span class="n">word</span> <span class="ow">in</span> <span class="n">word_list</span><span class="p">:</span>
<span class="n">h</span> <span class="o">=</span> <span class="n">hasher</span><span class="p">(</span><span class="n">word</span><span class="p">)</span> <span class="c1"># 3</span>
<span class="k">if</span> <span class="n">h</span> <span class="ow">not</span> <span class="ow">in</span> <span class="n">hash_dict</span><span class="p">:</span> <span class="c1"># 4</span>
<span class="n">hash_dict</span><span class="p">[</span><span class="n">h</span><span class="p">]</span> <span class="o">=</span> <span class="p">[]</span> <span class="c1"># 5</span>
<span class="n">hash_dict</span><span class="p">[</span><span class="n">h</span><span class="p">]</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">word</span><span class="p">)</span> <span class="c1"># 6</span>
<span class="k">return</span> <span class="p">[</span><span class="n">anagram</span> <span class="k">for</span> <span class="n">l</span> <span class="ow">in</span> <span class="n">hash_dict</span><span class="o">.</span><span class="n">values</span><span class="p">()</span> <span class="k">for</span> <span class="n">anagram</span> <span class="ow">in</span> <span class="n">l</span> <span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">l</span><span class="p">)</span> <span class="o">></span> <span class="mi">1</span><span class="p">]</span> <span class="c1"># 7</span>
<span class="k">if</span> <span class="vm">__name__</span> <span class="o">==</span> <span class="s1">'__main__'</span><span class="p">:</span>
<span class="k">print</span><span class="p">(</span><span class="n">anagram_finder</span><span class="p">([</span><span class="s2">"pool"</span><span class="p">,</span> <span class="s2">"loco"</span><span class="p">,</span> <span class="s2">"cool"</span><span class="p">,</span> <span class="s2">"stain"</span><span class="p">,</span> <span class="s2">"satin"</span><span class="p">,</span> <span class="s2">"pretty"</span><span class="p">,</span> <span class="s2">"nice"</span><span class="p">,</span> <span class="s2">"loop"</span><span class="p">]))</span>
</pre></div>
<p>In <code class="coderef">1</code>, we create the <code>hasher</code> function. The hash is simple, it sorts the string alphabetically using
<code>sorted</code>, which returns a list, which we then use as an iterable for <code>"".join</code> to create a string. We
do this because python lists are not hashable (because they are mutable).</p>
<p>In <code class="coderef">2</code>, inside the <code>anagram_finder</code> function, we create a <code>hash_dict</code>, a dictionary for all our hashes.
It must be pointed out that the dictionary, when adding new keys,
will hash those keys as well.<a class="footnote-reference" href="#howstringsarehashed" id="id1">[1]</a> The worst case for <code>hasher</code> is O(n) where n is the length
of the word in question, so no issue with the size of the list we're given.</p>
<p>In <code class="coderef">3</code> we actually call the <code>hasher</code> to hash the string. In <code class="coderef">4</code>, we check to see if this hash exists in the
keys of <code>hash_dict</code>. If not, then we create a new list so that we can append words to it in <code class="coderef">5</code>.</p>
<p>In the end, we <em>always</em> append the word to the list of that key in <code class="coderef">6</code>. This means, that every key will always have
<em>at least one</em> value stored in its list, and these values are the ones we don't one.</p>
<p>The simplified version of <code class="coderef">7</code> is as follows:</p>
<div class="highlight"><pre><span></span><span class="n">_ret</span> <span class="o">=</span> <span class="p">[]</span>
<span class="k">for</span> <span class="n">l</span> <span class="ow">in</span> <span class="n">hash_dict</span><span class="o">.</span><span class="n">values</span><span class="p">():</span>
<span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">l</span><span class="p">)</span> <span class="o">></span> <span class="mi">1</span><span class="p">:</span>
<span class="n">_ret</span> <span class="o">+=</span> <span class="n">l</span>
</pre></div>
<div class="section" id="the-pythonic-version">
<h4><a class="toc-backref" href="#id11">The Pythonic Version</a></h4>
<p>The above is great for explanation, but the pythonic version is much smaller:</p>
<div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">collections</span> <span class="kn">import</span> <span class="n">defaultdict</span>
<span class="k">def</span> <span class="nf">hasher</span><span class="p">(</span><span class="n">w</span><span class="p">):</span>
<span class="k">return</span> <span class="s2">""</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="nb">sorted</span><span class="p">(</span><span class="n">w</span><span class="p">))</span>
<span class="k">def</span> <span class="nf">anagram_finder</span><span class="p">(</span><span class="n">word_list</span><span class="p">):</span>
<span class="n">hash_dict</span> <span class="o">=</span> <span class="n">defaultdict</span><span class="p">(</span><span class="nb">list</span><span class="p">)</span>
<span class="k">for</span> <span class="n">word</span> <span class="ow">in</span> <span class="n">word_list</span><span class="p">:</span>
<span class="n">hash_dict</span><span class="p">[</span><span class="n">hasher</span><span class="p">(</span><span class="n">word</span><span class="p">)]</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">word</span><span class="p">)</span> <span class="c1"># 1</span>
<span class="k">return</span> <span class="p">[</span><span class="n">anagram</span> <span class="k">for</span> <span class="n">l</span> <span class="ow">in</span> <span class="n">hash_dict</span><span class="o">.</span><span class="n">values</span><span class="p">()</span> <span class="k">for</span> <span class="n">anagram</span> <span class="ow">in</span> <span class="n">l</span> <span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">l</span><span class="p">)</span> <span class="o">></span> <span class="mi">1</span><span class="p">]</span>
<span class="k">if</span> <span class="vm">__name__</span> <span class="o">==</span> <span class="s1">'__main__'</span><span class="p">:</span>
<span class="k">print</span><span class="p">(</span><span class="n">anagram_finder</span><span class="p">([</span><span class="s2">"pool"</span><span class="p">,</span> <span class="s2">"loco"</span><span class="p">,</span> <span class="s2">"cool"</span><span class="p">,</span> <span class="s2">"stain"</span><span class="p">,</span> <span class="s2">"satin"</span><span class="p">,</span> <span class="s2">"pretty"</span><span class="p">,</span> <span class="s2">"nice"</span><span class="p">,</span> <span class="s2">"loop"</span><span class="p">]))</span>
</pre></div>
<p>We've made the code significantly smaller by using a <code>defaultdict</code> in <code class="coderef">1</code>. <code>defaultdict</code> allows us to
give it a factory, in this case <code>list</code>, that will automatically create a list with a key if a key does not exist.
If it does exist, then it will return that list, and we can append to it.<a class="footnote-reference" href="#moreondefaultdict" id="id2">[3]</a></p>
<p>But wait, we forgot about the ordering.</p>
</div>
</div>
<div class="section" id="ordering-done-right">
<h3><a class="toc-backref" href="#id12">Ordering Done Right</a></h3>
<p>We have a quick fix for the ordering, and that is simply to loop through all the words in the initial list and include
only those that is in the list of anagrams. The solution is <em>still</em> O(n), but remains highly inefficient.
One thought might be to use the <code>collections.OrderedDict</code> class. But although that might <em>seem</em> to
work, the ordering will still not match the original in the case where anagrams are not next to each other.
For example, the following piece of code will return:</p>
<div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">collections</span> <span class="kn">import</span> <span class="n">OrderedDict</span>
<span class="k">def</span> <span class="nf">hasher</span><span class="p">(</span><span class="n">w</span><span class="p">):</span>
<span class="k">return</span> <span class="s2">""</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="nb">sorted</span><span class="p">(</span><span class="n">w</span><span class="p">))</span>
<span class="k">def</span> <span class="nf">anagram_finder</span><span class="p">(</span><span class="n">word_list</span><span class="p">):</span>
<span class="n">hash_dict</span> <span class="o">=</span> <span class="n">OrderedDict</span><span class="p">()</span>
<span class="k">for</span> <span class="n">word</span> <span class="ow">in</span> <span class="n">word_list</span><span class="p">:</span>
<span class="n">hash_dict</span><span class="o">.</span><span class="n">setdefault</span><span class="p">(</span><span class="n">hasher</span><span class="p">(</span><span class="n">word</span><span class="p">),</span> <span class="p">[])</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">word</span><span class="p">)</span>
<span class="k">return</span> <span class="p">[</span><span class="n">anagram</span> <span class="k">for</span> <span class="n">l</span> <span class="ow">in</span> <span class="n">hash_dict</span><span class="o">.</span><span class="n">values</span><span class="p">()</span> <span class="k">for</span> <span class="n">anagram</span> <span class="ow">in</span> <span class="n">l</span> <span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">l</span><span class="p">)</span> <span class="o">></span> <span class="mi">1</span><span class="p">]</span>
<span class="k">if</span> <span class="vm">__name__</span> <span class="o">==</span> <span class="s1">'__main__'</span><span class="p">:</span>
<span class="k">print</span><span class="p">(</span><span class="n">anagram_finder</span><span class="p">([</span><span class="s2">"nala"</span><span class="p">,</span> <span class="s2">"pool"</span><span class="p">,</span> <span class="s2">"loco"</span><span class="p">,</span> <span class="s2">"cool"</span><span class="p">,</span> <span class="s2">"stain"</span><span class="p">,</span> <span class="s2">"satin"</span><span class="p">,</span> <span class="s2">"pretty"</span><span class="p">,</span> <span class="s2">"nice"</span><span class="p">,</span> <span class="s2">"loop"</span><span class="p">,</span> <span class="s2">"laan"</span><span class="p">]))</span>
</pre></div>
<div class="highlight"><pre><span></span><span class="p">[</span><span class="s1">'nala'</span><span class="p">,</span> <span class="s1">'laan'</span><span class="p">,</span> <span class="s1">'pool'</span><span class="p">,</span> <span class="s1">'loop'</span><span class="p">,</span> <span class="s1">'loco'</span><span class="p">,</span> <span class="s1">'cool'</span><span class="p">,</span> <span class="s1">'stain'</span><span class="p">,</span> <span class="s1">'satin'</span><span class="p">]</span>
</pre></div>
<p><code>nala</code> and <code>laan</code> should not be next to each other. This is because <code>collections.OrderedDict</code>
remembers the order in which the <em>keys</em> were added.</p>
<p>So, in the end, I stuck to the following:</p>
<div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">collections</span> <span class="kn">import</span> <span class="n">defaultdict</span>
<span class="k">def</span> <span class="nf">hasher</span><span class="p">(</span><span class="n">w</span><span class="p">):</span>
<span class="k">return</span> <span class="s2">""</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="nb">sorted</span><span class="p">(</span><span class="n">w</span><span class="p">))</span>
<span class="k">def</span> <span class="nf">anagram_finder</span><span class="p">(</span><span class="n">word_list</span><span class="p">):</span>
<span class="n">hash_dict</span> <span class="o">=</span> <span class="n">defaultdict</span><span class="p">(</span><span class="nb">list</span><span class="p">)</span>
<span class="k">for</span> <span class="n">word</span> <span class="ow">in</span> <span class="n">word_list</span><span class="p">:</span>
<span class="n">hash_dict</span><span class="p">[</span><span class="n">hasher</span><span class="p">(</span><span class="n">word</span><span class="p">)]</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">word</span><span class="p">)</span>
<span class="k">return</span> <span class="p">[</span><span class="n">word</span> <span class="k">for</span> <span class="n">word</span> <span class="ow">in</span> <span class="n">word_list</span> <span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">hash_dict</span><span class="p">[</span><span class="n">hasher</span><span class="p">(</span><span class="n">word</span><span class="p">)])</span> <span class="o">></span> <span class="mi">1</span><span class="p">]</span>
<span class="k">if</span> <span class="vm">__name__</span> <span class="o">==</span> <span class="s1">'__main__'</span><span class="p">:</span>
<span class="k">print</span><span class="p">(</span><span class="n">anagram_finder</span><span class="p">([</span><span class="s2">"nala"</span><span class="p">,</span> <span class="s2">"pool"</span><span class="p">,</span> <span class="s2">"loco"</span><span class="p">,</span> <span class="s2">"cool"</span><span class="p">,</span> <span class="s2">"stain"</span><span class="p">,</span> <span class="s2">"satin"</span><span class="p">,</span> <span class="s2">"pretty"</span><span class="p">,</span> <span class="s2">"nice"</span><span class="p">,</span> <span class="s2">"loop"</span><span class="p">,</span> <span class="s2">"laan"</span><span class="p">]))</span>
</pre></div>
<p>With that, we solved the <em>ordering</em> problem and still managed to make it linear.</p>
</div>
<div class="section" id="profiling">
<h3><a class="toc-backref" href="#id13">Profiling</a></h3>
<p>We never know how well something will actually work unless we profile it. So I got a big list of words<a class="footnote-reference" href="#biglist" id="id3">[2]</a>
and go to profiling.</p>
<div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">collections</span> <span class="kn">import</span> <span class="n">defaultdict</span>
<span class="k">def</span> <span class="nf">hasher</span><span class="p">(</span><span class="n">w</span><span class="p">):</span>
<span class="k">return</span> <span class="s2">""</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="nb">sorted</span><span class="p">(</span><span class="n">w</span><span class="p">))</span>
<span class="k">def</span> <span class="nf">anagram_finder</span><span class="p">(</span><span class="n">word_list</span><span class="p">):</span>
<span class="n">hash_dict</span> <span class="o">=</span> <span class="n">defaultdict</span><span class="p">(</span><span class="nb">list</span><span class="p">)</span>
<span class="k">for</span> <span class="n">word</span> <span class="ow">in</span> <span class="n">word_list</span><span class="p">:</span>
<span class="n">hash_dict</span><span class="p">[</span><span class="n">hasher</span><span class="p">(</span><span class="n">word</span><span class="p">)]</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">word</span><span class="p">)</span>
<span class="k">return</span> <span class="p">[</span><span class="n">word</span> <span class="k">for</span> <span class="n">word</span> <span class="ow">in</span> <span class="n">word_list</span> <span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">hash_dict</span><span class="p">[</span><span class="n">hasher</span><span class="p">(</span><span class="n">word</span><span class="p">)])</span> <span class="o">></span> <span class="mi">1</span><span class="p">]</span>
<span class="k">if</span> <span class="vm">__name__</span> <span class="o">==</span> <span class="s1">'__main__'</span><span class="p">:</span>
<span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="s1">'wordsEn.txt'</span><span class="p">)</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span>
<span class="n">w_list</span> <span class="o">=</span> <span class="n">f</span><span class="o">.</span><span class="n">read</span><span class="p">()</span><span class="o">.</span><span class="n">split</span><span class="p">()</span>
<span class="n">anagram_finder</span><span class="p">(</span><span class="n">w_list</span><span class="p">)</span>
</pre></div>
<p>I used a library called <a class="reference external" href="https://github.com/what-studio/profiling">profiling</a>, and it can be installed with a simple
pip install. After profiling the above code, I found that my <code>hasher</code> function was actually making the the
whole process a lot slower, because it was being called <em>twice</em>.</p>
<img alt="Profiling without storing hash values" class="align-center" src="/images/anagrams_profile_01.png" />
<p>So, my first attempt (like a good pythonista) was to use <code>functools.lru_cache</code>. That attempt failed
spectacularly.</p>
<div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">collections</span> <span class="kn">import</span> <span class="n">defaultdict</span>
<span class="kn">from</span> <span class="nn">functools</span> <span class="kn">import</span> <span class="n">lru_cache</span>
<span class="nd">@lru_cache</span><span class="p">()</span>
<span class="k">def</span> <span class="nf">hasher</span><span class="p">(</span><span class="n">w</span><span class="p">):</span>
<span class="k">return</span> <span class="s2">""</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="nb">sorted</span><span class="p">(</span><span class="n">w</span><span class="p">))</span>
<span class="k">def</span> <span class="nf">anagram_finder</span><span class="p">(</span><span class="n">word_list</span><span class="p">):</span>
<span class="n">hash_dict</span> <span class="o">=</span> <span class="n">defaultdict</span><span class="p">(</span><span class="nb">list</span><span class="p">)</span>
<span class="k">for</span> <span class="n">word</span> <span class="ow">in</span> <span class="n">word_list</span><span class="p">:</span>
<span class="n">hash_dict</span><span class="p">[</span><span class="n">hasher</span><span class="p">(</span><span class="n">word</span><span class="p">)]</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">word</span><span class="p">)</span>
<span class="k">return</span> <span class="p">[</span><span class="n">word</span> <span class="k">for</span> <span class="n">word</span> <span class="ow">in</span> <span class="n">word_list</span> <span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">hash_dict</span><span class="p">[</span><span class="n">hasher</span><span class="p">(</span><span class="n">word</span><span class="p">)])</span> <span class="o">></span> <span class="mi">1</span><span class="p">]</span>
<span class="k">if</span> <span class="vm">__name__</span> <span class="o">==</span> <span class="s1">'__main__'</span><span class="p">:</span>
<span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="s1">'wordsEn.txt'</span><span class="p">)</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span>
<span class="n">w_list</span> <span class="o">=</span> <span class="n">f</span><span class="o">.</span><span class="n">read</span><span class="p">()</span><span class="o">.</span><span class="n">split</span><span class="p">()</span>
<span class="n">anagram_finder</span><span class="p">(</span><span class="n">w_list</span><span class="p">)</span>
</pre></div>
<p>This is because there were excessive calls being made all over the place. Even increasing the size of the cache from
<code>lru_cache()</code> which has a default if <code>128</code> to <code>lru_cache(11000)</code> which is roughly the size of the
word list I'm using. In fact, increasing the <code>lru_cache</code> size to such an amount slowed down the program so much
that I didn't wait for it to finish, but the root problem was the same; too many calls were being made all over the
place.</p>
<img alt="Using LRU cache this time" class="align-center" src="/images/anagrams_profile_02.png" />
<p>So, to compare the two programs, with lru and without lru, we can see that the program <em>without</em> lru was significantly
faster than the one <em>with</em> lru (4.77 seconds to 15.31 seconds. With the failure or <code>lru_cache</code> as a feasible
solution, I decided to just use a normal dictionary to store hashes. In our original attempt, <code>hasher</code> was being
called twice, once to add words to the dictionary, and then in the list comprehension. Why not just use a dictionary
to store the hash and the word?</p>
<div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">collections</span> <span class="kn">import</span> <span class="n">defaultdict</span>
<span class="k">def</span> <span class="nf">hasher</span><span class="p">(</span><span class="n">word</span><span class="p">):</span>
<span class="k">return</span> <span class="s2">""</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="nb">sorted</span><span class="p">(</span><span class="n">word</span><span class="p">))</span>
<span class="k">def</span> <span class="nf">anagram_finder</span><span class="p">(</span><span class="n">word_list</span><span class="p">):</span>
<span class="n">hash_dict</span> <span class="o">=</span> <span class="n">defaultdict</span><span class="p">(</span><span class="nb">list</span><span class="p">)</span>
<span class="n">hashes</span> <span class="o">=</span> <span class="p">{}</span>
<span class="k">for</span> <span class="n">word</span> <span class="ow">in</span> <span class="n">word_list</span><span class="p">:</span>
<span class="n">hashes</span><span class="p">[</span><span class="n">word</span><span class="p">]</span> <span class="o">=</span> <span class="n">hasher</span><span class="p">(</span><span class="n">word</span><span class="p">)</span>
<span class="n">hash_dict</span><span class="p">[</span><span class="n">hashes</span><span class="p">[</span><span class="n">word</span><span class="p">]]</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">word</span><span class="p">)</span>
<span class="k">return</span> <span class="p">[</span><span class="n">word</span> <span class="k">for</span> <span class="n">word</span> <span class="ow">in</span> <span class="n">word_list</span> <span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">hash_dict</span><span class="p">[</span><span class="n">hashes</span><span class="p">[</span><span class="n">word</span><span class="p">]])</span> <span class="o">></span> <span class="mi">1</span><span class="p">]</span>
<span class="k">if</span> <span class="vm">__name__</span> <span class="o">==</span> <span class="s1">'__main__'</span><span class="p">:</span>
<span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="s1">'wordsEn.txt'</span><span class="p">)</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span>
<span class="n">w_list</span> <span class="o">=</span> <span class="n">f</span><span class="o">.</span><span class="n">read</span><span class="p">()</span><span class="o">.</span><span class="n">split</span><span class="p">()</span>
<span class="n">anagram_finder</span><span class="p">(</span><span class="n">w_list</span><span class="p">)</span>
</pre></div>
<p>We just create a new dictionary, <code>hashes</code> to store all our hash and word pairs. This resulted in a speedup.</p>
<img alt="This time with the hashes dictionary" class="align-center" src="/images/anagrams_profile_03.png" />
</div>
</div>
<div class="section" id="a-vector-approach-to-hashing">
<h2><a class="toc-backref" href="#id14">A Vector Approach to Hashing</a></h2>
<p>Another approach to hashing would be to use a vector to determine the number of letters that appear in a word.
This is better explained through code:</p>
<div class="highlight"><pre><span></span><span class="k">def</span> <span class="nf">hasher</span><span class="p">(</span><span class="n">word</span><span class="p">):</span>
<span class="n">h</span> <span class="o">=</span> <span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="o">*</span> <span class="mi">26</span> <span class="c1"># 1</span>
<span class="k">for</span> <span class="n">c</span> <span class="ow">in</span> <span class="n">word</span><span class="p">:</span>
<span class="n">h</span><span class="p">[</span><span class="nb">ord</span><span class="p">(</span><span class="n">c</span><span class="p">)</span> <span class="o">-</span> <span class="nb">ord</span><span class="p">(</span><span class="s1">'a'</span><span class="p">)]</span> <span class="o">+=</span> <span class="mi">1</span> <span class="c1"># 2</span>
<span class="k">return</span> <span class="nb">tuple</span><span class="p">(</span><span class="n">h</span><span class="p">)</span> <span class="c1"># 3</span>
</pre></div>
<p>In <code class="coderef">1</code>, we create a list of length 26, having all its values initialized to 0. In <code class="coderef">2</code>, we calculate the rank of the
letter in the english alphabet using <code>ord(c) - ord(a)</code>. We use this rank as the index of our list. We then
add 1 to the value of the index. In other words we are merely counting the frequency of the letters in a word; very
similar to a histogram.</p>
<p>The above is O(n) compared to the O(n log(n)) solution that sorting a string gives us (n is the number of characters
in the word here). However, considering that the average size of a word in the english language is actually 5 and
the average size of a word in my <code>wordsEn.txt</code> file is 8, O(n) actually becomes the <em>smaller</em> problem here, and
the constant time that is required to create a list of 26 items is the bigger problem. In other words, its O(26) for
list creation and initiation. Its O(n) for creating the rank and its O(26) for creating the tuple.</p>
<p>Compare that to O(n log(n)) worst case for using <code>sorted</code>. In this particular use case since n is very low,
the better option is to use <code>sorted</code>.</p>
</div>
<div class="section" id="the-final-frontier">
<h2><a class="toc-backref" href="#id15">The Final Frontier</a></h2>
<p>The best solution to this problem however is not one of my own making, but rather one that a friend of mine
(Alexander) provided me with. My solution actually is superfluous in many cases, but the following cuts straight to
the point:</p>
<div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">collections</span> <span class="kn">import</span> <span class="n">defaultdict</span>
<span class="kn">from</span> <span class="nn">itertools</span> <span class="kn">import</span> <span class="n">izip</span>
<span class="k">def</span> <span class="nf">get_anagrams</span><span class="p">(</span><span class="n">words</span><span class="p">):</span>
<span class="n">normalized</span> <span class="o">=</span> <span class="p">[</span><span class="s1">''</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="nb">sorted</span><span class="p">(</span><span class="n">w</span><span class="p">))</span> <span class="k">for</span> <span class="n">w</span> <span class="ow">in</span> <span class="n">words</span><span class="p">]</span> <span class="c1"># 1</span>
<span class="n">d</span> <span class="o">=</span> <span class="n">defaultdict</span><span class="p">(</span><span class="nb">int</span><span class="p">)</span> <span class="c1"># 2</span>
<span class="k">for</span> <span class="n">n</span> <span class="ow">in</span> <span class="n">normalized</span><span class="p">:</span> <span class="c1"># 3</span>
<span class="n">d</span><span class="p">[</span><span class="n">n</span><span class="p">]</span> <span class="o">+=</span> <span class="mi">1</span> <span class="c1"># 4</span>
<span class="k">return</span> <span class="p">[</span><span class="n">w</span> <span class="k">for</span> <span class="n">w</span><span class="p">,</span> <span class="n">n</span> <span class="ow">in</span> <span class="n">izip</span><span class="p">(</span><span class="n">words</span><span class="p">,</span> <span class="n">normalized</span><span class="p">)</span> <span class="k">if</span> <span class="n">d</span><span class="p">[</span><span class="n">n</span><span class="p">]</span> <span class="o">></span> <span class="mi">1</span><span class="p">]</span> <span class="c1"># 5</span>
</pre></div>
<p>The genius of this approach is to use <code>collections.defaultdict</code> with <code>int</code> as the factory. This basically
means that you have a smaller memory footprint. So, in <code class="coderef">1</code>, we get a list of hashed values (which he calls normalized).
In <code class="coderef">2</code>, we create a <code>defaultdict</code> of int, this means that all the keys will have an initial value of 0.</p>
<p>In <code class="coderef">3</code>, we loop over all the normalized/hashed values. <code class="coderef">4</code> is where the magic happens. So, there will be duplicates
in <code>normalized</code>, and we are going to count the number of times each of those normalized values appear.</p>
<p>In <code class="coderef">5</code>, we zip over the initial word list, <code>words</code> and <code>normalized</code>. Remember, that <em>all</em> the words are
normalized. All we do then is simply check to see if the number of times the normalized value of the word appears in
<code>d</code>. If there's more than one occurrence, we add the word to the final list in this list comprehension.</p>
<div class="section" id="but-there-s-an-import-for-that">
<h3><a class="toc-backref" href="#id16">But There's An Import For That</a></h3>
<p>In the very beginning, I used <code>collections.Counter</code>, to provide the <em>worst</em> possible solution to the problem.
But, I think this time, we can actually use <code>collections.Counter</code> correctly, because <code class="coderef">2</code>, <code class="coderef">3</code> and <code class="coderef">4</code> can
be shortened to just <code>Counter(normalized)</code>. So the <em>final final</em> (gosh, that feels javaish) version is:</p>
<div class="highlight"><pre><span></span><span class="k">def</span> <span class="nf">get_anagrams</span><span class="p">(</span><span class="n">words</span><span class="p">):</span>
<span class="n">normalized</span> <span class="o">=</span> <span class="p">[</span><span class="s1">''</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="nb">sorted</span><span class="p">(</span><span class="n">w</span><span class="p">))</span> <span class="k">for</span> <span class="n">w</span> <span class="ow">in</span> <span class="n">words</span><span class="p">]</span>
<span class="n">d</span> <span class="o">=</span> <span class="n">Counter</span><span class="p">(</span><span class="n">normalized</span><span class="p">)</span>
<span class="k">return</span> <span class="p">[</span><span class="n">w</span> <span class="k">for</span> <span class="n">w</span><span class="p">,</span> <span class="n">n</span> <span class="ow">in</span> <span class="n">izip</span><span class="p">(</span><span class="n">words</span><span class="p">,</span> <span class="n">normalized</span><span class="p">)</span> <span class="k">if</span> <span class="n">d</span><span class="p">[</span><span class="n">n</span><span class="p">]</span> <span class="o">></span> <span class="mi">1</span><span class="p">]</span>
</pre></div>
</div>
</div>
<div class="section" id="lessons-learnt">
<h2><a class="toc-backref" href="#id17">Lessons Learnt</a></h2>
<ul class="simple">
<li>Hash Maps are very important. Learn to use them properly.</li>
<li>When making an algorithm don't think standard library, think of finding the best algorithmic solution.</li>
<li>After you've found the algorithmic solution try optimizing it using the standard library.</li>
<li>Quadratic time is worse than you can imagine.</li>
<li>Complexity analysis alone is a blunt instrument so always profile.</li>
<li>Just because an algorithm has better worst case complexity doesn't mean that
its the best one for the job.</li>
<li>Sometimes our brains stop working under pressure. Take a deep breath, stop worrying about what will happen if you
<em>don't</em> solved the problem and focus on all the good things that will happen if you <em>do</em>.</li>
</ul>
</div>
<div class="section" id="acknowledgments">
<h2><a class="toc-backref" href="#id18">Acknowledgments</a></h2>
<p><strong>Ashwin</strong> (@inspectorG4dget) for helping me answer this question and explaining the vector approach
to hashing. Ashwin's help has been paramount to my understanding.</p>
<p>To my interviewer for giving me a very good question, and to <strong>Ridwan</strong> (@hjr265)
for pointing out flaws in my solution and for encouraging me to write a blog post on this problem.</p>
<p><strong>Alexander</strong> (@metaprogrammer) for proof reading this post and providing me with the optimal solution to
this problem.<a class="footnote-reference" href="#whoisalexander" id="id4">[4]</a></p>
<hr class="docutils" />
<table class="docutils footnote" frame="void" id="howstringsarehashed" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#id1">[1]</a></td><td><a class="reference external" href="http://effbot.org/zone/python-hash.htm">Python Hash Algorithms</a> explains in detail
how the different hash functions in Python actually work.</td></tr>
</tbody>
</table>
<table class="docutils footnote" frame="void" id="biglist" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#id3">[2]</a></td><td><a class="reference external" href="http://www-01.sil.org/linguistics/wordlists/english/">wordsEn.txt</a> is the list I used.</td></tr>
</tbody>
</table>
<table class="docutils footnote" frame="void" id="moreondefaultdict" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#id2">[3]</a></td><td><a class="reference external" href="https://www.accelebrate.com/blog/using-defaultdict-python/">Using defaultdict in Python</a>
is a great writeup on how to use <code>defaultdict</code>. It explains the concept of a factory as well.</td></tr>
</tbody>
</table>
<table class="docutils footnote" frame="void" id="whoisalexander" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#id4">[4]</a></td><td>Alexander Kozlovsky is the creator of Pony ORM.</td></tr>
</tbody>
</table>
</div>
PyCharm: The Good Parts II2015-01-18T18:06:00+06:002015-01-18T18:06:00+06:00Nafiul Islamtag:nafiulis.me,2015-01-18:pycharm-the-good-parts-ii.html<div class="contents topic" id="table-of-contents">
<p class="topic-title first">Table of Contents</p>
<ul class="simple">
<li><a class="reference internal" href="#code-completion" id="id6">Code Completion</a><ul>
<li><a class="reference internal" href="#smart-completion" id="id7">Smart Completion</a><ul>
<li><a class="reference internal" href="#runtime-information" id="id8">Runtime Information</a></li>
<li><a class="reference internal" href="#docstrings" id="id9">Docstrings</a></li>
<li><a class="reference internal" href="#assert-statements" id="id10">Assert Statements</a></li>
</ul>
</li>
<li><a class="reference internal" href="#hippie-complete" id="id11">Hippie Complete</a></li>
<li><a class="reference internal" href="#spelling-correction" id="id12">Spelling Correction</a></li>
<li><a class="reference internal" href="#language-injection" id="id13">Language Injection</a></li>
</ul>
</li>
<li><a class="reference internal" href="#editor" id="id14">Editor</a><ul>
<li><a class="reference internal" href="#font" id="id15">Font</a></li>
<li><a class="reference internal" href="#braces-and-quotes" id="id16">Braces and Quotes</a></li>
<li><a class="reference internal" href="#case-sensitive-completion" id="id17">Case Sensitive Completion</a></li>
</ul>
</li>
<li><a class="reference internal" href="#up-next" id="id18">Up Next</a></li>
<li><a class="reference internal" href="#previously" id="id19">Previously</a></li>
<li><a class="reference internal" href="#update" id="id20">Update</a></li>
</ul>
</div>
<p>Before we begin, I must talk about what I mean by editing, and what I intend to cover. I believe
that the most important thing about editing is code completion. In this section I pay special attention to
python although PyCharm offers superb code completion for other languages like SQL and JavaScript.<a class="footnote-reference" href="#webtools" id="id1">[1]</a></p>
<p>I start off with code completion, because if you're like me, you have little time and want to dive into the thick of
things and code completion is what we want most out of an IDE.</p>
<p>Then, I talk about a couple of editor settings that help keep me sane.</p>
<div class="section" id="code-completion">
<h2><a class="toc-backref" href="#id6">Code Completion</a></h2>
<p>PyCharm's editor is very powerful and extensible. In this section we will go over some useful tools that I use to
write less error prone code faster.</p>
<div class="section" id="smart-completion">
<h3><a class="toc-backref" href="#id7">Smart Completion</a></h3>
<p>You get suggestions as you type. You can further enhance this by the following:</p>
<ul class="simple">
<li>Collecting runtime information</li>
<li>Adding docstrings to functions and methods</li>
<li>Invoking <code>assert</code> statements</li>
</ul>
<div class="section" id="runtime-information">
<h4><a class="toc-backref" href="#id8">Runtime Information</a></h4>
<p>Lets start with the easiest enhancement. All we need to do is check this box, and PyCharm will give you better
code completion as you debug your program more:</p>
<img alt="Runtime info" class="align-center" src="/images/charm_good_parts_07.png" />
<p>If you ever get wrong suggestions for completion clear the caches. Running your tests in debug mode will also give
PyCharm a better chance to get an understanding of the types that you're using.</p>
</div>
<div class="section" id="docstrings">
<h4><a class="toc-backref" href="#id9">Docstrings</a></h4>
<p>Writing docstrings helps PyCharm, but also helps developers gain a better understanding of the code that you have
written. PyCharm <em>understands</em> docstrings and uses the information in them to give you code completion and give
you warning signals if it thinks you're being inconsistent:</p>
<div class="highlight"><pre><span></span><span class="k">def</span> <span class="nf">foo</span><span class="p">(</span><span class="n">a</span><span class="p">,</span> <span class="n">b</span><span class="p">,</span> <span class="n">c</span><span class="p">):</span>
<span class="sd">"""</span>
<span class="sd"> :param a: A for apple</span>
<span class="sd"> :type a: str</span>
<span class="sd"> :param b: B for ball</span>
<span class="sd"> :type b: str</span>
<span class="sd"> :param c: C for cat</span>
<span class="sd"> :type c: str</span>
<span class="sd"> :return: The answer to everything is 42. Remember ye well!</span>
<span class="sd"> :rtype: int</span>
<span class="sd"> """</span>
<span class="k">return</span> <span class="mi">42</span>
<span class="k">if</span> <span class="vm">__name__</span> <span class="o">==</span> <span class="s1">'__main__'</span><span class="p">:</span>
<span class="n">foo</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">)</span>
</pre></div>
<p>In the example above, I've used PyCharm tells me that I'm being inconsistent:</p>
<img alt="PyCharm yells at me" class="align-center" src="/images/charm_good_parts_10.png" />
<p>But there's more! If you wrote docstrings, then PyCharm will give you a beautiful documentation popup about your
variables (by using <em>quick documentation</em>):<a class="footnote-reference" href="#keymap" id="id2">[2]</a></p>
<img alt="Beautiful docs" class="align-center" src="/images/charm_good_parts_11.png" />
</div>
<div class="section" id="assert-statements">
<h4><a class="toc-backref" href="#id10">Assert Statements</a></h4>
<p><code>assert</code> statements can help tell PyCharm what types you should expect. Please note that, <code>assert</code> statements
are not overhead free, unlike docstrings. I would advise using them in tests, so that the
<a class="reference internal" href="#runtime-information">debugger</a> can collect type information about your parameters. For example, the following will give code completions for
<code>a</code> but not for <code>b</code>:</p>
<div class="highlight"><pre><span></span><span class="k">def</span> <span class="nf">foo</span><span class="p">(</span><span class="n">a</span><span class="p">,</span> <span class="n">b</span><span class="p">,</span> <span class="n">c</span><span class="p">):</span>
<span class="k">assert</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">a</span><span class="p">,</span> <span class="nb">str</span><span class="p">)</span>
<span class="k">return</span> <span class="mi">42</span>
</pre></div>
<p>This is because PyCharm knows about <code>a</code>,
and that its a string, but it doesn't know about <code>b</code>:</p>
<img alt="Code completion for b" class="align-center" src="/images/charm_good_parts_12.png" />
</div>
</div>
<div class="section" id="hippie-complete">
<h3><a class="toc-backref" href="#id11">Hippie Complete</a></h3>
<p>Hippie Complete or Cyclic completion is best explained with a video:</p>
<div class="external youtube"><iframe width="640" height="360" src="//www.youtube.com/embed/jiaGFNnRL4o" frameborder="0"></iframe></div><p>Basically, it completes the word based on previous words you've typed <em>in the same document</em>.</p>
</div>
<div class="section" id="spelling-correction">
<h3><a class="toc-backref" href="#id12">Spelling Correction</a></h3>
<p>PyCharm also offers spelling correction as a quick fix. If you misspell "massachussets" like so:</p>
<img alt="Spelling" class="align-center" src="/images/charm_good_parts_16.png" />
<p>Then you can try quick fixing it with <kbd class="light">⌥</kbd> + <kbd class="light">return</kbd>. After you hit <kbd class="light">return</kbd> on the first option above, you
should see correction options:</p>
<img alt="Correction options" class="align-center" src="/images/charm_good_parts_17.png" />
<p><em>This was a request on reddit</em></p>
</div>
<div class="section" id="language-injection">
<h3><a class="toc-backref" href="#id13">Language Injection</a></h3>
<p>Language Inject basically means that you can <em>inject</em> other languages into Python strings, and have python
<em>recognize</em> those languages. This comes in really handy when you're trying to syntax highlighting for regex or
html, but it goes further than that often providing code completion. In order to get language injection, one must
invoke the <em>quick fix</em> command inside string delimiters (<code>''</code>).<a class="footnote-reference" href="#keymap" id="id3">[2]</a></p>
<p>Here is a video to demonstrate how language injection works:<a class="footnote-reference" href="#emmet" id="id4">[3]</a></p>
<div class="external youtube"><iframe width="640" height="360" src="//www.youtube.com/embed/ZmBe0CCQZk0" frameborder="0"></iframe></div></div>
</div>
<div class="section" id="editor">
<h2><a class="toc-backref" href="#id14">Editor</a></h2>
<p>This part covers the settings regarding the editor for better workflow. The options I set are based on my own preferences
and hence not scientifically tested but they <em>do</em> make <em>my</em> editing a lot faster. Take the following advice with a
healthy dose of skepticism since what might work for me, might not work for you.</p>
<div class="section" id="font">
<h3><a class="toc-backref" href="#id15">Font</a></h3>
<img alt="Consolas FTW!" class="align-center" src="/images/charm_good_parts_13.png" />
<p>Consolas is the only font that looks good on Windows, Mac and Linux. Consolas remains monospace regardless of
whether it is bold or italic, and the only font that looks consistently good in PyCharm. I have tested this on Mac OSX
Yosemite, Windows 8.1 and Ubuntu (14.04).<a class="footnote-reference" href="#gettingconsonlinux" id="id5">[4]</a></p>
<p>If you've had success with other fonts (especially in Linux), please do mention them in the comments.</p>
</div>
<div class="section" id="braces-and-quotes">
<h3><a class="toc-backref" href="#id16">Braces and Quotes</a></h3>
<p>A lot of the time, I forget to put parentheses around things or quotes around strings. By enabling the following
option, you can select text, and when you press <kbd class="light">"</kbd> or <kbd class="light">'</kbd> or <kbd class="light">(</kbd> or <kbd class="light">)</kbd>, instead of replacing the text with
the character entered, it will surround the selection with quotes or parentheses:</p>
<img alt="Parens!" class="align-center" src="/images/charm_good_parts_14.png" />
</div>
<div class="section" id="case-sensitive-completion">
<h3><a class="toc-backref" href="#id17">Case Sensitive Completion</a></h3>
<p>By default, PyCharm comes with <em>first letter</em> case sensitivity, meaning that if you were to type <code>e</code>,
<code>Exception</code> would not be offered as a viable completion. This is why I disable case sensitivity:</p>
<img alt="Case Sensitivity" class="align-center" src="/images/charm_good_parts_15.png" />
<p>Please note however, that on slower machines this does make code completion slower.</p>
</div>
</div>
<div class="section" id="up-next">
<h2><a class="toc-backref" href="#id18">Up Next</a></h2>
<p>I will likely be covering how to deal with interpreters.</p>
</div>
<div class="section" id="previously">
<h2><a class="toc-backref" href="#id19">Previously</a></h2>
<p>I talked about a couple of things in <a class="reference external" href="http://nafiulis.me/pycharm-the-good-parts-i.html">part one</a>.</p>
</div>
<div class="section" id="update">
<h2><a class="toc-backref" href="#id20">Update</a></h2>
<p><em>2015-01-19 11:41</em> : Added a section on <a class="reference internal" href="#spelling-correction">Spelling Correction</a></p>
<hr class="docutils" />
<table class="docutils footnote" frame="void" id="webtools" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#id1">[1]</a></td><td>I will cover javascript, html and other stuff in another section that deals with web tools. Don't worry I know that they are indispensable tools, and will make sure to give them the time they deserve.</td></tr>
</tbody>
</table>
<table class="docutils footnote" frame="void" id="keymap" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label">[2]</td><td><em>(<a class="fn-backref" href="#id2">1</a>, <a class="fn-backref" href="#id3">2</a>)</em> Use the <a class="reference external" href="http://nafiulis.me/pycharm-the-good-parts-i.html#keymap">keymap</a> to find your corresponding keyboard shortcut discussed in the previous part.</td></tr>
</tbody>
</table>
<table class="docutils footnote" frame="void" id="emmet" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#id4">[3]</a></td><td>Don't worry if you don't know what emmet does, I'll cover that when I cover web tools.</td></tr>
</tbody>
</table>
<table class="docutils footnote" frame="void" id="gettingconsonlinux" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#id5">[4]</a></td><td><div class="first last line-block">
<div class="line">Take a look at this answer if you want to use consolas on Ubuntu:</div>
<div class="line"><a class="reference external" href="http://askubuntu.com/questions/269757/consolas-not-visible-in-intellij-idea-on-ubuntu-12-04">http://askubuntu.com/questions/269757/consolas-not-visible-in-intellij-idea-on-ubuntu-12-04</a></div>
</div>
</td></tr>
</tbody>
</table>
</div>
PyCharm: The Good Parts I2015-01-17T12:09:00+06:002015-01-17T12:09:00+06:00Nafiul Islamtag:nafiulis.me,2015-01-17:pycharm-the-good-parts-i.html<div class="contents topic" id="table-of-contents">
<p class="topic-title first">Table of Contents</p>
<ul class="simple">
<li><a class="reference internal" href="#productivity-guide" id="id4">Productivity Guide</a></li>
<li><a class="reference internal" href="#starting-out" id="id5">Starting Out</a></li>
<li><a class="reference internal" href="#keyboard-shortcuts" id="id6">Keyboard Shortcuts</a><ul>
<li><a class="reference internal" href="#keymap" id="id7">Keymap</a></li>
<li><a class="reference internal" href="#find-action" id="id8">Find Action</a></li>
<li><a class="reference internal" href="#useful-shortcuts" id="id9">Useful Shortcuts</a></li>
</ul>
</li>
<li><a class="reference internal" href="#navigation" id="id10">Navigation</a><ul>
<li><a class="reference internal" href="#macro" id="id11">Macro</a><ul>
<li><a class="reference internal" href="#back-and-forward" id="id12">Back and Forward</a></li>
<li><a class="reference internal" href="#search-everywhere" id="id13">Search Everywhere</a></li>
<li><a class="reference internal" href="#switcher" id="id14">Switcher</a></li>
</ul>
</li>
<li><a class="reference internal" href="#micro" id="id15">Micro</a><ul>
<li><a class="reference internal" href="#structure" id="id16">Structure</a></li>
<li><a class="reference internal" href="#acejump" id="id17">AceJump</a></li>
</ul>
</li>
</ul>
</li>
<li><a class="reference internal" href="#up-next" id="id18">Up Next</a></li>
<li><a class="reference internal" href="#update" id="id19">Update</a></li>
</ul>
</div>
<p><strong>tldr:</strong> If you don't have the time to read the entire post, just read <a class="reference internal" href="#productivity-guide">Productivity Guide</a>, because I assume
that you want better productivity as your primary goal.</p>
<p>PyCharm is a very powerful IDE, and has a multitude of features that keep growing at a large pace. PyCharm is
essentially, a plugin built on top of the IntelliJ platform, Jetbrains' main IDE. Because PyCharm borrows
from a whole host of features available from IntelliJ, its easy to get lost. <a class="footnote-reference" href="#jb" id="id1">[1]</a></p>
<p>This post shows you some of the best features that PyCharm has to offer, and how to use those features properly.
I assume that you've installed plugins and know your way around the IDE. The table of contents should give you an
idea of what is covered in this post.</p>
<div class="section" id="productivity-guide">
<h2><a class="toc-backref" href="#id4">Productivity Guide</a></h2>
<p>You might not know this, but PyCharm tracks what features you use, and tells you how productive you are being in your
productivity guide:</p>
<img alt="How to get to the productivity guide" class="align-center" src="/images/charm_good_parts_01.png" />
<p>The productivity guide gives you detailed descriptions of what features can help you code faster. You can see below
that although I don't use all the features, I do use some quite a lot:</p>
<img alt="What features the Productivity guide I use" class="align-center" src="/images/charm_good_parts_02.png" />
</div>
<div class="section" id="starting-out">
<h2><a class="toc-backref" href="#id5">Starting Out</a></h2>
<p>PyCharm touts its project creation wizard a lot, and for Windows users is a godsend because you can easily create a
project and associate a virtualenv with it. However for *nix users, the best option is to start PyCharm from the
command line, and you can do this by creating a command line launcher:</p>
<img alt="Command Line Launcher" class="align-center" src="/images/charm_good_parts_08.png" />
<p>There is one thing that you need to know about starting from the command line is that PyCharm uses the interpreter that
the <code>python</code> command is linked to. So for example, if your <code>python</code> command in the shell links to
<code>/usr/bin/python</code>, then that interpreter will become the interpreter for your new project. However, if you want
to have the interpreter as a virtualenv, you have to create the virtualenv, activate it and then use the command line
launcher to initialize your project. If we do the following:</p>
<div class="highlight"><pre><span></span>$ mkdir SimpleProjectDir
$ <span class="nb">cd</span> SimpleProjectDir
$ charm .
</pre></div>
<p>It would create a PyCharm project in <code>SimpleProjectDir</code> and use the default interpreter in the shell like so:</p>
<img alt="Bad Interpreter" class="align-center" src="/images/charm_good_parts_09.png" />
<p>We can see that it is the same <code>python</code> that was linked to in our shell:</p>
<div class="highlight"><pre><span></span>$ which python
/usr/local/bin/python
</pre></div>
<p>However, if we were to create a virtualenv beforehand and then create a project using the command line launcher,
we would get the virtualenv as the interpreter instead:</p>
<div class="highlight"><pre><span></span>$ mkdir SimpleProjectDir
$ <span class="nb">cd</span> SimpleProjectDir
$ virtualenv .venv
New python executable in .venv/bin/python2.7
Also creating executable in .venv/bin/python
Installing setuptools, pip...done.
$ <span class="nb">source</span> .venv/bin/activate
<span class="o">(</span>.venv<span class="o">)</span>$ charm .
</pre></div>
<p>In simple terms, whatever <code>python</code> points to in the shell at the time of using the command line launcher,
PyCharm will adopt that interpreter as the project interpreter.</p>
</div>
<div class="section" id="keyboard-shortcuts">
<h2><a class="toc-backref" href="#id6">Keyboard Shortcuts</a></h2>
<div class="section" id="keymap">
<h3><a class="toc-backref" href="#id7">Keymap</a></h3>
<p>PyCharm allows you to see all your keyboard shortcuts using the <em>Keymap</em>. <em>Keymap</em> has two key features which aid
in development. One being the ability to find keyboard shortcuts based on what they're called:</p>
<img alt="Finding the Keymap based on what you are searching for" class="align-center" src="/images/charm_good_parts_04.png" />
<p>But it also allows you to find shortcuts based on the keys themselves, so that you can talk other people about them:</p>
<div class="external youtube"><iframe width="640" height="360" src="//www.youtube.com/embed/1MLGZyzOO6U" frameborder="0"></iframe></div><p>So, whenever I talk about an action like "Find Action" or "Search Anywhere", just search it up on your keymap to see
the corresponding keyboard shortcut.</p>
</div>
<div class="section" id="find-action">
<h3><a class="toc-backref" href="#id8">Find Action</a></h3>
<p>The best way to learn Keyboard shortcuts is to use <em>Find Action</em>. <a class="footnote-reference" href="#findaction" id="id2">[2]</a>
Every time you use an action using find action, the keyboard shortcut
is also denoted on the right:</p>
<img alt="Shortcuts in Find Action" class="align-center" src="/images/charm_good_parts_03.png" />
<p>Whatever you find yourself using over and over again through <em>Find Action</em>, just commit it to memory and you'll
be saving yourself some time.</p>
</div>
<div class="section" id="useful-shortcuts">
<h3><a class="toc-backref" href="#id9">Useful Shortcuts</a></h3>
<p>Use the Keymap to find what shortcuts these commands refer to: <a class="footnote-reference" href="#cantfind" id="id3">[3]</a></p>
<ul class="simple">
<li><strong>Find Action</strong></li>
<li><strong>Search Everywhere</strong></li>
<li>Find</li>
<li>Structure</li>
<li>Add Selection for Next Occurrence</li>
<li>Switcher</li>
<li>Check In Project</li>
<li>Syntax Aware Selection</li>
<li>New</li>
</ul>
<p>The first two features are must use shortcuts, and will aid you in learning more shortcuts as you see fit. The others
are shortcuts I use regularly, and find helpful.</p>
</div>
</div>
<div class="section" id="navigation">
<h2><a class="toc-backref" href="#id10">Navigation</a></h2>
<p>This section covers how to get from place to place. I've divided it into two sections. One being <a class="reference internal" href="#macro">Macro</a> and the
other being <a class="reference internal" href="#micro">Micro</a>. Macro is about moving between files, panels and tools. Micro is about moving from place to place
<em>within</em> a file.</p>
<div class="section" id="macro">
<h3><a class="toc-backref" href="#id11">Macro</a></h3>
<div class="section" id="back-and-forward">
<h4><a class="toc-backref" href="#id12">Back and Forward</a></h4>
<p>This is one of my most favorite features. <em>Back</em> and <em>Forward</em> allows you to go back to the place where your cursor was
previously. This is immensely powerful, and allows you to jump between places within files and places
between files. Let me demonstrate:</p>
<div class="external youtube"><iframe width="640" height="360" src="//www.youtube.com/embed/xGm0IBRDXeg" frameborder="0"></iframe></div></div>
<div class="section" id="search-everywhere">
<h4><a class="toc-backref" href="#id13">Search Everywhere</a></h4>
<p>We need to find stuff all the time like functions, files, classes and methods. The best way to search for
classes, files, and directories is to use <em>Search Everywhere</em> (press <kbd class="light">SHIFT</kbd> in quick succession):</p>
<img alt="Search Everywhere" class="align-center" src="/images/charm_good_parts_06.png" />
</div>
<div class="section" id="switcher">
<h4><a class="toc-backref" href="#id14">Switcher</a></h4>
<p>The other way is to use the <em>Switcher</em>, which gives you quick access to your most recently used panels and files:</p>
<img alt="The Switcher" class="align-center" src="/images/charm_good_parts_05.png" />
</div>
</div>
<div class="section" id="micro">
<h3><a class="toc-backref" href="#id15">Micro</a></h3>
<div class="section" id="structure">
<h4><a class="toc-backref" href="#id16">Structure</a></h4>
<p>In order to find things within files, its best to use <em>Find</em>, and for a more structural view, use <em>Structure</em>.
The <em>Structure</em> panel gives you a skeletal view of your file, with variables, classes, functions and methods.
It allows you to quickly navigate between important structures within your file:</p>
<div class="external youtube"><iframe width="640" height="360" src="//www.youtube.com/embed/hCbCiqAaMA4" frameborder="0"></iframe></div></div>
<div class="section" id="acejump">
<h4><a class="toc-backref" href="#id17">AceJump</a></h4>
<p>AceJump is a plugin that allows you to quickly jump from place to place. Its a clone of Emacs' AceJump mode.
There are two superb videos by the creator of AceJump, John Lindquist on how to use the plugins.</p>
<div class="external youtube"><iframe width="640" height="360" src="//www.youtube.com/embed/yK8eM50DsAY" frameborder="0"></iframe></div><div class="external youtube"><iframe width="640" height="360" src="//www.youtube.com/embed/i_XGEaceOLY" frameborder="0"></iframe></div><p>Installing a plugin is simple:</p>
<div class="external youtube"><iframe width="640" height="360" src="//www.youtube.com/embed/DvUdxZVrJSc" frameborder="0"></iframe></div></div>
</div>
</div>
<div class="section" id="up-next">
<h2><a class="toc-backref" href="#id18">Up Next</a></h2>
<p>In the next part, I will cover editing in PyCharm, with special respect to taking control of its powerful code
completion feature-set.</p>
</div>
<div class="section" id="update">
<h2><a class="toc-backref" href="#id19">Update</a></h2>
<p>On 2015-01-18 20:40</p>
<p><a class="reference external" href="http://nafiulis.me/pycharm-the-good-parts-ii.html">Part II</a> is out now!</p>
<hr class="docutils" />
<table class="docutils footnote" frame="void" id="jb" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#id1">[1]</a></td><td>The creator of PyCharm, IntelliJ, RubyMine and other IDEs.</td></tr>
</tbody>
</table>
<table class="docutils footnote" frame="void" id="findaction" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#id2">[2]</a></td><td><div class="first last line-block">
<div class="line"><kbd class="light">⌘</kbd> + <kbd class="light">SHIFT</kbd> + <kbd class="light">A</kbd> on Mac</div>
<div class="line"><kbd class="light">CTRL</kbd> + <kbd class="light">SHIFT</kbd> + <kbd class="light">A</kbd> on Windows and Linux</div>
</div>
</td></tr>
</tbody>
</table>
<table class="docutils footnote" frame="void" id="cantfind" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#id3">[3]</a></td><td>If you can't find any of these, then you're using an older version of PyCharm.</td></tr>
</tbody>
</table>
<hr class="docutils" />
<p><strong>Pingback</strong></p>
<ul class="simple">
<li><a class="reference external" href="https://news.ycombinator.com/item?id=8903642">Hacker News</a></li>
<li><a class="reference external" href="http://redd.it/2spolb">Reddit</a></li>
<li><a class="reference external" href="https://plus.google.com/110373111919831686916/posts/AMTf34fAzTs">Google+</a></li>
</ul>
</div>
Making a Static Blog with Pelican2015-01-06T14:51:00+06:002015-01-06T14:51:00+06:00Nafiul Islamtag:nafiulis.me,2015-01-06:making-a-static-blog-with-pelican.html<div class="contents topic" id="table-of-contents">
<p class="topic-title first">Table of Contents</p>
<ul class="simple">
<li><a class="reference internal" href="#getting-started" id="id15">Getting Started</a><ul>
<li><a class="reference internal" href="#installing-pelican" id="id16">Installing Pelican</a></li>
<li><a class="reference internal" href="#basic-setup" id="id17">Basic Setup</a></li>
<li><a class="reference internal" href="#your-first-post" id="id18">Your First Post</a></li>
</ul>
</li>
<li><a class="reference internal" href="#automation" id="id19">Automation</a><ul>
<li><a class="reference internal" href="#vanilla-python" id="id20">Vanilla Python</a></li>
<li><a class="reference internal" href="#fabric" id="id21">Fabric</a></li>
<li><a class="reference internal" href="#fabric-alternatives" id="id22">Fabric alternatives</a></li>
</ul>
</li>
<li><a class="reference internal" href="#themes" id="id23">Themes</a><ul>
<li><a class="reference internal" href="#using-a-theme" id="id24">Using a Theme</a></li>
<li><a class="reference internal" href="#creating-themes" id="id25">Creating Themes</a></li>
</ul>
</li>
<li><a class="reference internal" href="#settings" id="id26">Settings</a><ul>
<li><a class="reference internal" href="#the-two-settings-files" id="id27">The Two Settings Files</a></li>
<li><a class="reference internal" href="#how-settings-work" id="id28">How Settings Work</a></li>
</ul>
</li>
<li><a class="reference internal" href="#writing-content" id="id29">Writing Content</a></li>
<li><a class="reference internal" href="#previewing-content" id="id30">Previewing Content</a><ul>
<li><a class="reference internal" href="#devserver" id="id31">devserver</a></li>
<li><a class="reference internal" href="#livereload" id="id32">livereload</a></li>
</ul>
</li>
<li><a class="reference internal" href="#plugins" id="id33">Plugins</a><ul>
<li><a class="reference internal" href="#using-plugins" id="id34">Using Plugins</a></li>
<li><a class="reference internal" href="#creating-plugins" id="id35">Creating Plugins</a></li>
</ul>
</li>
<li><a class="reference internal" href="#hosting-on-github-pages" id="id36">Hosting on Github Pages</a><ul>
<li><a class="reference internal" href="#creating-the-repo" id="id37">Creating the repo</a></li>
<li><a class="reference internal" href="#ghp-import" id="id38">ghp-import</a></li>
<li><a class="reference internal" href="#custom-domains" id="id39">Custom domains</a></li>
</ul>
</li>
<li><a class="reference internal" href="#automating-publication" id="id40">Automating Publication</a></li>
<li><a class="reference internal" href="#acknowledgements" id="id41">Acknowledgements</a></li>
</ul>
</div>
<p>Pelican allows you to create a static blog. Most blog sites on the web are <em>dynamic</em> in the sense that the content of
the site live in a database. In order to view a post on a blog, the server has to <em>query</em> the database, get the
right content and then <em>convert</em> it into presentable HTML. However, in a static site, every page is pre-rendered
by the static blog generator. This means that your <em>entire</em> blog can be uploaded to a server.</p>
<p>Here's a non-exhaustive list of why a static blog generator is good:</p>
<ul class="simple">
<li><em>Cheap Scaling</em>: Pre-rendered static files can stand the onslaught of traffic if your
post makes it to the top of hackernews or reddit very cheaply. No queries are being made.</li>
<li><em>Any format</em>: You can use reStructuredText, Markdown etc. to write your posts.</li>
<li><em>Host Anywhere</em>: Static sites can be hosted easily on github pages, Amazon S3 and even dropbox.</li>
<li><em>Own Everything</em>: You have access to all your posts and all your themes. No company in the middle, just you and
your content.</li>
<li><em>Control</em>: Static site generators give you a lot of control over pretty much every aspect of your
site through templates and plugins, allowing you to quickly add complex functionality not found in popular
web blogging platforms.</li>
<li><em>Update Issues</em>: If you are using wordpress, you <em>have</em> to update your software when a new version comes out; otherwise,
you have a security risk. However, one can use the same version of a static blog generator indefinitely.</li>
</ul>
<p>Here's why pelican is a good choice in contrast with other generators: <a class="footnote-reference" href="#assumingpy" id="id1">[1]</a></p>
<ul class="simple">
<li><em>Plugins</em>: Pelican has a lot of plugins, that allow you to quickly add functionality.</li>
<li><em>Hackable</em>: Extending pelican as well as altering its behaviour is simple.</li>
<li><em>Themes</em>: Pelican has <em>a lot</em> of themes, and you can make your own.</li>
<li><em>Multi-platform</em>: Works well on Windows, OSX and Linux. <a class="footnote-reference" href="#windows" id="id2">[2]</a></li>
</ul>
<div class="line-block">
<div class="line"><br /></div>
<div class="line"><br /></div>
</div>
<p>I've been using Pelican as my static blog generator for the past year. Over this time,
I have come to recognize its strengths and weaknesses. Pelican allows you to quickly get a themed site up and
running in a matter of minutes. You can also choose to invest more time and create your own theme. In short, if you want
a low hassle setup, its there and if you want a more custom site you can have that too.</p>
<p>However, Pelican's initial setup barely scratches the surface of what's possible with this small yet powerful blog
generator. It has many plugins too for quickly adding extra functionality, like the support of other
document formats; <code>ipynb</code> and <code>asciidoc</code> for example. Plugins also allow you to support things
like MathJax in your pages and embed disqus comments. <a class="footnote-reference" href="#id13" id="id3">[3]</a></p>
<div class="section" id="getting-started">
<h2><a class="toc-backref" href="#id15">Getting Started</a></h2>
<p>Before diving in, I expect you to know a few basic things:</p>
<ul class="simple">
<li>How to use git</li>
<li>How to use bash or your shell of choice</li>
<li>How to use Python</li>
</ul>
<div class="section" id="installing-pelican">
<h3><a class="toc-backref" href="#id16">Installing Pelican</a></h3>
<p>Installation is very simple:</p>
<div class="highlight"><pre><span></span>$ pip install pelican
</pre></div>
<p>Feel free to use virtualenv if you want to but its not required.</p>
<p>Pelican works with both Python 2 and Python 3. However, I chose to use Python 2 because I use an automation tool called
Fabric, which only supports python 2 (for the time being).</p>
</div>
<div class="section" id="basic-setup">
<h3><a class="toc-backref" href="#id17">Basic Setup</a></h3>
<p>When first beginning with pelican, one can choose to use the pelican skeleton generator to create a basic structure
using <code>pelican-quickstart</code>. <a class="footnote-reference" href="#problemsquickstart" id="id4">[4]</a></p>
<div class="highlight"><pre><span></span>$ pelican-quickstart
Welcome to pelican-quickstart v3.5.0.
This script will <span class="nb">help</span> you create a new Pelican-based website.
Please answer the following questions so this script can generate the files
needed by Pelican.
> Where <span class="k">do</span> you want to create your new web site? <span class="o">[</span>.<span class="o">]</span>
> What will be the title of this web site? Test Site
> Who will be the author of this web site? Kevin Kevinson
> What will be the default language of this web site? <span class="o">[</span>en<span class="o">]</span>
> Do you want to specify a URL prefix? e.g., http://example.com <span class="o">(</span>Y/n<span class="o">)</span> n
> Do you want to <span class="nb">enable</span> article pagination? <span class="o">(</span>Y/n<span class="o">)</span>
> How many articles per page <span class="k">do</span> you want? <span class="o">[</span><span class="m">10</span><span class="o">]</span> <span class="m">8</span>
> Do you want to generate a Fabfile/Makefile to automate generation and publishing? <span class="o">(</span>Y/n<span class="o">)</span>
> Do you want an auto-reload <span class="p">&</span> simpleHTTP script to assist with theme and site development? <span class="o">(</span>Y/n<span class="o">)</span>
> Do you want to upload your website using FTP? <span class="o">(</span>y/N<span class="o">)</span>
> Do you want to upload your website using SSH? <span class="o">(</span>y/N<span class="o">)</span>
> Do you want to upload your website using Dropbox? <span class="o">(</span>y/N<span class="o">)</span>
> Do you want to upload your website using S3? <span class="o">(</span>y/N<span class="o">)</span>
> Do you want to upload your website using Rackspace Cloud Files? <span class="o">(</span>y/N<span class="o">)</span>
> Do you want to upload your website using GitHub Pages? <span class="o">(</span>y/N<span class="o">)</span>
Done. Your new project is available at /Users/quazinafiulislam/Desktop/testingzone
</pre></div>
<p>There's no need to change much in the default quickstart wizard. Default values are indicated by <code>[default]</code>
and <code>(y/N)</code>, where the capitalized letter is the default value. Just one thing to note here is URL prefix; don't
set it right now, set it when you actually want to take your site online, and we will go over how to do that later
in this tutorial.</p>
</div>
<div class="section" id="your-first-post">
<h3><a class="toc-backref" href="#id18">Your First Post</a></h3>
<p>After <code>pelican-quickstart</code> is done, this is what the directory should look like:</p>
<div class="highlight"><pre><span></span>.
├── Makefile
├── content/
├── develop_server.sh
├── fabfile.py
├── output/
├── pelicanconf.py
└── publishconf.py
</pre></div>
<p>Pelican supports reStructuredText out of the box. If you want to use another format like markdown then just install it:</p>
<div class="highlight"><pre><span></span>$ pip install markdown
</pre></div>
<p>More uncommon formats will be discussed later. With this out of the way, we can create our first post inside the
content folder. All our writing has to be inside the <code>content</code> folder, so we change our directory to the
<code>content</code> folder:</p>
<div class="highlight"><pre><span></span>$ <span class="nb">cd</span> content
</pre></div>
<p>My file is called <code>first_post.rst</code> and it looks like this:</p>
<div class="highlight"><pre><span></span><span class="gh">first_post</span>
<span class="gh">##########</span>
<span class="nc">:date:</span> <span class="nf">2014-12-13 18:32</span>
<span class="nc">:category:</span> <span class="nf">Test</span>
Hello World from Pelican!
</pre></div>
<p>Basically, all one needs is a category and a date. A markdown version of the same post would be:</p>
<div class="highlight"><pre><span></span>Title: first_post
Date: 2014-12-13 18:32
Category: Test
Hello world from Pelican!
</pre></div>
<p>This is what my project directory looks like now:</p>
<div class="highlight"><pre><span></span>.
├── Makefile
├── content
│ └── first_post.rst
├── develop_server.sh
├── fabfile.py
├── pelicanconf.py
├── pelicanconf.pyc
└── publishconf.py
</pre></div>
<p>Now changing back to our project directory, we can start the development server, and see the generated blog post:</p>
<div class="highlight"><pre><span></span>$ make devserver
</pre></div>
<p>Once this is done, we should be able to see our post on <code>localhost:8000</code>:</p>
<img alt="Our First post with pelican" class="align-center" src="/images/pelican_05.png" />
<p>We can stop the server through:</p>
<div class="highlight"><pre><span></span>$ make stopserver
</pre></div>
</div>
</div>
<div class="section" id="automation">
<h2><a class="toc-backref" href="#id19">Automation</a></h2>
<p>Although only a title, date and category are required for a post, pelican allows you to add a lot more information like
tags, authors and much more. However, creating a new file with the right date and time can be time consuming,
so we can use certain tools to automate the task for us.</p>
<div class="section" id="vanilla-python">
<h3><a class="toc-backref" href="#id20">Vanilla Python</a></h3>
<p>We can create simple scripts to automatically create our
entries with the right date and time. Here is an example script that I use called <code>make_entry.py</code>:</p>
<div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">sys</span>
<span class="kn">from</span> <span class="nn">datetime</span> <span class="kn">import</span> <span class="n">datetime</span>
<span class="n">TEMPLATE</span> <span class="o">=</span> <span class="s2">"""</span>
<span class="s2">{title}</span>
<span class="s2">{hashes}</span>
<span class="s2">:date: {year}-{month}-{day} {hour}:{minute:02d}</span>
<span class="s2">:tags:</span>
<span class="s2">:category:</span>
<span class="s2">:slug: {slug}</span>
<span class="s2">:summary:</span>
<span class="s2">:status: draft</span>
<span class="s2">"""</span>
<span class="k">def</span> <span class="nf">make_entry</span><span class="p">(</span><span class="n">title</span><span class="p">):</span>
<span class="n">today</span> <span class="o">=</span> <span class="n">datetime</span><span class="o">.</span><span class="n">today</span><span class="p">()</span>
<span class="n">slug</span> <span class="o">=</span> <span class="n">title</span><span class="o">.</span><span class="n">lower</span><span class="p">()</span><span class="o">.</span><span class="n">strip</span><span class="p">()</span><span class="o">.</span><span class="n">replace</span><span class="p">(</span><span class="s1">' '</span><span class="p">,</span> <span class="s1">'-'</span><span class="p">)</span>
<span class="n">f_create</span> <span class="o">=</span> <span class="s2">"content/{}_{:0>2}_{:0>2}_{}.rst"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span>
<span class="n">today</span><span class="o">.</span><span class="n">year</span><span class="p">,</span> <span class="n">today</span><span class="o">.</span><span class="n">month</span><span class="p">,</span> <span class="n">today</span><span class="o">.</span><span class="n">day</span><span class="p">,</span> <span class="n">slug</span><span class="p">)</span>
<span class="n">t</span> <span class="o">=</span> <span class="n">TEMPLATE</span><span class="o">.</span><span class="n">strip</span><span class="p">()</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">title</span><span class="o">=</span><span class="n">title</span><span class="p">,</span>
<span class="n">hashes</span><span class="o">=</span><span class="s1">'#'</span> <span class="o">*</span> <span class="nb">len</span><span class="p">(</span><span class="n">title</span><span class="p">),</span>
<span class="n">year</span><span class="o">=</span><span class="n">today</span><span class="o">.</span><span class="n">year</span><span class="p">,</span>
<span class="n">month</span><span class="o">=</span><span class="n">today</span><span class="o">.</span><span class="n">month</span><span class="p">,</span>
<span class="n">day</span><span class="o">=</span><span class="n">today</span><span class="o">.</span><span class="n">day</span><span class="p">,</span>
<span class="n">hour</span><span class="o">=</span><span class="n">today</span><span class="o">.</span><span class="n">hour</span><span class="p">,</span>
<span class="n">minute</span><span class="o">=</span><span class="n">today</span><span class="o">.</span><span class="n">minute</span><span class="p">,</span>
<span class="n">slug</span><span class="o">=</span><span class="n">slug</span><span class="p">)</span>
<span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="n">f_create</span><span class="p">,</span> <span class="s1">'w'</span><span class="p">)</span> <span class="k">as</span> <span class="n">w</span><span class="p">:</span>
<span class="n">w</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="n">t</span><span class="p">)</span>
<span class="k">print</span><span class="p">(</span><span class="s2">"File created -> "</span> <span class="o">+</span> <span class="n">f_create</span><span class="p">)</span>
<span class="k">if</span> <span class="vm">__name__</span> <span class="o">==</span> <span class="s1">'__main__'</span><span class="p">:</span>
<span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">sys</span><span class="o">.</span><span class="n">argv</span><span class="p">)</span> <span class="o">></span> <span class="mi">1</span><span class="p">:</span>
<span class="n">make_entry</span><span class="p">(</span><span class="n">sys</span><span class="o">.</span><span class="n">argv</span><span class="p">[</span><span class="mi">1</span><span class="p">])</span>
<span class="k">else</span><span class="p">:</span>
<span class="k">print</span> <span class="s2">"No title given"</span>
</pre></div>
<p>With this, making a new file can be created like so:</p>
<div class="highlight"><pre><span></span> $ python make_entry.py <span class="s2">"New Post"</span>
File created -> content/2014_12_13_new-post.rst
</pre></div>
</div>
<div class="section" id="fabric">
<h3><a class="toc-backref" href="#id21">Fabric</a></h3>
<p><em>Jump to the next section if you're not using Python 2</em> <a class="footnote-reference" href="#fabpy3support" id="id5">[5]</a></p>
<p>Using Vanilla Python is great, but imagine trying to make a script for <em>every single</em> task you need done. This would
be a nightmare. Of course, one can keep using if-else statements to add more argument parameters but that is tiring
and not very flexible (not to mention error prone). This is why one option to making scripts that can take in
different parameters is to use a handy library called <a class="reference external" href="http://www.fabfile.org/">Fabric</a>.</p>
<p>I use fabric for most of my tooling, since pelican already generates a <code>fabfile.py</code>. Getting a new
command line function in fabric is very easy:</p>
<ul class="simple">
<li>If you don't have a <code>fabfile.py</code>, create one.</li>
<li>Create a new function inside <code>fabfile.py</code></li>
<li>Run the function like so: <code>fab do_something</code></li>
<li>If the function has positional arguments, <code>fab do_something:True, False</code></li>
<li>If the function has optional keyword arguments <code>fab do_something:kill_p=True, be_happy=False</code></li>
</ul>
<p>Installing Fabric is easy:</p>
<div class="highlight"><pre><span></span>$ pip install Fabric
</pre></div>
<p>The body of the function to make an entry is the same as before:</p>
<div class="highlight"><pre><span></span><span class="c1"># TEMPLATE is declared before hand, and all the necessary imports made</span>
<span class="k">def</span> <span class="nf">make_entry</span><span class="p">(</span><span class="n">title</span><span class="p">):</span>
<span class="n">today</span> <span class="o">=</span> <span class="n">datetime</span><span class="o">.</span><span class="n">today</span><span class="p">()</span>
<span class="n">slug</span> <span class="o">=</span> <span class="n">title</span><span class="o">.</span><span class="n">lower</span><span class="p">()</span><span class="o">.</span><span class="n">strip</span><span class="p">()</span><span class="o">.</span><span class="n">replace</span><span class="p">(</span><span class="s1">' '</span><span class="p">,</span> <span class="s1">'-'</span><span class="p">)</span>
<span class="n">f_create</span> <span class="o">=</span> <span class="s2">"content/{}_{:0>2}_{:0>2}_{}.rst"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span>
<span class="n">today</span><span class="o">.</span><span class="n">year</span><span class="p">,</span> <span class="n">today</span><span class="o">.</span><span class="n">month</span><span class="p">,</span> <span class="n">today</span><span class="o">.</span><span class="n">day</span><span class="p">,</span> <span class="n">slug</span><span class="p">)</span>
<span class="n">t</span> <span class="o">=</span> <span class="n">TEMPLATE</span><span class="o">.</span><span class="n">strip</span><span class="p">()</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">title</span><span class="o">=</span><span class="n">title</span><span class="p">,</span>
<span class="n">hashes</span><span class="o">=</span><span class="s1">'#'</span> <span class="o">*</span> <span class="nb">len</span><span class="p">(</span><span class="n">title</span><span class="p">),</span>
<span class="n">year</span><span class="o">=</span><span class="n">today</span><span class="o">.</span><span class="n">year</span><span class="p">,</span>
<span class="n">month</span><span class="o">=</span><span class="n">today</span><span class="o">.</span><span class="n">month</span><span class="p">,</span>
<span class="n">day</span><span class="o">=</span><span class="n">today</span><span class="o">.</span><span class="n">day</span><span class="p">,</span>
<span class="n">hour</span><span class="o">=</span><span class="n">today</span><span class="o">.</span><span class="n">hour</span><span class="p">,</span>
<span class="n">minute</span><span class="o">=</span><span class="n">today</span><span class="o">.</span><span class="n">minute</span><span class="p">,</span>
<span class="n">slug</span><span class="o">=</span><span class="n">slug</span><span class="p">)</span>
<span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="n">f_create</span><span class="p">,</span> <span class="s1">'w'</span><span class="p">)</span> <span class="k">as</span> <span class="n">w</span><span class="p">:</span>
<span class="n">w</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="n">t</span><span class="p">)</span>
<span class="k">print</span><span class="p">(</span><span class="s2">"File created -> "</span> <span class="o">+</span> <span class="n">f_create</span><span class="p">)</span>
</pre></div>
<p>However, this time we can call it like so:</p>
<div class="highlight"><pre><span></span>$ fab make_entry:<span class="s2">"New Post"</span>
</pre></div>
</div>
<div class="section" id="fabric-alternatives">
<h3><a class="toc-backref" href="#id22">Fabric alternatives</a></h3>
<p>There are tonnes of alternatives that allow you to make a simple command line interface quickly:</p>
<ul class="simple">
<li><a class="reference external" href="http://click.pocoo.org/3/">Click</a> : A simple to use CLI by Armin Ronacher.</li>
<li><a class="reference external" href="https://github.com/willyg302/clip.py">Clip</a> : Like click, but newer, and has more features</li>
<li><a class="reference external" href="https://docs.python.org/3/library/argparse.html">Argparse</a> : This package comes with python by default. Its not
the easiest to use package out there, but if you don't want to install a new package, then this should work fine.</li>
</ul>
<p>All three work with python 3.</p>
</div>
</div>
<div class="section" id="themes">
<h2><a class="toc-backref" href="#id23">Themes</a></h2>
<div class="section" id="using-a-theme">
<h3><a class="toc-backref" href="#id24">Using a Theme</a></h3>
<p>Pelican has many themes to choose from its <a class="reference external" href="https://github.com/getpelican/pelican-themes">themes repository</a>. These
are the steps:</p>
<ul class="simple">
<li>Clone a theme repo, or if you want clone the entire <a class="reference external" href="https://github.com/getpelican/pelican-themes">themes repository</a>.</li>
<li>Inside <code>pelicancon.py</code> file, change the <code>THEME</code> variable, to the directory
of your theme.</li>
</ul>
<p>On a side note, if you're looking for a place where you can view a screenshot of all the different themes,
<a class="reference external" href="http://ptd.pronoiac.org/">this</a> site has nice screen-shots of each.</p>
<p>For example, if you wanted to use the <a class="reference external" href="https://github.com/kplaube/maggner-pelican">maggner-pelican</a> theme:</p>
<ul class="simple">
<li>Clone it into your project directory, <code>git clone https://github.com/kplaube/maggner-pelican.git</code></li>
<li>Change the <code>THEME</code> variable to <code>'maggner-pelican'</code> and you're done :)</li>
</ul>
</div>
<div class="section" id="creating-themes">
<h3><a class="toc-backref" href="#id25">Creating Themes</a></h3>
<p>Creating themes in Pelican is well documented in their documentation about <a class="reference external" href="http://docs.getpelican.com/en/3.5.0/themes.html">theme creation</a>. Although creating themes
is beyond the scope of this post (because there's a lot that could be said), here are a few pointers:</p>
<ul class="simple">
<li>Target one format, if you try to create a theme for multiple formats like both rst and md, you're doing to run
into a lot of issues.</li>
<li>Using preprocessors can be more effective than using raw css since they allow you create a single css file and provides
you with a lot more flexibility.</li>
<li>If you wish to embed disqus or google comments or other third party dynamic content, then I suggest that you use
separate include files instead of plugins.</li>
</ul>
</div>
</div>
<div class="section" id="settings">
<h2><a class="toc-backref" href="#id26">Settings</a></h2>
<div class="section" id="the-two-settings-files">
<h3><a class="toc-backref" href="#id27">The Two Settings Files</a></h3>
<p>By default, pelican comes with both a <code>pelicanconf.py</code> and <code>publishconf.py</code>. When you are writing content
and wish to see a preview, the settings in <code>pelicanconf.py</code> are used. When you choose to publish, for example to
github or some other place, the settings in <code>pelicanconf.py</code> and <code>publishconf.py</code> are used as
<code>pelicanconf.py</code> is imported in <code class="coderef">1</code> below:</p>
<div class="highlight"><pre><span></span><span class="ch">#!/usr/bin/env python</span>
<span class="c1"># -*- coding: utf-8 -*- #</span>
<span class="kn">from</span> <span class="nn">__future__</span> <span class="kn">import</span> <span class="n">unicode_literals</span>
<span class="c1"># This file is only used if you use `make publish` or</span>
<span class="c1"># explicitly specify it as your config file.</span>
<span class="kn">import</span> <span class="nn">os</span>
<span class="kn">import</span> <span class="nn">sys</span>
<span class="n">sys</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">os</span><span class="o">.</span><span class="n">curdir</span><span class="p">)</span>
<span class="kn">from</span> <span class="nn">pelicanconf</span> <span class="kn">import</span> <span class="o">*</span> <span class="c1"># 1</span>
<span class="p">(</span><span class="o">...</span><span class="p">)</span>
</pre></div>
</div>
<div class="section" id="how-settings-work">
<h3><a class="toc-backref" href="#id28">How Settings Work</a></h3>
<p>By convention all settings are capitalized:</p>
<div class="highlight"><pre><span></span><span class="ch">#!/usr/bin/env python</span>
<span class="c1"># -*- coding: utf-8 -*- #</span>
<span class="kn">from</span> <span class="nn">__future__</span> <span class="kn">import</span> <span class="n">unicode_literals</span>
<span class="n">AUTHOR</span> <span class="o">=</span> <span class="sa">u</span><span class="s1">'Nafiul Islam'</span>
<span class="n">SITENAME</span> <span class="o">=</span> <span class="sa">u</span><span class="s1">'fUNNY bLOG'</span>
<span class="n">SITEURL</span> <span class="o">=</span> <span class="s1">''</span>
</pre></div>
<p>If we take a look in the default theme, they are used like so:</p>
<div class="highlight"><pre><span></span><span class="cp">{%</span> <span class="k">if</span> <span class="nv">DISQUS_SITENAME</span> <span class="k">and</span> <span class="nv">SITEURL</span> <span class="k">and</span> <span class="nv">article.status</span> <span class="p">!</span><span class="o">=</span> <span class="s2">"draft"</span> <span class="cp">%}</span><span class="x"></span>
<span class="x"><div class="comments"></span>
<span class="x"><h2>Comments !</h2></span>
<span class="x"><div id="disqus_thread"></div></span>
<span class="x"><script type="text/javascript"></span>
<span class="x">var disqus_shortname = '</span><span class="cp">{{</span> <span class="nv">DISQUS_SITENAME</span> <span class="cp">}}</span><span class="x">';</span>
<span class="x">var disqus_identifier = '</span><span class="cp">{{</span> <span class="nv">article.url</span> <span class="cp">}}</span><span class="x">';</span>
<span class="x">var disqus_url = '</span><span class="cp">{{</span> <span class="nv">SITEURL</span> <span class="cp">}}</span><span class="x">/</span><span class="cp">{{</span> <span class="nv">article.url</span> <span class="cp">}}</span><span class="x">';</span>
<span class="x">(function() {</span>
<span class="x">var dsq = document.createElement('script'); dsq.type = 'text/javascript'; dsq.async = true;</span>
<span class="x">(...)</span>
</pre></div>
<p>Here, <code>SITEURL</code> AND <code>DISQUS_SITENAME</code> are both settings that are described in either <code>pelicanconf.py</code>
or <code>publishconf.py</code>. They appear as variables inside the jinja template. <a class="footnote-reference" href="#notmyideaarticlehtml" id="id6">[6]</a></p>
<p>Most settings that we put into our settings files are used in the theme, however, some settings like
<code>DEFAULT_PAGINATION</code> are used to determine other things.</p>
</div>
</div>
<div class="section" id="writing-content">
<h2><a class="toc-backref" href="#id29">Writing Content</a></h2>
<p>Pelican supports reStructuredText as its default document format. However, there is also support for markdown, ipython
notebook, asciidoc and probably much more. To write in any of these formats, one needs to install the plugin associated
with the file extension, such as <a class="reference external" href="https://github.com/getpelican/pelican-plugins/tree/master/asciidoc_reader">asciidoc_reader</a>
for the asciidoc format and <a class="reference external" href="https://github.com/danielfrg/pelican-ipynb">pelican-ipynb</a> for the ipython notebook
format. <a class="footnote-reference" href="#notallpluginsarefound" id="id7">[7]</a></p>
<p>I would suggest reStructuredText, since it has the most features and can
offer many ways to extend the syntax with features such as directives and interpreted roles. For example, reStructuredText
supports footnotes out of the box, whereas if one were to use markdown, one would need to install a separate plugin and
attach it to pelican. <a class="footnote-reference" href="#footnotesrst" id="id8">[8]</a></p>
<p>However, pelican has many third party plugins for markdown documents as well so if you prefer using markdown, then
please do so, I only recommend reStructuredText because its easy to extend.</p>
</div>
<div class="section" id="previewing-content">
<h2><a class="toc-backref" href="#id30">Previewing Content</a></h2>
<div class="section" id="devserver">
<h3><a class="toc-backref" href="#id31">devserver</a></h3>
<p>The default way to see a preview of the generated blog is to use <code>make devserver</code>, this regenerates your blog
every time that you change your <code>content</code> folder. This option is a horrible way to preview your blog for a couple
of reasons:</p>
<ul class="simple">
<li>You need to hit the refresh button in your browser to see the changes after a save. This can often get tiring.</li>
<li><code>make devserver</code> runs two processes in background for no good reason, the processes log data into the console
anyways, so why not just have a tool that generates your content and terminates when you hit <kbd class="light">ctrl</kbd> + <kbd class="light">C</kbd> ?</li>
<li>Furthermore, in the transformation from your chosen format to html, if you hit an error, the daemon that keeps
transforming your files to html also stops, and the error is often buried deep inside the logs</li>
</ul>
</div>
<div class="section" id="livereload">
<h3><a class="toc-backref" href="#id32">livereload</a></h3>
<p>We need a solution that does two things for us:</p>
<ul class="simple">
<li>Allows us to see live previews on each save without having to refresh the browser.</li>
<li>Does not stop unexpectedly from an error.</li>
<li>Has an easy to use CLI</li>
</ul>
<p>The solution is to use a neat little package called <code>livereload</code> and can be installed through:</p>
<div class="highlight"><pre><span></span>$ pip install livereload
</pre></div>
<p>Once this has been installed, we need to install the livereload plugin for our respective browser. For example in chrome:</p>
<img alt="Searching for the livereload extension" class="align-center" src="/images/pelican_01.png" />
<div class="line-block">
<div class="line"><br /></div>
</div>
<img alt="Installing the right plugin" class="align-center" src="/images/pelican_02.png" />
<p>In firefox, we head over to addons.mozilla.org and search for <code>livereload</code>:</p>
<img alt="Installing on Firefox" class="align-center" src="/images/pelican_03.png" />
<p>Once this has been installed, we need to run the livereload server. I do this by adding an extra function to my <code>fabfile.py</code> file:</p>
<div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">livereload</span>
<span class="k">def</span> <span class="nf">live_build</span><span class="p">(</span><span class="n">port</span><span class="o">=</span><span class="mi">8080</span><span class="p">):</span>
<span class="n">local</span><span class="p">(</span><span class="s1">'make clean'</span><span class="p">)</span> <span class="c1"># 1</span>
<span class="n">local</span><span class="p">(</span><span class="s1">'make html'</span><span class="p">)</span> <span class="c1"># 2</span>
<span class="n">os</span><span class="o">.</span><span class="n">chdir</span><span class="p">(</span><span class="s1">'output'</span><span class="p">)</span> <span class="c1"># 3</span>
<span class="n">server</span> <span class="o">=</span> <span class="n">livereload</span><span class="o">.</span><span class="n">Server</span><span class="p">()</span> <span class="c1"># 4</span>
<span class="n">server</span><span class="o">.</span><span class="n">watch</span><span class="p">(</span><span class="s1">'../content/*.rst'</span><span class="p">,</span> <span class="c1"># 5</span>
<span class="n">livereload</span><span class="o">.</span><span class="n">shell</span><span class="p">(</span><span class="s1">'pelican -s ../pelicanconf.py -o ../output'</span><span class="p">))</span> <span class="c1"># 6</span>
<span class="n">server</span><span class="o">.</span><span class="n">watch</span><span class="p">(</span><span class="s1">'../naffy/'</span><span class="p">,</span> <span class="c1"># 7</span>
<span class="n">livereload</span><span class="o">.</span><span class="n">shell</span><span class="p">(</span><span class="s1">'pelican -s ../pelicanconf.py -o ../output'</span><span class="p">))</span> <span class="c1"># 8</span>
<span class="n">server</span><span class="o">.</span><span class="n">watch</span><span class="p">(</span><span class="s1">'*.html'</span><span class="p">)</span> <span class="c1"># 9</span>
<span class="n">server</span><span class="o">.</span><span class="n">watch</span><span class="p">(</span><span class="s1">'*.css'</span><span class="p">)</span> <span class="c1"># 10</span>
<span class="n">server</span><span class="o">.</span><span class="n">serve</span><span class="p">(</span><span class="n">liveport</span><span class="o">=</span><span class="mi">35729</span><span class="p">,</span> <span class="n">port</span><span class="o">=</span><span class="n">port</span><span class="p">)</span> <span class="c1"># 11</span>
</pre></div>
<p>We have a lot to explain here so lets begin with <code class="coderef">1</code> and <code class="coderef">2</code>. In both <code class="coderef">1</code> and <code class="coderef">2</code>, we use the <code>local</code> function
from fabric. <code>local</code> allows us to run shell commands like <code>cd</code>. In this case, we are using the <code>make</code>
command because pelican comes with a Makefile. If we involve <code>make help</code> from the command line in our project
directory, we get the following:</p>
<div class="highlight"><pre><span></span>$ make <span class="nb">help</span>
Makefile <span class="k">for</span> a pelican Web site
Usage:
make html <span class="o">(</span>re<span class="o">)</span>generate the web site
make clean remove the generated files
</pre></div>
<p>So, in <code class="coderef">1</code> we clean out output folder, and in <code class="coderef">2</code> we generate our html files from our content.</p>
<p>In <code class="coderef">3</code>, we change the directory to our newly created output folder and in <code class="coderef">4</code> we create an instance
of the <code>livereload</code> server. In <code class="coderef">5</code>, we use the <code>server.watch</code> function to watch all the <code>rst</code> files
since my content is written in reStructuredText. We can change this to <code>../content/*.md</code> for markdown files for
example.</p>
<p><code>server.watch</code> had one required argument and two optional arguments. The first argument is <code>filepath</code>,
the second is <code>func</code>, a function that you run every time watched files are changed.
The last argument is <code>delay</code>, which delays function execution. <a class="footnote-reference" href="#id14" id="id9">[9]</a></p>
<p>In <code class="coderef">6</code>, we specify the function to be run as <code>livereload.shell('pelican -s ../pelicanconf.py -o ../output')</code>.
The <code>shell</code> function from <code>livereload</code> allows us to run shell functions every time a watched file is changed.
In this case, we ask pelican to remake our html files. Why was <code>make html</code> not used here? This is because the
<code>make</code> command looks for a Makefile and we are right now in our output directory.</p>
<p>In <code class="coderef">7</code>, the server is directed to watch the "naffy" directory. The "naffy" directory is the directory that
houses my theme; if any change is made to my theme, livereload refreshes the server. Replace "naffy" with whatever
folder you are housing your theme in. If you're using the default theme, then do not add <code class="coderef">7</code> and <code class="coderef">8</code>. <code class="coderef">8</code> is the
same as <code class="coderef">6</code>, we just rebuild the html.</p>
<p>In <code class="coderef">9</code> and <code class="coderef">10</code>, the server is further instructed to watch html and css files that change within the output directory.</p>
<p>Finally in <code class="coderef">11</code>, we tell livereload to serve on port <code>35729</code>. This is the default port for the livereload plugin <a class="footnote-reference" href="#chromedoesntwork" id="id10">[10]</a>.
The <code>port</code> variable is the port of the server serving the files, the <code>liveport</code> variable is a port that
livereload uses to make changes directly to the HTML file.</p>
<img alt="Livereload plugin preferences on FireFox" class="align-center" src="/images/pelican_04.png" />
</div>
</div>
<div class="section" id="plugins">
<h2><a class="toc-backref" href="#id33">Plugins</a></h2>
<div class="section" id="using-plugins">
<h3><a class="toc-backref" href="#id34">Using Plugins</a></h3>
<p>Most plugins in pelican tell you how to install them. For example, the <a class="reference external" href="https://github.com/getpelican/pelican-plugins/tree/master/render_math">render_math</a> plugin allows you to embed math
inside your posts. Here's the basic gist of what you need to do in order to install them:</p>
<ul class="simple">
<li>Download the plugin package into your plugins folder, which is set inside your <code>pelicanconf.py</code> file in the
<code>PLUGIN_PATHS</code> variable.</li>
<li>Then append the name of your plugin package inside your <code>pelicanconf.py</code> file. <a class="footnote-reference" href="#settingspy" id="id11">[11]</a></li>
<li>Follow any other additional instructions that your plugin might have.</li>
</ul>
<p>If just adding the name does not work, then one might consider adding <code><plugin_name>.<plugin_file></code>. <a class="footnote-reference" href="#pluginsimport" id="id12">[12]</a></p>
</div>
<div class="section" id="creating-plugins">
<h3><a class="toc-backref" href="#id35">Creating Plugins</a></h3>
<p>Creating plugins in pelican basically means that you create a function to do your processing. You then register that
function as a plugin. Creating plugins is
<a class="reference external" href="http://docs.getpelican.com/en/latest/plugins.html#how-to-create-plugins">well documented</a> and an example of creating
a reader is given.</p>
<p>In my use of pelican, more often than not, I've needed to create plugins that are process text in a given document, not
add support for new document formats. So, I'll show you how I created a plugin for adding keyboard keys like <kbd class="light">alt</kbd>
and <kbd class="light">⌘</kbd> .</p>
<p>I like to create my plugins as packages, but you can create them as simple files as well. So, inside our plugins folder,
create a new package, and inside the package, we create a new file for the plugin:</p>
<div class="highlight"><pre><span></span>$ <span class="nb">cd</span> plugins <span class="c1"># Or whatever your plugins folder is called</span>
$ mkdir keyboard
$ <span class="nb">cd</span> keyboard
$ touch __init__.py kb.py
$ ls
__init__.py kb.py
</pre></div>
<p>Inside <code>kb.py</code>, we can need a function that modifies existing code, and a function that registers the function:</p>
<div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">pelican</span> <span class="kn">import</span> <span class="n">signals</span>
<span class="kn">import</span> <span class="nn">logging</span>
<span class="kn">import</span> <span class="nn">re</span>
<span class="n">logging</span><span class="o">.</span><span class="n">Logger</span><span class="p">(</span><span class="vm">__name__</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">content_object_init</span><span class="p">(</span><span class="n">instance</span><span class="p">):</span>
<span class="sd">"""</span>
<span class="sd"> Provides the key plugin, make sure that you have Keys.css, http://michaelhue.com/keyscss/</span>
<span class="sd"> imported inside your HTML. How to use:</span>
<span class="sd"> So you hit [kb:CTRL] + [kb:ALT] + [kb:DEL] when in doubt</span>
<span class="sd"> Note, that light keyboard keys are enabled by default.</span>
<span class="sd"> """</span>
<span class="k">if</span> <span class="n">instance</span><span class="o">.</span><span class="n">_content</span> <span class="ow">is</span> <span class="ow">not</span> <span class="bp">None</span><span class="p">:</span> <span class="c1"># 1</span>
<span class="n">content</span> <span class="o">=</span> <span class="n">instance</span><span class="o">.</span><span class="n">_content</span>
<span class="n">new_content</span> <span class="o">=</span> <span class="n">re</span><span class="o">.</span><span class="n">sub</span><span class="p">(</span><span class="sa">r</span><span class="s2">"\[kb:(.+?)\]"</span><span class="p">,</span> <span class="sa">r</span><span class="s1">'<kbd class="light">\1</kbd>'</span><span class="p">,</span> <span class="n">content</span><span class="p">)</span> <span class="c1"># 2</span>
<span class="n">instance</span><span class="o">.</span><span class="n">_content</span> <span class="o">=</span> <span class="n">new_content</span>
<span class="k">return</span> <span class="n">instance</span>
<span class="k">def</span> <span class="nf">register</span><span class="p">():</span>
<span class="n">signals</span><span class="o">.</span><span class="n">content_object_init</span><span class="o">.</span><span class="n">connect</span><span class="p">(</span><span class="n">content_object_init</span><span class="p">)</span> <span class="c1"># 3</span>
</pre></div>
<p>In <code class="coderef">1</code>, we check if the article, page or blog post in question has content; essentially does it have a body. If we
don't have a body, then we skip this object. In <code class="coderef">2</code>, replace with the corresponding stylized version, so that typing
<code>[kb:alt]</code> will give you <kbd class="light">alt</kbd> .</p>
<p>In <code class="coderef">3</code>, we register the function with pelican. Registering this plugin is simple, inside <code>pelicanconf.py</code>, we
merely append to the list of <code>PLUGINS</code>:</p>
<div class="highlight"><pre><span></span><span class="p">(</span><span class="o">...</span><span class="p">)</span>
<span class="n">PLUGINS</span> <span class="o">+=</span> <span class="p">[</span><span class="s1">'keyboard.kb'</span><span class="p">]</span>
<span class="p">(</span><span class="o">...</span><span class="p">)</span>
</pre></div>
<p>I used <a class="reference external" href="http://michaelhue.com/keyscss/">key.css</a> to style my keys, and I simply added the css file to my theme's
<code>article.html</code>.</p>
<p>Once can create far more interesting plugins than the one I've shown, by looking at the plugin repository for
more interesting plugins or even reading the documentation (although its not exhaustive in its explanations).</p>
</div>
</div>
<div class="section" id="hosting-on-github-pages">
<h2><a class="toc-backref" href="#id36">Hosting on Github Pages</a></h2>
<p>Github is the best way to host your static blog. Some notable sites which get a substantial amount of traffic are
hosted on github like <a class="reference external" href="http://getbootstrap.com/">Bootstrap</a>, <a class="reference external" href="http://yeoman.io/">Yeoman</a>,
<a class="reference external" href="https://jakevdp.github.io/">Pythonic Perambulations</a> are hosted on github pages. If you don't have a github
account, creating one is simple and quick.</p>
<div class="section" id="creating-the-repo">
<h3><a class="toc-backref" href="#id37">Creating the repo</a></h3>
<p>In order to get a site name like <code>username.github.io</code>, one needs to create a repo with that name on github,
set the visibility to public and do not initialize with anything, you want an empty repo that you can push to.</p>
</div>
<div class="section" id="ghp-import">
<h3><a class="toc-backref" href="#id38">ghp-import</a></h3>
<p><code>ghp-import</code> is a simple plugin that allows you to easily push your content to a github repo. You can install it
using pip:</p>
<div class="highlight"><pre><span></span>$ pip install ghp-import
</pre></div>
<p>There are two steps to doing this:</p>
<div class="highlight"><pre><span></span>$ ghp-import output
$ git push git@github.com:<username>/<username>.github.io.git gh-pages:master
</pre></div>
<p>This will push the files in your <code>output</code> folder directly to the master branch of the <code><username>.github.io</code>
repository. It may take some time initially, but you will be able to see your fully generated blog on this domain.</p>
</div>
<div class="section" id="custom-domains">
<h3><a class="toc-backref" href="#id39">Custom domains</a></h3>
<p>In order to get your own top level domain like I have with <code>nafiulis.me</code>, you simply need to add a <code>CNAME</code>
file with your domain. The <code>CNAME</code> file must me located <em>inside</em> your output directory and must only have the
domain name. My <code>CNAME</code> file looks like this:</p>
<div class="highlight"><pre><span></span>nafiulis.me
</pre></div>
<p>The <code>CNAME</code> file must be included when you are pushing to your github repo.</p>
<p>Once this is done, you need to link your domain up with github, and this is wonderfully explained in this
<a class="reference external" href="http://davidensinger.com/2013/03/setting-the-dns-for-github-pages-on-namecheap/">article</a></p>
</div>
</div>
<div class="section" id="automating-publication">
<h2><a class="toc-backref" href="#id40">Automating Publication</a></h2>
<p>I use fabric to automate my publication, I just have two functions to do this:</p>
<div class="highlight"><pre><span></span><span class="k">def</span> <span class="nf">enter_dns_file</span><span class="p">():</span> <span class="c1"># 1</span>
<span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="s1">'output/CNAME'</span><span class="p">,</span> <span class="s1">'w'</span><span class="p">)</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span>
<span class="n">f</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="s1">'nafiulis.me'</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">github</span><span class="p">(</span><span class="n">publish_drafts</span><span class="o">=</span><span class="bp">False</span><span class="p">):</span> <span class="c1"># 2</span>
<span class="k">try</span><span class="p">:</span> <span class="c1"># 3</span>
<span class="k">if</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">exists</span><span class="p">(</span><span class="s1">'output/drafts'</span><span class="p">):</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">publish_drafts</span><span class="p">:</span>
<span class="n">local</span><span class="p">(</span><span class="s1">'rm -rf output/drafts'</span><span class="p">)</span>
<span class="k">except</span> <span class="ne">Exception</span><span class="p">:</span>
<span class="k">pass</span>
<span class="n">local</span><span class="p">(</span><span class="s1">'ghp-import output'</span><span class="p">)</span> <span class="c1"># 4</span>
<span class="n">local</span><span class="p">(</span><span class="s1">'git push'</span>
<span class="s1">'git@github.com:<username>/<username>.github.io.git</span>
<span class="s1">'gh-pages:master'</span><span class="p">)</span> <span class="c1"># 5</span>
<span class="n">local</span><span class="p">(</span><span class="s1">'rm -rf output'</span><span class="p">)</span> <span class="c1"># 6</span>
</pre></div>
<p>In <code class="coderef">1</code>, the function <code>enter_dns_file</code> is used to insert the <code>CNAME</code> file into the output folder before
pushing to github because in <code class="coderef">6</code>, we delete the entire output folder.</p>
<p>In <code class="coderef">2</code>, the <code>github</code> function is used to publish the content of the output folder to our github repo, with the
option of publishing drafts which is <code>False</code> by default. The <code>try-catch</code> block in <code class="coderef">3</code> is there to check if
we have any drafts in the first place, and if the <code>publish_drafts</code> variable is <code>True</code>, then we publish them.</p>
<p>From <code class="coderef">4</code> to <code class="coderef">5</code>, we push the content to the repository. Replace <code><username></code> with your username.</p>
</div>
<div class="section" id="acknowledgements">
<h2><a class="toc-backref" href="#id41">Acknowledgements</a></h2>
<p>I would like to thank <strong>Ashwini Chaudhury</strong>, <strong>Karan Sikka</strong>, <strong>Mahmud Ridwan</strong>, <strong>Sahib bin Mahboob</strong> and
<strong>Keiron Pizzey</strong> for proofreading this article before publication.</p>
<hr class="docutils" />
<table class="docutils footnote" frame="void" id="assumingpy" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#id1">[1]</a></td><td>Assuming that you are proficient in Python.</td></tr>
</tbody>
</table>
<table class="docutils footnote" frame="void" id="windows" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#id2">[2]</a></td><td>Don't use virtualenv, just use your base python when using windows. If using Windows,
make sure that you have 2.7.9 or 3.4.2, both of which have "ensurepip".</td></tr>
</tbody>
</table>
<table class="docutils footnote" frame="void" id="id13" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#id3">[3]</a></td><td><div class="first last line-block">
<div class="line">Pelican has its own plugin repository on github. Please note however, that some plugins are defunct and are not supported by newer versions of pelican.</div>
<div class="line"><a class="reference external" href="https://github.com/getpelican/pelican-plugins">https://github.com/getpelican/pelican-plugins</a></div>
</div>
</td></tr>
</tbody>
</table>
<table class="docutils footnote" frame="void" id="problemsquickstart" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#id4">[4]</a></td><td>If you cannot access this, restart your virtualenv if you are currently in one. Otherwise use restart your shell.</td></tr>
</tbody>
</table>
<table class="docutils footnote" frame="void" id="fabpy3support" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#id5">[5]</a></td><td><div class="first last line-block">
<div class="line">Check out the wall of superpowers if this has changed</div>
<div class="line"><a class="reference external" href="http://python3wos.appspot.com/">http://python3wos.appspot.com/</a></div>
</div>
</td></tr>
</tbody>
</table>
<table class="docutils footnote" frame="void" id="notmyideaarticlehtml" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#id6">[6]</a></td><td><div class="first last line-block">
<div class="line">This piece of code is taken from the default theme that comes with pelican called "notmyidea"</div>
<div class="line"><a class="reference external" href="https://github.com/getpelican/pelican/blob/3.5.0/pelican/themes/notmyidea/templates/article.html#L17">https://github.com/getpelican/pelican/blob/3.5.0/pelican/themes/notmyidea/templates/article.html#L17</a></div>
</div>
</td></tr>
</tbody>
</table>
<table class="docutils footnote" frame="void" id="notallpluginsarefound" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#id7">[7]</a></td><td>The ipython notebook plugin is one example of a plugin that you can't find on the official plugin
repository. This may be, because the plugin does not support the newest version, Pelican 3.5, since
the plugin says that it required pelican 3.4 to function.</td></tr>
</tbody>
</table>
<table class="docutils footnote" frame="void" id="footnotesrst" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#id8">[8]</a></td><td><div class="first last line-block">
<div class="line">reStructuredText also allows for more versatile footnotes as well. <a class="reference external" href="http://docutils.sourceforge.net/docs/user/rst/quickstart.html">Check out their quickstart guide</a>.</div>
</div>
</td></tr>
</tbody>
</table>
<table class="docutils footnote" frame="void" id="id14" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#id9">[9]</a></td><td><a class="reference external" href="https://livereload.readthedocs.org/en/latest/#shell">Livereload has documentation on the server class on its documentation website</a></td></tr>
</tbody>
</table>
<table class="docutils footnote" frame="void" id="chromedoesntwork" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#id10">[10]</a></td><td>I cannot find the same options for chrome, hence I use firefox to see live builds of the generated site</td></tr>
</tbody>
</table>
<table class="docutils footnote" frame="void" id="settingspy" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#id11">[11]</a></td><td>In some documentation regarding plugins, the pelicanconf.py file is called settings.py.</td></tr>
</tbody>
</table>
<table class="docutils footnote" frame="void" id="pluginsimport" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#id12">[12]</a></td><td>Sometimes, Pelican packages do not import the functions necessary inside their <code>__init__.py</code>
file. In this case you import one of the plugins's files. In most cases it is the file with a
<code>register</code> method.</td></tr>
</tbody>
</table>
</div>
Kotlin Koans I - The Beginning2014-11-22T21:32:00+06:002014-11-22T21:32:00+06:00Nafiul Islamtag:nafiulis.me,2014-11-22:kotlin-koans-i.html<div class="contents topic" id="table-of-contents">
<p class="topic-title first">Table of Contents</p>
<ul class="simple">
<li><a class="reference internal" href="#getting-started" id="id10">Getting Started</a><ul>
<li><a class="reference internal" href="#installing-kotlin" id="id11">Installing Kotlin</a></li>
<li><a class="reference internal" href="#hello-world-in-intellij-idea" id="id12">Hello World in IntelliJ IDEA</a></li>
<li><a class="reference internal" href="#dash-doctests" id="id13">Dash doctests</a></li>
<li><a class="reference internal" href="#cloning-the-koans" id="id14">Cloning the Koans</a></li>
</ul>
</li>
<li><a class="reference internal" href="#koans-part-i-introduction" id="id15">Koans Part I: Introduction</a><ul>
<li><a class="reference internal" href="#functions" id="id16">0: Functions</a></li>
<li><a class="reference internal" href="#collection-to-string" id="id17">1: Collection to String</a></li>
<li><a class="reference internal" href="#default-parameters" id="id18">2.1: Default Parameters</a></li>
<li><a class="reference internal" href="#collections-functions" id="id19">2.2: Collections functions</a></li>
<li><a class="reference internal" href="#lambdas" id="id20">3: Lambdas</a></li>
<li><a class="reference internal" href="#string-templates" id="id21">4: String templates</a></li>
<li><a class="reference internal" href="#null-safety" id="id22">5: Null safety</a></li>
<li><a class="reference internal" href="#smart-casts-recursion" id="id23">6: Smart Casts & Recursion</a></li>
<li><a class="reference internal" href="#data-classes" id="id24">7: Data Classes</a></li>
<li><a class="reference internal" href="#extension-functions" id="id25">8: Extension functions</a></li>
<li><a class="reference internal" href="#extending-collections" id="id26">9: Extending Collections</a></li>
<li><a class="reference internal" href="#object-expressions" id="id27">10: Object Expressions</a></li>
</ul>
</li>
<li><a class="reference internal" href="#takeaway" id="id28">Takeaway</a></li>
<li><a class="reference internal" href="#acknowledgements" id="id29">Acknowledgements</a></li>
</ul>
</div>
<p>Kotlin is a programming language created by Jetbrains. It runs on both the Java Virtual Machine (JVM)
and can be compiled to JavaScript.</p>
<p>I've been meaning to try out Kotlin for quite some time. After my foray into <a class="reference external" href="http://www.scala-lang.org/">Scala</a>,
which brought some very dramatic changes to the Java universe, my interest in JVM based languages was re-invigorated.</p>
<p>Back in 2013, Kotlin seemed like an interesting language because Jetbrains, a heavy investor in the JVM, was promoting it
as an <em>industry first</em> programming language. I knew Kotlin also was going to have <em>excellent</em> tooling support,
because after all Jetbrains is an IDE company. It wasn't until recently that Kotlin had a tutorial series called the
<a class="reference external" href="http://kotlinlang.org/docs/tutorials/koans.html">Kotlin Koans</a>.</p>
<p>So why should you learn Kotlin? Well that is what I will be trying to answer throughout this series. We will
go through the Kotlin Koans, and find out the differences between Java and Kotlin as well as other languages,
but in short <a class="footnote-reference" href="#mistake" id="id1">[1]</a>:</p>
<ul class="simple">
<li>No semicolons</li>
<li>First class functions, meaning they do not need to be declared <em>within</em> classes</li>
<li>Lambdas</li>
<li>An awesome collections library</li>
<li>Small runtime</li>
<li>data classes, like <code>case</code> classes in Scala</li>
<li>Good interoperability with existing Java code</li>
</ul>
<div class="section" id="getting-started">
<h2><a class="toc-backref" href="#id10">Getting Started</a></h2>
<div class="section" id="installing-kotlin">
<h3><a class="toc-backref" href="#id11">Installing Kotlin</a></h3>
<p>The simplest way to install Kotlin is to download <a class="reference external" href="https://www.jetbrains.com/idea/download/">IntelliJ IDEA</a>.
The community edition supports Kotlin so you can go ahead and download it. After this, we can install the Kotlin plugin,
which will automatically install the Kotlin runtime as well:</p>
<img alt="Installing the plugin. The steps are indicated by the numbering." class="align-center" src="/images/koans_04.png" />
</div>
<div class="section" id="hello-world-in-intellij-idea">
<h3><a class="toc-backref" href="#id12">Hello World in IntelliJ IDEA</a></h3>
<p>Once this has been installed, getting started with your first hello world project is simple. Hadi Hariri <a class="footnote-reference" href="#hadi" id="id2">[2]</a>
has created a bunch of videos on <a class="reference external" href="http://kotlinlang.org/docs/videos.html">getting started</a>. Here, he demonstrates
how to create a new Kotlin project:</p>
<div class="external youtube"><iframe width="640" height="360" src="//www.youtube.com/embed/iMMBrcabSOI" frameborder="0"></iframe></div></div>
<div class="section" id="dash-doctests">
<h3><a class="toc-backref" href="#id13">Dash doctests</a></h3>
<p>Kotlin also has <a class="reference external" href="http://kapeli.com/dash">Dash</a> <a class="footnote-reference" href="#dash" id="id4">[3]</a> doctests, provided by a third party:</p>
<img alt="Installing Kotlin doctests" class="align-center" src="/images/koans_05.png" />
<p>However, this merely is a copy of the online documentation available for <a class="reference external" href="http://kotlinlang.org/docs/reference/">Kotlin</a>,
and does not actually have a searchable API reference. The code completion in IDEA somewhat mitigates this though.
The Jetbrains team is still working on modernizing the reference for Kotlin at the moment.</p>
</div>
<div class="section" id="cloning-the-koans">
<h3><a class="toc-backref" href="#id14">Cloning the Koans</a></h3>
<p>The next part is just cloning the Koans repository. One can just use the command line like so:</p>
<div class="highlight"><pre><span></span>git clone https://github.com/jetbrains/workshop-jb
</pre></div>
<p>Or checkout from the IDEA start menu:</p>
<img alt="Checking out from git" class="align-center" src="/images/koans_06.png" />
<img alt="Setting the repo url" class="align-center" src="/images/koans_07.png" />
</div>
</div>
<div class="section" id="koans-part-i-introduction">
<h2><a class="toc-backref" href="#id15">Koans Part I: Introduction</a></h2>
<p>I'm going to go through all the exercises in the Koans. I will share how I solved these problems, with the challenges
I've had to face as well as examples of solutions themselves. One can merely continue reading to see a list of features
that Kotlin has, or one can follow along using the consecutive sections as an answer sheet.</p>
<div class="section" id="functions">
<h3><a class="toc-backref" href="#id16">0: Functions</a></h3>
<p>The goal of the Koans is to make all the tests pass. Lets head over to our first exercise:</p>
<img alt="Where the file is located" class="align-center" src="/images/koans_08.png" />
<p>Inside the file, we have <tt class="docutils literal">todoTask0</tt> and <tt class="docutils literal">task0</tt>. <code>todoTask0</code> gives us the information we need to complete the
task. We essentially need to make it return <tt class="docutils literal">"OK"</tt>. This seems easy enough:</p>
<div class="highlight"><pre><span></span><span class="k">fun</span> <span class="nf">task0</span><span class="p">():</span> <span class="n">String</span> <span class="p">{</span>
<span class="k">return</span> <span class="s">"OK"</span>
<span class="p">}</span>
</pre></div>
<p>That was easy. We have a function here that returns a <code>String</code>.</p>
</div>
<div class="section" id="collection-to-string">
<h3><a class="toc-backref" href="#id17">1: Collection to String</a></h3>
<p>In this task, we need to convert a given collection into a string. We are allowed to copy and paste the Java code and
transform it into Kotlin code:</p>
<blockquote>
You can just copy-paste it and agree to automatically convert it to Kotlin - but only in this task :).</blockquote>
<p>It is unfortunate, however, that <code>todoTask1</code> does not give us any extra information other than telling us to just
<em>"Rewrite JavaCode1.task1 to Kotlin"</em>. So, I decided to take a look at the test function located here:</p>
<img alt="Where the tests are located" class="align-center" src="/images/koans_09.png" />
<p>We can see that we are asked to turn the collection of numbers into a comma delimited string, with braces at the
beginning and at the end:</p>
<div class="highlight"><pre><span></span><span class="k">class</span> <span class="nc">_01_Functions</span><span class="p">()</span> <span class="p">{</span>
<span class="n">test</span> <span class="k">fun</span> <span class="nf">collection</span><span class="p">()</span> <span class="p">{</span>
<span class="n">Assert</span><span class="p">.</span><span class="n">assertEquals</span><span class="p">(</span><span class="s">"{1, 2, 3, 42, 555}"</span><span class="p">,</span> <span class="n">task1</span><span class="p">(</span><span class="n">listOf</span><span class="p">(</span><span class="m">1</span><span class="p">,</span> <span class="m">2</span><span class="p">,</span> <span class="m">3</span><span class="p">,</span> <span class="m">42</span><span class="p">,</span> <span class="m">555</span><span class="p">)))</span>
<span class="p">}</span>
<span class="p">}</span>
</pre></div>
<p>Copying the Java code from <tt class="docutils literal">JavaCode1.task1</tt> results in this:</p>
<div class="highlight"><pre><span></span><span class="k">val</span> <span class="py">sb</span> <span class="p">=</span> <span class="n">StringBuilder</span><span class="p">()</span>
<span class="n">sb</span><span class="p">.</span><span class="n">append</span><span class="p">(</span><span class="s">"{"</span><span class="p">)</span>
<span class="k">val</span> <span class="py">iterator</span> <span class="p">=</span> <span class="n">collection</span><span class="p">.</span><span class="n">iterator</span><span class="p">()</span>
<span class="k">while</span> <span class="p">(</span><span class="n">iterator</span><span class="p">.</span><span class="n">hasNext</span><span class="p">())</span> <span class="p">{</span>
<span class="k">val</span> <span class="py">element</span> <span class="p">=</span> <span class="n">iterator</span><span class="p">.</span><span class="n">next</span><span class="p">()</span>
<span class="n">sb</span><span class="p">.</span><span class="n">append</span><span class="p">(</span><span class="n">element</span><span class="p">)</span>
<span class="k">if</span> <span class="p">(</span><span class="n">iterator</span><span class="p">.</span><span class="n">hasNext</span><span class="p">())</span> <span class="p">{</span>
<span class="n">sb</span><span class="p">.</span><span class="n">append</span><span class="p">(</span><span class="s">", "</span><span class="p">)</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="n">sb</span><span class="p">.</span><span class="n">append</span><span class="p">(</span><span class="s">"}"</span><span class="p">)</span>
<span class="k">return</span> <span class="n">sb</span><span class="p">.</span><span class="n">toString</span><span class="p">()</span>
</pre></div>
<p>Not very different from the <em>actual</em> Java code. However, I chose to use the following function:</p>
<div class="highlight"><pre><span></span><span class="k">fun</span> <span class="nf">task1</span><span class="p">(</span><span class="n">collection</span><span class="p">:</span> <span class="n">Collection</span><span class="p"><</span><span class="n">Int</span><span class="p">>):</span> <span class="n">String</span> <span class="p">{</span>
<span class="k">val</span> <span class="py">sj</span> <span class="p">=</span> <span class="n">StringJoiner</span><span class="p">(</span><span class="s">", "</span><span class="p">,</span> <span class="s">"{"</span><span class="p">,</span> <span class="s">"}"</span><span class="p">)</span>
<span class="k">for</span> <span class="p">(</span><span class="n">item</span> <span class="k">in</span> <span class="n">collection</span><span class="p">)</span> <span class="p">{</span>
<span class="n">sj</span><span class="p">.</span><span class="n">add</span><span class="p">(</span><span class="n">item</span><span class="p">.</span><span class="n">toString</span><span class="p">())</span>
<span class="p">}</span>
<span class="k">return</span> <span class="n">sj</span><span class="p">.</span><span class="n">toString</span><span class="p">()</span>
<span class="p">}</span>
</pre></div>
<p>I don't know if my version has worse performance than the converted Java Code, but it sure passes the test:</p>
<img alt="Passing test 1" class="align-center" src="/images/koans_10.png" />
<p>In this I chose to use Kotlin's for loop that supports collections.</p>
</div>
<div class="section" id="default-parameters">
<h3><a class="toc-backref" href="#id18">2.1: Default Parameters</a></h3>
<p>In this exercise, we need to convert an overloaded Java class into a simple Kotlin function with default parameters.
The function in question is <code>foo</code>:</p>
<div class="highlight"><pre><span></span><span class="k">fun</span> <span class="nf">foo</span><span class="p">(</span><span class="n">name</span><span class="p">:</span> <span class="n">String</span><span class="p">):</span> <span class="n">String</span> <span class="p">=</span> <span class="n">todoTask2_1</span><span class="p">()</span>
</pre></div>
<p>At first this was confusing because it had been a long time since I had used a language that did <em>not</em> have default
parameters. Secondly, <code>foo</code> already seemed to have a function body. Thirdly, there seemed to be no instructions on
<em>what</em> the function did; you seemingly <em>had</em> to read the Java code to understand. I decided to go back to the tests again
and see what was up.</p>
<p>We have to rewrite the function <code>foo</code> to have three parameters:</p>
<ul class="simple">
<li><code>name</code> which is a <code>String</code>. Required</li>
<li><code>toUpperCase</code> which is a <code>Boolean</code> value. Optional, defaults to <code>false</code></li>
<li><code>number</code> which is a <code>Int</code>. Optional, defaults to <code>42</code></li>
</ul>
<p>With that out of the way, this is what I ended up with:</p>
<div class="highlight"><pre><span></span><span class="k">fun</span> <span class="nf">foo</span><span class="p">(</span><span class="n">name</span><span class="p">:</span> <span class="n">String</span><span class="p">,</span> <span class="n">number</span><span class="p">:</span> <span class="n">Int</span> <span class="p">=</span> <span class="m">42</span><span class="p">,</span> <span class="n">toUpperCase</span><span class="p">:</span> <span class="n">Boolean</span> <span class="p">=</span> <span class="k">false</span><span class="p">):</span> <span class="n">String</span> <span class="p">{</span>
<span class="k">return</span> <span class="k">if</span> <span class="p">(</span><span class="n">toUpperCase</span><span class="p">)</span> <span class="n">name</span><span class="p">.</span><span class="n">toUpperCase</span><span class="p">()+</span><span class="n">number</span><span class="p">.</span><span class="n">toString</span><span class="p">()</span> <span class="k">else</span> <span class="n">name</span><span class="p">+</span><span class="n">number</span><span class="p">.</span><span class="n">toString</span><span class="p">()</span> <span class="c1">// 1</span>
<span class="p">}</span>
<span class="k">fun</span> <span class="nf">task2_1</span><span class="p">():</span> <span class="n">String</span> <span class="p">{</span>
<span class="k">return</span> <span class="p">(</span><span class="n">foo</span><span class="p">(</span><span class="s">"a"</span><span class="p">)</span> <span class="p">+</span>
<span class="n">foo</span><span class="p">(</span><span class="s">"b"</span><span class="p">,</span> <span class="n">number</span> <span class="p">=</span> <span class="m">1</span><span class="p">)</span> <span class="p">+</span>
<span class="n">foo</span><span class="p">(</span><span class="s">"c"</span><span class="p">,</span> <span class="n">toUpperCase</span> <span class="p">=</span> <span class="k">true</span><span class="p">)</span> <span class="p">+</span>
<span class="n">foo</span><span class="p">(</span><span class="n">name</span> <span class="p">=</span> <span class="s">"d"</span><span class="p">,</span> <span class="n">number</span> <span class="p">=</span> <span class="m">2</span><span class="p">,</span> <span class="n">toUpperCase</span> <span class="p">=</span> <span class="k">true</span><span class="p">))</span>
<span class="p">}</span>
</pre></div>
<p>In <code class="coderef">1</code> I use a ternary operator, the equivalent in a full <code>if-else</code> statement would be:</p>
<div class="highlight"><pre><span></span><span class="k">if</span> <span class="p">(</span><span class="n">toUpperCase</span><span class="p">)</span> <span class="p">{</span>
<span class="k">return</span> <span class="n">name</span><span class="p">.</span><span class="n">toUpperCase</span><span class="p">()</span> <span class="p">+</span> <span class="n">number</span><span class="p">.</span><span class="n">toString</span><span class="p">()</span>
<span class="p">}</span> <span class="k">else</span> <span class="p">{</span>
<span class="k">return</span> <span class="n">name</span> <span class="p">+</span> <span class="n">number</span><span class="p">.</span><span class="n">toString</span><span class="p">()</span>
<span class="p">}</span>
</pre></div>
<p>Compared the above to the amount of Java code one would have to write:</p>
<div class="highlight"><pre><span></span><span class="kd">public</span> <span class="kd">class</span> <span class="nc">JavaCode2</span> <span class="kd">extends</span> <span class="n">JavaCode</span> <span class="o">{</span>
<span class="kd">private</span> <span class="kt">int</span> <span class="n">defaultNumber</span> <span class="o">=</span> <span class="mi">42</span><span class="o">;</span>
<span class="kd">public</span> <span class="n">String</span> <span class="nf">foo</span><span class="o">(</span><span class="n">String</span> <span class="n">name</span><span class="o">,</span> <span class="kt">int</span> <span class="n">number</span><span class="o">,</span> <span class="kt">boolean</span> <span class="n">toUpperCase</span><span class="o">)</span> <span class="o">{</span>
<span class="k">return</span> <span class="o">(</span><span class="n">toUpperCase</span> <span class="o">?</span> <span class="n">name</span><span class="o">.</span><span class="na">toUpperCase</span><span class="o">()</span> <span class="o">:</span> <span class="n">name</span><span class="o">)</span> <span class="o">+</span> <span class="n">number</span><span class="o">;</span>
<span class="o">}</span>
<span class="kd">public</span> <span class="n">String</span> <span class="nf">foo</span><span class="o">(</span><span class="n">String</span> <span class="n">name</span><span class="o">,</span> <span class="kt">int</span> <span class="n">number</span><span class="o">)</span> <span class="o">{</span>
<span class="k">return</span> <span class="n">foo</span><span class="o">(</span><span class="n">name</span><span class="o">,</span> <span class="n">number</span><span class="o">,</span> <span class="kc">false</span><span class="o">);</span>
<span class="o">}</span>
<span class="kd">public</span> <span class="n">String</span> <span class="nf">foo</span><span class="o">(</span><span class="n">String</span> <span class="n">name</span><span class="o">,</span> <span class="kt">boolean</span> <span class="n">toUpperCase</span><span class="o">)</span> <span class="o">{</span>
<span class="k">return</span> <span class="n">foo</span><span class="o">(</span><span class="n">name</span><span class="o">,</span> <span class="n">defaultNumber</span><span class="o">,</span> <span class="n">toUpperCase</span><span class="o">);</span>
<span class="o">}</span>
<span class="kd">public</span> <span class="n">String</span> <span class="nf">foo</span><span class="o">(</span><span class="n">String</span> <span class="n">name</span><span class="o">)</span> <span class="o">{</span>
<span class="k">return</span> <span class="n">foo</span><span class="o">(</span><span class="n">name</span><span class="o">,</span> <span class="n">defaultNumber</span><span class="o">);</span>
<span class="o">}</span>
<span class="o">}</span>
</pre></div>
</div>
<div class="section" id="collections-functions">
<h3><a class="toc-backref" href="#id19">2.2: Collections functions</a></h3>
<p>In <a class="reference internal" href="#collection-to-string">1: Collection to String</a>, we wrote some Kotlin to turn a collection of numbers into a string, enclosed
by braces. In this exercise, we just have to use on line of code to achieve the same thing using <code>joinToString</code>:</p>
<div class="highlight"><pre><span></span><span class="k">fun</span> <span class="nf">task2_2</span><span class="p">(</span><span class="n">collection</span><span class="p">:</span> <span class="n">Collection</span><span class="p"><</span><span class="n">Int</span><span class="p">>):</span> <span class="n">String</span> <span class="p">{</span>
<span class="k">return</span> <span class="n">collection</span><span class="p">.</span><span class="n">joinToString</span><span class="p">(</span><span class="s">", "</span><span class="p">,</span> <span class="s">"{"</span><span class="p">,</span> <span class="s">"}"</span><span class="p">);</span>
<span class="p">}</span>
</pre></div>
</div>
<div class="section" id="lambdas">
<h3><a class="toc-backref" href="#id20">3: Lambdas</a></h3>
<p>This exercise essentially asks us to use lambdas to search for a value, in this case <code>42</code> in a list of integers.
I could not find what I was looking for in the exercise file itself. For a person who is not used to Kotlin's lambda
syntax this exercise might seem a little confusing. I happened to find out how to use lambdas in Kotlin from a
<a class="reference external" href="http://jamie.mccrindle.org/2013/01/exploring-kotlin-standard-library-part-1.html">blog post</a>. In the beginning,
I ended up with the following:</p>
<div class="highlight"><pre><span></span><span class="k">fun</span> <span class="nf">task3</span><span class="p">(</span><span class="n">collection</span><span class="p">:</span> <span class="n">Collection</span><span class="p"><</span><span class="n">Int</span><span class="p">>):</span> <span class="n">Boolean</span> <span class="p">{</span>
<span class="k">return</span> <span class="n">collection</span> <span class="n">map</span> <span class="p">{</span> <span class="n">x</span> <span class="p">-></span> <span class="n">x</span> <span class="p">%</span> <span class="m">42</span> <span class="p">==</span> <span class="m">0</span><span class="p">}</span> <span class="n">contains</span> <span class="k">true</span>
<span class="p">}</span>
</pre></div>
<p>Here, we learn about Kotlin's new syntax for lambdas. One does not <em>have</em> to use braces, but they are allowed:</p>
<div class="highlight"><pre><span></span><span class="k">fun</span> <span class="nf">task3</span><span class="p">(</span><span class="n">collection</span><span class="p">:</span> <span class="n">Collection</span><span class="p"><</span><span class="n">Int</span><span class="p">>):</span> <span class="n">Boolean</span> <span class="p">{</span>
<span class="k">return</span> <span class="n">collection</span><span class="p">.</span><span class="n">map</span><span class="p">({</span> <span class="n">x</span> <span class="p">-></span> <span class="n">x</span> <span class="p">%</span> <span class="m">42</span> <span class="p">==</span> <span class="m">0</span><span class="p">}).</span><span class="n">contains</span><span class="p">(</span><span class="k">true</span><span class="p">)</span>
<span class="p">}</span>
</pre></div>
<p>However, a friend of mine pointed out that <code>map</code> is not needed. Instead, <code>any</code> <a class="footnote-reference" href="#arigatou" id="id5">[4]</a>
<a class="footnote-reference" href="#lambdasreference" id="id6">[5]</a> is a better solution:</p>
<div class="highlight"><pre><span></span><span class="k">fun</span> <span class="nf">task3</span><span class="p">(</span><span class="n">collection</span><span class="p">:</span> <span class="n">Collection</span><span class="p"><</span><span class="n">Int</span><span class="p">>):</span> <span class="n">Boolean</span> <span class="p">{</span>
<span class="k">return</span> <span class="n">collection</span> <span class="n">any</span> <span class="p">{</span><span class="n">i</span> <span class="p">-></span> <span class="n">i</span> <span class="p">%</span> <span class="m">42</span> <span class="p">==</span> <span class="m">0</span><span class="p">}</span>
<span class="p">}</span>
</pre></div>
<p>But, one can go even further. Instead of declaring a lambda that takes in a parameter, one can use the implicit iterator
created, <code>it</code> shown in <code class="coderef">1</code> below:</p>
<div class="highlight"><pre><span></span><span class="k">fun</span> <span class="nf">task3</span><span class="p">(</span><span class="n">collection</span><span class="p">:</span> <span class="n">Collection</span><span class="p"><</span><span class="n">Int</span><span class="p">>):</span> <span class="n">Boolean</span> <span class="p">{</span>
<span class="c1">// ---------------------vv------------------+</span>
<span class="k">return</span> <span class="n">collection</span> <span class="n">any</span> <span class="p">{</span> <span class="n">it</span> <span class="p">%</span> <span class="m">42</span> <span class="p">==</span> <span class="m">0</span> <span class="p">}</span> <span class="c1">// 1</span>
<span class="p">}</span>
</pre></div>
</div>
<div class="section" id="string-templates">
<h3><a class="toc-backref" href="#id21">4: String templates</a></h3>
<p>In this exercise, we need to make a regex to match the patterns in the tests. The regex for months is already made:</p>
<div class="highlight"><pre><span></span><span class="k">val</span> <span class="py">month</span> <span class="p">=</span> <span class="s">"(JAN|FEB|MAR|APR|MAY|JUN|JUL|AUG|SEP|OCT|NOV|DEC)"</span>
</pre></div>
<p>All that we need to do is use this variable inside the <code>task4()</code> function like so:</p>
<div class="highlight"><pre><span></span><span class="c1">// ||| |||</span>
<span class="c1">// -------------------vvv----------------------------------------vvv</span>
<span class="k">fun</span> <span class="nf">task4</span><span class="p">():</span> <span class="n">String</span> <span class="p">=</span> <span class="s">"""(\w*) (\w*) \((\d{2}) ${month} (\d{4})\)"""</span>
</pre></div>
<p>Above, we stick in the month variable, we could have also used <code>$month</code> instead of <code>${month}</code>.
The string that we're using is triple quoted, or a multi-line string.The difference is that in a multi-line
string, special characters don't need to be escaped.</p>
</div>
<div class="section" id="null-safety">
<h3><a class="toc-backref" href="#id22">5: Null safety</a></h3>
<p>This exercise is meant to demonstrate Kotlin's safety features. We have to change the body of <code>sendMessageToClient</code> to
do the following:</p>
<ul class="simple">
<li>If <code>client</code> is <strong>not</strong> <code>null</code>, then we get its <code>personalInfo</code></li>
<li>If <code>personalInfo</code> is <strong>not</strong> <code>null</code>, then we get the <code>email</code> from it</li>
<li>If both <code>email</code> and <code>message</code> are <strong>not</strong> <code>null</code>, then we send a message using <code>mailer.sendMessage</code>.
Otherwise, we return.</li>
</ul>
<p>Initially, I ended up with the following:</p>
<div class="highlight"><pre><span></span><span class="k">fun</span> <span class="nf">sendMessageToClient</span><span class="p">(</span><span class="n">client</span><span class="p">:</span> <span class="n">Client</span><span class="p">?,</span> <span class="n">message</span><span class="p">:</span> <span class="n">String</span><span class="p">?,</span> <span class="n">mailer</span><span class="p">:</span> <span class="n">Mailer</span><span class="p">)</span> <span class="p">{</span>
<span class="k">val</span> <span class="py">personalInfo</span> <span class="p">=</span> <span class="n">client</span><span class="o">?.</span><span class="n">personalInfo</span> <span class="c1">// 1</span>
<span class="k">val</span> <span class="py">email</span> <span class="p">=</span> <span class="n">personalInfo</span><span class="o">?.</span><span class="n">email</span> <span class="c1">// 2</span>
<span class="k">if</span> <span class="p">(</span><span class="n">email</span> <span class="p">!=</span> <span class="k">null</span> <span class="p">&&</span> <span class="n">message</span> <span class="p">!=</span> <span class="k">null</span><span class="p">)</span> <span class="p">{</span>
<span class="n">mailer</span><span class="p">.</span><span class="n">sendMessage</span><span class="p">(</span><span class="n">email</span><span class="p">,</span> <span class="n">message</span><span class="p">)</span>
<span class="p">}</span> <span class="k">else</span> <span class="p">{</span>
<span class="k">return</span> <span class="c1">// 3</span>
<span class="p">}</span>
<span class="p">}</span>
</pre></div>
<p>I use two statements, <code class="coderef">1</code> and <code class="coderef">2</code> to get the data I needed. This However, can be done in a better way:</p>
<div class="highlight"><pre><span></span><span class="k">val</span> <span class="py">email</span> <span class="p">=</span> <span class="n">client</span><span class="o">?.</span><span class="n">personalInfo</span><span class="o">?.</span><span class="n">email</span>
</pre></div>
<p>In <code class="coderef">3</code>, we just have an empty <code>return</code> statement. When a function returns nothing, it is actually returning
<code>Unit</code>. <code>Unit</code> is a singleton and does not have to be return explicitly <a class="footnote-reference" href="#breslavunit" id="id7">[6]</a>. In other words, this function
will work just fine:</p>
<div class="highlight"><pre><span></span><span class="k">fun</span> <span class="nf">sendMessageToClient</span><span class="p">(</span><span class="n">client</span><span class="p">:</span> <span class="n">Client</span><span class="p">?,</span> <span class="n">message</span><span class="p">:</span> <span class="n">String</span><span class="p">?,</span> <span class="n">mailer</span><span class="p">:</span> <span class="n">Mailer</span><span class="p">)</span> <span class="p">{</span>
<span class="k">val</span> <span class="py">email</span> <span class="p">=</span> <span class="n">client</span><span class="o">?.</span><span class="n">personalInfo</span><span class="o">?.</span><span class="n">email</span>
<span class="k">if</span> <span class="p">(</span><span class="n">email</span> <span class="p">!=</span> <span class="k">null</span> <span class="p">&&</span> <span class="n">message</span> <span class="p">!=</span> <span class="k">null</span><span class="p">)</span> <span class="p">{</span>
<span class="n">mailer</span><span class="p">.</span><span class="n">sendMessage</span><span class="p">(</span><span class="n">email</span><span class="p">,</span> <span class="n">message</span><span class="p">)</span>
<span class="p">}</span>
<span class="p">}</span>
</pre></div>
<p>In the end, we need <code>email</code> and <code>message</code> to use <code>mailer.sendMessage</code>. Even in something like Python,
one would need multiple if statements to get this done, and plenty of <code>hasattr</code> calls.</p>
</div>
<div class="section" id="smart-casts-recursion">
<h3><a class="toc-backref" href="#id23">6: Smart Casts & Recursion</a></h3>
<p>In this exercise, we need to write a recursive function using Kotlin's <code>when</code> keyword. The basic gist of the problem
is that we have:</p>
<ul class="simple">
<li>An abstract class called <code>Expr</code></li>
<li>A class called <code>Num</code> that represents a number.</li>
<li>A class called <code>Sum</code> that represents the <em>sum</em> of two <code>Num</code> types.</li>
</ul>
<p>With this, we need to create a function that:</p>
<ul class="simple">
<li>Returns the <em>expression</em> of a summation or a number. For example:<ul>
<li><code>print(Num(2))</code> should return <code>"2"</code></li>
<li><code>print(Sum(Sum(Num(1), Num(2)), Num(3)))</code> should return <code>"1 + 2 + 3"</code></li>
</ul>
</li>
</ul>
<p>Hence, if we are given a <code>Sum</code> object, we need to call the function <em>again</em> breaking the sum down into two parts,
the <code>left</code> and the <code>right</code>, as shown in <code>Sum</code>'s function definition:</p>
<div class="highlight"><pre><span></span><span class="k">class</span> <span class="nc">Sum</span><span class="p">(</span><span class="k">val</span> <span class="py">left</span><span class="p">:</span> <span class="n">Expr</span><span class="p">,</span> <span class="k">val</span> <span class="py">right</span><span class="p">:</span> <span class="n">Expr</span><span class="p">)</span> <span class="p">:</span> <span class="n">Expr</span><span class="p">()</span>
</pre></div>
<p>With that in mind, we arrive at the following function, which sadly is called <code>print</code>:</p>
<div class="highlight"><pre><span></span><span class="k">fun</span> <span class="nf">print</span><span class="p">(</span><span class="n">expr</span><span class="p">:</span> <span class="n">Expr</span><span class="p">):</span> <span class="n">String</span> <span class="p">{</span> <span class="c1">// 1</span>
<span class="k">when</span> <span class="p">(</span><span class="n">expr</span><span class="p">)</span> <span class="p">{</span> <span class="c1">// 2</span>
<span class="k">is</span> <span class="n">Num</span> <span class="p">-></span> <span class="k">return</span> <span class="n">expr</span><span class="p">.</span><span class="n">value</span><span class="p">.</span><span class="n">toString</span><span class="p">()</span> <span class="c1">// 3</span>
<span class="k">is</span> <span class="n">Sum</span> <span class="p">-></span> <span class="k">return</span> <span class="n">print</span><span class="p">(</span><span class="n">expr</span><span class="p">.</span><span class="n">left</span><span class="p">)</span> <span class="p">+</span> <span class="s">" + "</span> <span class="p">+</span> <span class="n">print</span><span class="p">(</span><span class="n">expr</span><span class="p">.</span><span class="n">right</span><span class="p">)</span> <span class="c1">// 4</span>
<span class="k">else</span> <span class="p">-></span> <span class="k">throw</span> <span class="n">IllegalArgumentException</span><span class="p">(</span><span class="s">"Unknown expression"</span><span class="p">)</span> <span class="c1">// 5</span>
<span class="p">}</span>
<span class="p">}</span>
</pre></div>
<p>In <code class="coderef">1</code> we declare the function <code>print</code> that takes in a type of <code>Expr</code> and returns a <code>String</code>.
In <code class="coderef">2</code>, <code>when</code> feels like a switch statement. If <code>expr</code> is of instance <code>Num</code>, then we return the
value of <code>expr</code> as a <code>String</code> in <code class="coderef">3</code>.</p>
<p>However, if <code>expr</code> is of instance <code>Sum</code>, then we break down the <code>expr</code>, and call <code>print</code>
on its <code>left</code> and <code>right</code> properties in <code class="coderef">4</code>.</p>
<p>Otherwise in <code class="coderef">5</code>, we throw an <code>IllegalArgumentException</code> error because <code>expr</code> is neither
a <code>Sum</code> or <code>Num</code>.</p>
<p>In <code class="coderef">4</code>, we could have also used string templates instead, like so:</p>
<div class="highlight"><pre><span></span><span class="k">fun</span> <span class="nf">print</span><span class="p">(</span><span class="n">expr</span><span class="p">:</span> <span class="n">Expr</span><span class="p">):</span> <span class="n">String</span> <span class="p">{</span>
<span class="k">when</span> <span class="p">(</span><span class="n">expr</span><span class="p">)</span> <span class="p">{</span>
<span class="k">is</span> <span class="n">Num</span> <span class="p">-></span> <span class="k">return</span> <span class="n">expr</span><span class="p">.</span><span class="n">value</span><span class="p">.</span><span class="n">toString</span><span class="p">()</span>
<span class="c1">// vvvvvvvvvvvvvvvvvvv vvvvvvvvvvvvvvvvvvvv</span>
<span class="k">is</span> <span class="n">Sum</span> <span class="p">-></span> <span class="k">return</span> <span class="s">"${print(expr.left)} + ${print(expr.right)}"</span>
<span class="k">else</span> <span class="p">-></span> <span class="k">throw</span> <span class="n">IllegalArgumentException</span><span class="p">(</span><span class="s">"Unknown expression"</span><span class="p">)</span>
<span class="p">}</span>
<span class="p">}</span>
</pre></div>
<p>In this example, we do not need to explicitly cast <code>expr</code> as <code>Num</code> or <code>Sum</code>, like we have to do in
Java:</p>
<div class="highlight"><pre><span></span><span class="o">(...)</span>
<span class="k">if</span> <span class="o">(</span><span class="n">expr</span> <span class="k">instanceof</span> <span class="n">Num</span><span class="o">)</span> <span class="o">{</span>
<span class="k">return</span> <span class="s">""</span> <span class="o">+</span> <span class="o">((</span><span class="n">Num</span><span class="o">)</span> <span class="n">expr</span><span class="o">).</span><span class="na">getValue</span><span class="o">();</span>
<span class="o">}</span>
<span class="o">(...)</span>
</pre></div>
</div>
<div class="section" id="data-classes">
<h3><a class="toc-backref" href="#id24">7: Data Classes</a></h3>
<p>Data classes are like <code>case</code> classes in Scala, they're for storing data, and have a lot of useful helper functions
baked in, like <code>equals</code> and <code>hashCode</code> and the obligatory <code>toString</code>. In this exercise, we really don't
have a lot to do, we just have to read through some of the code samples. One thing I feel that the samples missed out on
showing that <code>data</code> classes can also have optional parameters. The following code:</p>
<div class="highlight"><pre><span></span><span class="k">data</span> <span class="k">class</span> <span class="nc">Account</span><span class="p">(</span><span class="k">val</span> <span class="py">name</span><span class="p">:</span> <span class="n">String</span><span class="p">,</span> <span class="k">val</span> <span class="py">account_id</span><span class="p">:</span> <span class="n">String</span><span class="p">,</span> <span class="k">val</span> <span class="py">active</span><span class="p">:</span> <span class="n">Boolean</span> <span class="p">=</span> <span class="k">true</span><span class="p">)</span>
<span class="k">fun</span> <span class="nf">main</span><span class="p">(</span><span class="n">args</span><span class="p">:</span> <span class="n">Array</span><span class="p"><</span><span class="n">String</span><span class="p">>)</span> <span class="p">{</span>
<span class="k">val</span> <span class="py">ac</span> <span class="p">=</span> <span class="n">Account</span><span class="p">(</span><span class="s">"Nafiul Islam"</span><span class="p">,</span> <span class="s">"A2231200ODSSDF4%%32123"</span><span class="p">)</span>
<span class="n">print</span><span class="p">(</span><span class="s">"${ac.name}, ${ac.account_id}, ${ac.active}"</span><span class="p">)</span>
<span class="p">}</span>
</pre></div>
<p>Outputs:</p>
<div class="highlight"><pre><span></span>Nafiul Islam, A2231200ODSSDF4%%32123, true
</pre></div>
</div>
<div class="section" id="extension-functions">
<h3><a class="toc-backref" href="#id25">8: Extension functions</a></h3>
<p>The Kotlin team came up with a pretty neat euphemism for what is essentially <em>monkey-patching</em>. Extension functions
allow you to <em>patch</em> a class with extra methods. For example:</p>
<div class="highlight"><pre><span></span><span class="k">fun</span> <span class="nf">String</span><span class="p">.</span><span class="n">lastChar</span><span class="p">()</span> <span class="p">=</span> <span class="k">this</span><span class="p">.</span><span class="n">charAt</span><span class="p">(</span><span class="k">this</span><span class="p">.</span><span class="n">length</span> <span class="p">-</span> <span class="m">1</span><span class="p">)</span>
<span class="k">fun</span> <span class="nf">main</span><span class="p">(</span><span class="n">args</span><span class="p">:</span> <span class="n">Array</span><span class="p"><</span><span class="n">String</span><span class="p">>)</span> <span class="p">{</span>
<span class="n">print</span><span class="p">(</span><span class="s">"Hello World!"</span><span class="p">.</span><span class="n">lastChar</span><span class="p">())</span> <span class="c1">// Outputs: !</span>
<span class="p">}</span>
</pre></div>
<p>This is pretty neat, we can actually monkey patch stdlib classes. So, without much further ado, we need to create
extension functions on <code>Int</code>, to allow two new constructor methods for a <code>RationalNumber</code>. One being,
<code>Int.r()</code>, which allows us to create a rational number like so:</p>
<div class="highlight"><pre><span></span><span class="m">4.</span><span class="n">r</span><span class="p">()</span> <span class="c1">// Outputs RationalNumber(4, 1), in essence 4/1</span>
</pre></div>
<p>As well as:</p>
<div class="highlight"><pre><span></span><span class="n">Pair</span><span class="p">(</span><span class="m">10</span><span class="p">,</span> <span class="m">20</span><span class="p">)</span> <span class="c1">// Outputs RationalNumber(10, 20), in essence 10/20</span>
</pre></div>
<p>This is one way of implementing such auxiliary constructors:</p>
<div class="highlight"><pre><span></span><span class="k">data</span> <span class="k">class</span> <span class="nc">RationalNumber</span><span class="p">(</span><span class="k">val</span> <span class="py">numerator</span><span class="p">:</span> <span class="n">Int</span><span class="p">,</span> <span class="k">val</span> <span class="py">denominator</span><span class="p">:</span> <span class="n">Int</span><span class="p">)</span> <span class="c1">// 1</span>
<span class="k">fun</span> <span class="nf">Int</span><span class="p">.</span><span class="n">r</span><span class="p">():</span> <span class="n">RationalNumber</span> <span class="p">{</span>
<span class="k">return</span> <span class="n">RationalNumber</span><span class="p">(</span><span class="k">this</span><span class="p">,</span> <span class="m">1</span><span class="p">)</span> <span class="c1">// 2</span>
<span class="p">}</span>
<span class="k">fun</span> <span class="nf">Pair</span><span class="p"><</span><span class="n">Int</span><span class="p">,</span> <span class="n">Int</span><span class="p">>.</span><span class="n">r</span><span class="p">():</span> <span class="n">RationalNumber</span> <span class="p">{</span>
<span class="k">return</span> <span class="n">RationalNumber</span><span class="p">(</span><span class="k">this</span><span class="p">.</span><span class="n">first</span><span class="p">,</span> <span class="k">this</span><span class="p">.</span><span class="n">second</span><span class="p">)</span> <span class="c1">// 3</span>
<span class="p">}</span>
</pre></div>
<p>In <code class="coderef">1</code>, we have declared our <code>RationalNumber</code> data class. This is already given to us in the exercise.</p>
<p>In <code class="coderef">2</code>, we return an a <code>RationalNumber</code>, the first parameter is <code>this</code>, and the second <code>1</code>.
Basically, creating <code>4/1</code>.</p>
<p>In <code class="coderef">3</code>, we extend <code>Pair</code>, this is an inbuilt class from the stdlib. In this case, we're extending a pair of
two <code>Int</code> variables. Code completion brought up <code>first</code> and <code>second</code> fields for <code>Pair</code>. However,
the documentation did not say it had these fields:</p>
<img alt="Information on Pair" class="align-center" src="/images/koans_11.png" />
<p>Hitting <kbd class="light">⌘</kbd> + <kbd class="light">B</kbd> <a class="footnote-reference" href="#gotosource" id="id8">[7]</a> brought this up inside <code>Tuples.kt</code> (which appears to be part of the stdlib):</p>
<div class="highlight"><pre><span></span><span class="k">public</span> <span class="k">data</span> <span class="k">class</span> <span class="nc">Pair</span><span class="p"><</span><span class="k">out</span> <span class="n">A</span><span class="p">,</span> <span class="k">out</span> <span class="n">B</span><span class="p">>(</span>
<span class="k">public</span> <span class="k">val</span> <span class="py">first</span><span class="p">:</span> <span class="n">A</span><span class="p">,</span> <span class="c1">// 1</span>
<span class="k">public</span> <span class="k">val</span> <span class="py">second</span><span class="p">:</span> <span class="n">B</span> <span class="c1">// 2</span>
<span class="p">)</span> <span class="p">:</span> <span class="n">Serializable</span> <span class="p">{</span>
<span class="k">override</span> <span class="k">fun</span> <span class="nf">toString</span><span class="p">():</span> <span class="n">String</span> <span class="p">=</span> <span class="s">"($first, $second)"</span>
<span class="p">}</span>
</pre></div>
<p>We can see that in <code class="coderef">1</code> and <code class="coderef">2</code>, <code>first</code> and <code>second</code> is declared as variables.</p>
</div>
<div class="section" id="extending-collections">
<h3><a class="toc-backref" href="#id26">9: Extending Collections</a></h3>
<p>In this exercise, we have to extend collections. Following the convention of the other exercises before, we need to
translate some Java code to Kotlin code. In this case, the Java code does the following:</p>
<ul class="simple">
<li>Group strings present in <code>collection</code> by length.</li>
<li>Find the maximum size of all the groups, meaning find the group that has the most number of members. Stored in
<code>maximumSizeOfGroup</code>.</li>
<li>Return the members of the group with the highest number of members.</li>
</ul>
<p>After deciphering the Java version of the program in <code>JavaCode9.java</code>, I managed to figure out what needed to be done.
Here are the results:</p>
<div class="highlight"><pre><span></span><span class="k">fun</span> <span class="nf">doSomethingStrangeWithCollection</span><span class="p">(</span><span class="n">collection</span><span class="p">:</span> <span class="n">Collection</span><span class="p"><</span><span class="n">String</span><span class="p">>):</span> <span class="n">Collection</span><span class="p"><</span><span class="n">String</span><span class="p">>?</span> <span class="p">{</span>
<span class="k">val</span> <span class="py">groupsByLength</span> <span class="p">=</span> <span class="n">collection</span><span class="p">.</span> <span class="n">groupBy</span> <span class="p">{</span> <span class="n">s</span> <span class="p">-></span> <span class="n">s</span><span class="p">.</span><span class="n">length</span> <span class="p">}</span>
<span class="k">val</span> <span class="py">maximumSizeOfGroup</span> <span class="p">=</span> <span class="n">groupsByLength</span><span class="p">.</span> <span class="n">values</span><span class="p">().</span> <span class="n">map</span> <span class="p">{</span> <span class="n">group</span> <span class="p">-></span> <span class="n">group</span><span class="p">.</span><span class="n">size</span> <span class="p">}.</span> <span class="n">max</span><span class="p">()</span>
<span class="k">return</span> <span class="n">groupsByLength</span><span class="p">.</span> <span class="n">values</span><span class="p">().</span> <span class="n">firstOrNull</span> <span class="p">{</span> <span class="n">group</span> <span class="p">-></span> <span class="n">group</span><span class="p">.</span><span class="n">size</span> <span class="p">==</span> <span class="n">maximumSizeOfGroup</span> <span class="p">}</span>
<span class="p">}</span>
</pre></div>
</div>
<div class="section" id="object-expressions">
<h3><a class="toc-backref" href="#id27">10: Object Expressions</a></h3>
<p>This exercise shows us how to modify object instances on the fly. In this case, we need to modify <code>MouseAdapter</code>
to register the number of clicks that it receives. So, following the examples, we can see that we need to use
the following syntax:</p>
<div class="highlight"><pre><span></span><span class="k">val</span> <span class="py">foo</span><span class="p">:</span> <span class="n">Foo</span> <span class="p">=</span> <span class="k">object</span> <span class="err">: </span><span class="nc">Foo</span><span class="p">()</span> <span class="p">{</span>
<span class="k">override</span> <span class="k">fun</span> <span class="nf">foo</span><span class="p">()</span> <span class="p">{</span>
<span class="c1">// Code goes here</span>
<span class="p">}</span>
<span class="p">}</span>
</pre></div>
<p>With that set, lets modify our own instance of <code>MouseAdapter</code>:</p>
<div class="highlight"><pre><span></span><span class="k">fun</span> <span class="nf">task10</span><span class="p">(</span><span class="n">handleMouse</span><span class="p">:</span> <span class="p">(</span><span class="n">MouseListener</span><span class="p">)</span> <span class="p">-></span> <span class="n">Unit</span><span class="p">):</span> <span class="n">Int</span> <span class="p">{</span>
<span class="k">var</span> <span class="py">mouseClicks</span> <span class="p">=</span> <span class="m">0</span>
<span class="k">val</span> <span class="py">customAdapter</span> <span class="p">:</span> <span class="n">MouseAdapter</span> <span class="p">=</span> <span class="k">object</span> <span class="err">: </span><span class="nc">MouseAdapter</span><span class="p">()</span> <span class="p">{</span>
<span class="k">override</span> <span class="k">fun</span> <span class="nf">mouseClicked</span><span class="p">(</span><span class="n">e</span><span class="p">:</span> <span class="n">MouseEvent</span><span class="p">)</span> <span class="p">{</span> <span class="c1">// 1</span>
<span class="k">super</span><span class="p"><</span><span class="n">MouseAdapter</span><span class="p">>.</span><span class="n">mouseClicked</span><span class="p">(</span><span class="n">e</span><span class="p">)</span> <span class="c1">// 2</span>
<span class="n">mouseClicks</span> <span class="p">+=</span> <span class="m">1</span> <span class="c1">// 3</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="n">handleMouse</span><span class="p">(</span><span class="n">customAdapter</span><span class="p">)</span> <span class="c1">// 4</span>
<span class="k">return</span> <span class="n">mouseClicks</span> <span class="c1">// 5</span>
<span class="p">}</span>
</pre></div>
<p>In <code class="coderef">1</code> we override the function that handles mouse clicks. In <code class="coderef">2</code>, we make a call to the <code>super</code> function. When
doing this, I did not know how to make this call and found it well documented in the reference <a class="footnote-reference" href="#classesandinheritance" id="id9">[8]</a>.</p>
<p>In <code class="coderef">3</code>, we increase the number of mouse clicks in our <code>mouseClicks</code> variable. In <code class="coderef">4</code>, we register the handler
and finally in <code class="coderef">5</code>, we return the number of mouse clicks.</p>
</div>
</div>
<div class="section" id="takeaway">
<h2><a class="toc-backref" href="#id28">Takeaway</a></h2>
<p>The Kotlin Koans have a lot of material packed in. If you're coming from a language other than Java, it might be a
little difficult to understand what is going on. I wish it had more language agnostic instructions, as Kotlin is
targeting JavaScript platforms as well.</p>
<p>What I've learnt from the Introduction is that Kotlin is a ridiculously powerful language. Some of its features can
even make Python sweat. The parts I liked most were lambdas, null safety features and extension functions.</p>
<p>In total there are six parts to this series. We've just completed the first. I just can't wait for more :)</p>
</div>
<div class="section" id="acknowledgements">
<h2><a class="toc-backref" href="#id29">Acknowledgements</a></h2>
<p>A big thanks to the following people:</p>
<ul class="simple">
<li>Taro Nagasawa</li>
<li>Keiron Pizzey</li>
<li>Ilter Canberk</li>
<li>Sahib Bin Mahboob</li>
<li>Jeong Min-Lee</li>
<li>Ashwini Chaudhury</li>
<li>Hadi Hariri</li>
</ul>
<hr class="docutils" />
<table class="docutils footnote" frame="void" id="mistake" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#id1">[1]</a></td><td>If I've made a mistake in this list, or if you have any other suggestions that I've missed out, please do not hesitate to add.</td></tr>
</tbody>
</table>
<table class="docutils footnote" frame="void" id="hadi" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#id2">[2]</a></td><td>Hadi Hariri is an evangelist at Jetbrains.</td></tr>
</tbody>
</table>
<table class="docutils footnote" frame="void" id="dash" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#id4">[3]</a></td><td>Dash doctests are also supported by other tools on different platforms. Check out <a class="reference external" href="http://zealdocs.org/">Zeal</a>.</td></tr>
</tbody>
</table>
<table class="docutils footnote" frame="void" id="arigatou" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#id5">[4]</a></td><td>有難うございます。(Thank you very much) to Taro Nagasawa for pointing this out to me :)</td></tr>
</tbody>
</table>
<table class="docutils footnote" frame="void" id="lambdasreference" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#id6">[5]</a></td><td><div class="first last line-block">
<div class="line">Implicit iterators talked about towards the end of "Higher Order Functions"</div>
<div class="line"><a class="reference external" href="http://kotlinlang.org/docs/reference/lambdas.html">http://kotlinlang.org/docs/reference/lambdas.html</a></div>
</div>
</td></tr>
</tbody>
</table>
<table class="docutils footnote" frame="void" id="breslavunit" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#id7">[6]</a></td><td><div class="first last line-block">
<div class="line">Andrey Breslav, the lead designer of the language commented on Unit</div>
<div class="line"><a class="reference external" href="https://confluence.jetbrains.com/display/Kotlin/Functions?focusedCommentId=40704501#comment-40704501">https://confluence.jetbrains.com/display/Kotlin/Functions?focusedCommentId=40704501#comment-40704501</a></div>
</div>
</td></tr>
</tbody>
</table>
<table class="docutils footnote" frame="void" id="gotosource" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#id8">[7]</a></td><td>For Win/Linux, one can use <kbd class="light">CTRL</kbd> + <kbd class="light">B</kbd>, in essence "Go to Source".</td></tr>
</tbody>
</table>
<table class="docutils footnote" frame="void" id="classesandinheritance" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#id9">[8]</a></td><td>In the section called "Overriding Rules" <a class="reference external" href="http://kotlinlang.org/docs/reference/classes.html#overriding-rules">http://kotlinlang.org/docs/reference/classes.html#overriding-rules</a></td></tr>
</tbody>
</table>
</div>
Making dash doctests using sphinx and doc2dash2014-10-24T11:16:00+06:002014-10-24T11:16:00+06:00Nafiul Islamtag:nafiulis.me,2014-10-24:making-dash-doctests-using-sphinx-and-doc2dash.html<p>Dash is an offline documentation search engine. I've been using <a class="reference external" href="http://kapeli.com/dash">Dash</a> a lot recently.
It has very fast search and supports many libraries out there.
<a class="reference external" href="zealdocs.org/">Zeal</a> is a free and ope source alternative to Dash that uses Dash doctests for offline documentation
that supports all platforms. Other platform specific tools also exist such as
<a class="reference external" href="http://velocity.silverlakesoftware.com/">Velocity</a> for Windows and <a class="reference external" href="http://lovelydocs.io/">LovelyDocs</a> for
Android.</p>
<p>Dash supports many libraries however, libraries like <tt class="docutils literal">Fabric</tt> and <tt class="docutils literal">requests</tt> are not officially supported; their
dash compatible doctests can be generated though. So, I thought I'd make a quick tutorial on how to generate doctests
for third party libraries:</p>
<div class="external youtube"><iframe width="640" height="360" src="//www.youtube.com/embed/_B6drt8JEH4" frameborder="0"></iframe></div>A Todo app with flask and Pony2014-10-09T12:30:00+06:002014-10-09T12:30:00+06:00Nafiul Islamtag:nafiulis.me,2014-10-09:a-todo-app-with-flask-and-pony.html<div class="contents topic" id="table-of-contents">
<p class="topic-title first">Table of Contents</p>
<ul class="simple">
<li><a class="reference internal" href="#installation" id="id1">Installation</a></li>
<li><a class="reference internal" href="#routes" id="id2">Routes</a></li>
<li><a class="reference internal" href="#using-pony" id="id3">Using Pony</a><ul>
<li><a class="reference internal" href="#querying" id="id4">Querying</a></li>
<li><a class="reference internal" href="#creating-objects" id="id5">Creating Objects</a></li>
<li><a class="reference internal" href="#deleting-objects" id="id6">Deleting Objects</a></li>
</ul>
</li>
<li><a class="reference internal" href="#references" id="id7">References</a></li>
<li><a class="reference internal" href="#update" id="id8">Update</a></li>
<li><a class="reference internal" href="#p-s" id="id9">P.S.</a></li>
</ul>
</div>
<p><a class="reference external" href="http://ponyorm.com/">Pony</a> is a new ORM that has some very nice features especially when it comes to querying. It's
still in the works, missing features like migration and <code>py3</code> support (although version <code>0.6</code> supports <code>py3</code>
and already has a release candidate), but the creators tell me that both these features will be made available by
the end of this year. So, I decided to make a simple todo app with pony, <a class="reference external" href="http://flask.pocoo.org/">flask</a> and
<a class="reference external" href="http://flask-restful.readthedocs.org/en/latest/">flask-restful</a>. You can get the <a class="reference external" href="https://github.com/gamesbrainiac/Pony-Todo-API/tree/83287fc4fd99ebb6dc9e613571695a8c835c46b9">source code</a> from github.</p>
<div class="section" id="installation">
<h2><a class="toc-backref" href="#id1">Installation</a></h2>
<p>First, we need to set up our project and install a couple of requirements:</p>
<div class="highlight"><pre><span></span>pip install flask flask-restful pony
</pre></div>
</div>
<div class="section" id="routes">
<h2><a class="toc-backref" href="#id2">Routes</a></h2>
<p>We are going to be making an API for out todo list. A todo item will have three things:</p>
<ul class="simple">
<li><code>id</code>, the primary key</li>
<li><code>data</code>, what the item is about e.g. write a tutorial on pony</li>
<li><code>tags</code>, what type of item it is, e.g. work, important, home</li>
</ul>
<p>In order to do this, lets create our routes:</p>
<ul class="simple">
<li><code>/</code> will show you all the todo items on a <code>GET</code> request. One can add new items with a <code>PUT</code> request.</li>
<li><code>/<int:todo_id></code> will allow a <code>GET</code> request, and a <code>DELETE</code> request.
that shows you information regarding the todo item with that <code>id</code>.</li>
<li><code>/tags/</code>, will show you all the tags that are available, with links to those tag elements</li>
<li><code>/tags/<int:tag_id></code> will show you the corresponding tag.</li>
</ul>
<p>Thus, a <code>tag</code> must have an <code>id</code> and a value, which will likely be a string. Lets get to making our routes
then:</p>
<div class="highlight"><pre><span></span><span class="c1"># Inside app.py</span>
<span class="kn">from</span> <span class="nn">flask</span> <span class="kn">import</span> <span class="n">Flask</span>
<span class="kn">import</span> <span class="nn">flask.ext.restful</span> <span class="kn">as</span> <span class="nn">rest</span>
<span class="n">app</span> <span class="o">=</span> <span class="n">Flask</span><span class="p">(</span><span class="vm">__name__</span><span class="p">)</span>
<span class="n">api</span> <span class="o">=</span> <span class="n">rest</span><span class="o">.</span><span class="n">Api</span><span class="p">(</span><span class="n">app</span><span class="p">)</span>
<span class="c1"># Resource ######################################################################</span>
<span class="k">class</span> <span class="nc">Todos</span><span class="p">(</span><span class="n">rest</span><span class="o">.</span><span class="n">Resource</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">get</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="sd">"""Will give you all the todo items"""</span>
<span class="k">return</span> <span class="p">{}</span>
<span class="k">def</span> <span class="nf">put</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="sd">"""Payload contains information to create new todo item"""</span>
<span class="k">return</span> <span class="p">{}</span>
<span class="k">class</span> <span class="nc">TodoItem</span><span class="p">(</span><span class="n">rest</span><span class="o">.</span><span class="n">Resource</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">get</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">todo_id</span><span class="p">):</span>
<span class="sd">"""</span>
<span class="sd"> Get specific information on a Todo item</span>
<span class="sd"> :param todo_id: The Todo Item's ID, which is unique and a primary key</span>
<span class="sd"> :type todo_id: int</span>
<span class="sd"> """</span>
<span class="k">return</span> <span class="p">{}</span>
<span class="k">class</span> <span class="nc">Tags</span><span class="p">(</span><span class="n">rest</span><span class="o">.</span><span class="n">Resource</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">get</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="sd">"""Will show you all tags"""</span>
<span class="k">return</span> <span class="p">{}</span>
<span class="k">class</span> <span class="nc">TagItem</span><span class="p">(</span><span class="n">rest</span><span class="o">.</span><span class="n">Resource</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">get</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">tag_id</span><span class="p">):</span>
<span class="sd">"""</span>
<span class="sd"> Will show you information about a specific tag</span>
<span class="sd"> :param tag_id: ID for the tag</span>
<span class="sd"> :type tag_id: int</span>
<span class="sd"> """</span>
<span class="k">return</span> <span class="p">{}</span>
<span class="c1"># Routes #######################################################################</span>
<span class="n">api</span><span class="o">.</span><span class="n">add_resource</span><span class="p">(</span><span class="n">Todos</span><span class="p">,</span> <span class="s1">'/'</span><span class="p">,</span> <span class="n">endpoint</span><span class="o">=</span><span class="s1">'Home'</span><span class="p">)</span>
<span class="n">api</span><span class="o">.</span><span class="n">add_resource</span><span class="p">(</span><span class="n">TodoItem</span><span class="p">,</span> <span class="s1">'/<int:todo_id>'</span><span class="p">,</span> <span class="n">endpoint</span><span class="o">=</span><span class="s1">'TodoItem'</span><span class="p">)</span>
<span class="n">api</span><span class="o">.</span><span class="n">add_resource</span><span class="p">(</span><span class="n">Tags</span><span class="p">,</span> <span class="s1">'/tags/'</span><span class="p">,</span> <span class="n">endpoint</span><span class="o">=</span><span class="s1">'Tags'</span><span class="p">)</span>
<span class="n">api</span><span class="o">.</span><span class="n">add_resource</span><span class="p">(</span><span class="n">TagItem</span><span class="p">,</span> <span class="s1">'/tags/<int:tag_id>'</span><span class="p">,</span> <span class="n">endpoint</span><span class="o">=</span><span class="s1">'TagItem'</span><span class="p">)</span>
<span class="k">if</span> <span class="vm">__name__</span> <span class="o">==</span> <span class="s1">'__main__'</span><span class="p">:</span>
<span class="n">app</span><span class="o">.</span><span class="n">run</span><span class="p">(</span><span class="n">debug</span><span class="o">=</span><span class="bp">True</span><span class="p">)</span>
</pre></div>
<p>The <code>get</code> and <code>put</code> methods inside the <code>rest.Resource</code> objects indicate the types of methods that
you can use to interact with that route. The <code>api.add_resource(...)</code> function allows you to register the API
handler, i.e. a <code>rest.Resource</code> object and a route. the <code>endpoint</code> is optional.</p>
<p>The <code>endpoint</code> is an optional parameter and gives its route a reference, which is useful for redirection.</p>
</div>
<div class="section" id="using-pony">
<h2><a class="toc-backref" href="#id3">Using Pony</a></h2>
<p>Right now, all the API will return are empty json objects. Before we actually start returning stuff,
lets make our models:</p>
<div class="highlight"><pre><span></span><span class="c1"># Inside models.py</span>
<span class="kn">from</span> <span class="nn">pony</span> <span class="kn">import</span> <span class="n">orm</span> <span class="c1"># 1</span>
<span class="n">db</span> <span class="o">=</span> <span class="n">orm</span><span class="o">.</span><span class="n">Database</span><span class="p">()</span> <span class="c1"># 2</span>
<span class="k">class</span> <span class="nc">Todo</span><span class="p">(</span><span class="n">db</span><span class="o">.</span><span class="n">Entity</span><span class="p">):</span> <span class="c1"># 3</span>
<span class="n">_table_</span> <span class="o">=</span> <span class="s1">'Todos'</span> <span class="c1"># 4</span>
<span class="n">data</span> <span class="o">=</span> <span class="n">orm</span><span class="o">.</span><span class="n">Required</span><span class="p">(</span><span class="nb">unicode</span><span class="p">)</span> <span class="c1"># 5</span>
<span class="n">tags</span> <span class="o">=</span> <span class="n">orm</span><span class="o">.</span><span class="n">Set</span><span class="p">(</span><span class="s2">"Tag"</span><span class="p">)</span> <span class="c1"># 6</span>
<span class="k">class</span> <span class="nc">Tag</span><span class="p">(</span><span class="n">db</span><span class="o">.</span><span class="n">Entity</span><span class="p">):</span>
<span class="n">_table_</span> <span class="o">=</span> <span class="s1">'Tags'</span>
<span class="n">name</span> <span class="o">=</span> <span class="n">orm</span><span class="o">.</span><span class="n">Required</span><span class="p">(</span><span class="nb">unicode</span><span class="p">,</span> <span class="n">unique</span><span class="o">=</span><span class="bp">True</span><span class="p">)</span> <span class="c1"># 7</span>
<span class="n">tags</span> <span class="o">=</span> <span class="n">orm</span><span class="o">.</span><span class="n">Set</span><span class="p">(</span><span class="s2">"Todo"</span><span class="p">)</span> <span class="c1"># 8</span>
</pre></div>
<p>The first thing we do is import <code>pony.orm</code> in <code class="coderef">1</code> . In most of the documentation,
they import everything via <code>from pony.orm import *</code>, but since I like namespaces,
I prefer using <code>orm</code>. <code>pony.orm</code> houses everything you need to create and query objects.</p>
<p>We initialize the database using <code>orm.Database()</code> in <code class="coderef">2</code>.
This instance of the <code>pony.orm.Database</code> class stores all the data needed to create the tables that your models
define.</p>
<p>In <code class="coderef">3</code>, we create an <code>Entity</code> object by inheriting from <code>db.Entity</code>. Note that we use the <code>db</code>
instance, and not any import from <code>pony.orm</code>. Its similar to SQLAlchemy's <code>sqlalchemy.schema.MetaData</code>
object.</p>
<p>In <code class="coderef">4</code>, we set the table name to <code>Todos</code>, its a lot like <code>__tablename__</code> in SQLAlchemy.</p>
<p>In Pony, you can have either <code>Required</code> columns or <code>Optional</code> columns <a class="reference external" href="http://doc.ponyorm.com/entities.html#entity-attributes">(described here)</a>. In <code class="coderef">5</code>, we set <code>data</code> to be a <code>Required</code>
field of type <code>unicode</code>. Note that this <code>unicode</code> class is not from the <code>pony.orm</code> namespace.</p>
<p>You can also have many-to-many types in pony, using <code>Set</code> in <code class="coderef">6</code> and <code class="coderef">8</code>. In <code class="coderef">7</code>,
we set <code>unique</code> to be <code>True</code>. You have a couple of optional attributes that you can add to
<code>Required</code> or <code>Optional</code> fields.
<a class="reference external" href="http://doc.ponyorm.com/entities.html#required-and-optional">(Required and Optional Explained in more detail)</a></p>
<p>Notices any <code>id</code> anywhere? No? Well, thats because pony implicitly creates the <code>id</code> column for you.
<a class="reference external" href="http://doc.ponyorm.com/firststeps.html#defining-entities">(More about implicit ID creation)</a></p>
<p>Now, in <code>app.py</code> we need to bind a database and then
generate mappings:</p>
<div class="highlight"><pre><span></span><span class="c1"># Inside app.py</span>
<span class="p">(</span><span class="o">...</span><span class="p">)</span>
<span class="kn">from</span> <span class="nn">models</span> <span class="kn">import</span> <span class="o">*</span>
<span class="n">app</span> <span class="o">=</span> <span class="n">Flask</span><span class="p">(</span><span class="vm">__name__</span><span class="p">)</span>
<span class="n">api</span> <span class="o">=</span> <span class="n">rest</span><span class="o">.</span><span class="n">Api</span><span class="p">(</span><span class="n">app</span><span class="p">)</span>
<span class="n">db</span><span class="o">.</span><span class="n">bind</span><span class="p">(</span><span class="s1">'sqlite'</span><span class="p">,</span> <span class="s1">'todo_api.db'</span><span class="p">,</span> <span class="n">create_db</span><span class="o">=</span><span class="bp">True</span><span class="p">)</span>
<span class="n">db</span><span class="o">.</span><span class="n">generate_mapping</span><span class="p">(</span><span class="n">create_tables</span><span class="o">=</span><span class="bp">True</span><span class="p">)</span>
<span class="c1"># Resource #####################################################################</span>
<span class="p">(</span><span class="o">...</span><span class="p">)</span>
</pre></div>
<p>We first bind the objects, in this case <code>Tag</code> and <code>Todo</code> to the sqlite database.
<a class="reference external" href="http://doc.ponyorm.com/database.html#binding-the-database-object-to-a-specific-database">(More on binding databases)</a></p>
<p>We set <code>create_tables</code> and <code>create_db</code> to <code>True</code> because this is a new database,
and no tables have been created yet.
<a class="reference external" href="http://doc.ponyorm.com/database.html?highlight=bind#mapping-entities-to-the-database-tables">(More information regarding creating tables)</a></p>
<p><code>generate_mapping</code> generates all the SQL needed to create the tables. If you actually set
<code>orm.sql_debug(True)</code>, then all the SQL that is being generated will be logged into your console.</p>
<p>These are what the generated tables look like:</p>
<img alt="What the generated tables look like in Pony" class="align-center" src="images/pony_01.png" />
<div class="section" id="querying">
<h3><a class="toc-backref" href="#id4">Querying</a></h3>
<p>Lets get to finding stuff from the database. First, we need to show all the todo items in our database for our
<code>Todos.get</code> function:</p>
<div class="highlight"><pre><span></span><span class="c1"># Inside app.py</span>
<span class="k">class</span> <span class="nc">Todos</span><span class="p">(</span><span class="n">rest</span><span class="o">.</span><span class="n">Resource</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">get</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="sd">"""Will give you all the todo items"""</span>
<span class="k">with</span> <span class="n">orm</span><span class="o">.</span><span class="n">db_session</span><span class="p">:</span> <span class="c1"># 1</span>
<span class="k">return</span> <span class="p">{</span>
<span class="n">item</span><span class="o">.</span><span class="n">id</span><span class="p">:</span> <span class="p">{</span>
<span class="s1">'task'</span><span class="p">:</span> <span class="n">item</span><span class="o">.</span><span class="n">data</span><span class="p">,</span>
<span class="s1">'tags'</span><span class="p">:</span> <span class="p">[</span><span class="n">tag</span><span class="o">.</span><span class="n">id</span> <span class="k">for</span> <span class="n">tag</span> <span class="ow">in</span> <span class="n">item</span><span class="o">.</span><span class="n">tags</span><span class="p">]</span> <span class="c1"># 3</span>
<span class="p">}</span>
<span class="k">for</span> <span class="n">item</span> <span class="ow">in</span> <span class="n">Todo</span><span class="o">.</span><span class="n">select</span><span class="p">()</span> <span class="c1"># 2</span>
<span class="p">}</span>
</pre></div>
<p>We are using a context manager, <code>orm.db_session</code> in <code class="coderef">1</code>. This treats everything that pony does within the
code-block as a transaction, you don't have to worry about committing. We are going to return a list of all the items
according to their <code>id</code>, we have the <code>'task'</code> value in the return dictionary provide the information
regarding the <code>Todo</code> item. As for <code>'tags'</code> we loop through the values in the item's tags and give the
<code>id</code>. Note that in <code class="coderef">2</code>, we are looping through <em>all</em> the <code>Todo</code> items in the database, <code>Todo.select()</code>
is the same as <code>orm.select(tdo for tdo in Todo)</code>.</p>
<p>However, the <code>id</code> of the tags alone are not enough information about the tags associated with a certain
<code>Todo</code> item, it would be better if we could generate links to the tag pages. We can actually add a
<code>property</code> inside <code>Tag</code> as <code>url</code>:</p>
<div class="highlight"><pre><span></span><span class="c1"># Inside models.py</span>
<span class="p">(</span><span class="o">...</span><span class="p">)</span>
<span class="nd">@property</span>
<span class="k">def</span> <span class="nf">url</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">return</span> <span class="s2">"http://localhost:5000/tags/{}"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">id</span><span class="p">)</span>
<span class="p">(</span><span class="o">...</span><span class="p">)</span>
</pre></div>
<p>We use <code>property</code> because we will be able to access it as if it were any other attribute.</p>
<p>Now that we have added this property, we can call the <code>url</code> property instead of the <code>id</code>:</p>
<div class="highlight"><pre><span></span><span class="c1"># Inside app.py</span>
<span class="p">(</span><span class="o">...</span><span class="p">)</span>
<span class="s1">'tags'</span><span class="p">:</span> <span class="p">[</span><span class="n">tag</span><span class="o">.</span><span class="n">url</span> <span class="k">for</span> <span class="n">tag</span> <span class="ow">in</span> <span class="n">item</span><span class="o">.</span><span class="n">tags</span><span class="p">]</span> <span class="c1"># 3</span>
<span class="p">(</span><span class="o">...</span><span class="p">)</span>
</pre></div>
<p>Now that this is done, lets send a <code>GET</code> request to the <code>/</code> url:</p>
<div class="highlight"><pre><span></span>quazinafiulislam@Nafiuls-Mac: ~
$ http GET http://localhost:5000/
HTTP/1.0 <span class="m">200</span> OK
Content-Length: <span class="m">3</span>
Content-Type: application/json
Date: Thu, <span class="m">09</span> Oct <span class="m">2014</span> <span class="m">10</span>:32:00 GMT
Server: Werkzeug/0.9.6 Python/2.7.6
<span class="o">{}</span>
</pre></div>
</div>
<div class="section" id="creating-objects">
<h3><a class="toc-backref" href="#id5">Creating Objects</a></h3>
<p>So that (probably) works! We won't know until we've added a <code>Todo</code> item. We haven't inserted anything into our
database yet so lets do that. We are going to be accepted <code>PUT</code> requests for our <code>/</code> route. In order to
do this, we need to import the <code>json</code> module, as well as <code>flask.request</code>:</p>
<div class="highlight"><pre><span></span><span class="c1"># Inside app.py</span>
<span class="kn">import</span> <span class="nn">json</span> <span class="c1"># New import</span>
<span class="kn">from</span> <span class="nn">flask</span> <span class="kn">import</span> <span class="n">Flask</span><span class="p">,</span> <span class="n">request</span> <span class="c1"># New import</span>
<span class="kn">import</span> <span class="nn">flask.ext.restful</span> <span class="kn">as</span> <span class="nn">rest</span>
<span class="p">(</span><span class="o">...</span><span class="p">)</span>
</pre></div>
<p>Lets start by working on our <code>Todos.put</code> function. This is how the data is going to come in:</p>
<div class="highlight"><pre><span></span><span class="p">{</span>
<span class="nt">"data"</span><span class="p">:</span> <span class="s2">"Buy Milk!"</span><span class="p">,</span>
<span class="nt">"tags"</span><span class="p">:</span> <span class="p">[</span><span class="s2">"work"</span><span class="p">,</span> <span class="s2">"high"</span><span class="p">]</span>
<span class="p">}</span>
</pre></div>
<p>In order to make this function work, we need to do a couple of things:</p>
<ul class="simple">
<li>Extract <code>data</code> and <code>tags</code> from the json request.</li>
<li>Create a new <code>Todo</code> item.</li>
<li>Create new tags, if tags exist, then you need to use them instead of creating new ones.</li>
</ul>
<p>In order to fulfill those roles, this is the function I ended up with:</p>
<div class="highlight"><pre><span></span><span class="c1"># Inside app.py</span>
<span class="p">(</span><span class="o">...</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">put</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="sd">"""Payload contains information to create new todo item"""</span>
<span class="n">info</span> <span class="o">=</span> <span class="n">json</span><span class="o">.</span><span class="n">loads</span><span class="p">(</span><span class="n">request</span><span class="o">.</span><span class="n">data</span><span class="p">)</span> <span class="c1"># 1</span>
<span class="k">with</span> <span class="n">orm</span><span class="o">.</span><span class="n">db_session</span><span class="p">:</span> <span class="c1"># 2</span>
<span class="n">item</span> <span class="o">=</span> <span class="n">Todo</span><span class="p">(</span><span class="n">data</span><span class="o">=</span><span class="n">info</span><span class="p">[</span><span class="s1">'data'</span><span class="p">])</span> <span class="c1"># 3</span>
<span class="k">for</span> <span class="n">tag_name</span> <span class="ow">in</span> <span class="n">info</span><span class="p">[</span><span class="s1">'tags'</span><span class="p">]:</span> <span class="c1"># 4</span>
<span class="n">tag</span> <span class="o">=</span> <span class="n">Tag</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">name</span><span class="o">=</span><span class="n">tag_name</span><span class="p">)</span> <span class="c1"># 5</span>
<span class="k">if</span> <span class="n">tag</span> <span class="ow">is</span> <span class="bp">None</span><span class="p">:</span> <span class="c1"># 6</span>
<span class="n">tag</span> <span class="o">=</span> <span class="n">Tag</span><span class="p">(</span><span class="n">name</span><span class="o">=</span><span class="n">tag_name</span><span class="p">)</span> <span class="c1"># 7</span>
<span class="n">item</span><span class="o">.</span><span class="n">tags</span> <span class="o">+=</span> <span class="n">tag</span> <span class="c1"># 8</span>
<span class="k">return</span> <span class="p">{},</span> <span class="mi">200</span> <span class="c1"># 9</span>
<span class="p">(</span><span class="o">...</span><span class="p">)</span>
</pre></div>
<p>In <code class="coderef">1</code> we are loading the data from <code>request.data</code>, and turning that json string into a python dictionary.
After doing so, we start our database session by using the <code>orm.db_session</code> context manager in <code class="coderef">2</code>.</p>
<p>We start off in <code class="coderef">3</code> by creating a new <code>Todo</code> item. Note that our arguments are named/keyword arguments. We now
loop over all the values in values in <code>info['tags']</code> using <code>tag_name</code> as the temporary variable in <code class="coderef">4</code>.</p>
<p>In <code class="coderef">5</code>, we try to <em>retrieve</em> the tag if it exists in the database. If it doesn't <code class="coderef">6</code>,
then we create a new one in <code class="coderef">7</code>. We then append this tag to the item's list of tags in <code class="coderef">8</code>.</p>
<p>In <code class="coderef">9</code> we return an empty json object, with a 200 code, to say that everything happened smoothly. Note that in this
entire function, it feels like there's <em>no database</em>. I just find this amazing that I feel as if I'm working with
native python object stored in memory.</p>
<div class="highlight"><pre><span></span><span class="nx">quazinafiulislam</span><span class="err">@</span><span class="nx">Nafiuls</span><span class="o">-</span><span class="nx">Mac</span><span class="o">:</span> <span class="o">~</span>
<span class="nx">$</span> <span class="nx">http</span> <span class="nx">PUT</span> <span class="nx">http</span><span class="o">:</span><span class="c1">//localhost:5000/ data="Buy rice" tags:='["work", "high"]'</span>
<span class="nx">HTTP</span><span class="o">/</span><span class="mf">1.0</span> <span class="mi">200</span> <span class="nx">OK</span>
<span class="nx">Content</span><span class="o">-</span><span class="nx">Length</span><span class="o">:</span> <span class="mi">3</span>
<span class="nx">Content</span><span class="o">-</span><span class="nx">Type</span><span class="o">:</span> <span class="nx">application</span><span class="o">/</span><span class="nx">json</span>
<span class="nb">Date</span><span class="o">:</span> <span class="nx">Thu</span><span class="p">,</span> <span class="mi">09</span> <span class="nx">Oct</span> <span class="mi">2014</span> <span class="mi">11</span><span class="o">:</span><span class="mi">36</span><span class="o">:</span><span class="mi">04</span> <span class="nx">GMT</span>
<span class="nx">Server</span><span class="o">:</span> <span class="nx">Werkzeug</span><span class="o">/</span><span class="mf">0.9.6</span> <span class="nx">Python</span><span class="o">/</span><span class="mf">2.7.6</span>
<span class="p">{}</span>
</pre></div>
<p>Yay! So that worked. Or did it? Lets go back to our home page at <code>/</code> and lets see whats up:</p>
<div class="highlight"><pre><span></span><span class="nx">quazinafiulislam</span><span class="err">@</span><span class="nx">Nafiuls</span><span class="o">-</span><span class="nx">Mac</span><span class="o">:</span> <span class="o">~</span>
<span class="nx">$</span> <span class="nx">http</span> <span class="nx">GET</span> <span class="nx">http</span><span class="o">:</span><span class="c1">//localhost:5000/</span>
<span class="nx">HTTP</span><span class="o">/</span><span class="mf">1.0</span> <span class="mi">200</span> <span class="nx">OK</span>
<span class="nx">Content</span><span class="o">-</span><span class="nx">Length</span><span class="o">:</span> <span class="mi">166</span>
<span class="nx">Content</span><span class="o">-</span><span class="nx">Type</span><span class="o">:</span> <span class="nx">application</span><span class="o">/</span><span class="nx">json</span>
<span class="nb">Date</span><span class="o">:</span> <span class="nx">Sat</span><span class="p">,</span> <span class="mi">11</span> <span class="nx">Oct</span> <span class="mi">2014</span> <span class="mi">02</span><span class="o">:</span><span class="mi">33</span><span class="o">:</span><span class="mi">24</span> <span class="nx">GMT</span>
<span class="nx">Server</span><span class="o">:</span> <span class="nx">Werkzeug</span><span class="o">/</span><span class="mf">0.9.6</span> <span class="nx">Python</span><span class="o">/</span><span class="mf">2.7.6</span>
<span class="p">{</span>
<span class="s2">"1"</span><span class="o">:</span> <span class="p">{</span>
<span class="s2">"tags"</span><span class="o">:</span> <span class="p">[</span>
<span class="s2">"http://localhost:5000/tags/1"</span><span class="p">,</span>
<span class="s2">"http://localhost:5000/tags/2"</span>
<span class="p">],</span>
<span class="s2">"task"</span><span class="o">:</span> <span class="s2">"Buy rice"</span>
<span class="p">}</span>
<span class="p">}</span>
</pre></div>
<p>Now that we're done with the <code>Todo.get</code> and <code>Todo.put</code>, lets do <code>TodoItem.get</code>:</p>
<div class="highlight"><pre><span></span><span class="k">class</span> <span class="nc">TodoItem</span><span class="p">(</span><span class="n">rest</span><span class="o">.</span><span class="n">Resource</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">get</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">todo_id</span><span class="p">):</span>
<span class="sd">"""</span>
<span class="sd"> Get specific information on a Todo item</span>
<span class="sd"> :param todo_id: The Todo Item's ID, which is unique and a primary key</span>
<span class="sd"> :type todo_id: int</span>
<span class="sd"> """</span>
<span class="k">try</span><span class="p">:</span>
<span class="k">with</span> <span class="n">orm</span><span class="o">.</span><span class="n">db_session</span><span class="p">:</span>
<span class="n">todo</span> <span class="o">=</span> <span class="n">Todo</span><span class="p">[</span><span class="n">todo_id</span><span class="p">]</span> <span class="c1"># 1</span>
<span class="n">tags</span> <span class="o">=</span> <span class="p">[{</span><span class="n">tag</span><span class="o">.</span><span class="n">name</span><span class="p">:</span> <span class="n">tag</span><span class="o">.</span><span class="n">url</span><span class="p">}</span> <span class="k">for</span> <span class="n">tag</span> <span class="ow">in</span> <span class="n">todo</span><span class="o">.</span><span class="n">tags</span><span class="p">]</span> <span class="c1"># 2</span>
<span class="k">return</span> <span class="p">{</span>
<span class="s2">"task"</span><span class="p">:</span> <span class="n">todo</span><span class="o">.</span><span class="n">data</span><span class="p">,</span>
<span class="s2">"tags"</span><span class="p">:</span> <span class="n">tags</span>
<span class="p">}</span>
<span class="k">except</span> <span class="n">orm</span><span class="o">.</span><span class="n">ObjectNotFound</span><span class="p">:</span> <span class="c1"># 3</span>
<span class="k">return</span> <span class="p">{},</span> <span class="mi">404</span>
</pre></div>
<p>Since we are getting <code>todo_id</code> from the request itself, we can use it to get the <code>Todo</code> item with that
<code>id</code> in <code class="coderef">1</code>. This syntax is the same as <code>Todo.get(id=todo_id)</code>.</p>
<p>In <code class="coderef">2</code>, we are are creating a list of dictionaries from the tags in <code>todo.tags</code>. We are using <code>tag.name</code>
and <code>tag.url</code>. If you just wanted to get the names of of the tags, you could do either of these:</p>
<div class="highlight"><pre><span></span><span class="n">tag_names</span> <span class="o">=</span> <span class="nb">list</span><span class="p">(</span><span class="n">todo</span><span class="o">.</span><span class="n">tags</span><span class="o">.</span><span class="n">name</span><span class="p">)</span>
</pre></div>
<div class="highlight"><pre><span></span><span class="n">tag_names</span> <span class="o">=</span> <span class="p">[</span><span class="n">tag</span><span class="o">.</span><span class="n">name</span> <span class="k">for</span> <span class="n">tag</span> <span class="ow">in</span> <span class="n">todo</span><span class="o">.</span><span class="n">tags</span><span class="p">]</span>
</pre></div>
<p>If you think I'm pulling your leg, you can check the <a class="reference external" href="https://github.com/gamesbrainiac/Pony-Todo-API/tree/83287fc4fd99ebb6dc9e613571695a8c835c46b9">source code</a> and run it for yourself. In <code class="coderef">3</code> if we cannot find
the object, then we return a <code>404</code> error message back.</p>
<p>Now on to <code>Tags.get</code>:</p>
<div class="highlight"><pre><span></span><span class="k">class</span> <span class="nc">Tags</span><span class="p">(</span><span class="n">rest</span><span class="o">.</span><span class="n">Resource</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">get</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="sd">"""Will show you all tags"""</span>
<span class="k">with</span> <span class="n">orm</span><span class="o">.</span><span class="n">db_session</span><span class="p">:</span>
<span class="k">return</span> <span class="p">{</span>
<span class="n">tag</span><span class="o">.</span><span class="n">name</span><span class="p">:</span> <span class="n">tag</span><span class="o">.</span><span class="n">url</span>
<span class="k">for</span> <span class="n">tag</span> <span class="ow">in</span> <span class="n">Tag</span><span class="o">.</span><span class="n">select</span><span class="p">()</span>
<span class="p">}</span>
</pre></div>
<p>Nothing new here, very similar logic to our <code>/</code> url. Finally in our <code>TagItem.get</code> we have:</p>
<div class="highlight"><pre><span></span><span class="k">class</span> <span class="nc">TagItem</span><span class="p">(</span><span class="n">rest</span><span class="o">.</span><span class="n">Resource</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">get</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">tag_id</span><span class="p">):</span>
<span class="sd">"""</span>
<span class="sd"> Will show you information about a specific tag</span>
<span class="sd"> :param tag_id: ID for the tag</span>
<span class="sd"> :type tag_id: int</span>
<span class="sd"> """</span>
<span class="k">try</span><span class="p">:</span>
<span class="k">with</span> <span class="n">orm</span><span class="o">.</span><span class="n">db_session</span><span class="p">:</span>
<span class="n">tag</span> <span class="o">=</span> <span class="n">Tag</span><span class="p">[</span><span class="n">tag_id</span><span class="p">]</span>
<span class="n">todos</span> <span class="o">=</span> <span class="nb">list</span><span class="p">(</span><span class="n">tag</span><span class="o">.</span><span class="n">todos</span><span class="o">.</span><span class="n">data</span><span class="p">)</span> <span class="c1"># 1</span>
<span class="k">return</span> <span class="p">{</span>
<span class="s2">"tag"</span><span class="p">:</span> <span class="n">tag</span><span class="o">.</span><span class="n">name</span><span class="p">,</span>
<span class="s2">"tasks"</span><span class="p">:</span> <span class="n">todos</span>
<span class="p">}</span>
<span class="k">except</span> <span class="n">orm</span><span class="o">.</span><span class="n">ObjectNotFound</span><span class="p">:</span>
<span class="k">return</span> <span class="p">{},</span> <span class="mi">404</span>
</pre></div>
<p>This is similar logic to that of <code>TodoItem.get</code>. The difference here is that in <code class="coderef">1</code>,
we're getting all the data for the <code>todos</code> in a different way since we only need to know one thing.
The code in <code class="coderef">1</code> is the same as:</p>
<div class="highlight"><pre><span></span><span class="n">todos</span> <span class="o">=</span> <span class="p">[</span><span class="n">tag</span><span class="o">.</span><span class="n">data</span> <span class="k">for</span> <span class="n">tag</span> <span class="ow">in</span> <span class="n">todo</span><span class="o">.</span><span class="n">tags</span><span class="p">]</span>
</pre></div>
</div>
<div class="section" id="deleting-objects">
<h3><a class="toc-backref" href="#id6">Deleting Objects</a></h3>
<p>All that's left is to find a way to delete objects from the database.</p>
<div class="highlight"><pre><span></span><span class="k">def</span> <span class="nf">delete</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">todo_id</span><span class="p">):</span>
<span class="k">try</span><span class="p">:</span>
<span class="k">with</span> <span class="n">orm</span><span class="o">.</span><span class="n">db_session</span><span class="p">:</span>
<span class="n">todo</span> <span class="o">=</span> <span class="n">Todo</span><span class="p">[</span><span class="n">todo_id</span><span class="p">]</span> <span class="c1"># 1</span>
<span class="k">if</span> <span class="n">todo</span><span class="p">:</span>
<span class="n">tags</span> <span class="o">=</span> <span class="n">todo</span><span class="o">.</span><span class="n">tags</span><span class="o">.</span><span class="n">copy</span><span class="p">()</span> <span class="c1"># 2</span>
<span class="n">todo</span><span class="o">.</span><span class="n">delete</span><span class="p">()</span> <span class="c1"># 3</span>
<span class="k">for</span> <span class="n">tag</span> <span class="ow">in</span> <span class="n">tags</span><span class="p">:</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">tag</span><span class="o">.</span><span class="n">todos</span><span class="p">:</span> <span class="c1"># 4</span>
<span class="n">tag</span><span class="o">.</span><span class="n">delete</span><span class="p">()</span> <span class="c1"># 5</span>
<span class="k">except</span> <span class="n">orm</span><span class="o">.</span><span class="n">ObjectNotFound</span><span class="p">:</span>
<span class="k">return</span> <span class="p">{},</span> <span class="mi">400</span>
<span class="k">return</span> <span class="p">{},</span> <span class="mi">200</span>
</pre></div>
<p>In <code class="coderef">1</code> we get the todo item like before. In <code class="coderef">2</code>, we copy the tags, because when we delete them in <code class="coderef">3</code>,
we can no longer reference them through <code>todo.tags</code>, since the <code>todo</code> item is marked for deletion. If you
do not do this, then you are going to get a <code>OperationWithDeletedObjectError</code>.</p>
<p>In <code class="coderef">4</code> we check to see if the <code>tag</code> has todo items associated with it, if it doesn't then we delete the tag too
in <code class="coderef">5</code>.</p>
<p>Now, a demonstration of the API that we've built:</p>
<div class="external youtube"><iframe width="640" height="360" src="//www.youtube.com/embed/LxssYsNPZZk" frameborder="0"></iframe></div></div>
</div>
<div class="section" id="references">
<h2><a class="toc-backref" href="#id7">References</a></h2>
<ul class="simple">
<li><a class="reference external" href="http://ponyorm.com/">Official Pony ORM documentation</a></li>
<li><a class="reference external" href="http://flask-restful.readthedocs.org/en/latest/">Official Flask-RESTful documentation</a></li>
</ul>
</div>
<div class="section" id="update">
<h2><a class="toc-backref" href="#id8">Update</a></h2>
<p><em>Last Updated on 15 Oct 2014 7:00 PM</em></p>
<p>I'd like to thank the pony ORM authors for their help in making this tutorial.</p>
<p>You can also take a look at the <a class="reference external" href="https://github.com/gamesbrainiac/Pony-Todo-API/tree/Modularized">modularized version of the source code</a></p>
</div>
<div class="section" id="p-s">
<h2><a class="toc-backref" href="#id9">P.S.</a></h2>
<p>If you want to know what command line tool I used to make requests, I used <a class="reference external" href="https://github.com/jakubroztocil/httpie">httpie</a>.</p>
</div>
Testing out PyCharm's REST Client2014-10-09T07:55:00+06:002014-10-09T07:55:00+06:00Nafiul Islamtag:nafiulis.me,2014-10-09:testing-out-rest-with-pycharm.html<p>I've been trying out PyCharm's "Test RESTful Web Service" tool recently, just to check it out. I usually use
<code>requests</code>, <code>httpie</code> or <code>curl</code> if I have to use command line tools, but the tool I use most is
<a class="reference external" href="https://chrome.google.com/webstore/detail/advanced-rest-client/hgmloofddffdnphfgcellkdfbfbjeloo">Advanced REST client</a>.</p>
<p>Wanting to test out a sample API from <a class="reference external" href="http://flask-restful.readthedocs.org/en/latest/">Flask-Restful</a>,
I decided to take PyCharm's tool out for a spin. Here is the sample code from their <a class="reference external" href="http://flask-restful.readthedocs.org/en/latest/quickstart.html">quickstart</a> page:</p>
<div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">flask</span> <span class="kn">import</span> <span class="n">Flask</span><span class="p">,</span> <span class="n">request</span>
<span class="kn">from</span> <span class="nn">flask.ext.restful</span> <span class="kn">import</span> <span class="n">Resource</span><span class="p">,</span> <span class="n">Api</span>
<span class="n">app</span> <span class="o">=</span> <span class="n">Flask</span><span class="p">(</span><span class="vm">__name__</span><span class="p">)</span>
<span class="n">api</span> <span class="o">=</span> <span class="n">Api</span><span class="p">(</span><span class="n">app</span><span class="p">)</span>
<span class="n">todos</span> <span class="o">=</span> <span class="p">{}</span>
<span class="k">class</span> <span class="nc">TodoSimple</span><span class="p">(</span><span class="n">Resource</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">get</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">todo_id</span><span class="p">):</span>
<span class="k">return</span> <span class="p">{</span><span class="n">todo_id</span><span class="p">:</span> <span class="n">todos</span><span class="p">[</span><span class="n">todo_id</span><span class="p">]}</span>
<span class="k">def</span> <span class="nf">put</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">todo_id</span><span class="p">):</span>
<span class="n">todos</span><span class="p">[</span><span class="n">todo_id</span><span class="p">]</span> <span class="o">=</span> <span class="n">request</span><span class="o">.</span><span class="n">form</span><span class="p">[</span><span class="s1">'data'</span><span class="p">]</span>
<span class="k">return</span> <span class="p">{</span><span class="n">todo_id</span><span class="p">:</span> <span class="n">todos</span><span class="p">[</span><span class="n">todo_id</span><span class="p">]}</span>
<span class="n">api</span><span class="o">.</span><span class="n">add_resource</span><span class="p">(</span><span class="n">TodoSimple</span><span class="p">,</span> <span class="s1">'/<string:todo_id>'</span><span class="p">)</span>
<span class="k">if</span> <span class="vm">__name__</span> <span class="o">==</span> <span class="s1">'__main__'</span><span class="p">:</span>
<span class="n">app</span><span class="o">.</span><span class="n">run</span><span class="p">(</span><span class="n">debug</span><span class="o">=</span><span class="bp">True</span><span class="p">)</span>
</pre></div>
<img alt="How to get to the REST client" class="align-center" src="images/pycharm_rest_04.png" />
<p><code>GET</code> requests are easy enough. You don't need to change anything from the defaults:</p>
<img alt="Click the button indicated by the orange arrow to turn off the query string" class="align-center" src="images/pycharm_rest_01.png" />
<p>If you click the button indicated by the orange arrow, you will not have any query string in your URL. We will leave the
request body empty:</p>
<img alt="Keeping the request body empty" class="align-center" src="images/pycharm_rest_02.png" />
<p>After sending the request:</p>
<img alt="404 response" class="align-center" src="images/pycharm_rest_03.png" />
<p>The client supports syntax highlighting for multiple formats, like <code>html</code>, <code>json</code> and <code>xml</code>. In
this case, <code>html</code> has been highlighted. You can also check out response headers in the adjacent tab.</p>
<p><code>PUT</code>, <code>POST</code> and <code>DELETE</code> are just as easy, but they do not have the defaults that you'd expect.
For example, in <code>requests</code>, your content type is implicitly set for you to
<code>application/x-www-form-urlencoded</code>:</p>
<div class="highlight"><pre><span></span><span class="o">>>></span> <span class="kn">import</span> <span class="nn">requests</span>
<span class="o">>>></span> <span class="n">req</span> <span class="o">=</span> <span class="n">requests</span><span class="o">.</span><span class="n">put</span><span class="p">(</span><span class="s1">'http://127.0.0.1:5000/todo1'</span><span class="p">,</span>
<span class="n">data</span><span class="o">=</span><span class="p">{</span><span class="s1">'data'</span><span class="p">:</span> <span class="s1">'Make a tutorial on PyCharm</span><span class="se">\'</span><span class="s1">s REST client'</span> <span class="p">})</span>
<span class="o">>>></span> <span class="n">req</span>
<span class="o"><</span><span class="n">Response</span> <span class="p">[</span><span class="mi">200</span><span class="p">]</span><span class="o">></span>
<span class="o">>>></span> <span class="n">req</span><span class="o">.</span><span class="n">request</span><span class="o">.</span><span class="n">headers</span><span class="p">[</span><span class="s1">'Content-Type'</span><span class="p">]</span>
<span class="s1">'application/x-www-form-urlencoded'</span>
<span class="o">>>></span> <span class="n">req</span><span class="o">.</span><span class="n">content</span>
<span class="s1">'{</span><span class="se">\n</span><span class="s1"> "todo1": "Make a tutorial on PyCharm</span><span class="se">\'</span><span class="s1">s REST client"</span><span class="se">\n</span><span class="s1">}</span><span class="se">\n</span><span class="s1">'</span>
</pre></div>
<p>This took me some time to figure out in PyCharm's REST client. I <em>knew</em> I was doing something wrong,
but oddly enough it took me a while before I understood that it was because I had not set my content type in my
header. I think its because <code>curl</code>, <code>requests</code> and the chrome extension all set this implicitly for you.
The good news is that headers have code completion, but but not for values:</p>
<img alt="Setting content type" class="align-center" src="images/pycharm_rest_05.png" />
<p>This time, we set the text to the following, since we want to add a new todo list item:</p>
<img alt="Adding data to the request body" class="align-center" src="images/pycharm_rest_06.png" />
<p>Voilà, our request succeeds:</p>
<img alt="JSON request body" class="align-center" src="images/pycharm_rest_07.png" />
<p>You can do a lot more with the options that are available, like uploading from files,
adding request parameters and setting cookies.</p>
<div class="section" id="references">
<h2>References</h2>
<ul class="simple">
<li><a class="reference external" href="http://docs.python-requests.org/en/latest/user/advanced/#request-and-response-objects">Python Requests -Request and ResponseObjects</a></li>
<li><a class="reference external" href="http://stackoverflow.com/a/3121175/1624921">StackOverflow - How can I see the request headers made by curl when sending a request to the server?</a></li>
</ul>
</div>
<div class="section" id="p-s">
<h2>P.S.</h2>
<p>This tool is also available in other jetbrains IDEs as well, like webstorm and intellij.</p>
</div>
Understanding Schemas2014-10-08T15:44:00+06:002014-10-08T15:44:00+06:00Nafiul Islamtag:nafiulis.me,2014-10-08:understanding-schemas.html<div class="contents topic" id="table-of-contents">
<p class="topic-title first">Table of Contents</p>
<ul class="simple">
<li><a class="reference internal" href="#schemas-are-namespaces" id="id1">Schemas are namespaces</a></li>
<li><a class="reference internal" href="#inside-a-schema" id="id2">Inside a Schema</a></li>
<li><a class="reference internal" href="#creating-schemas" id="id3">Creating Schemas</a></li>
<li><a class="reference internal" href="#references" id="id4">References</a></li>
</ul>
</div>
<p>I've been brushing up on my SQL lately, and one key concept that I forgot about where schemas. I rarely used them
since I created most of my tables in the public schema (shame on me), but it was interesting to read up the
documentation on them again.</p>
<div class="section" id="schemas-are-namespaces">
<h2><a class="toc-backref" href="#id1">Schemas are namespaces</a></h2>
<p>In short, schemas are namespaces that house your functions, tables triggers etc. What's a namespace? Its like a
container of addresses. The reason why we use them is to prevent cluttering of the main/public namespace.</p>
<p>In fact, every table that you've ever created inside postgres lives inside a schema. For example,
in <code>psql</code> you can use <code>\d</code> and then hitting <kbd class="light">tab</kbd> to show you what schemas are available:</p>
<img alt="The dots indicate schemas or namespaces" class="align-center" src="images/schema_01.png" />
<p>The dots indicate that those options are schemas. There is always a schema involved when you create a table or a
function. Generally, all your functions are put into the <code>public</code> schema. So, in the above example, we can see
<code>messages</code> as a table, and it looks as though its not in a schema, but it is. One can also refer to it as
<code>public.messages</code>:</p>
<div class="highlight"><pre><span></span><span class="n">test</span><span class="o">=#</span> <span class="err">\</span><span class="n">d</span> <span class="k">public</span><span class="p">.</span><span class="n">messages</span>
<span class="k">Table</span> <span class="ss">"public.messages"</span>
<span class="k">Column</span> <span class="o">|</span> <span class="k">Type</span> <span class="o">|</span> <span class="n">Modifiers</span>
<span class="c1">--------+---------+-----------</span>
<span class="n">id</span> <span class="o">|</span> <span class="nb">integer</span> <span class="o">|</span> <span class="k">not</span> <span class="k">null</span>
<span class="k">data</span> <span class="o">|</span> <span class="nb">text</span> <span class="o">|</span>
<span class="nb">date</span> <span class="o">|</span> <span class="nb">date</span> <span class="o">|</span>
<span class="n">Indexes</span><span class="p">:</span>
<span class="ss">"messages_pkey"</span> <span class="k">PRIMARY</span> <span class="k">KEY</span><span class="p">,</span> <span class="n">btree</span> <span class="p">(</span><span class="n">id</span><span class="p">)</span>
<span class="n">test</span><span class="o">=#</span> <span class="err">\</span><span class="n">d</span> <span class="n">messages</span>
<span class="k">Table</span> <span class="ss">"public.messages"</span>
<span class="k">Column</span> <span class="o">|</span> <span class="k">Type</span> <span class="o">|</span> <span class="n">Modifiers</span>
<span class="c1">--------+---------+-----------</span>
<span class="n">id</span> <span class="o">|</span> <span class="nb">integer</span> <span class="o">|</span> <span class="k">not</span> <span class="k">null</span>
<span class="k">data</span> <span class="o">|</span> <span class="nb">text</span> <span class="o">|</span>
<span class="nb">date</span> <span class="o">|</span> <span class="nb">date</span> <span class="o">|</span>
<span class="n">Indexes</span><span class="p">:</span>
<span class="ss">"messages_pkey"</span> <span class="k">PRIMARY</span> <span class="k">KEY</span><span class="p">,</span> <span class="n">btree</span> <span class="p">(</span><span class="n">id</span><span class="p">)</span>
</pre></div>
</div>
<div class="section" id="inside-a-schema">
<h2><a class="toc-backref" href="#id2">Inside a Schema</a></h2>
<p>To see what tables are inside of a schema, you can keep on using tab completion, use a wildcard or use a query. Tab
completion is pretty self-explanatory. If you want to use a wildcard then use it like so:</p>
<div class="highlight"><pre><span></span><span class="n">exercises</span><span class="o">=#</span> <span class="err">\</span><span class="n">dt</span> <span class="n">cd</span><span class="p">.</span><span class="o">*</span>
<span class="n">List</span> <span class="k">of</span> <span class="n">relations</span>
<span class="k">Schema</span> <span class="o">|</span> <span class="n">Name</span> <span class="o">|</span> <span class="k">Type</span> <span class="o">|</span> <span class="k">Owner</span>
<span class="c1">--------+------------+-------+------------------</span>
<span class="n">cd</span> <span class="o">|</span> <span class="n">bookings</span> <span class="o">|</span> <span class="k">table</span> <span class="o">|</span> <span class="n">quazinafiulislam</span>
<span class="n">cd</span> <span class="o">|</span> <span class="n">facilities</span> <span class="o">|</span> <span class="k">table</span> <span class="o">|</span> <span class="n">quazinafiulislam</span>
<span class="n">cd</span> <span class="o">|</span> <span class="n">members</span> <span class="o">|</span> <span class="k">table</span> <span class="o">|</span> <span class="n">quazinafiulislam</span>
<span class="p">(</span><span class="mi">3</span> <span class="k">rows</span><span class="p">)</span>
</pre></div>
<p>We use the extra <code>t</code> in <code>\dt</code> to give us table information only. If you use used <code>\d cd.*</code>, then you'd
get a lot more information regarding each table. You can try it out for yourself.</p>
<p>Another way to get the same information is use to use a query:</p>
<div class="highlight"><pre><span></span><span class="k">SELECT</span> <span class="o">*</span> <span class="k">FROM</span> <span class="n">information_schema</span><span class="p">.</span><span class="n">tables</span> <span class="k">WHERE</span> <span class="n">table_schema</span> <span class="o">=</span> <span class="s1">'cd'</span>
</pre></div>
<p>Note that this is also a schema, and this schema also exists under the <code>public</code> schema. What do I mean by that,
well take a look at this:</p>
<div class="highlight"><pre><span></span><span class="n">exercises</span><span class="o">=#</span> <span class="err">\</span><span class="n">d</span> <span class="k">public</span><span class="p">.</span><span class="n">information_schema</span><span class="p">.</span><span class="n">tables</span>
<span class="k">View</span> <span class="ss">"information_schema.tables"</span>
<span class="k">Column</span> <span class="o">|</span> <span class="k">Type</span> <span class="o">|</span> <span class="n">Modifiers</span>
<span class="c1">------------------------------+-----------------------------------+-----------</span>
<span class="n">table_catalog</span> <span class="o">|</span> <span class="n">information_schema</span><span class="p">.</span><span class="n">sql_identifier</span> <span class="o">|</span>
<span class="n">table_schema</span> <span class="o">|</span> <span class="n">information_schema</span><span class="p">.</span><span class="n">sql_identifier</span> <span class="o">|</span>
<span class="k">table_name</span> <span class="o">|</span> <span class="n">information_schema</span><span class="p">.</span><span class="n">sql_identifier</span> <span class="o">|</span>
<span class="n">table_type</span> <span class="o">|</span> <span class="n">information_schema</span><span class="p">.</span><span class="n">character_data</span> <span class="o">|</span>
<span class="n">self_referencing_column_name</span> <span class="o">|</span> <span class="n">information_schema</span><span class="p">.</span><span class="n">sql_identifier</span> <span class="o">|</span>
<span class="n">reference_generation</span> <span class="o">|</span> <span class="n">information_schema</span><span class="p">.</span><span class="n">character_data</span> <span class="o">|</span>
<span class="k">user_defined_type_catalog</span> <span class="o">|</span> <span class="n">information_schema</span><span class="p">.</span><span class="n">sql_identifier</span> <span class="o">|</span>
<span class="k">user_defined_type_schema</span> <span class="o">|</span> <span class="n">information_schema</span><span class="p">.</span><span class="n">sql_identifier</span> <span class="o">|</span>
<span class="k">user_defined_type_name</span> <span class="o">|</span> <span class="n">information_schema</span><span class="p">.</span><span class="n">sql_identifier</span> <span class="o">|</span>
<span class="n">is_insertable_into</span> <span class="o">|</span> <span class="n">information_schema</span><span class="p">.</span><span class="n">yes_or_no</span> <span class="o">|</span>
<span class="n">is_typed</span> <span class="o">|</span> <span class="n">information_schema</span><span class="p">.</span><span class="n">yes_or_no</span> <span class="o">|</span>
<span class="n">commit_action</span> <span class="o">|</span> <span class="n">information_schema</span><span class="p">.</span><span class="n">character_data</span> <span class="o">|</span>
</pre></div>
<p>In essence, everything lives under the <code>public</code> schema.</p>
</div>
<div class="section" id="creating-schemas">
<h2><a class="toc-backref" href="#id3">Creating Schemas</a></h2>
<p>Creating schemas are as simple as creating tables. Here's an example:</p>
<div class="highlight"><pre><span></span><span class="n">test</span><span class="o">=#</span> <span class="k">CREATE</span> <span class="k">SCHEMA</span> <span class="n">happy_schema</span><span class="p">;</span>
<span class="k">CREATE</span> <span class="k">SCHEMA</span>
</pre></div>
<p>When you create a schema, you can also create tables and functions under it in one command:</p>
<div class="highlight"><pre><span></span><span class="n">test</span><span class="o">=#</span> <span class="k">CREATE</span> <span class="k">SCHEMA</span> <span class="n">ShoppingCenters</span>
<span class="n">test</span><span class="o">-#</span> <span class="k">CREATE</span> <span class="k">TABLE</span> <span class="n">Malls</span> <span class="p">(</span><span class="n">id</span> <span class="nb">integer</span> <span class="k">PRIMARY</span> <span class="k">KEY</span><span class="p">,</span> <span class="n">name</span> <span class="nb">VARCHAR</span><span class="p">(</span><span class="mi">100</span><span class="p">))</span>
<span class="n">test</span><span class="o">-#</span> <span class="k">CREATE</span> <span class="k">TABLE</span> <span class="n">SuperMarkets</span> <span class="p">(</span><span class="n">id</span> <span class="nb">integer</span> <span class="k">PRIMARY</span> <span class="k">KEY</span><span class="p">,</span> <span class="n">name</span> <span class="nb">VARCHAR</span><span class="p">(</span><span class="mi">100</span><span class="p">),</span> <span class="n">capacity</span> <span class="nb">integer</span><span class="p">);</span>
<span class="k">CREATE</span> <span class="k">SCHEMA</span>
<span class="n">test</span><span class="o">=#</span> <span class="err">\</span><span class="n">dt</span> <span class="n">shoppingcenters</span><span class="p">.</span><span class="o">*</span>
<span class="n">List</span> <span class="k">of</span> <span class="n">relations</span>
<span class="k">Schema</span> <span class="o">|</span> <span class="n">Name</span> <span class="o">|</span> <span class="k">Type</span> <span class="o">|</span> <span class="k">Owner</span>
<span class="c1">-----------------+--------------+-------+------------------</span>
<span class="n">shoppingcenters</span> <span class="o">|</span> <span class="n">malls</span> <span class="o">|</span> <span class="k">table</span> <span class="o">|</span> <span class="n">quazinafiulislam</span>
<span class="n">shoppingcenters</span> <span class="o">|</span> <span class="n">supermarkets</span> <span class="o">|</span> <span class="k">table</span> <span class="o">|</span> <span class="n">quazinafiulislam</span>
<span class="p">(</span><span class="mi">2</span> <span class="k">rows</span><span class="p">)</span>
</pre></div>
<p>This feels a lot like using python's <code>with</code> statement.</p>
</div>
<div class="section" id="references">
<h2><a class="toc-backref" href="#id4">References</a></h2>
<ul class="simple">
<li><a class="reference external" href="http://stackoverflow.com/questions/15644152/list-tables-in-a-postgresql-schema">List tables in a Postgresql Schema</a></li>
<li><a class="reference external" href="http://www.postgresql.org/docs/9.1/static/ddl-schemas.html">Postgresql Schema</a></li>
</ul>
</div>
Why you should be using Pyenv2014-07-15T00:48:00+06:002014-07-15T00:48:00+06:00Nafiul Islamtag:nafiulis.me,2014-07-15:why-you-should-be-using-pyenv.html<p><strong>TL;DR -> Install Pyenv its good for you. You'll thank me later.</strong></p>
<p>So, a lot of people have issues with managing their different Python versions. There are plenty of "pythons" out
there. There's <code>cpython2</code>, <code>pypy</code>, <code>stackless</code>, <code>anaconda</code>, <code>cpython3</code> ...</p>
<p>The list could go on forever.</p>
<p>Thus it isn't a surprise to find people struggling with trying to leverage different versions of Python. I mean,
things can get messy, when you have two versions of Python installed. I for example have five versions of Python
installed.</p>
<p>Luckily, there's a tool out there that allows you to seamlessly change between the different versions of Python on
your *nix operating system (sorry Windows folks, no dice). Its called <a class="reference external" href="https://github.com/yyuu/pyenv">Pyenv</a> (you
probably guessed that one).</p>
<p>On a mac, <code>pyenv</code> is pretty easy to install, using homebrew with:</p>
<div class="highlight"><pre><span></span>brew install pyenv
</pre></div>
<p>For linux, you can use the following <a class="reference external" href="https://github.com/yyuu/pyenv-installer">installer</a>. Just to top it off here
are the benefits:</p>
<ul class="simple">
<li>Manage multiple versions of the Python interpreter, this includes not only the language versions but also different
interpreters entirely.</li>
<li>Stay up to date with all the latest versions of Python that's coming out.</li>
<li>Change seamlessly between the different versions of Python, you have complete control over what the <code>python</code>
alias means.</li>
</ul>
Watch out iPython here comes b(i)python!2014-05-20T20:21:00+06:002014-05-20T20:21:00+06:00Nafiul Islamtag:nafiulis.me,2014-05-20:watch-out-ipython-here-comes-bpython.html<img alt="Like what you see?" class="align-center" src="images/bpython.png" />
<div class="contents topic" id="table-of-contents">
<p class="topic-title first">Table of Contents</p>
<ul class="simple">
<li><a class="reference internal" href="#getting-started" id="id1">Getting Started</a></li>
<li><a class="reference internal" href="#a-few-quick-tips" id="id2">A Few Quick Tips</a></li>
<li><a class="reference internal" href="#sweet-code-completion" id="id3">Sweet code completion</a></li>
<li><a class="reference internal" href="#don-t-worry-if-you-mess-up" id="id4">Don't worry if you mess up</a></li>
<li><a class="reference internal" href="#share-your-code-on-the-fly" id="id5">Share your code on the fly</a></li>
<li><a class="reference internal" href="#contacting-the-team" id="id6">Contacting the team</a></li>
<li><a class="reference internal" href="#but-what-if-you-wanted-both" id="id7">But what if you wanted both?</a></li>
<li><a class="reference internal" href="#update" id="id8">Update</a></li>
</ul>
</div>
<p>So if you've been using Python for some time now, you've probably heard of <a class="reference external" href="http://ipython.org/">iPython</a>. This is
pretty much the go to shell for most pythonistas who want to get a better command line debug experience. I find
<code>ipdb</code> to be a godsend when I'm quickly trying to diagnose the problem.</p>
<p>But what about when you're trying to <em>explore</em> a library or even write some code quickly with plenty of
auto-completion, or perhaps quickly demonstrate something, you might want to chose <code>bpython</code>. And if you wanted
the best of both worlds, both <code>ipython</code> and <code>bpython</code>, there's <code>bipython</code>. But first,
lets take a look at what <code>bpython</code> can offer us:</p>
<div class="section" id="getting-started">
<h2><a class="toc-backref" href="#id1">Getting Started</a></h2>
<p><code>bpython</code> works on <code>*nix</code> systems with ease, for windows users all you need to is use
<a class="reference external" href="http://www.lfd.uci.edu/~gohlke/pythonlibs/#curses">this installer</a> to install the package. But <code>*nix</code> users,
its much simpler:</p>
<div class="highlight"><pre><span></span>pip install bpython
</pre></div>
<p>In my tests <code>bpython</code> worked with <strong>both python 2 and python 3 as well as pypy</strong> because its written in pure python and
the only dependency <code>bpython</code> has is on <code>pygments</code>:</p>
<img alt="bpython works on Python 3!" class="align-center" src="images/bpython_works_on_3.png" />
<p>Once you've installed <code>bpython</code>, you can then just involve it from the command line:</p>
<img alt="How you start it :)" class="align-center" src="images/bpython_start.png" />
</div>
<div class="section" id="a-few-quick-tips">
<h2><a class="toc-backref" href="#id2">A Few Quick Tips</a></h2>
<ul class="simple">
<li>The first thing I learn to do with any shell or any new software for that matter is how to get out out of it. The best
way to get out of <code>bpython</code> is just to hit <code>CTRL</code> + <code>D</code>, and that will take you out.</li>
<li><code>bpython</code> will automatically add the contents of your session to your console.</li>
<li>You can save you session with <code>CTRL</code> + <code>S</code>.</li>
<li>You can clear the <code>bpython</code> screen with <code>CTRL</code> + <code>L</code>.</li>
</ul>
</div>
<div class="section" id="sweet-code-completion">
<h2><a class="toc-backref" href="#id3">Sweet code completion</a></h2>
<p>I love code completion. How can you not love something that pretty much writes your code for you? As a person who's
always looking out for better tools, <code>bpython</code> hits the spot. It's got the best code completion I've seen in
<em>any</em>
shell, it even puts <code>fish</code> to shame:</p>
<div class="external youtube"><iframe width="640" height="360" src="//www.youtube.com/embed/ghohKayNeds" frameborder="0"></iframe></div><p>What I love about <code>bpython</code> is how fast the code completion pops up and the way documentation just pops up when you
want to use a function.</p>
</div>
<div class="section" id="don-t-worry-if-you-mess-up">
<h2><a class="toc-backref" href="#id4">Don't worry if you mess up</a></h2>
<p>When I'm trying to reproduce something on the command line, I often make mistakes (although with <code>bpython</code>'s
code completion it's a lot harder to do). I'm trying to import <code>asyncio</code> here:</p>
<div class="highlight"><pre><span></span><span class="o">>>></span> <span class="kn">import</span> <span class="nn">asycnio</span>
<span class="n">Traceback</span> <span class="p">(</span><span class="n">most</span> <span class="n">recent</span> <span class="n">call</span> <span class="n">last</span><span class="p">):</span>
<span class="n">File</span> <span class="s2">"<input>"</span><span class="p">,</span> <span class="n">line</span> <span class="mi">1</span><span class="p">,</span> <span class="ow">in</span> <span class="o"><</span><span class="n">module</span><span class="o">></span>
<span class="ne">ImportError</span><span class="p">:</span> <span class="n">No</span> <span class="n">module</span> <span class="n">named</span> <span class="s1">'asycnio'</span>
</pre></div>
<p>Now, when you're trying to send a console session to someone, these mistakes are not only annoying but also can get
in the way of the person trying to help you, so you can just redo the last line with <code>CTRL</code> + <code>r</code>:</p>
<div class="external youtube"><iframe width="640" height="360" src="//www.youtube.com/embed/JSYV9h2Mrr4" frameborder="0"></iframe></div><p>Please note that this does not negate what you did, it just clears the screen of the last command that you input. So
if you imported <code>datetime</code>, but you actually wanted to import <code>requests</code> (yes,
I don't know how to spell but bear with me), then <code>datetime</code> will still be usable from the shell:</p>
<div class="external youtube"><iframe width="640" height="360" src="//www.youtube.com/embed/_h9BlN4s_wM" frameborder="0"></iframe></div></div>
<div class="section" id="share-your-code-on-the-fly">
<h2><a class="toc-backref" href="#id5">Share your code on the fly</a></h2>
<p>If you want to share your session with someone, you can do that in just one keystroke <code>F8</code>. It will prompt you
for confirmation, after which you can view your paste on bpython's code hosting site:</p>
<img alt="Pastebin" class="align-center" src="images/bpython_natsu.png" />
<p>After you've said yes to the prompt, it will generate a link for you, so
<a class="reference external" href="http://bpaste.net/show/NnLYEXislfYLVZTKVgAE/">this</a> is the one it generated for my session.</p>
</div>
<div class="section" id="contacting-the-team">
<h2><a class="toc-backref" href="#id6">Contacting the team</a></h2>
<p>These are just <em>some</em> of the awesome things you can do with <code>bpython</code>. There's probably more. I don't know them
all since I'm still exploring what you can do with it.</p>
<p>You can find out more about by talking to the guys on their <code>irc</code> in <code>#bpython</code> on <code>irc.freenode.net</code>.
Find out more about the community around <code>bpython</code> <a class="reference external" href="http://bpython-interpreter.org/community/">here</a>.</p>
<p>I initially though that <code>bpyhon</code> was not properly maintained, but if you take a look at <a class="reference external" href="https://bitbucket.org/bobf/bpython">their project page</a>, it seems that its being kept up to date:</p>
<img alt="Its up to date :)" class="align-center" src="images/bpython_bb.png" />
</div>
<div class="section" id="but-what-if-you-wanted-both">
<h2><a class="toc-backref" href="#id7">But what if you wanted both?</a></h2>
<p>Well you'd have to use <code>bipython</code> then. Its still in the works, and there's no proper highlighting for
errors right now like in <code>ipython</code>, but its heading towards being a package that unifies both
<code>ipython</code> and <code>bpython</code>. So, you need to install <code>ipython</code>, <code>bipython</code>:</p>
<div class="highlight"><pre><span></span>pip install ipython bipython
</pre></div>
<p>And you're done with the installations but running <code>bipython</code> is not as simple. At this time,
<code>bipython</code> does not work with python 3, but it does work with python 2. Have a look:</p>
<div class="external youtube"><iframe width="640" height="360" src="//www.youtube.com/embed/w9JcgM2YN4I" frameborder="0"></iframe></div></div>
<div class="section" id="update">
<h2><a class="toc-backref" href="#id8">Update</a></h2>
<p><em>Last Updated on 15 Nov 2014 1:11 PM</em></p>
<p>Information about <tt class="docutils literal">bipython</tt> added.</p>
</div>
Rifts in the Community2014-04-29T17:54:00+06:002014-04-29T17:54:00+06:00Nafiul Islamtag:nafiulis.me,2014-04-29:rifts-in-the-community.html<div class="contents topic" id="table-of-contents">
<p class="topic-title first">Table of Contents</p>
<ul class="simple">
<li><a class="reference internal" href="#diving-deeper" id="id2">Diving Deeper</a></li>
<li><a class="reference internal" href="#challenges" id="id3">Challenges</a></li>
<li><a class="reference internal" href="#solutions" id="id4">Solutions</a><ul>
<li><a class="reference internal" href="#pool-our-resources" id="id5">Pool our resources</a></li>
<li><a class="reference internal" href="#admitting-our-problems" id="id6">Admitting our problems</a></li>
<li><a class="reference internal" href="#make-it-easier-to-contribute" id="id7">Make it easier to contribute</a></li>
</ul>
</li>
<li><a class="reference internal" href="#resources" id="id8">Resources</a><ul>
<li><a class="reference internal" href="#the-global-interpreter-lock-gil" id="id9">The Global Interpreter Lock (GIL)</a></li>
<li><a class="reference internal" href="#pypy" id="id10">PyPy</a></li>
<li><a class="reference internal" href="#id1" id="id11">Pyston</a></li>
<li><a class="reference internal" href="#python-2-to-3" id="id12">Python 2 to 3</a></li>
</ul>
</li>
</ul>
</div>
<p>I love Python. Its the first programming language that I really stuck to. At the time when I was first starting, Python
had a lot going for it:</p>
<ul class="simple">
<li>MIT taught its Introductory CS Course in Python</li>
<li>Python was widely used in the Scientific community</li>
<li>It was very intuitive and easy to use off the bat</li>
<li>The community on StackOverflow was super nice too</li>
</ul>
<p>So, my journey began with MITx's Introduction to Computer Science course, and although it was the first time such a
course was offered on <a class="reference external" href="https://www.edx.org/">edx</a>, it was pretty darn awesome. The course was mostly taught by John
Guttag. Most of the lectures made sense, and the homework questions were a lot of fun to complete. I made a lot of
diverse friends too; one guy, a plumber but learning code so that he could help his son learn code (what an awesome
dad!).</p>
<p>I never finished the course, but it gave me enough to get started with Python and programming. I started asking
questions on StackOverflow, and soon started answering them too. I soon found one of the hidden gems of the site,
its <a class="reference external" href="http://chat.stackoverflow.com/">chat-room</a>. I was enthralled by the community that grew around Python,
it was and continued to be a super helpful community.</p>
<div class="section" id="diving-deeper">
<h2><a class="toc-backref" href="#id2">Diving Deeper</a></h2>
<p>However, as I started diving deeper into things I soon came to realize a few of Python's limitations. First and
foremost was that Python was very slow in some cases, especially when it came to raw loops or dealing with numbers. I
personally was super grateful for Python's Integer promotion (helped me a lot with a couple of assignments and
Project Euler too). However, I also realized there were many cases where Python can be optimised. The Python
community responded to these problems by dividing the problem into different parts:</p>
<ul class="simple">
<li>To make numerical data crunching (for lack of a better term) faster, the community came up with numpy.</li>
<li>For wrapping third party C libraries, there came <a class="reference external" href="http://www.swig.org/">swig by David Beazley</a> and
<a class="reference external" href="http://cython.org/">cython</a> which is an evolutionary form of the Pyrex project. Please note that Cython is
<em>python specific</em> whereas swig can be used with other programming languages as well. If you'd like to know more
about these two libraries, I feel that <a class="reference external" href="https://www.youtube.com/watch?v=mv0kfiepn3s">this video</a> does a good job
at exploring what they are, and highlights the features that make them different.</li>
<li>Python Implementation, faster implementations have popped up like <strong>Pypy</strong> (a faster implementation of
Python thanks to JIT). <strong>Jython/IronPython</strong>, both of which take advantage of their respective VMs,
bypassing Python's GIL. Most recently,
<a class="reference external" href="https://tech.dropbox.com/2014/04/introducing-pyston-an-upcoming-jit-based-python-implementation/">Pyston</a>.
There are others, but they have not gained enough traction.</li>
</ul>
<p>Now, I can tell you that numpy (and Python's Data tools in general) was a huge success,
and hence Python's scientific community has thrived. I can
honestly say this with pride, that Python's data tools and libraries are arguably the best of the breed. PyData,
which is a conference dedicated to the discussion of Python's data tools (and boy are there <em>plenty</em> of them) has
been growing both in popularity and influence.</p>
</div>
<div class="section" id="challenges">
<h2><a class="toc-backref" href="#id3">Challenges</a></h2>
<p>Now the tools that the scientific community employ are third party libraries,
they do not offer changes to the <em>implementation</em> of Python itself. However, the projects that are <em>trying</em> to make
headway have two really big challenges, and the two are actually quite interconnected. So,
what does making python <em>faster</em> mean? In essence there are two parts to it:</p>
<ul class="simple">
<li>Making Python code itself execute faster</li>
<li>Better support for concurrency in Python</li>
</ul>
<p>Now in order to meet the first goal, pypy implemented a JIT compiler and it WORKS! You can check it out yourself.
Pyston on the other hand is trying to achieve JIT but through a different means (<a class="reference external" href="http://pypy.readthedocs.org/en/latest/faq.html#could-we-use-llvm">which has been attempted by pypy</a>, but was not successful; it still remains a
theoretical possibility). The first challenge here is make an implementation successful enough to replace CPython. I
feeling that if the two camps, Pyston with its Dropbox backing and Pypy with its years of experience were to combine
their efforts, then we might be able to see good results even faster. I know its just conjecture,
but its something that might divide our community between two sides, ones that use pypy and the other which uses
Pyston. Right now, Pyston is working towards supporting only Python 2.</p>
<p>Secondly, trying to get rid of the GIL has be a major problem (this is the reason why threading will often slow down
your python programs), and as Alex Gaynor put it, a subtle reason for the division in our community:</p>
<blockquote>
I think there's been little uptake because Python 3 is fundamentally unexciting. It doesn't have the super big
ticket items people want, such as removal of the GIL or better performance (for which many are using PyPy).</blockquote>
<p>Most attempts at making Python faster work mostly with Python 2.7.*. Pypy 3 is not fully stable yet, and pypy-stm,
which is pypy's attempt at removing the GIL (iirc) is aimed at Python 2 and not 3. Thus we come to the second big
problem, and that is division between python 2 and 3.</p>
</div>
<div class="section" id="solutions">
<h2><a class="toc-backref" href="#id4">Solutions</a></h2>
<div class="section" id="pool-our-resources">
<h3><a class="toc-backref" href="#id5">Pool our resources</a></h3>
<p>We need to get the experts in the Python community together. We also need to get the companies that use python to
back up a project that can make a python implementation fast enough to compete with Javascript's v8 engine,
and one that supports concurrency well.</p>
</div>
<div class="section" id="admitting-our-problems">
<h3><a class="toc-backref" href="#id6">Admitting our problems</a></h3>
<p>Everyone knows that Python 3 has issues. Every single time someone says "Unicode", I scream "Python 3" inside my head.
Right now the Python community is in a state of denial regarding Python 3:</p>
<div class="external">
<blockquote class="twitter-tweet" lang="en"><p>Guys i get it. I should not say anything bad about Python 3 because
clearly I don't see the big picture.</p>— Armin Ronacher (@mitsuhiko)
<a href="https://twitter.com/mitsuhiko/statuses/461156792685559808">April 29, 2014</a></blockquote>
<script async src="//platform.twitter.com/widgets.js" charset="utf-8"></script>
</div><p>We need to get out of it, and start working words solving the problem.</p>
</div>
<div class="section" id="make-it-easier-to-contribute">
<h3><a class="toc-backref" href="#id7">Make it easier to contribute</a></h3>
<p>Pythonistas <em>want</em> to help. We need to have more resources describing the problems that we face right now. Pypy
project does an excellent job of that, and the effort they put into their FAQs are laudable with the exception of
describing what rPython is:</p>
<blockquote>
RPython is a restricted subset of Python that is amenable to static analysis. Although there are additions to the
language and some things might surprisingly work, this is a rough list of restrictions that should be considered.
Note that there are tons of special cased restrictions that you’ll encounter as you go. The exact definition is
“RPython is everything that our translation tool-chain can accept” :)</blockquote>
</div>
</div>
<div class="section" id="resources">
<h2><a class="toc-backref" href="#id8">Resources</a></h2>
<p>If you want to get started on understanding what these different technologies are, then these resources will help:</p>
<div class="section" id="the-global-interpreter-lock-gil">
<h3><a class="toc-backref" href="#id9">The Global Interpreter Lock (GIL)</a></h3>
<p><a class="reference external" href="https://www.youtube.com/watch?v=ph374fJqFPE">David Beazley's talk on the GIL</a> was pretty much the only talk I
needed to get started, and (sadly) I did not do much more than that.</p>
</div>
<div class="section" id="pypy">
<h3><a class="toc-backref" href="#id10">PyPy</a></h3>
<p><a class="reference external" href="https://www.youtube.com/watch?v=l_HBRhcgeuQ">Beazley's Keynote on Pypy</a> was an excellent talk (if you haven't
realized already, I adore Beazley's talks). He really dives into the guts on Pypy on this one,
and shows you how it works from the inside.</p>
<p><a class="reference external" href="https://www.youtube.com/watch?v=Q9wf63flICs">Pypy and Software Transactional Memory</a> is a good talk,
but not the best. Its made by members of Pypy's core team, and introduces what STM is. Although this talk leaves a
lot to be desired of, you can still walk away with some key points regarding what STM is and why its useful.</p>
</div>
<div class="section" id="id1">
<h3><a class="toc-backref" href="#id11">Pyston</a></h3>
<p>This is a new attempt at making a faster python implementation, backed by Dropbox. You can take a look at their
progress on their <a class="reference external" href="https://github.com/dropbox/pyston">github project page</a>.</p>
</div>
<div class="section" id="python-2-to-3">
<h3><a class="toc-backref" href="#id12">Python 2 to 3</a></h3>
<p><a class="reference external" href="http://lucumr.pocoo.org/2013/5/21/porting-to-python-3-redux/">Armin Ronacher's blog post on porting from 2 to 3</a>,
really good stuff. He's been a strong critic of Python 3, but I feel that he justifies is position well and regularly.</p>
</div>
</div>
I've joined the dark side2014-03-23T20:45:00+06:002014-03-23T20:45:00+06:00Nafiul Islamtag:nafiulis.me,2014-03-23:i've-joined-the-dark-side.html<div class="contents topic" id="table-of-contents">
<p class="topic-title first">Table of Contents</p>
<ul class="simple">
<li><a class="reference internal" href="#some-developer-apps" id="id2">Some developer apps</a><ul>
<li><a class="reference internal" href="#oh-my-zsh" id="id3">Oh-My-ZSH</a></li>
<li><a class="reference internal" href="#homebrew" id="id4">Homebrew</a></li>
<li><a class="reference internal" href="#pyenv" id="id5">pyenv</a></li>
<li><a class="reference internal" href="#sublime-text-3" id="id6">Sublime Text 3</a></li>
<li><a class="reference internal" href="#pycharm" id="id7">PyCharm</a></li>
<li><a class="reference internal" href="#keka" id="id8">Keka</a></li>
<li><a class="reference internal" href="#appcleaner" id="id9">AppCleaner</a></li>
</ul>
</li>
<li><a class="reference internal" href="#the-switch" id="id10">The Switch</a></li>
</ul>
</div>
<p>So, after debating about it for quite some time, I finally got myself a Mac! Now, I got myself a Mac book pro 15 inch
and it pretty much cost me a fortune. So far, I've been playing around with it, and I gotta say, that I would prefer
a Mac over windows for Python development, any day. But before I tell you why its so great, let me share a couple of
tools that I've found immensely useful. So, here's a list that any Python dev would find useful on a new Mac:</p>
<div class="section" id="some-developer-apps">
<h2><a class="toc-backref" href="#id2">Some developer apps</a></h2>
<div class="section" id="oh-my-zsh">
<h3><a class="toc-backref" href="#id3">Oh-My-ZSH</a></h3>
<img alt="Oh my ZSH!" class="align-center" src="images/omzsh.png" />
<p>This is a must install for me on <em>any</em> *nix system. It gives me some <em>awesome</em> auto-completion, and some very
beautiful themes. Frankly, I don't even think I want <em>need</em> to justify why one should install this awesome tool,
just give it a tool, if you don't like it, then you can always change back. Check it out on <a class="reference external" href="https://github.com/robbyrussell/oh-my-zsh">oh-my-zsh</a>.</p>
</div>
<div class="section" id="homebrew">
<h3><a class="toc-backref" href="#id4">Homebrew</a></h3>
<p>This is such a wonderful package manager for Mac. I was initially split between Macports and Homebrew, but most of
the people I talked to, overwhelmingly favored Homebrew. What <em>really</em> got me to pick Homebrew was that Pyenv supported
a direct installation via Homebrew (Pyenv is next on the list). With homebrew installed with a quick:</p>
<div class="highlight"><pre><span></span>ruby -e <span class="s2">"</span><span class="k">$(</span>curl -fsSL https://raw.github.com/Homebrew/homebrew/go/install<span class="k">)</span><span class="s2">"</span>
</pre></div>
<p>And bam! All installed. I just copied and pasted the text from the <a class="reference external" href="http://brew.sh/">homebrew website</a>, so I did not
even notice that I was using ruby. You gotta love *nix systems for having python and ruby installed by default. With
brew, I made my first install:</p>
<div class="highlight"><pre><span></span>brew install pyenv
</pre></div>
</div>
<div class="section" id="pyenv">
<h3><a class="toc-backref" href="#id5">pyenv</a></h3>
<p>This allows you to manage different versions of Python with ease. Really useful if you want to work with different
versions of python on the command line. Check it out <a class="reference external" href="https://github.com/yyuu/pyenv">here</a>.</p>
</div>
<div class="section" id="sublime-text-3">
<h3><a class="toc-backref" href="#id6">Sublime Text 3</a></h3>
<img alt="Sublime Text 3" class="align-center" src="images/st3.png" />
<p>Wonderful text editor, I use this for small projects or a quick edit. For more intensive projects,
I use pycharm. Although their website says that ST3 is still in beta, its been pretty stable for me, and the package
manager for ST3 works very well. I suggest using ST3 because its starts up instantly,
compared to ST2. If you're going to be using this, then I suggest you install the package manager as soon as you've
installed ST3, <a class="reference external" href="https://sublime.wbond.net/installation">instructions here</a>.</p>
</div>
<div class="section" id="pycharm">
<h3><a class="toc-backref" href="#id7">PyCharm</a></h3>
<p>Well, I was bound to install this at <em>some</em> point. In my honest opinion, this is the Python IDE to beat. Just make sure
you create the <code>charm</code> Terminal variable, through "Create Command Line Launcher" option:</p>
<img alt="Creating PyCharm command line launcher" class="align-center" src="images/commandlinelauncher.png" />
</div>
<div class="section" id="keka">
<h3><a class="toc-backref" href="#id8">Keka</a></h3>
<img alt="Keka" class="align-center" src="images/keka.png" />
<p>This is for unpacking archives. Really simple to use tool. Its available on the Mac Store as well.</p>
</div>
<div class="section" id="appcleaner">
<h3><a class="toc-backref" href="#id9">AppCleaner</a></h3>
<img alt="App Cleaner" class="align-center" src="images/appcleaner.png" />
<p>Simple app that completely uninstalls your installed apps. Works like a charm. Get it
<a class="reference external" href="http://www.freemacsoft.net/appcleaner/">on their site</a>. I recommend this as a must install since a lot of the
times, your
apps leave behind settings, that come back to bite you if you reinstall.</p>
<p>And thats a wrap. You can find loads of other tools available, and <code>git</code> comes installed by default. If you
want to install mercurial, then its a simple <code>brew install hg</code>.</p>
</div>
</div>
<div class="section" id="the-switch">
<h2><a class="toc-backref" href="#id10">The Switch</a></h2>
<p>For me, it was <em>very</em> painful at first. The keyboard was just plain <em>wrong</em>, it had the <code>control</code> key in the
<em>wrong</em> place. I say <em>wrong</em> because the standard keyboard for windows and Linux is very different. I just wish Apple
had followed the general keyboard layout, it would make migration so much simpler. However, this was probably the
only problem I had with my new Mac Book pro, and everything else is pretty neat. The best thing about Mac is that the
platform had <em>many</em> apps for windows virtualisation. My pick would be Vmware Fusion, it allows you to run Windows
apps as if they were mac applications.</p>
<p>The only qualm I have right now is that I haven't been able to find a download manager that comes close to Internet
Download Manager (IDM). I'm using the free version of Folx. It sucks. It looks nice, but it sucks.</p>
<p>Other than that, I feel that a Mac was exactly what I needed since although Ubuntu is nice,
it lacks some good commercial software for screen recording, which is something that I <em>really</em> need.</p>
<p>If you make the switch to Mac, and you're an open source developer, I assure you that you won't regret it. Funnily
enough, three days after I got my mac, my PC just died. I took it to the repair guy,
and he told me that the motherboard was broke. I had just spend 50 bucks on repairs a week ago. The motherboard would
cost me another $250.</p>
<p>I won't my fixing my PC any time soon.</p>
</div>
Updating Pycharm 3.0.2 on Ubuntu2013-12-12T20:21:00+06:002013-12-12T20:21:00+06:00Nafiul Islamtag:nafiulis.me,2013-12-12:updating-pycharm-3.0.2-on-ubuntu.html<p>Now, I've just come out of an issue with Jetbrains PyCharm 3. The problem was this,
updating the software to <code>3.0.2</code>. I had <code>3.0.1</code> installed. Now, when the update <em>initially</em>
came around, the IDE told me, that I had to download the <em>entire</em> file all over again from
the jetbrains website.</p>
<p>So, I downloaded the newest version, and it turned out that the update was broken,
when I went to unzip the software, Archive Manager lashed out saying that it was a broken
archive file. So, I had another guy check it out, and he got broken archive too.</p>
<p>After a couple of days, jetbrains <em>realised</em> that they messed up so, they put in a patch.
I downloaded the patch, and this time <em>again</em>, I had another problem. What was it this time,
it turns out that I did not have rights to update Pycharm (imagine that!). So, in order to
update it, I had to find out <em>where</em> it was installed.</p>
<div class="highlight"><pre><span></span>whereis pycharm
</pre></div>
<p>Now, the file was installed <code>/opt/pycharm-3.0.1/bin/pycharm.sh</code>. So, I went in there,
and opened it as root:</p>
<div class="highlight"><pre><span></span>sudo bash /opt/pycharm-3.0.1/bin/pycharm.sh
</pre></div>
<p>After that, I just checked for updates, by clicking the check link to the bottom left of the dialogue
box:</p>
<img alt="Where to look for the update link" class="align-center" src="images/pycharm.png" />
<p>The update worked after that. So, if you find yourself in a similar predicament, then I strongly advise
you to run the <code>pycharm.sh</code> file as root.</p>
Fabric Entry Creation2013-12-11T00:58:00+06:002013-12-11T00:58:00+06:00Nafiul Islamtag:nafiulis.me,2013-12-11:fabric-entry-creation.html<p>So, in my attempts to create a functioning static blog generator using pelican,
I've been using fabric to create my entries so that I can easily sort them according
to their dates.</p>
<p>Here's the code:</p>
<div class="highlight"><pre><span></span><span class="k">def</span> <span class="nf">make_entry</span><span class="p">(</span><span class="n">title</span><span class="o">=</span><span class="s1">'default'</span><span class="p">):</span>
<span class="n">today</span> <span class="o">=</span> <span class="n">datetime</span><span class="o">.</span><span class="n">datetime</span><span class="o">.</span><span class="n">today</span><span class="p">()</span>
<span class="n">slug</span> <span class="o">=</span> <span class="n">title</span><span class="o">.</span><span class="n">lower</span><span class="p">()</span><span class="o">.</span><span class="n">strip</span><span class="p">()</span><span class="o">.</span><span class="n">replace</span><span class="p">(</span><span class="s1">' '</span><span class="p">,</span> <span class="s1">'-'</span><span class="p">)</span>
<span class="n">f_create</span> <span class="o">=</span> <span class="s2">"content/{}_{:0>2}_{:0>2}_{}.rst"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">today</span><span class="o">.</span><span class="n">year</span><span class="p">,</span> <span class="n">today</span><span class="o">.</span><span class="n">month</span><span class="p">,</span> <span class="n">today</span><span class="o">.</span><span class="n">day</span><span class="p">,</span> <span class="n">slug</span><span class="p">)</span>
<span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="n">f_create</span><span class="p">,</span> <span class="s1">'w'</span><span class="p">)</span> <span class="k">as</span> <span class="n">w</span><span class="p">:</span>
<span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="s1">'template.txt'</span><span class="p">,</span> <span class="s1">'r'</span><span class="p">)</span> <span class="k">as</span> <span class="n">r</span><span class="p">:</span>
<span class="n">s</span> <span class="o">=</span> <span class="n">r</span><span class="o">.</span><span class="n">read</span><span class="p">()</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">title</span><span class="o">=</span><span class="n">title</span><span class="p">,</span>
<span class="n">hashes</span><span class="o">=</span><span class="s1">'#'</span> <span class="o">*</span> <span class="nb">len</span><span class="p">(</span><span class="n">title</span><span class="p">),</span>
<span class="n">year</span><span class="o">=</span><span class="n">today</span><span class="o">.</span><span class="n">year</span><span class="p">,</span>
<span class="n">month</span><span class="o">=</span><span class="n">today</span><span class="o">.</span><span class="n">month</span><span class="p">,</span>
<span class="n">day</span><span class="o">=</span><span class="n">today</span><span class="o">.</span><span class="n">day</span><span class="p">,</span>
<span class="n">hour</span><span class="o">=</span><span class="n">today</span><span class="o">.</span><span class="n">hour</span><span class="p">,</span>
<span class="n">minute</span><span class="o">=</span><span class="n">today</span><span class="o">.</span><span class="n">minute</span><span class="p">,</span>
<span class="n">slug</span><span class="o">=</span><span class="n">slug</span><span class="p">)</span>
<span class="n">w</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="n">s</span><span class="p">)</span>
<span class="k">print</span><span class="p">(</span><span class="s2">"File created -> "</span> <span class="o">+</span> <span class="n">f_create</span><span class="p">)</span>
</pre></div>
<p>Now, when I was doing this, I initally used the <tt class="docutils literal">datetime.date</tt> class, but that did not
have the <tt class="docutils literal">hour</tt> and the <tt class="docutils literal">minute</tt> attributes in the class. I also did not want to bother with
<tt class="docutils literal">slug</tt> creation, so I got that build in too.</p>
<p>I tried to get all the template string inside the function, but later on decided to refactor it out to
a <tt class="docutils literal">template.txt</tt>, which I can more easily use and change:</p>
<div class="highlight"><pre><span></span>{title}
{hashes}
:date: {year}-{month}-{day} {hour}:{minute}
:tags:
:category:
:slug: {slug}
:author: Nafiul Islam
#Post goes here
</pre></div>
<p>The great thing about fabric is how easy it makes writing command line arguments. So,
if I wanted to create a file with say <tt class="docutils literal">{year}_{month}_{day}_{title}</tt>, I'd just do this:</p>
<div class="highlight"><pre><span></span>fab make_entry:'My awesome title here'
</pre></div>
<p>Now that <em>is</em> awesome! :D</p>