tag:milinda.svbtle.com,2014:/feedMilinda Pathirage2015-09-01T07:54:16-07:00Milinda Pathiragehttps://milinda.svbtle.comSvbtle.comtag:milinda.svbtle.com,2014:Post/copying-additional-files-with-gradle-application-plugin2015-09-01T07:54:16-07:002015-09-01T07:54:16-07:00Copying Additional Files With Gradle Application Plugin<p>When building application distributions for Java apps you often need to bundle default configuration files, other resource, etc. into your application distribution. If you are using <a href="http://gradle.org">Gradle</a> with <a href="https://docs.gradle.org/current/userguide/application_plugin.html">Gradle Application Plugin</a> for creating the application distribution for your project, you can use following code fragment in your Gradle project to copy addition files.</p>
<pre><code class="prettyprint lang-groovy">applicationDistribution.from("src/main/resources/conf") {
into "conf"
}
</code></pre>
<p>Above code fragment copies content of <em>conf</em> directory into application distributions <em>conf</em> directory located in the root directory of your distribution.</p>
tag:milinda.svbtle.com,2014:Post/monitoring-play-web-applications2015-05-20T09:37:23-07:002015-05-20T09:37:23-07:00Publishing Play Web Application Metrics to InfluxDB<p><a href="https://github.com/milinda/metrics-play">metrics-play</a> which is a fork of <a href="https://github.com/kenshoo/metrics-play">metrics-play</a> project can be used to publish Play web application metrics including JVM metrics to InfluxDB via Graphite protocol.</p>
<p>To add support for publishing Play application metrics to InfluxDB, first you have to add <em>metrics-play</em> with Graphite reporter into you Play app dependencies like below:</p>
<pre><code class="prettyprint">libraryDependencies ++= Seq(
"com.kenshoo" %% "metrics-play" % "2.3.0_0.2.1-graphite",
javaJdbc,
javaEbean,
cache,
javaWs
)
</code></pre>
<p>And then you have to add the Play plugin <code class="prettyprint">com.kenshoo.play.metrics.MetricsPlugin</code> to your play.plugins file. Then in the Play application configuration, you can enable and configure the Graphite reporter like below:</p>
<pre><code class="prettyprint">metrics {
graphite {
enabled = true
period = 1
unit = MINUTES
host = localhost
port = 2003
}
}
</code></pre>
<p>Above steps assume that you have configured InfluxDB Graphite input plugin like below.</p>
<pre><code class="prettyprint">[input_plugins]
# Configure the graphite api
[input_plugins.graphite]
enabled = true
# address = "0.0.0.0" # If not set, is actually set to bind-address.
port = 2003
database = "play-influxdb" # store graphite data in this database
udp_enabled = true # enable udp interface on the same port as the tcp interface
</code></pre>
<p>Current implementation of this plugin doesn’t support route level metrics. I am hoping to add them based on <a href="https://github.com/typesafehub/play-plugins/tree/master/statsd">StatsD plugin</a> for Play.</p>
tag:milinda.svbtle.com,2014:Post/cluster-and-service-monitoring-using-grafana-influxdb-and-collecd2015-04-30T11:25:24-07:002015-04-30T11:25:24-07:00Server Monitoring Solution Using Grafana, InfluxDB and collectd<p>Couple of days ago, I wanted to add couple of new nodes to Ganglia deployment I maintained to monitor <a href="http://www.hathitrust.org/htrc">HTRC</a> services and cluster nodes. Even though everything looks okay after installing and configuring Ganglia monitor daemons in new machines, I couldn’t get them to publish monitoring data to Ganglia’s gmetad. Worse thing was, I couldn’t find any errors (I am not sure whether I looked at the correct location, but I couldn’t find anything). I first tried to install <a href="http://pcp.io">Performance Co-Pilot</a> with Netflix <a href="https://github.com/Netflix/vector">Vector</a>, but couldn’t figure out how to setup a central metric collection server. Even though PCP and Vector combination looked really great, having to type the node host name every time I wanted to monitor a server was not what I wanted.</p>
<p>So I decided to give a try to <a href="http://grafana.org">Grafana</a>, <a href="http://influxdb.com">InfluxDB</a> and <a href="https://collectd.org">collectd</a> combination. I was able to get this setup working within couple hours with several dashboards for a subset of servers. Below is a screenshot of one of the dashboards.</p>
<p><a href="https://svbtleusercontent.com/fckfkivngpv9g.png"><img src="https://svbtleusercontent.com/fckfkivngpv9g_small.png" alt="Screen Shot 2015-04-30 at 1.50.02 PM.png"></a></p>
<p>In this post, I am going to discuss how to get these three tools working for a scalable and flexible monitoring solution for small scale cluster of nodes. </p>
<p>First of all you have to install latest InfluxDB in one of your nodes and do necessary firewall configurations to open up InfluxDB admin console port and InfluxDB back end port. You can find installation instructions <a href="http://influxdb.com/download/">here</a>. Next thing is to install collectd in one of your nodes. If you are using Ubuntu, <a href="https://www.digitalocean.com/community/tutorials/how-to-configure-collectd-to-gather-system-metrics-for-graphite-on-ubuntu-14-04">this</a> is a good document on how to install and configure collectd. </p>
<p>Your next task is to publish stats collected by collectd daemons to InfluxDB instance you just deployed. Sometime back we had to use collectd to InfluxDB proxy to get this done. But since 0.8.4 version InfluxDB supports native collectd protocol. To make this setup work, you have to enable collectd input plugin for InfluxDB. To do this you have to add following configurations to <em>input_plugins</em> section of InfluxDB configuration file /opt/influxdb/shared/config.toml.</p>
<pre><code class="prettyprint">[input_plugins.collectd]
enabled = true
port = 8096
database = "collectd"
typesdb = "/usr/share/collectd/types.db"
</code></pre>
<p><strong>typesdb</strong> definition is used by collectd to understand data it receives. You can copy this file from one of your servers running collectd or you can install collectd on the server you are running InfluxDB. Above example shows the scenario where we have collectd installed on the same server as InfluxDB. Just restart InfluxDB after necessary configurations are done. You can use whatever the <em>port</em> you like in above configuration. Also make sure to create database called <strong>collectd</strong> in InfluxDB via admin web ui or using REST API like below.</p>
<pre><code class="prettyprint">curl -X POST 'http://influxdb-host-name:8086/db?u=root&p=root' \
-d '{"name": "collectd"}'
</code></pre>
<p>Next step is to configure collectd to send data directly to InfluxDB instead of storing data as rrd files. In collectd configuration file (/etc/collectd/collectd.con in Ubuntu and /etc/collectd.conf in RedHat EL) enable <em>network</em> plugin and configure it to send metrics to InfluxDB as shown below.</p>
<pre><code class="prettyprint">Loadplugin network
...
<Plugin "network">
Server "influxdb-host-name" "8096"
</Plugin "network">
</code></pre>
<p>Other thing is you can disable <em>rrdtool</em> plugin because you no longer need that (We are sending metrics to InfluxDB). Restart collectd to make the configuration changes effective.</p>
<p>Next you can go to InfluxDB web ui and list the time series (<em>using ‘list series’ query</em>) you have in your database. If you are getting data from collectd, you should see something like below:</p>
<p><a href="https://svbtleusercontent.com/5xdl3gjspy66wq.png"><img src="https://svbtleusercontent.com/5xdl3gjspy66wq_small.png" alt="Screen Shot 2015-04-30 at 2.21.22 PM.png"></a></p>
<p>Final step is to deploy Grafana somewhere can create some dashboard widgets to visualize time series data you are interested in like I have shown in the first image. More information on how to configure InfluxDB data source for Grafana can be found <a href="http://docs.grafana.org/datasources/influxdb/">here</a>.</p>
<h3 id="related-posts_3">Related Posts <a class="head_anchor" href="#related-posts_3">#</a>
</h3>
<ul>
<li><a href="http://milinda.svbtle.com/monitoring-play-web-applications">Publishing Play Web Application Metrics to InfluxDB</a></li>
</ul>
tag:milinda.svbtle.com,2014:Post/interesting-resources-on-writing2015-04-17T18:33:14-07:002015-04-17T18:33:14-07:00Interesting Resources on Writing<p>Starting to participate in #The100DayProject by writing every day for 100 days got me into research more about writing. Writing is major part of the life as grad student. But I was far behind my writing and I wanted to improve by writing frequently. Its well known that writing is hard and it always be hard for lot of us. But writing more and more will make you better at it. Following two articles contains some interesting ideas and tips on how to get better at writing.</p>
<ul>
<li><a href="http://zettelkasten.de/posts/identity-schedule-serious-writing/">Make Writing a Part of Your Identity</a></li>
<li><a href="http://thesiswhisperer.com/2011/03/24/how-to-write-1000-words-a-day-and-not-go-bat-shit-crazy/">How to write 1000 words a day</a></li>
</ul>
<p>‘<a href="http://zettelkasten.de/posts/identity-schedule-serious-writing/">Make Writing a Part of Your Identity</a>’ has some interesting ideas about how to make writing a habit and why it is important to make writing a habit. I strongly suggest that article to anyone who wants to be better at writing. The article contains couple of major ideas.</p>
<ul>
<li>You have to track writing if you want to improve writing.</li>
<li>You have to make writing a habit and articles has some great advice on how to make writing a habit.</li>
<li>Article introduces concept called <strong>Knowledge Cycle</strong> where writing process is a cycle of steps – research, read, take note and write.</li>
</ul>
<p>’<a href="http://thesiswhisperer.com/2011/03/24/how-to-write-1000-words-a-day-and-not-go-bat-shit-crazy/">How to write 1000 words a day</a>’ is mostly about large writing projects such as thesis. But below tips from that post will be helpful to anyone who is working on a writing project whether it is big or small.</p>
<ul>
<li>
<strong>Spend less time at your desk:</strong> Don’t spend whole day writing. Dedicate less than a quarter a day or even less than that depending on the size of your project for writing, take a break and return later to clean it up.</li>
<li>
<strong>Remember the two hour role:</strong> Article about ‘<a href="http://www.kenrockwell.com/business/two-hour-rule.htm">The Two-Hour Rule</a>’</li>
<li>
<strong>Make writing new stuff the first thing you do when you sit down to your desk:</strong> IMO, this can be valid if you have to write full-time or for a task such as thesis writing. But in other cases this may not be completely applicable.</li>
<li>
<strong>Start in the middle:</strong> Don’t attempt to write introductions, conclusions or important transitions first.</li>
<li>
<strong>Write as fast as you can, not as well as you can:</strong> Worrying too much about sentences you write can slow down the process. So write as fast as you can and come back later to clean it up.</li>
<li>
<strong>Leve it to rest… then re-write:</strong> When you write fast most of the things you have written may looks crappy. Take a break and then start re-writing.</li>
</ul>
<p>Even though some points may not be applicable if you are working on a large writing project, I found above tips such as 'writing as fast as we can’ and ‘take a break and clean it up’ really helpful for any kind of writing.</p>
tag:milinda.svbtle.com,2014:Post/versioning-rest-apis2015-04-09T20:39:08-07:002015-04-09T20:39:08-07:00Versioning REST APIs<p>Yesterday, a discussion happened around versioning REST APIs resulted in an interesting sequence of events where one person even started to attack my self and one other person via personally e-mails. So I wanted to explore more about versioning REST APIs to understand the problem and solutions better. Lets start by looking at why versioning is needed.</p>
<p>As Troy Hunt discussed in his popular <a href="http://www.troyhunt.com/2014/02/your-api-versioning-is-wrong-which-is.html">post</a> on API versioning, the main reason is evolution of software. Its hard (may be even impossible) to get software <em>right</em> in the first release. As world moves on, new requirements may come. So introducing a new version is unavoidable.</p>
<p>When your API get used by various clients, you may have to maintain multiple versions. It’s not realistic to expect that everyone will migrate to new version within short time period. </p>
<p>There are multiple popular ways to version a REST API and there are proponents and opponents for each of these solutions.</p>
<ul>
<li>Include the API version in the URL like <a href="http://services.digg.com/2.0/comment.bury">http://services.digg.com/2.0/comment.bury</a>
</li>
<li>Adding a custom request header. For example Azure <a href="https://msdn.microsoft.com/en-us/library/azure/dd894041.aspx">services accept custom header</a> <code class="prettyprint">x-ms-version: 2014-02-14</code> to determine which API version to choose when responding to a client in some situations.</li>
<li>Modifying the <em>Accept</em> header to include the version like <code class="prettyprint">Accept: application/vnd.musicstore-v1+json</code>
</li>
</ul>
<p>People have different views about above mentioned solutions. Some people argue that, you should always keep the resource URL as a constant. Then people argues that adding customer header sucks because it’s not the ideal way (semantically) to describe a resource. People says <code class="prettyprint">Accept</code> header sucks because it is harder to test. You can no longer give a clickable URL to test the API. But as Troy has already discussed in his <a href="http://www.troyhunt.com/2014/02/your-api-versioning-is-wrong-which-is.html">article</a> there is no right and wrong way. You should go for the most practical solution for your situation.</p>
<p>If you are interested in this topic, you should read Troy’s post. And there are some other interesting posts as well.</p>
<ul>
<li>
<a href="http://www.lexicalscope.com/blog/2012/03/12/how-are-rest-apis-versioned/">How are REST APIs versioned?</a> - Lists how popular APIs do versioning.</li>
<li><a href="http://barelyenough.org/blog/2008/05/versioning-rest-web-services/">Versioning REST Web Services</a></li>
<li>
<a href="http://blog.steveklabnik.com/posts/2011-07-03-nobody-understands-rest-or-http">Nobody Understands REST or HTTP</a><br>
</li>
</ul>
tag:milinda.svbtle.com,2014:Post/cql-continuous-query-language2015-04-07T19:16:03-07:002015-04-07T19:16:03-07:00CQL - Continuous Query Language<p>In today’s data driven economy, organizations depend heavily on data analytics to stay competitive. Advances in <em>Big Data</em> related technologies transformed how organizations interact with data and as a result more and more data is generated at ever increasing rates. And most of these data is available as continuous streams and organizations utilizes stream processing technologies to extract insights in real-time (or as data arrives). As a result of this change in how we collect and process data stream processing platforms like Apache Storm, Spark Streaming and Apache Samza were created based on about a decade of experience using <em>Big Data</em> processing technologies such as Hadoop. </p>
<p>But these modern platforms lack support for SQL like declarative query languages and require sound knowledge on imperative style programming and distributed systems to effectively utilize them. But for broader adoption, support for SQL like continuous query languages or SQL with streaming extensions is required. In this post I’m going to discuss one such language known as <a href="http://dl.acm.org/citation.cfm?id=1146463">CQL</a> for querying data streams invented roughly 10 years ago. Theoretical framework and SQL extensions discussed in CQL paper is still important and we are using concepts from CQL as a foundation for <a href="https://issues.apache.org/jira/browse/SAMZA-390">Apache Samza’s Streaming SQL</a> implementation.</p>
<h1 id="what-is-cql_1">What is CQL? <a class="head_anchor" href="#what-is-cql_1">#</a>
</h1>
<p>CQL is not SQL, but a SQL based declarative language for querying streaming and stored relations (a.k.a. database tables). Abstract semantics of CQL relies on three types of operations – <em>stream-to-relations</em>, <em>relation-to-relation</em> and <em>relation-to-stream</em> – on two types of data – <strong>streams</strong> and <strong>relations</strong>. </p>
<h2 id="streams-and-relations_2">Streams and relations <a class="head_anchor" href="#streams-and-relations_2">#</a>
</h2>
<ul>
<li>
<strong>Stream</strong> - (possibly infinite) bag of elements <em><s></s></em>, where <em>s</em> is a tuple and <em>t</em> is the timestamp of the element</li>
<li>
<strong>Relation</strong> - time instant to finite but unbounded bag of tuples mapping. This is different from general definition of relation where there is no notion of time and relations in the context of CQL is know as <em>instantaneous relations</em> which varies with the time.</li>
</ul>
<h2 id="operators_2">Operators <a class="head_anchor" href="#operators_2">#</a>
</h2>
<ul>
<li>
<strong>Stream-to-relation</strong> - produces a (instantaneous) relation from a stream. Window operator (there are different types such as sliding and tumbling) is the only stream-to-relation operator available in CQL.</li>
<li>
<strong>Relation-to-relation</strong> - produces a relation from one or more relations. Selection, projection and aggregation operators in CQL are <em>relaiton-to-relation</em> operators.</li>
<li>
<strong>Relation-to-stream</strong> - produces a stream from a relation. Difference between previous and current instantaneous relation is used to convert a relation to a stream. </li>
</ul>
<p>Stream-to-stream operators are absent and they should be constructed by combining three types of operators defined above. Below figure from CQL <a href="http://dl.acm.org/citation.cfm?id=1146463">paper</a> is the best visualization of abstract semantics defined in CQL.</p>
<p><a href="https://svbtleusercontent.com/3oufi0345afvqa.png"><img src="https://svbtleusercontent.com/3oufi0345afvqa_small.png" alt="cql-semantics.png"></a></p>
<h1 id="why-cql-is-interesting_1">Why CQL is interesting? <a class="head_anchor" href="#why-cql-is-interesting_1">#</a>
</h1>
<p>Operators like <em>join</em> and some aggregation operators available in SQL are blocking and impossible to evaluate over streams. So, a window operator which divide the stream into possibly overlapping subsets is used after stream scan to reduce the scope of the query to a window extent. </p>
<p>In CQL, the concept of window is embedded into the semantics using the concept <em>instantaneous relation</em> and this allows query execution engines to implement operators such as joins and aggregations as they are operating on general relations. In addition to that, CQL allows integration of stored relations to streaming queries without any magic because once a stream is converted to an instantaneous relation, we are basically working on relations. </p>
<p>In addition to above mentioned semantic features, query execution strategy explained in CQL is also interesting.</p>
<h1 id="cql-query-execution_1">CQL Query Execution <a class="head_anchor" href="#cql-query-execution_1">#</a>
</h1><h2 id="streams-and-insertdelete-streams_2">Streams and Insert/Delete Streams <a class="head_anchor" href="#streams-and-insertdelete-streams_2">#</a>
</h2>
<p>In CQL runtime stream is represented as a sequence of timestamped <em>insert</em> tuples. And time-varying relation (bag of tuples) is represented as timestamped <em>insert</em> and <em>delete</em> tuples. These insertions and deletions represent the changing state of a relation. This technique makes easy to implement incremental processing of streams.</p>
<p>Synopses are used to maintain the intermediate state such as current contents of a sliding window or current state of a relation for join operation.</p>
<p>More information about CQL query execution can be found in Section 12 of CQL <a href="http://dl.acm.org/citation.cfm?id=1146463">paper</a>.</p>
<h1 id="limitations_1">Limitations <a class="head_anchor" href="#limitations_1">#</a>
</h1>
<p>Coming soon.</p>
tag:milinda.svbtle.com,2014:Post/freshet-cql-based-clojure-dsl-for-streaming-queries2015-01-05T08:25:10-08:002015-01-05T08:25:10-08:00Freshet - CQL based Clojure DSL for Streaming Queries (Draft)<p><strong><em>This blog post is still a draft.</em></strong></p>
<p>Interest on continuous queries on streams of data has increased over the last couple of years due to the need of deriving actionable information as soon as possible to be competitive in the fast moving world. Existing Big Data technologies designed to handle batch processing couldn’t handle today’s near real-time requirements and distributed stream processing systems like Yahoo’s <a href="http://incubator.apache.org/s4/">S4</a>, Twitter’s <a href="https://storm.apache.org/">Storm</a>, <a href="https://spark.apache.org/streaming/">Spark Streaming</a> and LinkedIn’s <a href="http://samza.incubator.apache.org/">Samza</a> were introduced into the fast growing Big Data eco-system to tackle real-time requirements. These systems are robust, fault tolerant and scalable to handle massive volumes of streaming data, but lack first class support for SQL like querying capabilities. All of these frameworks provide <a href="http://samza.incubator.apache.org/learn/documentation/0.8/api/overview.html">high-level</a> <a href="https://storm.apache.org/documentation/Trident-tutorial.html">programming</a> <a href="https://spark.apache.org/docs/latest/streaming-programming-guide.html">API</a>s in JVM compatible languages. </p>
<p>In the golden era of stream processing research, a lot of work has been done on query engines and languages for stream processing. But we have yet to adapt these work on streaming query languages to above mentioned distributed stream processing systems widely in use today. </p>
<p>Also with the transition from batch to real-time Big Data, different architectures were proposed to handle the <a href="http://lambda-architecture.net/">integration of batch and real-time systems (Lambda Archiecture)</a> as well as to <a href="http://www.kappa-architecture.com/">revolutionized the way we built today’s systems (Kappa Architecture)</a>. Even though there aren’t any standards (like SQL and Relational Algebra for DBs) on implementing these architectures, <a href="https://github.com/twitter/summingbird">Summingbird</a> implements <a href="http://lambda-architecture.net/">Lambda Architecture</a> based on <a href="http://en.wikipedia.org/wiki/Monoid">monoids</a>. Also there are <a href="http://spark-summit.org/2014/talk/applying-the-lambda-architecture-with-spark">other ways</a> to implement <em>Lambda Architecture</em> such as Spark’s Scala API for streaming and batch processing. Even though it is possible to implement <a href="http://www.kappa-architecture.com/">Kappa Architecture</a> manually using above mentioned frameworks, there aren’t any high-level frameworks like Summingbird for this purpose. <a href="https://github.com/milinda/Freshet">Freshet</a> tries to fill this gap by adapting continuous query semantics and execution planning methods discussed by Arasu et. al. in their paper <a href="https://cs.uwaterloo.ca/%7Edavid/cs848/stream-cql.pdf">The CQL Continuous Query Language: Semantic Foundations and Query Execution</a> to implement <em>Kappa Architecture</em> on top of Apache Samza.</p>
<p>Before going into details about <a href="https://github.com/milinda/Freshet">Freshet</a> , it’s important to discuss <em>Kappa Architecture</em> and <em>CQL</em>. These are the fundamental ideas and technologies which Freshet is based on.</p>
<h1 id="kappa-architecture_1">Kappa Architecture <a class="head_anchor" href="#kappa-architecture_1">#</a>
</h1>
<p><em>Kappa Architecture</em> which has the notion of – Everything Is A Stream – is <a href="http://radar.oreilly.com/2014/07/questioning-the-lambda-architecture.html">proposed</a> as an alternative to Lambda Architecture. In the link above, the author argues that, stream processing is a generalization of data-flow DAGs with support for check-pointing intermediate results and continuous output to the end user. And he emphasizes that we can actually use current distributed stream processing framework like Aapache Samza combine with message queue like <a href="http://kafka.apache.org/">Kafka</a>, which retains ordered data, to implement use cases handled by Lambda Architecture. Reprocessing is accomplished by replaying the stream through new versions of stream processing code or for a completely new algorithm.</p>
<h1 id="cql-continuous-query-language_1">CQL - Continuous Query Language <a class="head_anchor" href="#cql-continuous-query-language_1">#</a>
</h1>
<p><a href="https://cs.uwaterloo.ca/%7Edavid/cs848/stream-cql.pdf">CQL</a> - aka Continuous Query Language - is a SQL- based declarative language for expressing queries over data streams and time varying relations. CQL’s abstract semantics are based on two data types - <strong>streams</strong> and <strong>relations</strong> - and three types of operations - <em>stream-to-relations</em>, <em>relation-to-relation</em> and <em>relation-to-stream</em>. In CQL, stream is a infinite bag of tuples and relation is a mapping from time Τ to a finite but unbounded bag of tuples. This special variant of the standard relation is called <em>instantaneous relation</em> in the context of CQL, because a relation <strong>R</strong> in CQL represent a finite but unbounded bag of elements at a given time instance τ. CQL takes advantage of well understood relational semantics and keeps the language simple and queries compact by introducing minimal changes to SQL.</p>
<ul>
<li>Window specification derived from SQL-99 to transform streams to relations</li>
<li>Three new operators to transform time varying relations into streams.</li>
</ul>
<h2 id="cql-sample-emfiltering-a-streamem_2">CQL Sample - <em>Filtering A Stream</em> <a class="head_anchor" href="#cql-sample-emfiltering-a-streamem_2">#</a>
</h2>
<pre><code class="prettyprint">SELECT Rstream(*)
FROM PosSpeedStr [Now]
WHERE speed > 65
</code></pre>
<p>CQL uses SQL for relation-to-relation transformations, but relations in CQL are different from relations in SQL. CQL relations vary with time. CQL introduces two new concepts: insert/delete streams, which encode both streams and relations in a unified way, and synopses, which contain state (e.g. a counter or buffer of messages) for an operator.</p>
<p>Another important thing is the fact that we can use traditional database relations in CQL queries which enables us to do things like stream-relation joins common in real world applications.</p>
<h1 id="freshet_1">Freshet <a class="head_anchor" href="#freshet_1">#</a>
</h1>
<p>Let’s come back to <a href="https://github.com/milinda/Freshet">Freshet</a> . <a href="https://github.com/milinda/Freshet">Freshet</a> is a first step towards a complete implementation of Kappa Architecture based on <a href="https://cs.uwaterloo.ca/%7Edavid/cs848/stream-cql.pdf">CQL</a> to support continuous queries. Freshet implements a subset (select, windowing, aggregates) of CQL on top of Apache Samza. Freshet implements RStream and IStream relation-to-stream operators, tuple and time based sliding windows to convert streams to relations and basic relation to relation operators for implementing business logic. Following CQL, Freshet uses <em>insert/delete</em> stream to model instantaneous relations.</p>
<p><a href="https://svbtleusercontent.com/bzsmth0xzik1jq.jpg"><img src="https://svbtleusercontent.com/bzsmth0xzik1jq_small.jpg" alt="freshet-arch.jpg"></a></p>
<p>As shown in above figure, Freshet is built out of five main logical components.</p>
<ul>
<li>
<strong>Query DSL</strong>: Implemented as a Clojure DSL and used to express CQL queries against streams. Queries expressed in Freshet DSL will get compiled in to streaming relation algebra model and then will get converted into an execution plan that consists of a set of operators written as Samza <a href="http://samza.incubator.apache.org/learn/documentation/0.8/api/overview.html">stream tasks</a> connected together as a DAG via Kafka queues.</li>
<li>
<strong>Query Compiler</strong>: Compile SQL model generated from DSL to intermediate representation which can be converted in to execution plan.</li>
<li>
<strong>Execution Planner</strong>: Generate execution plans (Samza jobs connected via input, intermediate and output streams to form a DAG) based on intermediate representation and current status of the Freshet cluster.</li>
<li>
<strong>Scheduler</strong>: Does the actual scheduling of Samza Jobs.</li>
<li>
<strong>Query Operators</strong>: Samza stream tasks. Implement
CQL operators like window, select, aggregate, and view generation operators like rstream, istream. These operators, connected via intermediate streams, perform stream processing according to the query express in Freshet DSL.</li>
</ul>
<h1 id="freshet-dsl_1">Freshet DSL <a class="head_anchor" href="#freshet-dsl_1">#</a>
</h1>
<p>Freshet Clojure DSL is inspired by <a href="http://sqlkorma.com/">Korma</a> Clojure DSL for SQL. Freshet DSL follows the same style as Korma DSL. There are two main constructs in the current Freshet DSL. <strong>defstream</strong> and <strong>select</strong> queries. These are two forms I am planning to support in the initial version. Other constructs will be added later.</p>
<h2 id="defstream_2">defstream <a class="head_anchor" href="#defstream_2">#</a>
</h2>
<p>Used to define a new stream. Streams defined using <em>defstream</em> represent Kafka topics in the current implementation. New modifiers will be added to <em>defstream</em> in future to support different input sources. The most important modifier for <em>defstream</em> is the <em>stream-fields</em> modifier, which modifies the stream definition with a field name/type mapping. Clojure <em>keywords</em> are used to specify field names and these keywords will get converted to strings internally. There are pre-defined keywords for specifying types like :string, :integer. Below is how we define a wikipedia activity stream to use in stream queries.</p>
<pre><code class="prettyprint lang-clojure">(defstream wikipedia-activity
(stream-fields [:title :string
:user :string
:diff-bytes :integer
:diff-url :string
:unparsed-flags :string
:summary :string
:is-minor :boolean
:is-unpatrolled :boolean
:is-special :boolean
:is-talk :boolean
:is-new :boolean
:is-bot-edit :boolean
:timestamp :long])
(ts :timestamp))
</code></pre>
<h2 id="select_2">select <a class="head_anchor" href="#select_2">#</a>
</h2>
<p>Used to define select queries over streams. Stream filtering using <em>where</em> and <em>aggregators</em> will be supported in the initial version and <em>joins</em> will be added next. Below is a sample select query which filters a stream.</p>
<pre><code class="prettyprint lang-clojure">(select wikipedia-activity
(modifiers :istream)
(window (unbounded))
(where (> :diff-bytes 100)))
</code></pre>
<h1 id="query-execution_1">Query Execution <a class="head_anchor" href="#query-execution_1">#</a>
</h1>
<p>Freshet follows the same execution semantics as CQL and flow of execution is shown below.</p>
<p><a href="https://svbtleusercontent.com/csdnd7s46rbvq.jpg"><img src="https://svbtleusercontent.com/csdnd7s46rbvq_small.jpg" alt="freshet-query-execution.jpg"></a></p>
<p>Window operator converts input stream into a <em>time varying relation</em> and the <em>time varying relation</em> is encoded as <em>insert/delete</em> stream to make it easy to implement relational operators as streaming operators. The relational part of the query is converted to a DAG of Samza operators, which will operate on insert/delete streams according to the query definition. Finally the stream materializer will materialize the stream according to specification of the original query. </p>
<h1 id="why-samza_1">Why Samza <a class="head_anchor" href="#why-samza_1">#</a>
</h1>
<p>Freshet chose Samza for its initial implementation mainly because</p>
<ul>
<li>Samza is fully integrated with Kafka</li>
<li>Samza supports and encourages <a href="http://samza.incubator.apache.org/learn/documentation/0.8/container/state-management.html">stateful stream processing</a>
</li>
<li>Samza’s local storage is really useful for implementing CQL synopses</li>
</ul>
<p>Samza’s property based job configuration is the only limitation when it comes to Freshet. A Storm-like topology builder would have come in handy for Freshet-like layers on top of Samza.</p>
<h1 id="current-status-and-future-work_1">Current Status and Future Work <a class="head_anchor" href="#current-status-and-future-work_1">#</a>
</h1>
<p>I am working on bridging the Clojure DSL and the CQL operator layer (Samza stream tasks) currently. I plan to do the initial release within couple of weeks. After the initial release of Freshet, I am planning to contribute to Apache Samza’s <a href="https://issues.apache.org/jira/browse/SAMZA-390">Stream Query implementation</a>, which is also based on CQL. Once finished, Freshet can be updated to use Apache Samza’s CQL operators directly, rather than having its own.</p>
tag:milinda.svbtle.com,2014:Post/good-reads-october-9th-20142014-10-09T16:46:48-07:002014-10-09T16:46:48-07:00Good Reads: October 9th 2014<ul>
<li> <a href="http://www.mmds.org/#ver21">Mining of Massive Datasets</a> - This is a very interesting book which covers many topics in Big Data including Map-Reduce, Recommendation Systems, Mining Social-Network Graphs, Dimensionality Reduction and Large-Scale Machine Learning. I am currently in the 2nd chapter, but found lot of interesting things related to Map-Reduce such as modeling relation algebra using Map-Reduce which is really interesting. Those who are interested in large scale data mining can also follow the free online <a href="https://www.coursera.org/course/mmds">course</a> from the authors of this book.</li>
<li> <a href="https://colah.github.io/posts/2014-10-Visualizing-MNIST/">Visualizing MNIST: An Exploration of Dimensionality Reduction</a> - This is also a really interesting post about dimensionality reduction in the context of machine learning. The post is well written, even someone not familiar with machine learning, deep learning and dimensionality reduction can read and understand underlying concepts with the help of awesome visualization you will find there.</li>
<li>
<a href="http://cs.ulb.ac.be/public/_media/teaching/infoh417/sql2alg_eng.pdf">Translating SQL into the Relational Algebra</a> - Talks about basic concepts of translating SQL to Relation Algebra. If you are new to concepts of SQL to Relation Algebra translation, this document contains brief descriptions, examples and corner cases about most of the translation tasks such as translating subqueries and joins.</li>
</ul>
tag:milinda.svbtle.com,2014:Post/good-reads-september-30th-20142014-09-30T07:05:43-07:002014-09-30T07:05:43-07:00Good Reads: September 30th 2014<ul>
<li>
<a href="http://www.bailis.org/blog/linearizability-versus-serializability/">Linearizability versus Serializability</a> - Clarifies the differences between <em>linearizability</em> and <em>serializability</em>, two important properties about interleavings of operations in databases and distributed systems. </li>
<li>
<a href="http://muratbuffalo.blogspot.com/2014/09/paper-summary-high-availability.html">Paper Summary: High-availability distributed logging with BookKeeper</a> - Distributed logging with high availability and many distributed readers are interested in reading the logs.</li>
<li>
<a href="https://www.youtube.com/watch?v=fU9hR3kiOK0">Turning the database inside out with Apache Samza</a> - Different way of thinking about databases and how we develop database backed applications. Propose the idea of applying <em>Stream</em> abstraction everywhere, from database to backend web services to the UI.</li>
</ul>
tag:milinda.svbtle.com,2014:Post/academic-writing-with-markdown-pandoc-and-emacs2014-09-25T17:50:03-07:002014-09-25T17:50:03-07:00Academic Writing With Markdown, Pandoc and Emacs<p>LaTeX is the de-facto standard for academic writing. And there are several editors available for LaTeX. But the problem with these editors and generally editing LaTeX in any editor like Emacs is the fact that LaTeX is not writer friendly. LaTeX commands will dominate your document and can be distracting most of the time. </p>
<p>What we want is <em>writer-friendly</em> LaTeX editor as described <a href="http://blog.johnjcamilleri.com/2012/02/latex-editor-manifesto/">here</a><sup>1</sup>. But we don’t have such a editor with features described in [1]. One alternative I found<sup>2</sup> is to use combination of <a href="http://johnmacfarlane.net/pandoc/demo/example9/pandocs-markdown.html">Markdown</a>, <a href="http://johnmacfarlane.net/pandoc/">Pandoc</a> and LaTeX.</p>
<p>Basic process is as follows:</p>
<ol>
<li>Create LaTeX <a href="https://gist.github.com/milinda/96eef25b4a6e4a432b55">header</a> and <a href="https://gist.github.com/milinda/1f25ffbbd7f881d242cf">footer</a> files, where header includes up to the abstract and footer include bibliography and document end tag. Any package imports or new command definitions can go in header.</li>
<li>Write the main content in <a href="http://johnmacfarlane.net/pandoc/demo/example9/pandocs-markdown.html">Pandoc Markdown</a>.</li>
<li>Convert Markdown file to LaTeX using <a href="http://johnmacfarlane.net/pandoc/">Pandoc</a>.</li>
<li>Append generated LaTeX file and footer latex file to the header and use preferred LaTex to PDF converter.</li>
</ol>
<p><a href="https://gist.github.com/milinda/d9272e67711d324f18ba">md2latex.sh</a> is a very simple script which automates the above process. This was based on the script from original article [2]. </p>
<p>You can use <a href="http://www.iawriter.com/mac/">iA Writer</a>, <a href="http://www.hogbaysoftware.com/products/writeroom">WriteRoom</a>, <a href="http://writemonkey.com">WriteMonkey</a>, <a href="http://writeapp.net">Write App</a> or Emacs with <em>writeroom-mode</em> for distraction free Markdown editing. </p>
<p>This is how <em>writeroom-mode</em> looks in Emacs.</p>
<p><a href="https://svbtleusercontent.com/x78m3nw7qo6ubg.jpg"><img src="https://svbtleusercontent.com/x78m3nw7qo6ubg_small.jpg" alt="writeroom-mode.jpg"></a></p>
<p><small>[2] - I couldn’t find the reference link to the original idea. Will post it as soon as I find it.</small></p>