ANTLYPLS.COM

blog.antlypls.com

Getting started with Spark Streaming using Docker - Thoughts Resampled

http://blog.antlypls.com/blog/2015/10/05/getting-started-with-spark-streaming-using-docker

Getting started with Spark Streaming using Docker. Stream processing technologies have been getting a lot of attention lately. You’ve already might heard about Kafka. And its streaming extension. If you’ve always wanted to try Spark Streaming, but never found a time to give it a shot, this post provides you with easy steps on how to get development setup with Spark and Kafka using Docker. Note: This walkthrough covers OS X and uses Homebrew. If you want just to get code, you can find complete example here.

blog.antlypls.com

Processing JSON data with Spark SQL - Thoughts Resampled

http://blog.antlypls.com/blog/2016/01/30/processing-json-data-with-sparksql

Processing JSON data with Spark SQL. Provides built-in support for variety of data formats, including JSON. Each new release of Spark contains enhancements that make use of DataFrames API with JSON data more convenient. Same time, there are a number of tricky aspects that might lead to unexpected results. In this post I’ll show how to use Spark SQL to deal with JSON. Examples below show functionality for Spark 1.6 which is latest version at the moment of writing. Also you read JSON data from. Will not lo...

blog.antlypls.com

Running spark-shell in browser with Apache Mesos and Marathon - Thoughts Resampled

http://blog.antlypls.com/blog/2015/11/13/running-spark-shell-in-browser-with-apache-mesos-and-marathon

Running spark-shell in browser with Apache Mesos and Marathon. I’d like to share a small trick on how to run. As a web app using Mesos. This might be useful for debugging or just to try some spark code. And to do that you don’t need to install spark on a cluster. All you need are Mesos cluster, Marathon framework running on top of Mesos, and Java runtime installed on Mesos slaves. Another secret ingredient is a gotty tool. It simply wraps terminal session into a web app. So the idea is to run. Allows wri...

blog.antlypls.com

How to Write Data into Parquet - Thoughts Resampled

http://blog.antlypls.com/blog/2015/12/02/how-to-write-data-into-parquet

How to Write Data into Parquet. Column oriented data stores have proven its success for many analytical purposes. Such success was shown by RCFile. Formats and their wide adoption in many distributed data processing tools in Hadoop ecosystem. Another columnar format that has gained popularity lately is Apache Parquet. Which is now top level Apache project. It is supported by many data processing tools including Spark. Provide support for parquet format. For demo purposes I simply use protobuf. Frankly, i...

blog.antlypls.com

Blog Archive - Thoughts Resampled

http://blog.antlypls.com/blog/archives

When Data Driven App Smells Bad. Spark SQL and Parquet files. Processing JSON data with Spark SQL. How to Write Data into Parquet. Running spark-shell in browser with Apache Mesos and Marathon. Getting started with Spark Streaming using Docker. When Data Driven App Smells Bad. Spark SQL and Parquet files. Processing JSON data with Spark SQL. How to Write Data into Parquet.

blog.antlypls.com

Spark SQL and Parquet files - Thoughts Resampled

http://blog.antlypls.com/blog/2016/02/29/spark-sql-and-parquet-files

Spark SQL and Parquet files. This is a short note on how to deal with Parquet files with Spark. I showed how to write parquet files using just parquet library. But Spark SQL has built-in support. For Parquet data format, which makes processing data in parquet files easy using simple DataFrames API. Reading DataFrame from parquet is simple as:. And writing data to parquet files:. More advanced topics like partitioning and schema merging will be covered later. Posted by Anatoliy Plastinin.

blog.antlypls.com

When Data Driven App Smells Bad - Thoughts Resampled

http://blog.antlypls.com/blog/2016/04/29/when-data-driven-app-smells-bad

When Data Driven App Smells Bad. Recently I have been involved in reviewing one “ Big Data. Rdquo; project. I can’t say that volume of data processed by that app was that big, but the project was treated as real big data project by people who developed it. They used Hadoop. Just because that is what you use for big data. And even project’s codename had phrase “Big Data” in it. R with UI implemented in PHP and Shiny Apps. Written with python, R or etc. Then those notebooks are transformed into a bunch...

blog.antlypls.com

spark-shell without Spark - Thoughts Resampled

http://blog.antlypls.com/blog/2016/11/10/spark-shell-without-spark

Is an interactive shell that comes with spark’s distribution. Shell is useful for learning api, quick experiments, prototyping and etc. But to do that you don’t need to have cluster or even have spark distribution installed. What you need to have is a basic knowledge of scala. File with following content:. It defines two spark components as dependencies:. Later for testing. You can import other spark libraries that you need like MLLib, Streaming and etc. Let’s start with setting up a. Have fun playing wi...