engineer.crowdworks.jp
CW Tech Meetup #02: Ruby on Rails Tech Meetup を開催しました - クラウドワークス エンジニアブログ
http://engineer.crowdworks.jp/entry/2016/10/28/160415
CW Tech Meetup #02: Ruby on Rails Tech Meetup を開催しました. 先日クラウドワークスでは、Ruby on Railsについて利用したアプリケーション開発からRailsそのものについて気軽に情報交換できる場を作りたいということで CW Tech Meetup #02. Upgrade Forkwell to Rails5 the faster. さんより Upgrade Forkwell to Rails5 the faster をご講演いただきました。 The State of Sprockets. さんより The State of Sprockets をご講演いただきました。 今後クラウドワークスでは CW Tech Meetup #03. CW Tech Meetup #02: Ruby on Rails Tech Meetup を開催しました. Lambda CloudWatch Events KMS で AWS. レポート 総勢27名 クラウドワークス開発合宿を行いました @湯河原温泉旅館 おんやど恵.
davidzchen.com
dzc / Projects
http://davidzchen.com/projects.html
David Z. Chen. Open source projects that I am or have been involved with. My contributions to open source projects. Googles own build system. Bazel has built-in support for building both client and server software, including apps for both Android and iOS, and provides an extensible framework for develop your own build rules. Bazel currently has support for over 15 programming languages, including C/C , Java, Python, Go, Rust, D, Jsonnet, and Scala. Http:/ google.github.io/jsonnet/doc/. Https:/ git-wip-us...
wecode.wepay.com
Building WePay's data warehouse using BigQuery and Airflow
https://wecode.wepay.com/posts/wepays-data-warehouse-bigquery-airflow
Go to wepay.com. Building WePay's data warehouse using BigQuery and Airflow. Jul 5, 2016. Over the coming weeks, we’ll be writing a series of posts describing how we’ve built and run WePay’s data warehouse. This series will cover our usage of Google Cloud Platform, BigQuery, and Apache Airflow (incubating), as well as how we handle security, data quality checks, and our plans for the future. We run MySQL as our primary OLTP. Multi-tenancy issues, where one user would severely degrade the entire cluster.
confluentinc.wordpress.com
November 2014 – Confluent
https://confluentinc.wordpress.com/2014/11
The value of Apache Kafka in Big Data ecosystem. Confluent will be at QCon NYC next week. Using logs to build a solid data infrastructure (or: why dual writes are a bad idea). How I Learned to Stop Worrying and Love the Schema, part 1. Compatibility Testing For Apache Kafka. Martin Kleppmann on Bottled Water: Real-time integ…. On Bottled Water: Real-time integ…. On Bottled Water: Real-time integ…. On Stream processing, Event sourc…. The Immutable Stack…. On Turning the database inside-ou…. But adopting t...
erikbern.com
Pinterest open sources Pinball · Erik Bernhardsson
https://erikbern.com/2015/03/14/pinterest-open-sources-pinball
Pinterest open sources Pinball. Pinterest just open sourced Pinball. Which seems like an interesting Luigi. Alternative. There’s two blog posts: Pinball: Building workflow management. From 2014) and Open-sourcing Pinball. From this week). The author has a comment in the comments thread. Luigi was not available in public, when Pinball starts. So not sure the pros and cons between Pinball and Luigi. Flexible workflow scheduling policy, easy failure handling. We provide rich UI for you to easily manage your...
techlife.cookpad.com
巨大なバッチを分割して構成する 〜SQLバッチフレームワークBricolage〜 - クックパッド開発者ブログ
http://techlife.cookpad.com/entry/2015/06/27/154407
好きなRubyのメソッドは10年前からString#slice(re, nth)ですが、 最近はRubyよりCoffeeScriptとSQLのほうが書く量が多くて悩んでいます。 今日はわたしが開発している たべみる の背後で働いている 巨大バッチの構成について話したいと思います。 バレンタイン の検索頻度グラフ 2014年 2015年. これは一般的なRailsアプリに比べれば遅いほうですが、 Amazon Redshiftに直接アクセスして6年分のデータ 10億件を余裕で越えます に対して分析を行っていることを考えると、実は非常に高速と言える速度なのです。 このような分析を高速に実行できるようにするために、 背後では事前に集計を済ませた、いわゆる サマリーテーブル を大量に作成しています。 たべみるバッチは日次 1日に1回の頻度 で動き、 次のような仕組みでサマリーテーブルを更新します。 まず元データをcookpad.comのメインデータベースであるMySQLと、 Treasure Data Hadoop から取得してRedshiftに入れます。 Item id , r.item id as.
iinteractive.com
Infinity Interactive | Capabilities
http://www.iinteractive.com/capabilities.html
You may not know our language, but we know yours. But not only yours. We speak the language of everyone involved in your project and translate it. We translate your business needs into solutions the CTO and CFO can get behind. We translate technical limitations into alternative approaches. We translate the designer’s vision into an implementable, usable and effective face for your customer. Software development—User-facing and behind the scenes. At the end of the day, what matters is your customers&rsquo...
chatwithengineers.com
Demystifying Different Roles in Data Team – chatwithengineers
https://chatwithengineers.com/2016/08/21/demystifying-different-roles-in-data-team
Demystifying Different Roles in Data Team. One question I often get is how to build a data team in a company. This is mostly asked by people who are doing a startup or thinking about starting one seriously. This question is followed by another set of questions. Two common ones are:. What does a data scientist or data analyst or data engineer do? How are all these positions different? If I have to hire one data person, should I hire a data scientist or a data engineer? Beginning of Data Organization.