Tag Archives: Kafka

Video: Maxwell generates Kafka events from MySQL binlogs

Maxwell is a new open source project that replays MySQL binlogs as JSON events, focusing on replicating events to Kafka. This talk will discuss Maxwell, Mysql binlogs, Kafka, and the systems that Zendesk is building on top of stream-based architecture.

How Apache Kafka is transforming Hadoop, Spark,Storm

Video: Apache Kafka with Spark Streaming_ Real Time Analytics Redefined

Spring XD for Data Ingestion

via Spring 😄 for Data Ingestion | Java Code Geeks.

Spring 😄 is a powerful tool that is an installable set of Spring Boot services that run either standalone, on top of YARN or on top of EC2. Spring 😄 also includes an admin UI website and a command line tool for job and stream management. Spring 😄 is a powerful set of services that work with a variety of datasources.

For ideal usage, it should be running inside an Apache Spark or Hadoop cluster. In the first section we will setup 😄 to run on a Centos/RHEL machine with the necessary data services. These are for the infrastructure required to run and also for data ingestion. You can integrated your existing RDBMS, MongoDB, Kafka, Apache Spark, Hadoop, REST, RabbitMQ and other services.

You can also install 😄 on Mac, Windows and other Linux distributions. For basic usage on a developer machine, just download Spring 😄 from the Spring.IO website and run xd/xd/bin/xd-standalone and that will be sufficient for running data ingestion.

Video: Real-Time Visualization with Kafka, Storm, Redis, node.js & d3.js

In this meetup, Byron will demonstrate a realtime dashboard for streaming events.
This application involves several technologies like Kafka, Storm, Redis, node.js and d3.js.

This lecture is a preview of a lecture that will be held later this month in Strata conference in Santa Clara CA.

The lecture covers:
– Demo of the application
– Explain how the above technologies helped us achieve a realtime dashboard for mass data
– Go a bit deeper to code samples of each part.

NodeJS client for Apache Kafka 0.8

via SOHU-Co/kafka-node · GitHub.

Kafka-node is a Node.js client with Zookeeper integration for Apache Kafka. It only supports the latest version of Kafka 0.8 which is still under development, so this module is not production ready so far.

The Zookeeper integration does the following jobs:

  • Loads broker metadata from Zookeeper before we can communicate with the Kafka server
  • Watches broker state, if broker changes, the client will refresh broker and topic metadata stored in the client

Java Backend Developer

Role : Java Backend Developer
Location: Beaverton, OR

Note: Interested candidates can send me there updated resume at kdinesh@prokarma.com or you can also reach me at (402) 905 9212. Please share or like this post.

Needed for this Position
• Experience developing applications to run in a large-scale environment.
• Experience integrating and designing NoSQL technologies inside a real-time analytics stack. This can include MongoDB, Cassandra, HBase or Couchbase.
• Experience working with real-time streaming data service technologies such as Kinesis, Kafka, or Flume.
• Experience with distributed real-time computational (CEP) tools like Storm, Tibco Streambase, etc
• Experience designing and developing on horizontally and highly scalable cloud-based architecture, AWS cloud infrastructure experience strongly preferred.
• Experience developing in a continuous integration environment using Jenkins, Bamboo, or TeamCity CI frameworks.
• Experience writing automated unit and integration tests using JUnit or TestNG testing frameworks.

Preferred if you have:
• Experience working in an Agile development environment.
• Experience developing ecommerce based web applications.
• Experience with cloud DevOps tools like Puppet, Chef or Vagrant.
• Experience with version control tools such as GIT/Stash or SVN.
• Experience with Linux (CentOS, Ubuntu, etc).
• Experience using and integrating with software & logging analysis tools such as New Relic & Splunk.
• Experience developing in a TDD environment.
• Experience with indexing technologies such as SOLR or Elastic Search.
• Experience working with Hadoop or other MapReduce technologies.
• Experience in a Blue/Green deployment model.
• Experience in a continuous delivery environment.