Kafka Kerberos Enable and Testing.

linux centos redhat cloudera kafka kerberos cluster

Apache Kafka is a distributed streaming platform. Kafka 2.0 supports Kerberos authentication, Enabling Kerberos Authentication Using the Wizard on cloudera manager. Courtesy - Apache Kafka

READ MORE

Cloudera Manager - Duplicate entry 'zookeeper' for key 'NAME'.

linux centos redhat cloudera kafka zookeeper cluster

We had recently built a cluster using cloudera API’s and had all the services running on it with Kerberos enabled. Next we had a requirement to add another kafka cluster to our already exsisting cluster in cloudera manager. Since it is a quick task to get the zookeeper and kafka up and running. We decided to get this done using the cloudera manager instead of the API’s. But we faced the Duplicate entry 'zookeeper' for key 'NAME' issue as described in the bug below.

https://issues.cloudera.org/browse/DISTRO-790

READ MORE

Parcel Not Distributing Cloudera CDH.

linux centos redhat cloudera hadoop cluster

We were deploying one of the cluster on our lab environment which is used by everyone. So the lab has it own share of stale information on it.

READ MORE

Creating /etc/hosts file in Chef.

linux centos redhat chef hadoop cluster

We had a cluster environment which we needed to update the /etc/hosts file. Which would help communicate between the server over a private network. Our servers have multiple interfaces and we need them to communicate between each other using the private network.

READ MORE

Enable Kerberos Using Cloudera API.

linux cloudera hadoop cloudera-api kerberos

Python API for cloudera is really nice, apart from getting the cluster setup, we can also do configuration and automation. We use a lot of automation using Chef/Ansible, but cloudera API give more control over the cluster.

READ MORE

Setting Up HDFS Services Using Cloudera API [Part 3]

linux cloudera hadoop cloudera-api zookeeper hdfs

This is the second follow up post. In the earlier post

READ MORE

Setting Up Zookeeper Services Using Cloudera API [Part 2]

linux cloudera hadoop cloudera-api zookeeper

This is the second follow up post. In the earlier post Setting Up Cloudera Manager Services Using Cloudera API [Part 1] we install the cloudera management services. Now we will be installing Zookeeper service to the cluster.

READ MORE

Setting Up Cloudera Manager Services Using Cloudera API [Part 1]

linux cloudera hadoop cloudera-api

Cloudera API is a very convenient way to setup a cluster and do more.

READ MORE

Getting Started with Cloudera API

linux cloudera hadoop cloudera-api

This is a basic steps to get connected with cloudera manager.

READ MORE

Basic Testing On Hadoop Environment [Cloudera]

linux cloudera hadoop testing

These are a set of testing which we can do on a Hadoop environment. These are basic testing to make sure the environment is setup correctly.

READ MORE

Setting Hue to Listen on `0.0.0.0` [Cloudera]

linux cloudera hadoop

We were working on setting up a cluster, but the Hue URL was set to a private IP of the server. As we had setup all the nodes to access each other using a private IP. But we wanted Hue to bind to public interface so that it can be accessed within the network.

READ MORE

Nagios - Service Group Summary ERROR

linux monitoring zabbix nagios nagiosxi error

We were working on nagios and found that after our migration, service group summary was not working. You might get below error on the screen and the solution is similar for both issues.

READ MORE

Zabbix History Table Clean Up

linux monitoring zabbix nagios nagiosxi

Zabbix history table gets really big, and if you are in a situation where you want to clean it up. Then we can do so, using the below steps.

READ MORE

Windows Testing Using Kitchen Chef

windows ubuntu chef kitchen testing

Kitchen-Vagrant has the capability to spin up a windows instance for testing. To make it work you will need the vagrant-winrm to be installted on the workstation.

READ MORE

Package Installer for Cygwin [apt-cyg].

linux windows cygwin

After a longtime I was on my windows machine and had to make it feel more like my linux machine. So install the thing what everyone else does cygwin. Surpise my custom .bashrc and .vimrc worked without any issues. Good !! had the bashrc update vimrc update, we are back to linux .. like :)

READ MORE

Issues - Monitoring MongoDB using Nagios XI.

linux mongo mongodb ubuntu nagios nagiosxi

Monitoring for mongodb using nagiosxi is straight forword but you might have some issues when we are setting up. Here are few issues which might come up using mongodb version 3.

READ MORE

Installing CouchDB on Ubuntu 14 LTS.

linux couchdb ubuntu nagios

CouchDB is a database that completely embraces the web. Store your data with JSON documents. Access your documents and query your indexes with your web browser, via HTTP. Index, combine, and transform your documents with JavaScript. CouchDB works well with modern web and mobile apps. You can even serve web apps directly out of CouchDB. And you can distribute your data, or your apps, efficiently using CouchDB’s incremental replication. CouchDB supports master-master setups with automatic conflict detection.

READ MORE

Installing MongoDB on Ubuntu 14 LTS.

linux mongo mongodb ubuntu nagios

MongoDB is an open-source document database, and leading NoSQL database. MongoDB is written in c++. Below is a brief document about installing a mongodb on a test node to try it out.

READ MORE

Encrypted Data Bags - Chef

chef linux ubuntu centos kitchen

Data Bags are a way to store information on the chef-server which all the cookbooks can access. Few more additional advantages are that we can encrypt the data-bags as well, this will help in keeping any sensitive information like user/password.

READ MORE

Bootstrap Windows - knife-windows - Chef

windows chef chef-client

knife-windows plugin adds additional functionality to the Chef Knife CLI tool for configuring / interacting with nodes running Microsoft Windows.

READ MORE

knife Quick Reference

centos rhel nagios opensource monitoring

This is a quick reference for few of the commands which I use often.

READ MORE

Remove Old Files using find Command

ubuntu centos linux commands

GNU find searches the directory tree rooted at each given file name by evaluating the given expression from left to right, according to the rules of precedence, until the outcome is known (the left hand side is false for and operations, true for or), at which point find moves on to the next file name. Remove old files which are older than a specific time using find Command.

READ MORE

Moving RRD file from 32bit to 64bit Architecture

ubuntu centos linux rrd nagios

When we were working on a nagios monitoring system we were migrating from a 32bit nagios to a 64bit Architecture. Most of the graphs are not working as the RRD was from an older 32bit architecture.

READ MORE

Migrating Nagios XI to a New Server on Centos6.6

centos rhel nagiosxi nagios opensource monitoring

Migrating an old Nagios backup to a new nagiosxi server. Migration is simple in Nagios XI, you a backup and restore it. Once that is done we need to update/add the clients with IP of the new server, so that both Nagios get alerts. After a while once we are comfortable with the new server we can decommission the old one.

READ MORE

Setting up ssl https On Nagios XI Server

centos linux nagios nagiosxi ssl https monitoring

HTTPS is a protocol for secure communication over a computer network which is widely used on the Internet. HTTPS consists of communication over Hypertext Transfer Protocol (HTTP) within a connection encrypted by Transport Layer Security or its predecessor, Secure Sockets Layer. The main motivation for HTTPS is authentication of the visited website and protection of the privacy and integrity of the exchanged data. Intro Courtesy Wikipedia

READ MORE

Standalone Chef Server / Workstation / Client Setup on CentOS 6

centos linux chef chefdk

The standalone installation of Chef server creates a working installation on a single server. This installation is also useful when you are installing Chef server in a virtual machine, for proof-of-concept deployments, or as a part of a development or testing loop.

READ MORE

Update hosts file in Windows 8

windows

Host file contains IP followed by the FQDN which can be used to reach that IP address. Host file takes precedence over your DNS servers. In Microsoft operating systems, the HOSTS file is located in the following location: C:\Windows\System32\Drivers\etc

READ MORE

Creating Documents Using pandoc

centos linux win documents

Pandoc is an opensource utility to create documents from markdown. We can create PDF, Doc, doc, html and other formats. And can be also used to convert html to doc, html to pdf, markdown to pdf and many more.

READ MORE

Setup/Configuration Nagios XI on Centos6.6

centos rhel nagios nagiosxi monitoring

Nagios monitors your entire IT infrastructure to ensure systems, applications, services, and business processes are functioning properly. In the event of a failure, Nagios can alert technical staff of the problem, allowing them to begin remediation processes before outages affect business processes, end-users, or customers.

READ MORE

RPM Command Cheat Sheet

centos rhel rpms

RPM (Redhat Package Manager) is the most popular package utility and is used mostly on RHEL, Centos and Fedora. RPM helps user/admins to build, install, query, verify, update, and remove/erase individual software packages.

READ MORE

Creating a large file in Windows / Linux

centos windows

Was working on a monitoring project, need to create a large file to test notifications. Here is how we can do that.

READ MORE

Cloudera Manager Setup Using Chef [CentOS 6.6]

centos chef chefdk cloudera-manager cdh

This cookbook [currently as of now] can be used to setup a Cloudera Manager Server (Management Server) running on MySQL database. But the intended use for this cookbook [rather a wishlist] is to do more. Simply put Auto Deployment of a Cloudera Hadoop Cluster using Chef, Python and Cloudera API. This will help create cluster for a development/test/preproduction/production environment on a click of a button.

READ MORE DOWNLOAD PDF

Chef Workstation Setup on Windows Machine.

windows chef chefdk

The Chef Development Kit (ChefDK) brings the best-of-breed development tools built by the awesome Chef community to your workstation with just a few clicks. Download your package and start coding Chef in seconds.

READ MORE

Check Port on Remote Server CentOS 6.6/RHEL 6

centos linux nc

Checking port available on a remote machine using nc command instead of telnet. Same command can be used to check on a remote server as well, change the 127.0.0.1 with IP address of the server.

READ MORE

LUKS Disk encryption for CentOS 6.6/RHEL 6

centos linux luks disk encryption

Linux Unified Key Setup-on-disk-format (or LUKS) allows you to encrypt partitions on your Linux computer. This is particularly important when it comes to mobile computers and removable media. LUKS allows multiple user keys to decrypt a master key, which is used for the bulk encryption of the partition.

READ MORE

Mysql Database Moving Data Directory to New Location.

mysql centos linux

How to move an existing data directory in mysql to a new location. We were running out of space and had to move the existing data directory to a new drive. Below are the steps to move the data directory to new location.

READ MORE

Mysql Database Disk Usage.

mysql centos linux

We were running out of disk space on one of the databases server, we need to get information on what the current table/database usage was. Below are few commands for mysql server tables usages.

READ MORE

Installing `python 2.7.x` on Centos 6.5/6.6

python centos linux install

By default centos comes with python 2.6. In most of the cases we might need python 2.7 or later to be installed. Below are few ways to install python 2.7 on centos 6.x.

READ MORE

Zabbix Installation Script on Centos/Redhat 6.x

zabbix snmp trap centos linux install online nagios monitoring

Installation on zabbix is quit simple and straight forward, but when you are install multiple servers and want to automate installation it is better to use a script. I would have preferred an ansible setup, but we have non of that on the server, so had to make do with good old shell script.

READ MORE

Zabbix Database Backup Script for `postgres` and `mysql`.

zabbix snmp trap centos linux backup mysql postgres nagios monitoring

After the zabbix setup is done, we would like to take backup of the database. Here we are doing a complete backup. But if we are looking for only configuration backup then we can ignore the history* tables from the database, but for now we are taking complete backup. Backups are taken everyday and last 3 days backups are stored and rest for removed.

READ MORE

Installing Zabbix Version 2.4 Offline (Zabbix Server without Internet).

zabbix snmp trap centos linux offline nagios monitoring

There might be situations where you have a remote/zabbix server which does not have internet connectivity, due to security or other reasons. So we create a custom repo on the remote/zabbix server so that we can install zabbix using rpms

READ MORE DOWNLOAD PDF

Zabbix Template Creation using CSV file.

zabbix snmp trap centos linux offline nagios monitoring

In zabbix we dont have a better way to capture snmptraps. we have to manually create an item and corresponfding trigger to handle a trap arriving from the device.

READ MORE

Installing SNMP Builder using `zabbix-extra` on Zabbix Version 2.4.

zabbix snmp trap centos linux nagios monitoring

SNMP Builder/Extra is an add-in for Zabbix. It provides new web interface components to browse MIB trees and values. SNMP OIDs can then be automatically converted into Zabbix items and inserted into a template. The underlying snmpbuilder script uses calls to NetSNMP in order to communicate with devices on the network.

READ MORE DOWNLOAD PDF

Setting up SNMP Trapper for Zabbix.

zabbix snmp trap centos linux

Receiving SNMP traps is the opposite to querying SNMP-enabled devices. In this case the information is sent from a SNMP-enabled device and is collected or “trapped” by Zabbix. Usually traps are sent upon some condition change and the agent connects to the server on port 162 (as opposed to port 161 on the agent side that is used for queries). Using traps may detect some short problems that occur amidst the query interval and may be missed by the query data.

READ MORE DOWNLOAD PDF

Explain - System Security Services Daemon (SSSD) Config.

sssd linux pam sssd-configuration auth authentication security

The System Security Services Daemon (SSSD) provides a set of daemons to manage access to remote directories and authentication mechanisms. It provides Name Service Switch (NSS) and Pluggable Authentication Modules(PAM) interfaces toward the system and a pluggable back end system to connect to multiple different account sources.

READ MORE DOWNLOAD PDF

Setting up my Linux Environment.

setup linux vim vimrc bashrc bash pathogen

This is a basic environment setup for my Centos/RHEL. We are setting up vim and bash. vim we are setting pathogen for bundling and NerdTree for file navigation.

READ MORE

YUM Repository Creation on HTTPD Web Server.

linux hadoop webserver http httpd yum rhel centos

Setting up yum repos on RHEL using httpd. We will be setting up httpd and yum repo on top of it. So that we can access yum over http.

READ MORE DOWNLOAD PDF

Access Filter Setup with SSSD

linux hadoop sssd access-filter rhel centos security

If using access_provider = ldap, this option is mandatory. It specifies an LDAP search filter criteria that must be met for the user to be granted access on this host. If access_provider = ldap and this option is not set, it will result in all users being denied access. Use access_provider = allow to change this default behaviour.

READ MORE DOWNLOAD PDF

Update Cloudera Manager to specific version [5.4.5]

linux hadoop upgrade cloudera manager

Upgrading Cloudera Manager 5 to the Latest Cloudera Manager, In most cases it is possible to complete the following upgrade without shutting down most CDH services, although you may need to stop some dependent services. CDH daemons can continue running, unaffected, while Cloudera Manager is upgraded. The upgrade process does not affect your CDH installation. After upgrading Cloudera Manager you may also want to upgrade CDH 5 clusters to CDH 5.4.5 or latest.

READ MORE DOWNLOAD PDF

Getting started with Hive with Kerberos.

linux hadoop hive kerberos ad ldap cloudera security

Apache Hive is a powerful data warehousing application built on top of Hadoop; it enables you to access your data using Hive QL, a language that is similar to SQL. Install Hive on your client machine(s) from which you submit jobs; you do not need to install it on the nodes in your Hadoop cluster. If Kerberos authentication is used, authentication is supported between the Thrift client and HiveServer2, and between HiveServer2 and secure HDFS.

READ MORE DOWNLOAD PDF

Redhat Intergration with Active Directory using SSSD.

linux hadoop sssd active-directory ad ldap rhel centos security

There are inherent structural differences between how Windows and Linux handle system users. The user schemas used in Active Directory and standard LDAPv3 directory services also differ significantly. When using an Active Directory identity provider with SSSD to manage system users, it is necessary to reconcile Active Directory-style users to the new SSSD users. There are two ways to achieve it:

READ MORE DOWNLOAD PDF

Simple Steps to Intergrate RHEL with Active Directory using SSSD.

linux hadoop sssd ad active-directory rhel centos security

There are inherent structural differences between how Windows and Linux handle system users. The user schemas used in Active Directory and standard LDAPv3 directory services also differ significantly. When using an Active Directory identity provider with SSSD to manage system users, it is necessary to reconcile Active Directory-style users to the new SSSD users. There are two ways to achieve it:

READ MORE DOWNLOAD PDF

Setting `iptables` - `Port` forwarding from one interface to another.

linux iptables port-forwarding interface network

iptables is a user-space application program that allows a system administrator to configure the tables provided by the Linux kernel firewall (implemented as different Netfilter modules) and the chains and rules it stores. Different kernel modules and programs are currently used for different protocols; iptables applies to IPv4, ip6tables to IPv6, arptables to ARP, and ebtables to Ethernet frames.

READ MORE DOWNLOAD PDF

Setting up Pentaho Data Integration 5.4.1 with Hadoop Cluster (Clouder Manager)

linux hadoop pentaho data-integration cloudera manager

Pentaho Data Integration (PDI) is a suite of open source Business Intelligence (BI) products which provide data integration, OLAP services, reporting, dashboarding, data mining and ETL capabilities. Here we are setting up pentaho server has the below steps. Will be referring to Pentaho Data Integration as PDI from now on.

READ MORE DOWNLOAD PDF

Installing `squid` as a sibling to an already existing Parent `squid`.

linux squid proxy

Squid is a caching and forwarding web proxy. It has a wide variety of uses, from speeding up a web server by caching repeated requests; to caching web, DNS and other computer network lookups for a group of people sharing network resources; to aiding security by filtering traffic.

READ MORE DOWNLOAD PDF

Creating a two-node RHEL cluster with Virtual IP using CMAN and Pacemaker.

linux ha high-availability cman pacemaker heartbeat haproxy

CMAN v3 is a Corsync plugin that monitors the names and number of active cluster nodes in order to deliver membership and quorum information to clients (such as the Pacemaker daemons). We are using this to have a VIP (Virtual IP) being made high available.

READ MORE DOWNLOAD PDF

No valid credentials provided Mechanism level Failed to find any Kerberos tgt

hadoop kerberos error krb5

We have 2 domains forests in our environment, ABC and XYZ. We were not able to authenticate normal users from either of the domains. Most of the information is there on the Cloudera Website. You might want to check on the site first, if you see any thing similar.

READ MORE DOWNLOAD PDF

Interface Forwarding - from `eth1` to `eth0` on `EDGE` node.

linux hadoop interface network

Adding route to all the slaves which reside on a private network to communicate with External Server directly using an EDGE node using Interface Forwarding.

READ MORE DOWNLOAD PDF

Installing `ansible` on RHEL 6.6.

linux ansible hadoop rhel centos

Ansible is a radically simple IT automation engine that automates cloud provisioning, configuration management, application deployment, intra-service orchestration, and many other IT needs. For more detail, hop over to docs.ansible.com.

READ MORE DOWNLOAD PDF

Mounting RAID10 using `parted`.

linux gparted parted mount raid

GUID Partition Table (GPT) is a standard for the layout of the partition table on a physical hard disk, using globally unique identifiers (GUID). Although it forms a part of the Unified Extensible Firmware Interface (UEFI) standard (Unified EFI Forum proposed replacement for the PC BIOS), it is also used on some BIOS systems because of the limitations of master boot record (MBR) partition tables, which use 32 bits for storing logical block addresses (LBA) and size information.

READ MORE DOWNLOAD PDF

Setup and Configure NFS Mounts on Linux Server

linux nfs mount

A Network File System (NFS) allows remote hosts to mount file systems over a network and interact with those file systems as though they are mounted locally. This enables system administrators to consolidate resources onto centralized servers on the network.

READ MORE DOWNLOAD PDF

Ansible Playbook - Setup Zookeeper Using `tarball`.

linux ansible hadoop zookeeper

This is a simple zookeeper playbook, to quickly start zookeeper running on a single or more nodes, in a clustered mode.

READ MORE DOWNLOAD PDF

Ansible Playbook - Setup Storm Cluster.

linux ansible storm hadoop zookeeper

This is a simple Storm Cluster Setup. We are using a dedicated Zookeeper Cluster/Node, instead of the standalone zkserver. Below is how we will deploy our cluster.

READ MORE DOWNLOAD PDF

Ansible Playbook - Setup Kafka Cluster.

linux ansible kafka hadoop zookeeper

This is a simple Kafka setup. In this setup we are running kafka over a dedicated zookeeper service. (NOT the standalone zookeeper which comes with kafka)

READ MORE DOWNLOAD PDF

Ansible Playbook - Setup Hadoop CDH5 Using `tarball`.

linux ansible hadoop tarball

Setting up Hadoop using Ansible, we will be using cdh5 tarball for installation of the cluster.

READ MORE DOWNLOAD PDF

Streaming Data Processing - Storm Vs Spark.

linux storm spark data-processing

Apache Spark is an in-memory distributed data analysis platform– primarily targeted at speeding up batch analysis jobs, iterative machine learning jobs, interactive query and graph processing.

READ MORE DOWNLOAD PDF

Creating a passwordless entry on Centos 6.5.

linux centos rhel

You can enter a linux system without entring a password using below steps. We will be creating a ssh jey which will be share between the servers, which will be used to authenticate. This is also a secure way of connecting to server, as the private key is inside the user home directory and can only be accessed by the user.

READ MORE DOWNLOAD PDF

Creating a Multi-node Cassandra Cluster on Centos 6.5.

hadoop linux cassandra centos rhel

The Apache Cassandra database is the right choice when you need scalability and high availability without compromising performance. Linear scalability and proven fault-tolerance on commodity hardware or cloud infrastructure make it the perfect platform for mission-critical data. Cassandra’s support for replicating across multiple datacenters is best-in-class, providing lower latency for your users and the peace of mind of knowing that you can survive regional outages.

READ MORE DOWNLOAD PDF

Installing SpagoBI 5.1 on Centos 6.5 -Tomcat 7 with MySQL 5.6.

linux spagobi bi spago centos tomcat mysql

The SpagoBI project is a free software/open source initiative by the SpagoBI Labs of Engineering Group. It aims to realize the most complete 100% open source business intelligence suite, aggregating developers, integrators, companies, users and passionate people in an open community.

READ MORE DOWNLOAD PDF

KVM Installation on CentOS 6.x.

linux kvm centos rhel virtual-machine vmware

KVM is a kernel-based Virutal Machine which grows quickly in maturity and popularity in the Linux server market. Red Hat officially dropped Xen in favor of KVM since RHEL 6. With KVM being officially supported by Red Hat, installing KVM on RedHat-based systems should be a breeze.

READ MORE DOWNLOAD PDF

Performance Tuning HBase and Hadoop.

linux hadoop hbase performance-tuning

Using HBase in production often requires that you turn many knobs to make it hum as expected. More Here http://hbase.apache.org/0.94/book/performance.html

READ MORE DOWNLOAD PDF

Setting you Hbase Cluster on Hadoop (YARN). Ubuntu 12.04 LTS

hadoop hbase linux ubuntu yarn

HBase is an open source, non-relational, distributed database modeled after Google’s BigTable and written in Java. It is developed as part of Apache Software Foundation’s Apache Hadoop project and runs on top of HDFS (Hadoop Distributed Filesystem), providing BigTable-like capabilities for Hadoop. That is, it provides a fault-tolerant way of storing large quantities of sparse data (small amounts of information caught within a large collection of empty or unimportant data, such as finding the 50 largest items in a group of 2 billion records, or finding the non-zero items representing less than 0.1% of a huge collection).

READ MORE DOWNLOAD PDF

VMware Workstation 10 Error `Not enough physical memory is available to power this virtual machine`

windows vmware error virtual-machine

If you are using VMWare Workstation (or VMWare player) on Windows 8.1 and have just update Windows, specifically KB2995388, you may receive this error message when attempt to start a virtual machine.

READ MORE DOWNLOAD PDF

Unable to locate package oracle-java7-installer - Ubuntu 13

linux ubuntu java jdk

Was installing Java today, this is an easy install thanks to ppa:webupd8team/java, but when I tried it was not working, but has worked for me all this while.

READ MORE DOWNLOAD PDF

Installing `zabbix-java-gateway` on Centos 6.5

zabbix linux java zabbix-java-gateway centos rhel nagios monitoring

Zabbix 2.0 adds native support for monitoring JMX applications by introducing a new Zabbix daemon called Zabbix Java gateway. Zabbix Java gateway is a daemon written in Java. When Zabbix server wants to know the value of a particular JMX counter on a host, it asks Zabbix Java gateway, which uses the JMX management API to query the application of interest remotely. The application does not need any additional software installed, it just has to be started with -Dcom.sun.management.jmxremote option on the command line.

READ MORE DOWNLOAD PDF

How to setup HAProxy

linux ha load-balancing high-avaliability proxy nginx haproxy

HAProxy is the Reliable, High Performance TCP/HTTP Load Balancer and it works nicely with Deveo Cluster setup.

READ MORE DOWNLOAD PDF

Using `npm` behind a proxy

linux npm nodejs proxy

npm stands for Node Package Manager, and is the default package manager for the JavaScript runtime environment Node.js.

READ MORE DOWNLOAD PDF

Integrating NODEJS and Kafka

linux webserver nodejs kafka-node kafka integrate

Below are the list required to integrate NodeJS and Kafka. This is a simple HOWTO to get started.

READ MORE DOWNLOAD PDF

Sending JSON to NodeJS to Multiple Topics in Kafka - using kafka-node.

linux webserver nodejs kafka-node json kafka

Kafka-node is a Node.js client with Zookeeper integration for Apache Kafka 0.8.1 and later. The Zookeeper integration does the following jobs:

READ MORE DOWNLOAD PDF

NodeJS Kafka Producer - Using `kafka-node`

linux webserver nodejs kafka-node kafka

Now that we have Kafka and NodeJS ready. Lets some data to our Kafka Cluster.

READ MORE DOWNLOAD PDF

Sending JSON to NodeJS to Kafka.

linux webserver nodejs kafka kafka-node

What we are trying to achieve ?

READ MORE DOWNLOAD PDF

Installing NodeJS on Centos 6.6.

linux webserver nodejs centos rhel

Node.js is an open-source, cross-platform runtime environment for developing server-side web applications. Node.js applications are written in JavaScript and can be run within the Node.js runtime on OS X, Microsoft Windows, Linux, FreeBSD, IBM AIX, IBM System z and IBM i.

READ MORE DOWNLOAD PDF

Installing KAFKA Single Node - Quick Start.

linux hadoop kafka quick-start

Apache Kafka is publish-subscribe messaging rethought as a distributed commit log. Kafka is a distributed, partitioned, replicated commit log service. It provides the functionality of a messaging system, but with a unique design. What does all that mean?

READ MORE DOWNLOAD PDF

sudo Permission On Server - effective uid is not 0, is sudo installed setuid root?

linux setuid root permissions

Got into this issue on the server, when trying to install mysql-server.

READ MORE DOWNLOAD PDF

Custom Webserver/NodeJS sysctl.conf file.

linux webserver nodejs performance tuning kernel sysctl

Have updated and explanation in the conf file below.

READ MORE DOWNLOAD PDF

Creating server parameter sysctl.conf for WebServers / FTP Server.

linux webserver ftp-server kernel sysctl performance tuning

sysctl is an interface that allows you to make changes to a running Linux kernel. With /etc/sysctl.conf you can configure various Linux networking and system settings such as:

  • Limit network-transmitted configuration for IPv4
  • Limit network-transmitted configuration for IPv6
  • Turn on execshield protection
  • Prevent against the common syn flood attack
  • Turn on source IP address verification
  • Prevents a cracker from using a spoofing attack against the IP address of the server.
  • Logs several types of suspicious packets, such as spoofed packets, source-routed packets, and redirects.
READ MORE DOWNLOAD PDF

Creating Virtual Interface and Assign Multiple IP Addresses.

linux vip network multiple-ip

Current IP of the server : 10.130.18.11. Virtual IPs being assigned to server : 10.130.18.22, 10.130.18.23, 10.130.18.24 to our server..

Go to network-scripts directory and copy the existing ifcfg-eth0 file. Make sure you are using a static ip for your server.

READ MORE DOWNLOAD PDF

Changing Timezone in Ubuntu server.

linux ubuntu timezone

Changing Timezone in Ubuntu server.

READ MORE DOWNLOAD PDF

Hadoop `sysctl.conf` parameters.

hadoop linux kernel sysctl performance tuning

Performace tuning Hadoop at kernel level. sysctl is an interface that allows you to make changes to a running Linux kernel. With /etc/sysctl.conf you can configure various Linux networking and system settings such as:

  • Limit network-transmitted configuration for IPv4
  • Limit network-transmitted configuration for IPv6
  • Turn on execshield protection
  • Prevent against the common syn flood attack
  • Turn on source IP address verification
  • Prevents a cracker from using a spoofing attack against the IP address of the server.
  • Logs several types of suspicious packets, such as spoofed packets, source-routed packets, and redirects.
READ MORE DOWNLOAD PDF

Setting SUN Java for Ubuntu.

linux install-java java ubuntu jdk

The Java Development Kit (JDK) is an implementation of either one of the Java SE, Java EE or Java ME platforms released by Oracle Corporation in the form of a binary product aimed at Java developers on Solaris, Linux, Mac OS X or Windows. This is currently not avaiable on apt-get repo, so here is quick way to setup java on ubuntu server/desktop with tar ball.

READ MORE DOWNLOAD PDF

Simple Script to create a user on centos 6.5

linux create user centos rhel

Default passwd is set as username@123 As we are setting the chage, user needs to change the passwd on first login.

READ MORE DOWNLOAD PDF

Performance Tuning for `nginx`

linux nginx performance tuning load-balancing

Nginx (pronounced engine x) is a web server with a strong focus on high concurrency, performance and low memory usage. It can also act as a reverse proxy server for HTTP, HTTPS, SMTP, POP3, and IMAP protocols, as well as a load balancer and an HTTP cache.

READ MORE DOWNLOAD PDF

Installing and Initial setup of Tsung Load Testing CentOS

load-testing performance seige tsung testing

Tsung (formerly known as idx-Tsunami) is a stress testing tool written in the Erlang language and distributed under the GPL license. It can currently stress test HTTP, WebDAV, LDAP, MySQL, PostgreSQL, SOAP and XMPP servers. Tsung can simulate hundreds of simultaneous users on a single system. It can also function in a clustered environment.

READ MORE DOWNLOAD PDF

How To Configure Swappiness

hadoop linux swappiness swap sysctl config

Swappiness is a Linux kernel parameter that controls the relative weight given to swapping out runtime memory, as opposed to dropping pages from the system page cache. Swappiness can be set to values between 0 and 100 inclusive. A low value causes the kernel to avoid swapping, a higher value causes the kernel to try to use swap space. The default value is 60, and for most desktop systems, setting it to 100 may affect the overall performance, whereas setting it lower (even 0) may decrease response latency.

READ MORE DOWNLOAD PDF

Setup Seige on Centos 6.5, Kernel 2.6, CPU x86_64

load-testing performance seige tsung testing

Siege is an http load testing and benchmarking utility. It was designed to let web developers measure their code under duress, to see how it will stand up to load on the internet. Siege supports basic authentication, cookies, HTTP, HTTPS and FTP protocols. It lets its user hit a server with a configurable number of simulated clients. Those clients place the server under siege.

READ MORE DOWNLOAD PDF

HA Setup - `heartbeat` for `nginx`.

linux heartbeat nginx http httpd replication load-balancing high-availability

Heartbeat is a daemon that provides cluster infrastructure (communication and membership) services to its clients. This allows clients to know about the presence (or disappearance!) of peer processes on other machines and to easily exchange messages with them. NGINX is a free, open-source, high-performance HTTP server and reverse proxy, as well as an IMAP/POP3 proxy server. NGINX is known for its high performance, stability, rich feature set, simple configuration, and low resource consumption.

READ MORE DOWNLOAD PDF

SpagoBI Tomcat Clustering Using `mod_jk` and `httpd` on Centos - In-Memory Session Replication.

linux spago-bi bi tomcat http httpd replication load-balancing high-availability

In an earlier post we did a basic session based replication, but the session was not redundant.

READ MORE DOWNLOAD PDF

Setting up Tomcat Cluster for SpagoBI 5.1.

linux spago bi tomcat http spago-bi

Setting up SpagoBI 5.1 on Tomcat with MySQL post.

READ MORE DOWNLOAD PDF

Setting up `NGINX` for `HTTP` load balancing.

linux nginx http httpd load-balancing high-availability

NGINX is a free, open-source, high-performance HTTP server and reverse proxy, as well as an IMAP/POP3 proxy server. NGINX is known for its high performance, stability, rich feature set, simple configuration, and low resource consumption.

READ MORE DOWNLOAD PDF

httpd HA using Heartbeat.

linux heartbeat high-availability httpd http

httpd is the Apache HyperText Transfer Protocol (HTTP) server program. It is designed to be run as a standalone daemon process. When used like this it will create a pool of child processes or threads to handle requests. Heartbeat is a daemon that provides cluster infrastructure (communication and membership) services to its clients. This allows clients to know about the presence (or disappearance!) of peer processes on other machines and to easily exchange messages with them.

READ MORE DOWNLOAD PDF

Setting up Local HBase on Top of HDFS.

hadoop hdfs hbase hadoop-config hbase-config

First lets setup the Hbase Configuration Files. For pseudo-distributed replace with 'localhost'

READ MORE DOWNLOAD PDF

Zabbix Installation 2.4 - CentOS 6.5

zabbix centos nagios monitoring

Zabbix is the ultimate enterprise-level software designed for monitoring availability and performance of IT infrastructure components. Zabbix is open source and comes at no cost.

READ MORE DOWNLOAD PDF

Enable Authorization on HBase.

hadoop hbase hadoop-config hbase-config security authentication

Add this below tag to all the Master and Region Server.

READ MORE DOWNLOAD PDF

Zabbix Hadoop Monitoring.

zabbix csv-processing linux mib python hadoop xml nagios monitoring

This script can be used to monitor Namenode Parameters. This script can be used to Generate Zabbix Import XML or Send monitoring data to Zabbix server.

READ MORE DOWNLOAD PDF

Fetch Monitoring Data from Item to CSV file.

zabbix csv-processing iReasoning linux mib python nagios monitoring

This section describes how to get monitoring data from Zabbix server through the command line and export these to a CSV formatted file.

READ MORE DOWNLOAD PDF

Zabbix Item Creation using OID and Name CSV files.

zabbix csv-processing iReasoning linux mib python nagios monitoring

This script is to create a xml import file from a csv file. We use 2 csv file for this.

READ MORE DOWNLOAD PDF

Send KPI Data to Zabbix from XML KPI File

zabbix xml-processing linux mib python zbxsend nagios monitoring

Send Zabbix metric by reading from a XML file. This needs package zbxsend from : https://github.com/zubayr/zbxsend

READ MORE DOWNLOAD PDF

Zabbix Import File from iReasoning Export CSV

zabbix csv-processing iReasoning linux mib nagios monitoring

Module used to generate Zabbix import xml for snmp traps, Below are steps to create a Zabbix Import for Traps. These traps are captured from the snmptrap function in Zabbix.

READ MORE DOWNLOAD PDF

SFTP Data Collector

sftp ftp data-collector linux python script github

Easy way to collect files recursively over a sftp server is to connect to the server over scp and do scp -r. Problem was that the device we were connecting to did not support recursive :( over a regEx expression.

READ MORE DOWNLOAD PDF