Posts Tagged “linux”

Kafka Kerberos Enable and Testing.

Apache Kafka is a distributed streaming platform. Kafka 2.0 supports Kerberos authentication, Enabling Kerberos Authentication Using the Wizard on cloudera manager. Courtesy - Apache Kafka

Written on May 16, 2017
linux centos redhat cloudera kafka kerberos cluster


Cloudera Manager - Duplicate entry 'zookeeper' for key 'NAME'.

We had recently built a cluster using cloudera API’s and had all the services running on it with Kerberos enabled. Next we had a requirement to add another kafka cluster to our already exsisting cluster in cloudera manager. Since it is a quick task to get the zookeeper and kafka up and running. We decided to get this done using the cloudera manager instead of the API’s. But we faced the Duplicate entry 'zookeeper' for key 'NAME' issue as described in the bug below.

https://issues.cloudera.org/browse/DISTRO-790

Written on May 14, 2017
linux centos redhat cloudera kafka zookeeper cluster


Parcel Not Distributing Cloudera CDH.

We were deploying one of the cluster on our lab environment which is used by everyone. So the lab has it own share of stale information on it.

Written on March 8, 2017
linux centos redhat cloudera hadoop cluster


Creating /etc/hosts file in Chef.

We had a cluster environment which we needed to update the /etc/hosts file. Which would help communicate between the server over a private network. Our servers have multiple interfaces and we need them to communicate between each other using the private network.

Written on March 3, 2017
linux centos redhat chef hadoop cluster


Enable Kerberos Using Cloudera API.

Python API for cloudera is really nice, apart from getting the cluster setup, we can also do configuration and automation. We use a lot of automation using Chef/Ansible, but cloudera API give more control over the cluster.

Written on February 26, 2017
linux cloudera hadoop cloudera-api kerberos


Setting Up HDFS Services Using Cloudera API [Part 3]

This is the second follow up post. In the earlier post

Written on February 8, 2017
linux cloudera hadoop cloudera-api zookeeper hdfs


Setting Up Zookeeper Services Using Cloudera API [Part 2]

This is the second follow up post. In the earlier post Setting Up Cloudera Manager Services Using Cloudera API [Part 1] we install the cloudera management services. Now we will be installing Zookeeper service to the cluster.

Written on February 2, 2017
linux cloudera hadoop cloudera-api zookeeper


Setting Up Cloudera Manager Services Using Cloudera API [Part 1]

Cloudera API is a very convenient way to setup a cluster and do more.

Written on January 25, 2017
linux cloudera hadoop cloudera-api


Getting Started with Cloudera API

This is a basic steps to get connected with cloudera manager.

Written on January 14, 2017
linux cloudera hadoop cloudera-api


Basic Testing On Hadoop Environment [Cloudera]

These are a set of testing which we can do on a Hadoop environment. These are basic testing to make sure the environment is setup correctly.

Written on January 13, 2017
linux cloudera hadoop testing


Setting Hue to Listen on `0.0.0.0` [Cloudera]

We were working on setting up a cluster, but the Hue URL was set to a private IP of the server. As we had setup all the nodes to access each other using a private IP. But we wanted Hue to bind to public interface so that it can be accessed within the network.

Written on January 12, 2017
linux cloudera hadoop


Nagios - Service Group Summary ERROR

We were working on nagios and found that after our migration, service group summary was not working. You might get below error on the screen and the solution is similar for both issues.

Written on October 12, 2016
linux monitoring zabbix nagios nagiosxi error


Zabbix History Table Clean Up

Zabbix history table gets really big, and if you are in a situation where you want to clean it up. Then we can do so, using the below steps.

Written on October 10, 2016
linux monitoring zabbix nagios nagiosxi


Package Installer for Cygwin [apt-cyg].

After a longtime I was on my windows machine and had to make it feel more like my linux machine. So install the thing what everyone else does cygwin. Surpise my custom .bashrc and .vimrc worked without any issues. Good !! had the bashrc update vimrc update, we are back to linux .. like :)

Written on September 29, 2016
linux windows cygwin


Issues - Monitoring MongoDB using Nagios XI.

Monitoring for mongodb using nagiosxi is straight forword but you might have some issues when we are setting up. Here are few issues which might come up using mongodb version 3.

Written on September 15, 2016
linux mongo mongodb ubuntu nagios nagiosxi


Installing CouchDB on Ubuntu 14 LTS.

CouchDB is a database that completely embraces the web. Store your data with JSON documents. Access your documents and query your indexes with your web browser, via HTTP. Index, combine, and transform your documents with JavaScript. CouchDB works well with modern web and mobile apps. You can even serve web apps directly out of CouchDB. And you can distribute your data, or your apps, efficiently using CouchDB’s incremental replication. CouchDB supports master-master setups with automatic conflict detection.

Written on September 15, 2016
linux couchdb ubuntu nagios


Installing MongoDB on Ubuntu 14 LTS.

MongoDB is an open-source document database, and leading NoSQL database. MongoDB is written in c++. Below is a brief document about installing a mongodb on a test node to try it out.

Written on September 14, 2016
linux mongo mongodb ubuntu nagios


Encrypted Data Bags - Chef

Data Bags are a way to store information on the chef-server which all the cookbooks can access. Few more additional advantages are that we can encrypt the data-bags as well, this will help in keeping any sensitive information like user/password.

Written on September 6, 2016
chef linux ubuntu centos kitchen


Remove Old Files using find Command

GNU find searches the directory tree rooted at each given file name by evaluating the given expression from left to right, according to the rules of precedence, until the outcome is known (the left hand side is false for and operations, true for or), at which point find moves on to the next file name. Remove old files which are older than a specific time using find Command.

Written on August 23, 2016
ubuntu centos linux commands


Moving RRD file from 32bit to 64bit Architecture

When we were working on a nagios monitoring system we were migrating from a 32bit nagios to a 64bit Architecture. Most of the graphs are not working as the RRD was from an older 32bit architecture.

Written on August 22, 2016
ubuntu centos linux rrd nagios


Setting up ssl https On Nagios XI Server

HTTPS is a protocol for secure communication over a computer network which is widely used on the Internet. HTTPS consists of communication over Hypertext Transfer Protocol (HTTP) within a connection encrypted by Transport Layer Security or its predecessor, Secure Sockets Layer. The main motivation for HTTPS is authentication of the visited website and protection of the privacy and integrity of the exchanged data. Intro Courtesy Wikipedia

Written on August 17, 2016
centos linux nagios nagiosxi ssl https monitoring


Standalone Chef Server / Workstation / Client Setup on CentOS 6

The standalone installation of Chef server creates a working installation on a single server. This installation is also useful when you are installing Chef server in a virtual machine, for proof-of-concept deployments, or as a part of a development or testing loop.

Written on August 16, 2016
centos linux chef chefdk


Creating Documents Using pandoc

Pandoc is an opensource utility to create documents from markdown. We can create PDF, Doc, doc, html and other formats. And can be also used to convert html to doc, html to pdf, markdown to pdf and many more.

Written on August 15, 2016
centos linux win documents


Check Port on Remote Server CentOS 6.6/RHEL 6

Checking port available on a remote machine using nc command instead of telnet. Same command can be used to check on a remote server as well, change the 127.0.0.1 with IP address of the server.

Written on April 1, 2016
centos linux nc


LUKS Disk encryption for CentOS 6.6/RHEL 6

Linux Unified Key Setup-on-disk-format (or LUKS) allows you to encrypt partitions on your Linux computer. This is particularly important when it comes to mobile computers and removable media. LUKS allows multiple user keys to decrypt a master key, which is used for the bulk encryption of the partition.

Written on February 23, 2016
centos linux luks disk encryption


Mysql Database Moving Data Directory to New Location.

How to move an existing data directory in mysql to a new location. We were running out of space and had to move the existing data directory to a new drive. Below are the steps to move the data directory to new location.

Written on February 4, 2016
mysql centos linux


Mysql Database Disk Usage.

We were running out of disk space on one of the databases server, we need to get information on what the current table/database usage was. Below are few commands for mysql server tables usages.

Written on February 4, 2016
mysql centos linux


Installing `python 2.7.x` on Centos 6.5/6.6

By default centos comes with python 2.6. In most of the cases we might need python 2.7 or later to be installed. Below are few ways to install python 2.7 on centos 6.x.

Written on February 3, 2016
python centos linux install


Zabbix Installation Script on Centos/Redhat 6.x

Installation on zabbix is quit simple and straight forward, but when you are install multiple servers and want to automate installation it is better to use a script. I would have preferred an ansible setup, but we have non of that on the server, so had to make do with good old shell script.

Written on January 6, 2016
zabbix snmp trap centos linux install online nagios monitoring


Zabbix Database Backup Script for `postgres` and `mysql`.

After the zabbix setup is done, we would like to take backup of the database. Here we are doing a complete backup. But if we are looking for only configuration backup then we can ignore the history* tables from the database, but for now we are taking complete backup. Backups are taken everyday and last 3 days backups are stored and rest for removed.

Written on January 6, 2016
zabbix snmp trap centos linux backup mysql postgres nagios monitoring


Installing Zabbix Version 2.4 Offline (Zabbix Server without Internet).

There might be situations where you have a remote/zabbix server which does not have internet connectivity, due to security or other reasons. So we create a custom repo on the remote/zabbix server so that we can install zabbix using rpms

Written on December 29, 2015
zabbix snmp trap centos linux offline nagios monitoring


Zabbix Template Creation using CSV file.

In zabbix we dont have a better way to capture snmptraps. we have to manually create an item and corresponfding trigger to handle a trap arriving from the device.

Written on December 10, 2015
zabbix snmp trap centos linux offline nagios monitoring


Installing SNMP Builder using `zabbix-extra` on Zabbix Version 2.4.

SNMP Builder/Extra is an add-in for Zabbix. It provides new web interface components to browse MIB trees and values. SNMP OIDs can then be automatically converted into Zabbix items and inserted into a template. The underlying snmpbuilder script uses calls to NetSNMP in order to communicate with devices on the network.

Written on December 9, 2015
zabbix snmp trap centos linux nagios monitoring


Setting up SNMP Trapper for Zabbix.

Receiving SNMP traps is the opposite to querying SNMP-enabled devices. In this case the information is sent from a SNMP-enabled device and is collected or “trapped” by Zabbix. Usually traps are sent upon some condition change and the agent connects to the server on port 162 (as opposed to port 161 on the agent side that is used for queries). Using traps may detect some short problems that occur amidst the query interval and may be missed by the query data.

Written on December 3, 2015
zabbix snmp trap centos linux


Explain - System Security Services Daemon (SSSD) Config.

The System Security Services Daemon (SSSD) provides a set of daemons to manage access to remote directories and authentication mechanisms. It provides Name Service Switch (NSS) and Pluggable Authentication Modules(PAM) interfaces toward the system and a pluggable back end system to connect to multiple different account sources.

Written on November 9, 2015
sssd linux pam sssd-configuration auth authentication security


Setting up my Linux Environment.

This is a basic environment setup for my Centos/RHEL. We are setting up vim and bash. vim we are setting pathogen for bundling and NerdTree for file navigation.

Written on November 6, 2015
setup linux vim vimrc bashrc bash pathogen


YUM Repository Creation on HTTPD Web Server.

Setting up yum repos on RHEL using httpd. We will be setting up httpd and yum repo on top of it. So that we can access yum over http.

Written on November 3, 2015
linux hadoop webserver http httpd yum rhel centos


Access Filter Setup with SSSD

If using access_provider = ldap, this option is mandatory. It specifies an LDAP search filter criteria that must be met for the user to be granted access on this host. If access_provider = ldap and this option is not set, it will result in all users being denied access. Use access_provider = allow to change this default behaviour.

Written on October 23, 2015
linux hadoop sssd access-filter rhel centos security


Update Cloudera Manager to specific version [5.4.5]

Upgrading Cloudera Manager 5 to the Latest Cloudera Manager, In most cases it is possible to complete the following upgrade without shutting down most CDH services, although you may need to stop some dependent services. CDH daemons can continue running, unaffected, while Cloudera Manager is upgraded. The upgrade process does not affect your CDH installation. After upgrading Cloudera Manager you may also want to upgrade CDH 5 clusters to CDH 5.4.5 or latest.

Written on October 22, 2015
linux hadoop upgrade cloudera manager


Getting started with Hive with Kerberos.

Apache Hive is a powerful data warehousing application built on top of Hadoop; it enables you to access your data using Hive QL, a language that is similar to SQL. Install Hive on your client machine(s) from which you submit jobs; you do not need to install it on the nodes in your Hadoop cluster. If Kerberos authentication is used, authentication is supported between the Thrift client and HiveServer2, and between HiveServer2 and secure HDFS.

Written on October 20, 2015
linux hadoop hive kerberos ad ldap cloudera security


Redhat Intergration with Active Directory using SSSD.

There are inherent structural differences between how Windows and Linux handle system users. The user schemas used in Active Directory and standard LDAPv3 directory services also differ significantly. When using an Active Directory identity provider with SSSD to manage system users, it is necessary to reconcile Active Directory-style users to the new SSSD users. There are two ways to achieve it:

Written on October 6, 2015
linux hadoop sssd active-directory ad ldap rhel centos security


Simple Steps to Intergrate RHEL with Active Directory using SSSD.

There are inherent structural differences between how Windows and Linux handle system users. The user schemas used in Active Directory and standard LDAPv3 directory services also differ significantly. When using an Active Directory identity provider with SSSD to manage system users, it is necessary to reconcile Active Directory-style users to the new SSSD users. There are two ways to achieve it:

Written on October 6, 2015
linux hadoop sssd ad active-directory rhel centos security


Setting `iptables` - `Port` forwarding from one interface to another.

iptables is a user-space application program that allows a system administrator to configure the tables provided by the Linux kernel firewall (implemented as different Netfilter modules) and the chains and rules it stores. Different kernel modules and programs are currently used for different protocols; iptables applies to IPv4, ip6tables to IPv6, arptables to ARP, and ebtables to Ethernet frames.

Written on October 6, 2015
linux iptables port-forwarding interface network


Setting up Pentaho Data Integration 5.4.1 with Hadoop Cluster (Clouder Manager)

Pentaho Data Integration (PDI) is a suite of open source Business Intelligence (BI) products which provide data integration, OLAP services, reporting, dashboarding, data mining and ETL capabilities. Here we are setting up pentaho server has the below steps. Will be referring to Pentaho Data Integration as PDI from now on.

Written on October 6, 2015
linux hadoop pentaho data-integration cloudera manager


Installing `squid` as a sibling to an already existing Parent `squid`.

Squid is a caching and forwarding web proxy. It has a wide variety of uses, from speeding up a web server by caching repeated requests; to caching web, DNS and other computer network lookups for a group of people sharing network resources; to aiding security by filtering traffic.

Written on October 6, 2015
linux squid proxy


Creating a two-node RHEL cluster with Virtual IP using CMAN and Pacemaker.

CMAN v3 is a Corsync plugin that monitors the names and number of active cluster nodes in order to deliver membership and quorum information to clients (such as the Pacemaker daemons). We are using this to have a VIP (Virtual IP) being made high available.

Written on October 6, 2015
linux ha high-availability cman pacemaker heartbeat haproxy


Interface Forwarding - from `eth1` to `eth0` on `EDGE` node.

Adding route to all the slaves which reside on a private network to communicate with External Server directly using an EDGE node using Interface Forwarding.

Written on October 6, 2015
linux hadoop interface network


Installing `ansible` on RHEL 6.6.

Ansible is a radically simple IT automation engine that automates cloud provisioning, configuration management, application deployment, intra-service orchestration, and many other IT needs. For more detail, hop over to docs.ansible.com.

Written on October 6, 2015
linux ansible hadoop rhel centos


Mounting RAID10 using `parted`.

GUID Partition Table (GPT) is a standard for the layout of the partition table on a physical hard disk, using globally unique identifiers (GUID). Although it forms a part of the Unified Extensible Firmware Interface (UEFI) standard (Unified EFI Forum proposed replacement for the PC BIOS), it is also used on some BIOS systems because of the limitations of master boot record (MBR) partition tables, which use 32 bits for storing logical block addresses (LBA) and size information.

Written on September 30, 2015
linux gparted parted mount raid


Setup and Configure NFS Mounts on Linux Server

A Network File System (NFS) allows remote hosts to mount file systems over a network and interact with those file systems as though they are mounted locally. This enables system administrators to consolidate resources onto centralized servers on the network.

Written on September 30, 2015
linux nfs mount


Ansible Playbook - Setup Zookeeper Using `tarball`.

This is a simple zookeeper playbook, to quickly start zookeeper running on a single or more nodes, in a clustered mode.

Written on June 16, 2015
linux ansible hadoop zookeeper


Ansible Playbook - Setup Storm Cluster.

This is a simple Storm Cluster Setup. We are using a dedicated Zookeeper Cluster/Node, instead of the standalone zkserver. Below is how we will deploy our cluster.

Written on June 16, 2015
linux ansible storm hadoop zookeeper


Ansible Playbook - Setup Kafka Cluster.

This is a simple Kafka setup. In this setup we are running kafka over a dedicated zookeeper service. (NOT the standalone zookeeper which comes with kafka)

Written on June 16, 2015
linux ansible kafka hadoop zookeeper


Ansible Playbook - Setup Hadoop CDH5 Using `tarball`.

Setting up Hadoop using Ansible, we will be using cdh5 tarball for installation of the cluster.

Written on June 16, 2015
linux ansible hadoop tarball


Streaming Data Processing - Storm Vs Spark.

Apache Spark is an in-memory distributed data analysis platform– primarily targeted at speeding up batch analysis jobs, iterative machine learning jobs, interactive query and graph processing.

Written on June 1, 2015
linux storm spark data-processing


Creating a passwordless entry on Centos 6.5.

You can enter a linux system without entring a password using below steps. We will be creating a ssh jey which will be share between the servers, which will be used to authenticate. This is also a secure way of connecting to server, as the private key is inside the user home directory and can only be accessed by the user.

Written on March 19, 2015
linux centos rhel


Creating a Multi-node Cassandra Cluster on Centos 6.5.

The Apache Cassandra database is the right choice when you need scalability and high availability without compromising performance. Linear scalability and proven fault-tolerance on commodity hardware or cloud infrastructure make it the perfect platform for mission-critical data. Cassandra’s support for replicating across multiple datacenters is best-in-class, providing lower latency for your users and the peace of mind of knowing that you can survive regional outages.

Written on March 19, 2015
hadoop linux cassandra centos rhel


Installing SpagoBI 5.1 on Centos 6.5 -Tomcat 7 with MySQL 5.6.

The SpagoBI project is a free software/open source initiative by the SpagoBI Labs of Engineering Group. It aims to realize the most complete 100% open source business intelligence suite, aggregating developers, integrators, companies, users and passionate people in an open community.

Written on March 4, 2015
linux spagobi bi spago centos tomcat mysql


KVM Installation on CentOS 6.x.

KVM is a kernel-based Virutal Machine which grows quickly in maturity and popularity in the Linux server market. Red Hat officially dropped Xen in favor of KVM since RHEL 6. With KVM being officially supported by Red Hat, installing KVM on RedHat-based systems should be a breeze.

Written on February 18, 2015
linux kvm centos rhel virtual-machine vmware


Performance Tuning HBase and Hadoop.

Using HBase in production often requires that you turn many knobs to make it hum as expected. More Here http://hbase.apache.org/0.94/book/performance.html

Written on February 13, 2015
linux hadoop hbase performance-tuning


Setting you Hbase Cluster on Hadoop (YARN). Ubuntu 12.04 LTS

HBase is an open source, non-relational, distributed database modeled after Google’s BigTable and written in Java. It is developed as part of Apache Software Foundation’s Apache Hadoop project and runs on top of HDFS (Hadoop Distributed Filesystem), providing BigTable-like capabilities for Hadoop. That is, it provides a fault-tolerant way of storing large quantities of sparse data (small amounts of information caught within a large collection of empty or unimportant data, such as finding the 50 largest items in a group of 2 billion records, or finding the non-zero items representing less than 0.1% of a huge collection).

Written on February 13, 2015
hadoop hbase linux ubuntu yarn


Unable to locate package oracle-java7-installer - Ubuntu 13

Was installing Java today, this is an easy install thanks to ppa:webupd8team/java, but when I tried it was not working, but has worked for me all this while.

Written on February 11, 2015
linux ubuntu java jdk


Installing `zabbix-java-gateway` on Centos 6.5

Zabbix 2.0 adds native support for monitoring JMX applications by introducing a new Zabbix daemon called Zabbix Java gateway. Zabbix Java gateway is a daemon written in Java. When Zabbix server wants to know the value of a particular JMX counter on a host, it asks Zabbix Java gateway, which uses the JMX management API to query the application of interest remotely. The application does not need any additional software installed, it just has to be started with -Dcom.sun.management.jmxremote option on the command line.

Written on February 9, 2015
zabbix linux java zabbix-java-gateway centos rhel nagios monitoring


How to setup HAProxy

HAProxy is the Reliable, High Performance TCP/HTTP Load Balancer and it works nicely with Deveo Cluster setup.

Written on February 9, 2015
linux ha load-balancing high-avaliability proxy nginx haproxy


Using `npm` behind a proxy

npm stands for Node Package Manager, and is the default package manager for the JavaScript runtime environment Node.js.

Written on February 7, 2015
linux npm nodejs proxy


Integrating NODEJS and Kafka

Below are the list required to integrate NodeJS and Kafka. This is a simple HOWTO to get started.

Written on February 7, 2015
linux webserver nodejs kafka-node kafka integrate


Sending JSON to NodeJS to Multiple Topics in Kafka - using kafka-node.

Kafka-node is a Node.js client with Zookeeper integration for Apache Kafka 0.8.1 and later. The Zookeeper integration does the following jobs:

Written on February 7, 2015
linux webserver nodejs kafka-node json kafka


NodeJS Kafka Producer - Using `kafka-node`

Now that we have Kafka and NodeJS ready. Lets some data to our Kafka Cluster.

Written on February 6, 2015
linux webserver nodejs kafka-node kafka


Sending JSON to NodeJS to Kafka.

What we are trying to achieve ?

Written on February 6, 2015
linux webserver nodejs kafka kafka-node


Installing NodeJS on Centos 6.6.

Node.js is an open-source, cross-platform runtime environment for developing server-side web applications. Node.js applications are written in JavaScript and can be run within the Node.js runtime on OS X, Microsoft Windows, Linux, FreeBSD, IBM AIX, IBM System z and IBM i.

Written on February 6, 2015
linux webserver nodejs centos rhel


Installing KAFKA Single Node - Quick Start.

Apache Kafka is publish-subscribe messaging rethought as a distributed commit log. Kafka is a distributed, partitioned, replicated commit log service. It provides the functionality of a messaging system, but with a unique design. What does all that mean?

Written on February 6, 2015
linux hadoop kafka quick-start


sudo Permission On Server - effective uid is not 0, is sudo installed setuid root?

Got into this issue on the server, when trying to install mysql-server.

Written on February 5, 2015
linux setuid root permissions


Custom Webserver/NodeJS sysctl.conf file.

Have updated and explanation in the conf file below.

Written on February 3, 2015
linux webserver nodejs performance tuning kernel sysctl


Creating server parameter sysctl.conf for WebServers / FTP Server.

sysctl is an interface that allows you to make changes to a running Linux kernel. With /etc/sysctl.conf you can configure various Linux networking and system settings such as:

  • Limit network-transmitted configuration for IPv4
  • Limit network-transmitted configuration for IPv6
  • Turn on execshield protection
  • Prevent against the common syn flood attack
  • Turn on source IP address verification
  • Prevents a cracker from using a spoofing attack against the IP address of the server.
  • Logs several types of suspicious packets, such as spoofed packets, source-routed packets, and redirects.
Written on February 3, 2015
linux webserver ftp-server kernel sysctl performance tuning


Creating Virtual Interface and Assign Multiple IP Addresses.

Current IP of the server : 10.130.18.11. Virtual IPs being assigned to server : 10.130.18.22, 10.130.18.23, 10.130.18.24 to our server..

Go to network-scripts directory and copy the existing ifcfg-eth0 file. Make sure you are using a static ip for your server.

Written on February 2, 2015
linux vip network multiple-ip


Changing Timezone in Ubuntu server.

Changing Timezone in Ubuntu server.

Written on January 29, 2015
linux ubuntu timezone


Hadoop `sysctl.conf` parameters.

Performace tuning Hadoop at kernel level. sysctl is an interface that allows you to make changes to a running Linux kernel. With /etc/sysctl.conf you can configure various Linux networking and system settings such as:

  • Limit network-transmitted configuration for IPv4
  • Limit network-transmitted configuration for IPv6
  • Turn on execshield protection
  • Prevent against the common syn flood attack
  • Turn on source IP address verification
  • Prevents a cracker from using a spoofing attack against the IP address of the server.
  • Logs several types of suspicious packets, such as spoofed packets, source-routed packets, and redirects.
Written on January 28, 2015
hadoop linux kernel sysctl performance tuning


Setting SUN Java for Ubuntu.

The Java Development Kit (JDK) is an implementation of either one of the Java SE, Java EE or Java ME platforms released by Oracle Corporation in the form of a binary product aimed at Java developers on Solaris, Linux, Mac OS X or Windows. This is currently not avaiable on apt-get repo, so here is quick way to setup java on ubuntu server/desktop with tar ball.

Written on January 27, 2015
linux install-java java ubuntu jdk


Simple Script to create a user on centos 6.5

Default passwd is set as username@123 As we are setting the chage, user needs to change the passwd on first login.

Written on January 27, 2015
linux create user centos rhel


Performance Tuning for `nginx`

Nginx (pronounced engine x) is a web server with a strong focus on high concurrency, performance and low memory usage. It can also act as a reverse proxy server for HTTP, HTTPS, SMTP, POP3, and IMAP protocols, as well as a load balancer and an HTTP cache.

Written on January 22, 2015
linux nginx performance tuning load-balancing


How To Configure Swappiness

Swappiness is a Linux kernel parameter that controls the relative weight given to swapping out runtime memory, as opposed to dropping pages from the system page cache. Swappiness can be set to values between 0 and 100 inclusive. A low value causes the kernel to avoid swapping, a higher value causes the kernel to try to use swap space. The default value is 60, and for most desktop systems, setting it to 100 may affect the overall performance, whereas setting it lower (even 0) may decrease response latency.

Written on January 20, 2015
hadoop linux swappiness swap sysctl config


HA Setup - `heartbeat` for `nginx`.

Heartbeat is a daemon that provides cluster infrastructure (communication and membership) services to its clients. This allows clients to know about the presence (or disappearance!) of peer processes on other machines and to easily exchange messages with them. NGINX is a free, open-source, high-performance HTTP server and reverse proxy, as well as an IMAP/POP3 proxy server. NGINX is known for its high performance, stability, rich feature set, simple configuration, and low resource consumption.

Written on January 6, 2015
linux heartbeat nginx http httpd replication load-balancing high-availability


SpagoBI Tomcat Clustering Using `mod_jk` and `httpd` on Centos - In-Memory Session Replication.

In an earlier post we did a basic session based replication, but the session was not redundant.

Written on January 4, 2015
linux spago-bi bi tomcat http httpd replication load-balancing high-availability


Setting up Tomcat Cluster for SpagoBI 5.1.

Setting up SpagoBI 5.1 on Tomcat with MySQL post.

Written on January 4, 2015
linux spago bi tomcat http spago-bi


Setting up `NGINX` for `HTTP` load balancing.

NGINX is a free, open-source, high-performance HTTP server and reverse proxy, as well as an IMAP/POP3 proxy server. NGINX is known for its high performance, stability, rich feature set, simple configuration, and low resource consumption.

Written on January 3, 2015
linux nginx http httpd load-balancing high-availability


httpd HA using Heartbeat.

httpd is the Apache HyperText Transfer Protocol (HTTP) server program. It is designed to be run as a standalone daemon process. When used like this it will create a pool of child processes or threads to handle requests. Heartbeat is a daemon that provides cluster infrastructure (communication and membership) services to its clients. This allows clients to know about the presence (or disappearance!) of peer processes on other machines and to easily exchange messages with them.

Written on January 3, 2015
linux heartbeat high-availability httpd http


Zabbix Hadoop Monitoring.

This script can be used to monitor Namenode Parameters. This script can be used to Generate Zabbix Import XML or Send monitoring data to Zabbix server.

Written on November 25, 2014
zabbix csv-processing linux mib python hadoop xml nagios monitoring


Fetch Monitoring Data from Item to CSV file.

This section describes how to get monitoring data from Zabbix server through the command line and export these to a CSV formatted file.

Written on October 16, 2014
zabbix csv-processing iReasoning linux mib python nagios monitoring


Zabbix Item Creation using OID and Name CSV files.

This script is to create a xml import file from a csv file. We use 2 csv file for this.

Written on October 15, 2014
zabbix csv-processing iReasoning linux mib python nagios monitoring


Send KPI Data to Zabbix from XML KPI File

Send Zabbix metric by reading from a XML file. This needs package zbxsend from : https://github.com/zubayr/zbxsend

Written on October 15, 2014
zabbix xml-processing linux mib python zbxsend nagios monitoring


Zabbix Import File from iReasoning Export CSV

Module used to generate Zabbix import xml for snmp traps, Below are steps to create a Zabbix Import for Traps. These traps are captured from the snmptrap function in Zabbix.

Written on October 14, 2014
zabbix csv-processing iReasoning linux mib nagios monitoring


SFTP Data Collector

Easy way to collect files recursively over a sftp server is to connect to the server over scp and do scp -r. Problem was that the device we were connecting to did not support recursive :( over a regEx expression.

Written on October 14, 2014
sftp ftp data-collector linux python script github