Setting up Hadoop 1.22 on Centos 6.4 with OpenJDK1.6

Before we begin I would like to give a little intro.  We have been working on a lot of “Big Data” and machine learning type projects for some future products and services we intend to offer.  So we like to share some of our initial findings and code into this big realm.  Haven’t heard of Hadoop yet?
As the title indicates we will be installing Hadoop 1.22 on a Centos 6.4 linux server.  This will be a “singe node” install, I will write a separate articles on installing hadoop on a cluster.  If you are just getting into Hadoop this tutorial will help get your first hadoop installation out of the way.

First let’s get started, with some java.

 Installing OpenJDK 1.6

Luckily hadoop works and runs well (from what we have tested) on the openjdk.  No need to download any jdk binaries from Oracle.

yum install java-1.6.0-openjdk.x86_64

Hadoop will run just fine with the vanilla open jdk.  However for maven to run properly we are going need the devel jdk also installed.

yum install java-1.6.0-openjdk-devel.x86_64

Let’s make sure java is registered on your system run.

First let’s see if the correct version is setup in our path.

java -version

java version "1.6.0_28"
OpenJDK Runtime Environment (IcedTea6 1.13.0pre) (rhel-1.66.1.13.0.el6-x86_64)
OpenJDK 64-Bit Server VM (build 23.25-b01, mixed mode)

javac -version

IF you do not see this version of Java, you probably had a previous install of java on your system.  Luckily we are using CentOS so we can easily change this by running the alternatives config.

alternatives -config java
There is 1 program that provides 'java'.
Selection Command
 -----------------------------------------------
 *+ 1 /usr/lib/jvm/jre-1.6.0-openjdk.x86_64/bin/java

If you had any previous versions of java installed make sure the 1.6 JDK is selected.  We will need this version for our later examples.

Now that we have all that java stuff out of the way let’s get down to installing and configuring hadoop.

Note: As with anything there are many ways to do this, Robe de Mariée 2014when we get into installing Hadoop into a cluster it will save you a lot of time.  However sometimes it’s best to do the hard way first to familiarize your self with something new.

Before we begin let’s create some credentials for Hadoop to use.

useradd hadoop

passwd hadoop

You do not have to create a specific user account for hadoop to run properly, of course root works just fine.  You can even enable key based login for your cluster.

Note: For this demo we ran everything as the hadoop account with no key login.

Let’s create a directory for all the hadoop binaries(okay mostly jars) to live in.

Create a Hadoop Directory

mkdir /opt/hadoop

cd /opt/hadoop

wget http://apache.cs.utah.edu/hadoop/common/hadoop-1.2.1/hadoop-1.2.1-bin.tar.gz

Note: Default minimum installs of CentOS don’t include wget.

yum install wget

tar -xzf Hado -C hadoop

Chown -R /opt/hadoop hadoop

 Edit Hadoop Configs

Now that we have everything extracted and proper ownership applied there are a few hadoop configs we will need to change.

 vi conf/core-site.xml

#Add the following inside the configuration tag
<property>
    <name>fs.default.name</name>
    <value>hdfs://localhost:9000/</value>
</property>
<property>
    <name>dfs.permissions</name>
    <value>false</value>
</property>

Edit hdfs-site.xml

 vi conf/hdfs-site.xml

# Add the following inside the configuration tag
<property>
 <name>dfs.data.dir</name>
 <value>/opt/hadoop/hadoop/dfs/name/data</value>
 <final>true</final>
</property>
<property>
 <name>dfs.name.dir</name>
 <value>/opt/hadoop/hadoop/dfs/name</value>
 <final>true</final>
</property>
<property>
 <name>dfs.replication</name>
 <value>2</value>
</property>

Edit mapred-site.xml

 vi conf/mapred-site.xml

# Add the following inside the configuration tag
<property>
        <name>mapred.job.tracker</name>
 <value>localhost:9001</value>
</property>

Edit hadoop-env.sh

 vi conf/hadoop-env.sh

export JAVA_HOME=/usr/lib/jvm/jre-1.6.0-openjdk.x86_64/

Set JAVA_HOME path as per your system configuration for java.

Let’s format our first namenode!

 su - hadoop
 cd /opt/hadoop/hadoop
 bin/hadoop namenode -format

 Start Hadoop

 bin/start-all.sh

Each Service Has It’s Own Status Page

  http://hnode1.vaurent.com:50030/   for the Jobtracker
  http://hnode1.vaurent.com:50070/   for the Namenode
  http://hnode1.vaurent.com:50060/   for the Tasktracker

To stop Hadoop

bin/stop-all.sh

That about sums it all up. I will update this article with links to setting up Maven and running our first Hadoop test with Apache Pig.

Enable SUDO for RHEL & CENTOS

Sudo is an arguably safer alternative to logging in (or using the su command) to the root account. Sudo allows you to partition and delegate superuser commands (functions) without giving a user total "root" power on the system. Here are a few other advantages:

• Privileged commands are logged. It is a simple way to audit who did what at what point in time.
• It is more efficient to use sudo over su, or to log in as root, in reference to keystrokes.
• You don't have to change the root password when an administrator has his root functions revoked, leaves the company, changes roles, etc. The change part is easy, but coordinating the new password with every other administrator can be a hassle.

# Is sudo installed?
Login with the root user.

Let's first determine if the sudo package is installed.

# rpm -q sudo

If the package is not installed, we can retrieve/install it with the following command:

# yum install sudo

# Create a normal user
Create the user and add to the wheel group. The wheel group is usually predefined as the container for administrator accounts.

# useradd -G wheel -c "Test User" testNew

Create a password for the user.

# passwd testNew
Changing password for user testNew.
New UNIX password: P@$$w0rd
Retype new UNIX password: P@$$w0rd
passwd: all authentication tokens updated successfully.

# Or modify an existing user
Add an existing user (the user testMod in my example) to the wheel group.

# usermod -aG wheel testMod

# Modify the sudoers file
Use the visudo command to safely modify the sudoers file.

# visudo

Search for the Allows people in group wheel to run all commands directive and uncomment the second line to enable the wheelgroup to run all commands.

...
## Allows people in group wheel to run all commands
%wheel  ALL=(ALL)       ALL
...

Save the file.

# Test with a privileged command (logged in as a normal user)
We will first attempt to run the visudo command with our normal user account. As expected, the operation will fail.

$ /usr/sbin/visudo
visudo: /etc/sudoers: Permission denied

Now we will run the command within the context of sudo to temporarily elevate the privileges of our normal user.

$ sudo -i visudo
[sudo] password for test: P@$$w0rd

# Verify the command is logged
Check the secure log to verify the event is recorded.

$ sudo grep visudo /var/log/secure
...
Aug  21 20:01:20 centos sudo:     test : TTY=pts/0 ; PWD=/home/test ; USER=root ; COMMAND=/bin/bash -c visudo
...

This is just a single use case of how to implement sudo. I encourage you to check out the man pages and other documentation to see how you can tailor it to your specific environment.

WildFly – A New Improved JBoss Application Server for Linux

As we all know that JBoss AS has been renamed to WildFly.

WildFly 8 is Red Hat‘s Java EE 7 compliant open source application server. The Main features are as below:

Java EE 7 Compatibility: The biggest change in this is that now WildFly 8 is official Java EE7 Certified.
High Performance Web Server: Undertow is new high performance web server written in Java. Now this has been implemented in WildFly 8. This is really designed for high throughput and scalability and can handle millions of connections. Undertow’s lifecycle is completely controlled by the embedding application. This is extremely lightweight with core jar having size of 1MB and embedded server using less than 4MB of heap space. This is really great.
3Port Reduction:  Since it is using Undertow which support for Upgrading HTTP, which will allow multiple protocols to be multiplexed over single HTTP port. WildFly 8 has moved nearly all of its protocols to be multiplexed over two HTTP ports: one is management and another one is application port. This is really a big change and benefit to cloud providers (such as OpenShift) who run hundreds to thousands of instances on a single server. In total, it has two default ports for configuration and they are 9990 (Web Administration Console) and 8080 (Application Console).
Management Role Based Access Control & Auditing: This is the new and interesting thing implemented in WildFly 8. By using this we can create different users and can assign those one to different roles as per requirements. I’ll show you later with screen shots.
Logging:The management API now supports the ability to list and view the available log files on a server. Now, we have attribute called “add-logging-api-dependencies” available for any kind of deployments in which we want to skip container logging. This will disable the adding of the implicit server logging dependencies. We have another option i.e. we can use a jboss-deployment-structure.xml to exclude the logging subsystem. Using this, it will help to stop the logging subsystem from going through any deployment.

We can also make use of another parameter i.e. use-deployment-logging-config for enabling/ disabling processing of logging configuration files within a deployment.
 Note: System Property that we were using for disabling per logging has been deprecated from this version.

Clustering: Again Big change is one clustering. All Features related to Clustering support had been changed in WildFly 8 and these includes as below:

• Distributed web session has been optimized for it with new Java Based Web Server i.e. Undertow.
• mod_cluster support for Undertow.
• Optimized Distributed SSO(Single Sign-On) capabilities and support for Undertow.
• New/optimized distributed @Stateful EJB caching implementation.
• WildFly 8 added some new public clustering API.
• For creating singleton services it provides new public APIs.

 CLI Improvements: CLI Configuration has also been improved. You know All admin love to work on CLI ;). So, now we can create alias for particular server and then can use that alias whenever want to connect to that server using connect command.
There are still lots of enhancements and updates done in WildFly 8. You can check all these at:http://wildfly.org/news/2014/02/11/WildFly8-Final-Released/

Installation of WildFly 8 in Linux

Before moving ahead with Installation make sure that you have Java EE 7 installed on your system. WildFly 8 will not work with previous revisions. Please follow the below guide to install Java EE 7 in the Linux systems (Install JDK/JRE 7u25 in Linux).

Step 1: Downloading WildFly 8

[root@anuppc-02]# wget http://download.jboss.org/wildfly/8.0.0.Final/wildfly-8.0.0.Final.zip
[root@anuppc-02 jboss]# wget http://download.jboss.org/wildfly/8.1.0.CR1/wildfly-8.1.0.CR1.tar.gz
--2014-05-11 19:00:31--  http://download.jboss.org/wildfly/8.1.0.CR1/wildfly-8.1.0.CR1.tar.gz
Resolving download.jboss.org... 23.67.250.131, 23.67.250.122
Connecting to download.jboss.org|23.67.250.131|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 115039521 (110M) [application/x-gzip]
Saving to: `wildfly-8.1.0.CR1.tar.gz'

100%[======================================================================================================>] 115,039,521  975K/s   in 2m 10s

2014-05-11 19:02:41 (867 KB/s) - `wildfly-8.1.0.CR1.tar.gz' saved [115039521/115039521]

[root@anuppc-02 data3]#

Step 2: Extract it

# tar -xvf yourfile.tar [to extract to current directory]
# tar -C /myfolder -zxvf yourfile.tar.gz [ to extract to another directory]
$ tar -zxvf wildfly-8.1.0.CR1.tar.gz

Step 3: Setting Environment variable

Now set some environment variables. You can set these on system wise or within your configuration files. Here I am setting within configuration files standalone.sh and standalone.conf in ‘bin‘ folder.
[root@anuppc-02 data]# cd wildfly-8.0.0.Final
[root@anuppc-02 data]# cd bin/ 
# Add following 2 lines to     standalone.sh / standalone.conf
JBOSS_HOME="/var/cemp/data3/wildfly-8.1.0.CR1"
JAVA_HOME="/usr/java/jdk1.8.0_05"
Note: For whole system wide, you can set it under ‘/etc/profile‘ file.

standalone.xml
<socket-binding-group name="standard-sockets" default-interface="public" port-offset="${jboss.socket.binding.port-offset:0}">
    <socket-binding name="management-http" interface="management" port="${jboss.management.http.port:9990}"/>
    <socket-binding name="management-https" interface="management" port="${jboss.management.https.port:9993}"/>
    <socket-binding name="ajp" port="${jboss.ajp.port:8009}"/>
    <socket-binding name="http" port="${jboss.http.port:9080}"/> <!-- 8080 to 9080 -->
    <socket-binding name="https" port="${jboss.https.port:9443}"/> <!-- 8443 to 9443 -->
    <socket-binding name="txn-recovery-environment" port="4712"/>
    <socket-binding name="txn-status-manager" port="4713"/>
    <outbound-socket-binding name="mail-smtp">
        <remote-destination host="localhost" port="25"/>
    </outbound-socket-binding>
 </socket-binding-group>
</server>

Step 4: Starting WildFly 8

Now start server i.e. for standalone mode use ‘standalone.sh‘ and for domain mode use ‘domain.sh‘.
[root@anuppc-02 bin]# ./standalone.sh
[root@anuppc-02 bin]# ./domain.sh
But, here I am starting in standalone mode. By default it will get start by ‘standalone.xml‘ file, But You can also start with some other configuration using ‘–server-config‘ option.
As below I am starting server with ‘standalone-full-ha.xml‘ and this file is present in “$JBOSS_HOME/standalone(profile)/configuration/”.
[root@anuppc-02 bin]# ./standalone.sh --server-config standalone-full-ha.xml

Step 5: Acessing WildFly 8

Now you can point your browser to ‘http://localhost:9080‘ (if using the default configured http port) which brings you to the Welcome Screen.
From here, you can access WildFly community documentation guides and enhanced web-based Administration Console access.

Step 6: Managing WildFly 8

WildFly 8 provides two administrative consoles for managing running instance:

• web-based Administration Console
• command-line interface

Before connecting to administration console or remotely using the command line, you will need to create a new user using the ‘add-user.sh‘ script in the bin folder.
Next, go to ‘bin‘ directory, set ‘JBOSS_HOME‘ in add-user.sh (if variable is not set on system bases) and create user as below.
[root@anuppc-02 bin]# ./add-user.sh
Once starting the script you will be guided through the process to add a new user:

Sample Output
What type of user do you wish to add?
 a) Management User (mgmt-users.properties)
 b) Application User (application-users.properties)
(a):
Enter the details of the new user to add.
Using realm 'ManagementRealm' as discovered from the existing property files.
Username : admin
The username 'admin' is easy to guess
Are you sure you want to add user 'admin' yes/no? yes
Password recommendations are listed below. To modify these restrictions edit the add-user.properties configuration file.
 - The password should not be one of the following restricted values {root, admin, administrator}
 - The password should contain at least 8 characters, 1 alphanumeric character(s), 1 digit(s), 1 non-alphanumeric symbol(s)
 - The password should be different from the username
Password :
Re-enter Password :
What groups do you want this user to belong to? (Please enter a comma separated list, or leave blank for none)[  ]:
About to add user 'admin' for realm 'ManagementRealm'
Is this correct yes/no? yes
Added user 'admin' to file '/data/wildfly-8.0.0.Final/standalone/configuration/mgmt-users.properties'
Added user 'admin' to file /data/wildfly-8.0.0.Final/domain/configuration/mgmt-users.properties'
Added user 'admin' with groups  to file /data/wildfly-8.0.0.Final/standalone/configuration/mgmt-groups.properties'
Added user 'admin' with groups  to file /data/wildfly-8.0.0.Final/domain/configuration/mgmt-groups.properties'
Is this new user going to be used for one AS process to connect to another AS process?
e.g. for a slave host controller connecting to the master or for a Remoting connection for server to server EJB calls.
yes/no? yes
To represent the user add the following to the server-identities definition
Press any key to continue . . .
Now access the web-based Administration Console at ‘http://localhost:9990/console‘ and enter the new created username and password to directly access the Management Console.

If you prefer to handle your server from the CLI, run the ‘jboss-cli.sh‘ script from the ‘bin‘ directory that offers the same capabilities available via the web-based UI.
[root@anuppc-02 bin]# cd bin
[root@anuppc-02 bin]# ./jboss-cli.sh --connectConnected to standalone controller at localhost:9999

For more information, follow the official WildFly 8 documentation at https://docs.jboss.org/author/display/WFLY8/Documentation.

MBean Servers in Weblogic

Weblogic comes with three own MBeanServers, which are exported via RMI/IIOP as JSR-160 connectors. They can be looked up via a certain JNDI name as shown in the table below. Additionally, there is the ubiquitous PlatformMBeanServer.
MBean Server                                  JNDI Name
Domain Runtime MBean Server          weblogic.management.mbeanservers.domainruntime
Runtime MBean Server                  weblogic.management.mbeanservers.runtime
Edit MBean Server                          weblogic.management.mbeanservers.edit
PlatformMBeanServer                          —

The Runtime MBean Server specifies an individual application server, whereas the Domain Runtime MBean Server exposes the MBeans for all servers in a cluster. The Edit MBean Server is used for accessing and modifying the domain configuration.
Two access modes: There are two ways how the MBeanServers mentioned above can be exported for remote access:

1. Via RMI/IIOP exported by Weblogic. This way, the three Weblogic MBeanServers (those with an JNDI name) can be exported (but not the PlatformMBeanServer directly). Advantage of this export is that it can be enabled with the admin console and that it includes the complete Weblogic security stack.
2. Via RMI/JRMP exported by the JVM. This allows for the PlatformMBeanServer to be exported (but not the other, Weblogic specific MBeanServers). It gets enabled as usual by setting certain java defines as startup options and is secured the JDK way. A forthcoming blog will clarify how to setup security for JDK exported JSR-160 connectors.

Both methods are explained in detail in the following sections.
RMI/IIOP exported by Weblogic: First of all, some configuration items must be set to enable IIOP exported MBeans. In the admin console, check that following properties are set:

• First, IIOP must be enabled: Domain (‘wl_server’) / Environment / Servers -> Server (‘examplesServer’) -> Protocols -> IIOP -> Enable IIOP
• Allow for anonymous read access, if you want to monitor without sending credentials: Domain (‘wl_server’) -> Security -> General -> Anonymous Admin Lookup Enabled
• If you want to secure IIOP access (or want to have write access), set the name and password of the default user: Domain (‘wl_server’) / Environment / Servers -> Server (‘examplesServer’) -> Protocols -> IIOP -> (Advanced) -> “Default IIOP Username” and “Default IIOP Password”

You need to restart the server if you change one of the options above. The JMX service URL for accessing Weblogic JMX connectors via IIOP looks like
service:jmx:rmi:///jndi/iiop://<server address>:<port>/<jndi name>
where the JNDI name is one of the MBeanServer’s JNDI name as described above. For example:
service:jmx:iiop:///jndi/iiop://bhut:7001/weblogic.management.mbeanservers.runtime

RMI/JRMP exported by JVM: As an alternative the usual way for exporting MBeans via RMI/JRMP can be used. This includes to set some java defines as startup options (probably within setDomainEnv.sh) like:
JAVA_OPTIONS="$JAVA_OPTIONS \ 
    -Dcom.sun.management.jmxremote \
    -Dcom.sun.management.jmxremote.port=9999 \
    -Dcom.sun.management.jmxremote.ssl=false \
    -Dcom.sun.management.jmxremote.authenticate=false"
jconsole service:jmx:rmi:///jndi/rmi://target:9999/jmxrmi
Unfortunately, by default, only the PlatformMBeanServer gets exported without any Weblogic specific MBean. However, there is a way to get to the Weblogic MBeans as described in the next section.

Using PlatformMBeanServer for Weblogic MBeans: For monitoring purposes having Weblogic runtime MBeans and JVM MXBeans in different MBeanServers which are exportable in different ways is quite annoying. However, for Weblogic there is a solution by configuring WLS to use the PlatformMBeanServer as it’s MBeanServer. With this configuration, your are able to access Weblogic MBeans from RMI/JRMP Service URLs (as it is used with ‘normal’ JMX clients like jconsole) and Java MXBeans from the RMI/IIOP connector used by Weblogic.

For this to work, you need to set Domain (‘wl_server’) > Configuration > General > (Advanced) > Platform MBeanServer enabled in the admin console for both WLS 9 and 10. Afterwards a server restart is required.

For WLS 10 an additional configuration parameter has to be set in order to let WLS use the PlatformMBeanServer for its runtime MBeans. The attribute PlatformMBeanServerUsed needs to be set to true on the JMXMBean (it is false by default). Unfortunately I didn’t find a way to set this attribute via the admin console, but only via the Weblogic scripting environment WLST. Assuming that your current working directory is you WLS 10 root directory, you should use wlst.sh to fire up the WLST in interactive mode and when the server is not running:
$ common/bin/wlst.sh
.....
wls:/offline> readDomain('samples/domains/wl_server')
wls:/offline/wl_server>cd('JMX')
wls:/offline/wl_server/JMX>ls()
drw-   NO_NAME_0
wls:/offline/wl_server/JMX>cd('NO_NAME_0')
wls:/offline/wl_server/JMX/NO_NAME_0>ls()
-rw-   CompatibilityMBeanServerEnabled               true
-rw-   DomainMBeanServerEnabled                      true
-rw-   EditMBeanServerEnabled                        true
-rw-   InvocationTimeoutSeconds                      0
-rw-   ManagementEJBEnabled                          true
-rw-   Name                                          null
-rw-   Notes                                         null
-rw-   PlatformMBeanServerEnabled                    false
-rw-   PlatformMBeanServerUsed                       false
-rw-   RuntimeMBeanServerEnabled                     true
wls:/offline/wl_server/JMX/NO_NAME_0>set('PlatformMBeanServerUsed','true')
wls:/offline/wl_server/JMX/NO_NAME_0>set('PlatformMBeanServerEnabled','true')
wls:/offline/wl_server/JMX/NO_NAME_0>updateDomain()
wls:/offline/wl_server/JMX/NO_NAME_0>closeDomain()
wls:/offline>exit()
The path to the domain (samples/domains/wl_server), an the name of the JMX-Bean (NO_NAME_0) might differ at your side, but I think, the idea is clear. As shown above you also can set PlatformMBeanServerEnabled via WLST as well.

Fire up the server and you should be ready for accessing WLS MBeans and JDK MXMBeans from within the same MBeanServer on Weblogic 10.

BTW, setting PlatformMBeanServerUsed via jmx4perl itself doesn’t work, because Weblogic needs some extra boilerplate (which I didn’t dive into) before configuration can be changed via JMX:
Part 1: Source:

• How to implement a custom MBean to manage configuration associated with an application.
• How to package the resulting code and configuration as part of the application's ear file.
• How to register MBeans upon application startup, and unregistered them upon application stop (or undeployment).
• How to use generic JMX clients such as JConsole to browse and edit our application's MBean.

Part 2: Source:

• How to add localized descriptions to our MBean, MBean attributes, MBean operations and MBean operation parameters.
• How to specify meaningful name to our MBean operation parameters.
• We also touched on future enhancements that will simplify how we can implement localized MBeans.

Part 3: Source:

• How to localize our MBean based on the calling client's locale as opposed to the server's locale.
• How to take advantage of the default resource mapping provided by WebLogic Server to localize MBean descriptions.
• How to write a simple client that access our MBean using a locale other than the server's locale.
• How to set the locale associated with JConsole.
• In this new blog entry we will demonstrate:

Part 4 : Source 

• How to emit an AttributeChangeNotification each time a new property is added or changed through our MBean.
• How to properly localize our notifications.
• How to observe notification using JConsole.
• How to write a client that subscribes to the notification emitted by our MBean.