What I have leant

Yum

yum install <package-name> #install package
yum remove <package-name> #remove package
yum erase <package-name> #delete package
yum search <keyword> #search if exsit package containing keyword
yum repolist #show repository list
yum update #update installed packages

Good site searching for repos: Linux packages search

How to add a yum repository mannually

Yum repository directory is located at /etc/yum.repos.d/, so if you have a yum .repo file, just copy this file into the directory.

And there are some repositories wrapped as rpm files, such as postgreSQL. What you need to do is to download the rpm files and install them.
See Also: Adding, Enabling, and Disabling a Yum Repository

RPM

rpm -i <package-name>.rpm #install rpm package
rpm -qa | grep <package> #check if installed

PS

ps #show process
ps -l #Display information associated with the following keywords: uid, pid, ppid, flags, cpu, pri, nice, vsz=SZ, rss, wchan, state=S, paddr=ADDR, tty, time, and command=CMD
ps aux | grep <keyword> #show details of process

Kill

kill [signal] PID #kill specified process using pid, most used is '-9' to kill process.

kill -l #display all the [signal]s
kill `ps | grep <keyword> | awk '{print $1}'` #kill process by <keyword>

TOP

top #display all process

top -user <user> #display <user>'s process

top -pid <pid> #display process by given <pid>

service

service <service-name> status #show service status(running or stopped)

service <service-name> start #start service

service <service-name> stop #stop service

hostname

hostname #show hostname
hostname --fqd #show fully qualified domain name(FQDN)
hostname NEW_NAME #change hostname temporary

See also: How to Change the Hostname of a Linux System

restart the network

/etc/init.d/network restart        #!Red Hat Specific, may not applicable in                                        other linux

Add third-party repositories

EPEL: see here

User

Add user

userdel user #delete user

useradd -G <group-name> <username> #add user to non-existed group
useradd -g <group-name> <username> #add user to primary group
usermod -a -G <group-name> <username> #add a existing user to existing group
usermod -g <group-name> <username> #change user's primary group to <group-name>
passwd <username> #setup password for user
id <username> #show user information

tar

tar -zxvf <file-name>.tar.gz #extract tar.gz file

Netstat

netstat -a #list all ports of TCP/UDP

netstat -at #list TCP connection

netstat -au #list UDP connection

netstat -l #list all listening ports

Catalog

Install HDP2.0 using Sandbox;
Prepare for the Storm-yarn depolyment;
Set up Storm on your cluster

All the steps are based on Mac OS 10.8, Windows & Linux can be applied as well with some tiny differences.

Install HDP2.0 using Sandbox

What you need?

VMware Fusion/Play 5 (Virtualbox 4.2 can work as well)
Hortonworks Sandbox 2.0

Steps

1.Open VMware Fusion.

2.Click File -> Import.

3.The file browser opens. Select the appropriate Sandbox appliance file.

Click Open.

4.Import Library opens. Unless you have specific needs, the default values are fine.

Click Import.

5.The appliance is imported. A console window opens and the VM shows up in the Virtual Machine Library.

6.Start the VM. When the Sandbox has finished starting up, the console displays the login instructions.

Press fn+control+option+f5 to start command line.

7.Use a browser on your host machine to open the URL displayed on the console. You will see the index page. Now the HDP2.0 has been settled.

Prepare for the Storm-yarn depolyment

(To distinguish the commands between local Mac OS and VM, I assume the Mac OS command line starts with '$', and VM command line starts with '#')

1. Open the Terminal in your Mac

2. ssh from your Mac to VM;

$ ssh root@<Your VM IP>

Password is hadoop in default.

3. Disable selinux using the command:

# setenforce 0

4. Edit the SELinux configuration file:

# vi /etc/selinux/config

Change SELINUX=enforcing to SELINUX=disabled

5. Stop the iptables firewall and disable it.

# stop iptables
# service iptables stop
# chkconfig iptables off

6. Install the wget package

# yum -y install wget

7. Get the repo for Ambari and copy it to /etc/yum.repos.d

# wget http://public-repo-1.hortonworks.com/ambari-beta/centos6/1.x/beta/ambari.repo
# cp ambari.repo /etc/yum.repos.d

8. Install Oracle Java7 Development environment.

Download jdk file from Oracle Website to Mac local storage, choose the Linux x64 rpm package.
Copy downloaded rpm into the VM, let's say we downloaded jdk into ~/Download

$ scp ~/Download/jdk-7u<version>-linux-x64.rpm root@<Your VM IP>:/tmp

Install the jdk file

# rpm -ivh jdk-7u<version>-linux-x64.rpm

Set JAVA_HOME & PATH

# vi ~/.bash_profile

before

export PATH

JAVA_HOME=/usr/java/jdk1.7.0_45/
export JAVA_HOME
PATH=$PATH:$HOME/bin
PATH=$JAVA_HOME/bin:$PATH

Test your Installation to see if jdk7 installed successfully.(if it say java 1.6 instead of 1.7, check previous steps again)

# java -version

9. Install ntpd, start service and sync time

# yum -y install ntp
# service ntpd start

10. Run the Ambari server setup

# ambari-server setup -s -j /usr/java/jdk1.7<version>/

11. Start Ambari server & agent

# ambari-server start
# ambari-agent start

12. Install Maven 3.11

# wget http://mirror.symnds.com/software/Apache/maven/maven-3/3.1.1/binaries/apache-maven-3.1.1-bin.tar.gz
# tar -zxvf apache-maven-3.1.1-bin.tar.gz
# mkdir -p /usr/lib/maven
# mv apache-maven-3.1.1 /usr/lib/maven
# vi ~/.bash_profile

Add Scripts before export PATH:

PATH=$PATH:/usr/lib/maven/bin

13. Get a copy of the repository for Storm on YARN from GitHub

# wget https://github.com/anfeng/storm-yarn/archive/master.zip

# unzip master.zip

14. Edit the pom.xml repos and Hadoop version to point at Hortonworks.

# cd storm-yarn-master
# vi pom.xml

Uncommand some lines and make some lines commanded as below:

Set up Storm on your cluster

1. Create a work folder to hold working files for Storm. Let's say '~/workspace/storm'

# mkdir -p ~/workspace/storm

2. Copy storm.zip to work folder. Go to your work folder and unzip storm.zip.

# cp lib/storm.zip ~/workspace/storm
# cd ~/workspace/storm
# unzip storm.zip

3. Add storm-0.9.0-wip21 and storm-yarn-master bin folders to path

# vi ~/.bash_profile

Add Scripts before export PATH

PATH=$PATH:$HOME/workspace/storm/storm-0.9.0-wip21/bin:$HOME/storm-yarn-master/bin

4. Add root user to hdfs group

# usermod -G hdfs root

5. Add storm.zip to hdfs /lib/storm/0.9.0-wip[*]/storm.zip

# sudo -u hdfs hadoop fs -put ~/storm-yarn-master/lib/storm.zip /lib/storm/0.9.0-wip[*]/storm.zip

You may encounter some permission problems, try this:

# cp ~/storm-yarn-master/lib/storm.zip /tmp/storm.zip

# chown hdfs:hdfs /tmp/storm.zip

# sudo -u hdfs hadoop fs -put /tmp/storm.zip /lib/storm/0.9.0-wip[*]/storm.zip

6. Start Maven in the storm-yarn-master folder.

# cd storm-yarn-master
# mvn package

7. Start Storm

# storm-yarn launch

You may encounter permission issues when launch storm, try step 4 in this section to set root to hdfs group

8. Get the stormconfig with the yarn application id. (application id should be like application_numbers_numbers)

# yarn application -list

9. We store the storm.yaml file in the .storm directory so the storm command can find it when it is submitting jobs.

# storm-yarn getStormConfig -appId <application id> -output ~/.storm/storm.yaml

10. Try running two of the sample topologies:

Word Count:

# storm jar lib/storm-starter-0.0.1-SNAPSHOT.jar storm.starter.WordCountTopology

Exclamation:

# storm jar lib/storm-starter-0.0.1-SNAPSHOT.jar storm.starter.ExclamationTopology

11. Storm UI monitoring tool

Still not working yet...to be continue.