NTP server install on linux

Posted on June 24, 2015June 25, 2015 by coder

I installed NTP server on Ubuntu Server 10.04 using:

sudo apt-get install ntp

1	sudo apt-get install ntp

The NTP daemon seems working and listening on the 123 port.

However, I was not able to get the time from another machine:

sudo ntpdate -u my_ntp_server

1	sudo ntpdate -u my_ntp_server

–

Remove previous version

sudo aptitude remove ntpdate

1	sudo aptitude remove ntpdate

tail -f /var/log/syslog

#-------------------------------------------------
ntpq -c lpeer  
# Example output. 
-europium.canoni 193.79.237.14 2 u 63 64 37 97.375 71.020 1.875
-dtype.org 69.25.96.13 2 u 2 64 77 86.956 69.178 1.804

tail -f /var/log/syslog

#-------------------------------------------------

ntpq -c lpeer

# Example output.

-europium.canoni 193.79.237.14 2 u 63 64 37 97.375 71.020 1.875

-dtype.org 69.25.96.13 2 u 2 64 77 86.956 69.178 1.804

—
If you have ntpd installed on another machine, you can use your first server in the servers list of your ntp.conf file, or can synchronize one time with the -q option, as follows:

ntpd -q [IP address of your server]

1	ntpd -q [IP address of your server]

NFS Server and Client Installation on CentOS 7

Posted on June 22, 2015July 3, 2018 by coder

The NFS server exposes the internal directory as the /export folder, which is bind-mounted on the host. The Docker host then mounts this folder using the NFS protocol to its /mnt folder. Then a so-called infrastructure container is created which binds the mount folder.

For Ubuntu :

apt-get install nfs-kernel-server mkdir /export chmod 777 /export

1	apt-get install nfs-kernel-server mkdir /export chmod 777 /export

mount --bind /opt/test/db /export

1	mount --bind /opt/test/db /export

$ mount -t nfs 127.0.0.1:/export /mnt
$ exportfs -a
$ service nfs-kernel-server restart

$ mount -t nfs 127.0.0.1:/export /mnt

$ exportfs -a

$ service nfs-kernel-server restart

NFS server on CentOS 7.

Network File System (NFS) is a popular distributed filesystem protocol that enables users to mount remote directories on their server.

hostname server1.example.com and IP as 192.168.0.100

Serverside

yum install nfs-utils

#directory that will be shared by NFS:

mkdir /var/nfsshare

#Change the permissions of the folder as follows:

chmod -R 755 /var/nfsshare
chown nfsnobody:nfsnobody /var/nfsshare

yum install nfs-utils

#directory that will be shared by NFS:

mkdir /var/nfsshare

#Change the permissions of the folder as follows:

chmod -R 755 /var/nfsshare

chown nfsnobody:nfsnobody /var/nfsshare

Now create the

mkdir  /var/nfsshare as shared folder,

systemctl enable rpcbind
systemctl enable nfs-server
systemctl enable nfs-lock
systemctl enable nfs-idmap
systemctl start rpcbind
systemctl start nfs-server
systemctl start nfs-lock
systemctl start nfs-idmap

mkdir /var/nfsshare as shared folder,

systemctl enable rpcbind

systemctl enable nfs-server

systemctl enable nfs-lock

systemctl enable nfs-idmap

systemctl start rpcbind

systemctl start nfs-server

systemctl start nfs-lock

systemctl start nfs-idmap

Now we will share the NFS directory over the network a follows:

nano /etc/exports

We will make two sharing points /home and /var/nfsshare. Edit the exports file as follows:

/var/nfsshare    192.168.0.101(rw,sync,no_root_squash,no_all_squash)
/home            192.168.0.101(rw,sync,no_root_squash,no_all_squash)

1 2	/var/nfsshare 192.168.0.101(rw,sync,no_root_squash,no_all_squash) /home 192.168.0.101(rw,sync,no_root_squash,no_all_squash)

Note 192.168.0.101 is the IP of client machine, if you wish that any other client should access it you need to add the it IP wise other wise you can add “*” instead of IP for all IP access.

Condition is that it must be pingable at both ends.

start the NFS service:

systemctl restart nfs-server

1	systemctl restart nfs-server

Again we need to add the NFS service override in CentOS 7 firewall-cmd public zone service as:

firewall-cmd --permanent --zone=public --add-service=nfs
firewall-cmd --permanent --zone=public --add-service=mountd
firewall-cmd --permanent --zone=public --add-service=rpc-bind
firewall-cmd --reload

firewall-cmd --permanent --zone=public --add-service=nfs

firewall-cmd --permanent --zone=public --add-service=mountd

firewall-cmd --permanent --zone=public --add-service=rpc-bind

firewall-cmd --reload

Now we are ready with the NFS server part.

Client Side

yum install nfs-utils

# Now create the NFS directory mount points:

mkdir -p /mnt/nfs/home
mkdir -p /mnt/nfs/var/nfsshare

# Mount the NFS shared home directory in the client machine as shown below:

mount -t nfs 192.168.0.100:/home /mnt/nfs/home/

# Multi Mount
mount -t nfs 192.168.0.100:/var/nfsshare /mnt/nfs/var/nfsshare/

df -kh

[root@client1 ~]# df -kh
Filesystem                    Size  Used Avail Use% Mounted on
192.168.0.100:/var/nfsshare   39G  980M   38G   3% /mnt/nfs/var/nfsshare
192.168.0.100:/home           19G   33M   19G   1% /mnt/nfs/home

# ! Connected 

touch /mnt/nfs/var/nfsshare/test_nfs

yum install nfs-utils

# Now create the NFS directory mount points:

mkdir -p /mnt/nfs/home

mkdir -p /mnt/nfs/var/nfsshare

# Mount the NFS shared home directory in the client machine as shown below:

mount -t nfs 192.168.0.100:/home /mnt/nfs/home/

# Multi Mount

mount -t nfs 192.168.0.100:/var/nfsshare /mnt/nfs/var/nfsshare/

df -kh

[root@client1 ~]# df -kh

Filesystem Size Used Avail Use% Mounted on

192.168.0.100:/var/nfsshare 39G 980M 38G 3% /mnt/nfs/var/nfsshare

192.168.0.100:/home 19G 33M 19G 1% /mnt/nfs/home

# ! Connected

touch /mnt/nfs/var/nfsshare/test_nfs

Permanent NFS mounting

nano /etc/fstab

1	nano /etc/fstab

Add the entries like this:

[...]
192.168.0.100:/home    /mnt/nfs/home   nfs defaults 0 0
192.168.0.100:/var/nfsshare    /mnt/nfs/var/nfsshare   nfs defaults 0 0

[...]

192.168.0.100:/home /mnt/nfs/home nfs defaults 0 0

192.168.0.100:/var/nfsshare /mnt/nfs/var/nfsshare nfs defaults 0 0

Note 192.168.0.100 is the server NFS-share IP address, it will vary in your case.

This will make the permanent mount of the NFS-share.

Anwendung	NFS
Darstellung	XDR
Sitzung	(Sun-) RPC
Transport	(UDP)	TCP
Netzwerk	IP (IPv4, IPv6)
Netzzugang	Ethernet	Token
Ring	FDDI	…
Das Network File System – abgekürzt NFS (auch Network File Service) –
 ist ein von Sun Microsystems entwickeltes Protokoll, das den 
Zugriff auf Dateien über ein Netzwerk ermöglicht. 
Dabei werden die Dateien nicht wie z. B. bei FTP übertragen,
 sondern die Benutzer können auf Dateien, die sich auf einem 
entfernten Rechner befinden, so zugreifen, als ob sie auf ihrer
 lokalen Festplatte abgespeichert wären.

Bei diesem UNIX-Netzwerkprotokoll handelt es sich um einen
 Internet-Standard (RFC 1094, RFC 1813, RFC 3530, RFC 7530),
 der auch als verteiltes Dateisystem (englisch distributed file system)
 bezeichnet wird.


# Server-Adresse: 10.0.0.1
# NFSv2, NFSv3:
# Exportiert /path/to/directory an alle IPs von 10.0.0.0 bis 10.0.255.255,
# und zwar zum Lesen/Schreiben (rw), asynchronem Zugriff (Daten werden
# nicht sofort geschrieben) und auch von Ports über 1024 aus (insecure)
#
# Erreichbar als: 10.0.0.1:/path/to/directory
#
### Linux-Systeme
/path/to/directory        10.0.0.0/16(rw,async,insecure)

### FreeBSD
/path/to/directory        -network 10.0.0.0/16
# NFSv4:
# Benötigt zur optimalen Funktion eine Freigabe mit der Option fsid=0.
# Diese wird als root-Freigabe genutzt und ist als die Freigabe / zu 
# erreichen. Die anderen Freigaben liegen unterhalb davon. Ansonsten 
# ist optional eine Authentifizierung/Verschlüsselung mit Kerberos 
# möglich.
#
### Linux-Systeme:
# Erreichbar als 10.0.0.1:/
# Wird diese Freigabe eingehängt, so sind alle darunterliegenden 
# Freigaben logischerweise zugänglich.
/path/to/nfsv4/root       10.0.0.0/16(rw,async,insecure,fsid=0)
# Erreichbar als 10.0.0.1:/export1
/path/to/nfsv4/root/export1 10.0.0.0/16(rw,async,insecure)

### FreeBSD
# Root-Punkt spezifizieren (unter Linux der mit fsid=0 markierte Punkt)
V4: /path/to/nfsv4/root    -network 10.0.0.0/16
# Freigaben angeben
/path/to/nfsv4/root/export1 -network 10.0.0.0/16
Der Client kann eine Freigabe manuell mounten oder ggf. mit einem Eintrag in der Datei fstab automatisieren.


REf: Wikipedia. direct content

Anwendung NFS

Darstellung XDR

Sitzung (Sun-) RPC

Transport (UDP) TCP

Netzwerk IP (IPv4, IPv6)

Netzzugang Ethernet Token

Ring FDDI …

Das Network File System – abgekürzt NFS (auch Network File Service) –

ist ein von Sun Microsystems entwickeltes Protokoll, das den

Zugriff auf Dateien über ein Netzwerk ermöglicht.

Dabei werden die Dateien nicht wie z. B. bei FTP übertragen,

sondern die Benutzer können auf Dateien, die sich auf einem

entfernten Rechner befinden, so zugreifen, als ob sie auf ihrer

lokalen Festplatte abgespeichert wären.

Bei diesem UNIX-Netzwerkprotokoll handelt es sich um einen

Internet-Standard (RFC 1094, RFC 1813, RFC 3530, RFC 7530),

der auch als verteiltes Dateisystem (englisch distributed file system)

bezeichnet wird.

# Server-Adresse: 10.0.0.1

# NFSv2, NFSv3:

# Exportiert /path/to/directory an alle IPs von 10.0.0.0 bis 10.0.255.255,

# und zwar zum Lesen/Schreiben (rw), asynchronem Zugriff (Daten werden

# nicht sofort geschrieben) und auch von Ports über 1024 aus (insecure)

# Erreichbar als: 10.0.0.1:/path/to/directory

### Linux-Systeme

/path/to/directory 10.0.0.0/16(rw,async,insecure)

### FreeBSD

/path/to/directory -network 10.0.0.0/16

# NFSv4:

# Benötigt zur optimalen Funktion eine Freigabe mit der Option fsid=0.

# Diese wird als root-Freigabe genutzt und ist als die Freigabe / zu

# erreichen. Die anderen Freigaben liegen unterhalb davon. Ansonsten

# ist optional eine Authentifizierung/Verschlüsselung mit Kerberos

# möglich.

### Linux-Systeme:

# Erreichbar als 10.0.0.1:/

# Wird diese Freigabe eingehängt, so sind alle darunterliegenden

# Freigaben logischerweise zugänglich.

/path/to/nfsv4/root 10.0.0.0/16(rw,async,insecure,fsid=0)

# Erreichbar als 10.0.0.1:/export1

/path/to/nfsv4/root/export1 10.0.0.0/16(rw,async,insecure)

### FreeBSD

# Root-Punkt spezifizieren (unter Linux der mit fsid=0 markierte Punkt)

V4: /path/to/nfsv4/root -network 10.0.0.0/16

# Freigaben angeben

/path/to/nfsv4/root/export1 -network 10.0.0.0/16

Der Client kann eine Freigabe manuell mounten oder ggf. mit einem Eintrag in der Datei fstab automatisieren.

REf: Wikipedia. direct content

Installing hadoop 2.4.0 on Ubuntu server

Posted on June 20, 2015July 14, 2018 by coder

First of all, THESE ARE IMPORTANT FOR INDRODUCTION

*Ubuntu server 14.04
*VM in VirtualBox with 1GB RAM
*SWAP SPACE : SELECT more than 2 GB

Up-to-date system.

kutayzorlu@coder_telekom:~$ sudo apt-get update 
kutayzorlu@coder_telekom:~$ sudo apt-get upgrade

1 2	kutayzorlu@coder_telekom:~$ sudo apt-get update kutayzorlu@coder_telekom:~$ sudo apt-get upgrade

Moreover, it is advisable not to run Hadoop services through a general-purpose user, so the next step consists in adding a group hadoop and a user hadoop-user belonging to that group

kutayzorlu@coder_telekom:~$ sudo addgroup hadoop
kutayzorlu@coder_telekom:~$ sudo adduser --ingroup hadoop hadoop-user

1 2	kutayzorlu@coder_telekom:~$ sudo addgroup hadoop kutayzorlu@coder_telekom:~$ sudo adduser --ingroup hadoop hadoop-user

Installing Java
The mentioned tutorials suggest a potentially unsafe procedure in order to install the jdk through `apt-get`,

kutayzorlu@coder_telekom:~$ wget   "http://server12.kutayzorlu.com/7u45-b18/jdk-7u45-linux-x64.tar.gz"
...
kutayzorlu@coder_telekom:~$ tar -xvzf jdk-7-linux-x64.tar.gz
kutayzorlu@coder_telekom:~$ sudo mkdir /usr/local/java
kutayzorlu@coder_telekom:~$ sudo cp -r jdk1.7.0_45 /usr/local/java
kutayzorlu@coder_telekom:~$ sudo update-alternatives --install "/usr/bin/javac" "javac" "/usr/local/java/jdk1.7.0_45/bin/javac" 1
kutayzorlu@coder_telekom:~$ sudo update-alternatives --set javac /usr/local/java/jdk1.7.0_45/bin/javac

kutayzorlu@coder_telekom:~$ wget "http://server12.kutayzorlu.com/7u45-b18/jdk-7u45-linux-x64.tar.gz"

...

kutayzorlu@coder_telekom:~$ tar -xvzf jdk-7-linux-x64.tar.gz

kutayzorlu@coder_telekom:~$ sudo mkdir /usr/local/java

kutayzorlu@coder_telekom:~$ sudo cp -r jdk1.7.0_45 /usr/local/java

kutayzorlu@coder_telekom:~$ sudo update-alternatives --install "/usr/bin/javac" "javac" "/usr/local/java/jdk1.7.0_45/bin/javac" 1

kutayzorlu@coder_telekom:~$ sudo update-alternatives --set javac /usr/local/java/jdk1.7.0_45/bin/javac

Finally, a couple of environment variables should be set up so that the java executables are in $PATH and hadoop knows where java has been installed: this is easily accomplished adding

JAVA_HOME=/usr/local/java/jdk1.7.0_45
PATH=$PATH:$HOME/bin:$JAVA_HOME/bin
export JAVA_HOME
export PATH

JAVA_HOME=/usr/local/java/jdk1.7.0_45

PATH=$PATH:$HOME/bin:$JAVA_HOME/bin

export JAVA_HOME

export PATH

at the end of /etc/profile <<< We need to edit that to set automatically JAVA PATH

kutayzorlu@coder_telekom:~$ . /etc/profile
kutayzorlu@coder_telekom:~$ javac -version
javac 1.7.0_45

kutayzorlu@coder_telekom:~$ . /etc/profile

kutayzorlu@coder_telekom:~$ javac -version

javac 1.7.0_45

Setup SSH

All communications with Hadoop are encrypted via SSH, thus the corresponding server should be installed:

kutayzorlu@coder_telekom:~$ sudo apt-get install openssh-server

1	kutayzorlu@coder_telekom:~$ sudo apt-get install openssh-server

and the hadoop-user must be associated to a key pair and subsequently granting its access to the local machine:

kutayzorlu@coder_telekom:~$ su - hadoop-user
hadoop-user@coder_telekom:~$ ssh-keygen -t rsa -P ""    
#<<<<<<<<<<<< Generating public/private rsa key pair. 
The key's randomart image is: 
hadoop-user@coder_telekom:~$ cat $HOME/.ssh/id_rsa.pub >> $HOME/.ssh/authorized_keys

kutayzorlu@coder_telekom:~$ su - hadoop-user

hadoop-user@coder_telekom:~$ ssh-keygen -t rsa -P ""

#<<<<<<<<<<<< Generating public/private rsa key pair.

The key's randomart image is:

hadoop-user@coder_telekom:~$ cat $HOME/.ssh/id_rsa.pub >> $HOME/.ssh/authorized_keys

Now hadoop-user should be able to access via ssh to localhost without ( WE SET THE PASS = “null”)providing a password:

hadoop-user@coder_telekom:~$ ssh localhost
The authenticity of host 'localhost (::1)' can't be established.
Last login: 
$

hadoop-user@coder_telekom:~$ ssh localhost

The authenticity of host 'localhost (::1)' can't be established.

Last login:

Disable IPV6

Hadoop and IPV6 do not agree on the meaning of 0.0.0.0 address,

LOOK AT : ??>>>>> /etc/sysctl.conf

net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1

net.ipv6.conf.all.disable_ipv6 = 1

net.ipv6.conf.default.disable_ipv6 = 1

net.ipv6.conf.lo.disable_ipv6 = 1

REBOOT SYSTEM

CONTROL

cat /proc/sys/net/ipv6/conf/all/disable_ipv6  # should be 1,
#1 meaning that IPV6 is actually disabled.

1 2	cat /proc/sys/net/ipv6/conf/all/disable_ipv6 # should be 1, #1 meaning that IPV6 is actually disabled.

Hadoop

Download and install Hadoop

Download hadoop-2.4.0.tar.gz, unpack it and move the results in /usr/local, adding a symlink using the more friendly name hadoop and changing ownership of the directory contents to the hadoop-user user:

kutayzorlu@coder_telekom:~$ wget wget http://apache.mirrors.pair.com/hadoop/common/hadoop-2.4.0/hadoop-2.4.0.tar.gz
...
kutayzorlu@coder_telekom:~$ tar -xzvf hadoop-2.4.0.tar.gz
kutayzorlu@coder_telekom:~$ sudo mv hadoop-2.4.0 /usr/local
kutayzorlu@coder_telekom:~$ cd /usr/local
kutayzorlu@coder_telekom:/usr/local$ sudo ln -s hadoop-2.4.0 hadoop
kutayzorlu@coder_telekom:/usr/local$ sudo chown -R hadoop-user:hadoop hadoop-2.4.0

kutayzorlu@coder_telekom:~$ wget wget http://apache.mirrors.pair.com/hadoop/common/hadoop-2.4.0/hadoop-2.4.0.tar.gz

...

kutayzorlu@coder_telekom:~$ tar -xzvf hadoop-2.4.0.tar.gz

kutayzorlu@coder_telekom:~$ sudo mv hadoop-2.4.0 /usr/local

kutayzorlu@coder_telekom:~$ cd /usr/local

kutayzorlu@coder_telekom:/usr/local$ sudo ln -s hadoop-2.4.0 hadoop

kutayzorlu@coder_telekom:/usr/local$ sudo chown -R hadoop-user:hadoop hadoop-2.4.0

Setup the dedicated user environment
Switch to the `hadoop-user` user and add the following lines at the end of `~/.bashrc`:

# Set Hadoop-related environment variables
export HADOOP_PREFIX=/usr/local/hadoop
export HADOOP_HOME=/usr/local/hadoop
export HADOOP_MAPRED_HOME=${HADOOP_HOME}
export HADOOP_COMMON_HOME=${HADOOP_HOME}
export HADOOP_HDFS_HOME=${HADOOP_HOME}
export YARN_HOME=${HADOOP_HOME}
export HADOOP_CONF_DIR=${HADOOP_HOME}/etc/hadoop
# Native Path
export HADOOP_COMMON_LIB_NATIVE_DIR=${HADOOP_PREFIX}/lib/native
export HADOOP_OPTS="-Djava.library.path=$HADOOP_PREFIX/lib"
#Java path
export JAVA_HOME='/usr/local/java/jdk1.7.0_45'
# Add Hadoop bin/ directory to PATH
export PATH=$PATH:$HADOOP_HOME/bin:$JAVA_PATH/bin:$HADOOP_HOME/sbin

# Set Hadoop-related environment variables

export HADOOP_PREFIX=/usr/local/hadoop

export HADOOP_HOME=/usr/local/hadoop

export HADOOP_MAPRED_HOME=${HADOOP_HOME}

export HADOOP_COMMON_HOME=${HADOOP_HOME}

export HADOOP_HDFS_HOME=${HADOOP_HOME}

export YARN_HOME=${HADOOP_HOME}

export HADOOP_CONF_DIR=${HADOOP_HOME}/etc/hadoop

# Native Path

export HADOOP_COMMON_LIB_NATIVE_DIR=${HADOOP_PREFIX}/lib/native

export HADOOP_OPTS="-Djava.library.path=$HADOOP_PREFIX/lib"

#Java path

export JAVA_HOME='/usr/local/java/jdk1.7.0_45'

# Add Hadoop bin/ directory to PATH

export PATH=$PATH:$HADOOP_HOME/bin:$JAVA_PATH/bin:$HADOOP_HOME/sbin

To have the new environment variables in place,

1- reload .bashrc through “source .bashrc"

2- then open /usr/local/hadoop/etc/hadoop/hadoop-env.sh ,

3- uncomment the line setting JAVA_HOME and set its value to the jdk directory:

export JAVA_HOME=/usr/local/java/jdk1.7.0_45

1	export JAVA_HOME=/usr/local/java/jdk1.7.0_45

Configure Hadoop

Before being able to actually use the hadoop file system it is necessary to modify some configuration files inside /usr/local/hadoop/etc/hadoop
All such files follow the an XML format, and the updates should concern the top-level node configuration
(likely empty after the hadoop installation). Specifically:

in yarn-site.xml :

    yarn.nodemanager.aux-services
    mapreduce_shuffle
  
  
    yarn.nodemanager.aux-services.mapreduce.shuffle.class
    org.apache.hadoop.mapred.ShuffleHandler

yarn.nodemanager.aux-services

mapreduce_shuffle

yarn.nodemanager.aux-services.mapreduce.shuffle.class

org.apache.hadoop.mapred.ShuffleHandler

look in core-site.xml:

fs.default.name hdfs://localhost:9000

1
2

fs.default.name
hdfs://localhost:9000

look in mapred-site.xml

mapreduce.framework.name yarn

1
2

mapreduce.framework.name
yarn

nano hdfs-site.xml:

    dfs.replication
    1
  
  
    dfs.namenode.name.dir
    file:/usr/local/hadoop/yarn_data/hdfs/namenode
  
  
    dfs.datanode.data.dir
    file:/usr/local/hadoop/yarn_data/hdfs/datanode

dfs.replication

dfs.namenode.name.dir

file:/usr/local/hadoop/yarn_data/hdfs/namenode

dfs.datanode.data.dir

file:/usr/local/hadoop/yarn_data/hdfs/datanode

Run these commands.

hadoop-user@coder_telekom:~$ mkdir -p /usr/local/hadoop/yarn_data/hdfs/namenode
hadoop-user@coder_telekom:~$ mkdir -p /usr/local/hadoop/yarn_data/hdfs/datanode

1 2	hadoop-user@coder_telekom:~$ mkdir -p /usr/local/hadoop/yarn_data/hdfs/namenode hadoop-user@coder_telekom:~$ mkdir -p /usr/local/hadoop/yarn_data/hdfs/datanode

Formatting the distributed file system

USER should be hadoop user

hadoop-user@coder_telekom:~$ hdfs namenode -format
...

1 2	hadoop-user@coder_telekom:~$ hdfs namenode -format ...

Find the these 2 file and run

start-dfs.sh

start-yarn.sh

hadoop-user@coder_telekom:~$ start-dfs.sh
8/08/10 13:18:22 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Starting namenodes on [localhost]
localhost: starting namenode, logging to /usr/local/hadoop-2.4.0/logs/hadoop-hadoop-user-namenode-coder_telekom.out
localhost: starting datanode, logging to /usr/local/hadoop-2.4.0/logs/hadoop-hadoop-user-datanode-coder_telekom.out
Starting secondary namenodes [0.0.0.0]
0.0.0.0: starting secondarynamenode, logging to /usr/local/hadoop-2.4.0/logs/hadoop-hadoop-user-secondarynamenode-coder_telekom.out
8/08/10 13:18:22 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
hadoop-user@coder_telekom:~$ start-yarn.sh
starting yarn daemons
starting resourcemanager, logging to /usr/local/hadoop/logs/yarn-hadoop-user-resourcemanager-coder_telekom.out
localhost: starting nodemanager, logging to /usr/local/hadoop-2.4.0/logs/yarn-hadoop-user-nodemanager-coder_telekom.out

....

hadoop-user@coder_telekom:~$ hdfs dfs -mkdir /user
hadoop-user@coder_telekom:~$ hdfs dfs -mkdir /user/hadoop-user

... 
hadoop-user@coder_telekom:~$ hadoop jar /usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.4.0.jar pi 10 1000 ... Job Finished in 22.136 seconds Estimated value of Pi is 3.142857....

hadoop-user@coder_telekom:~$ start-dfs.sh

8/08/10 13:18:22 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

Starting namenodes on [localhost]

localhost: starting namenode, logging to /usr/local/hadoop-2.4.0/logs/hadoop-hadoop-user-namenode-coder_telekom.out

localhost: starting datanode, logging to /usr/local/hadoop-2.4.0/logs/hadoop-hadoop-user-datanode-coder_telekom.out

Starting secondary namenodes [0.0.0.0]

0.0.0.0: starting secondarynamenode, logging to /usr/local/hadoop-2.4.0/logs/hadoop-hadoop-user-secondarynamenode-coder_telekom.out

8/08/10 13:18:22 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

hadoop-user@coder_telekom:~$ start-yarn.sh

starting yarn daemons

starting resourcemanager, logging to /usr/local/hadoop/logs/yarn-hadoop-user-resourcemanager-coder_telekom.out

localhost: starting nodemanager, logging to /usr/local/hadoop-2.4.0/logs/yarn-hadoop-user-nodemanager-coder_telekom.out

....

hadoop-user@coder_telekom:~$ hdfs dfs -mkdir /user

hadoop-user@coder_telekom:~$ hdfs dfs -mkdir /user/hadoop-user

...

hadoop-user@coder_telekom:~$ hadoop jar /usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.4.0.jar pi 10 1000 ... Job Finished in 22.136 seconds Estimated value of Pi is 3.142857....

... 
# For stop daemons
$ stop-dfs.sh 
$ stop-yarn.sh.

...

# For stop daemons

$ stop-dfs.sh

$ stop-yarn.sh.

Hadoop security File and directory permissions

Posted on June 20, 2015June 20, 2015 by coder

Permissions for both HDFS and local fileSystem paths

The following table lists various paths on HDFS and local filesystems (on all nodes) and recommended permissions:

Filesystem	Path	User:Group	Permissions
local	`dfs.namenode.name.dir`	hdfs:hadoop	`drwx------`
local	`dfs.datanode.data.dir`	hdfs:hadoop	`drwx------`
local	$HADOOP_LOG_DIR	hdfs:hadoop	`drwxrwxr-x`
local	$YARN_LOG_DIR	yarn:hadoop	`drwxrwxr-x`
local	`yarn.nodemanager.local-dirs`	yarn:hadoop	`drwxr-xr-x`
local	`yarn.nodemanager.log-dirs`	yarn:hadoop	`drwxr-xr-x`
local	container-executor	root:hadoop	`--Sr-s--*`
local	`conf/container-executor.cfg`	root:hadoop	`r-------*`
hdfs	/	hdfs:hadoop	`drwxr-xr-x`
hdfs	/tmp	hdfs:hadoop	`drwxrwxrwxt`
hdfs	/user	hdfs:hadoop	`drwxr-xr-x`
hdfs	`yarn.nodemanager.remote-app-log-dir`	yarn:hadoop	`drwxrwxrwxt`
hdfs	`mapreduce.jobhistory.intermediate-done-dir`	mapred:hadoop	`drwxrwxrwxt`
hdfs	`mapreduce.jobhistory.done-dir`	mapred:hadoop	`drwxr-x---`

For More please Visit that page :
http://hadoop.apache.org/docs/r2.7.0/hadoop-project-dist/hadoop-common/SecureMode.html#Proxy_user

How to add init script as a service

Posted on June 20, 2015June 20, 2015 by coder

1. You need to create a script file inside of

/etc/init.d/SCRIPT_NAME_kutayzorlu

1	/etc/init.d/SCRIPT_NAME_kutayzorlu

We will add this script to chkconfig

chkconfig --add SCRIPT_NAME_kutayzorlu

1	chkconfig --add SCRIPT_NAME_kutayzorlu

After list the CHKconfig LIST

[root@kutayzorlu ~]# chkconfig --list SCRIPT_NAME_kutayzorlu
SCRIPT_NAME_kutayzorlu      0:off   1:off   2:off   3:on    4:on    5:on    6:off

1 2	[root@kutayzorlu ~]# chkconfig --list SCRIPT_NAME_kutayzorlu SCRIPT_NAME_kutayzorlu 0:off 1:off 2:off 3:on 4:on 5:on 6:off

Start the service

[root@kutayzorlu ~]# service SCRIPT_NAME_kutayzorlu start
 [  OK  ]

1 2	[root@kutayzorlu ~]# service SCRIPT_NAME_kutayzorlu start [ OK ]

Example Init Script

#!/bin/bash
#
#       /etc/rc.d/init.d/<servicename>
#
#       <description of the *service*>
#       <any general comments about this init script>
#
# <tags -- see below for tag definitions.  *Every line* from the top
#  of the file to the end of the tags section must begin with a #
#  character.  After the tags section, there should be a blank line.
#  This keeps normal comments in the rest of the file from being
#  mistaken for tags, should they happen to fit the pattern.>

# Source function library.
. /etc/init.d/functions

<define any local shell functions used by the code that follows>

start() {
        echo -n "Starting <servicename>: "
        <start daemons, perhaps with the daemon function>
        touch /var/lock/subsys/<servicename>
        return <return code of starting daemon>
}

stop() {
        echo -n "Shutting down <servicename>: "
        <stop daemons, perhaps with the killproc function>
        rm -f /var/lock/subsys/<servicename>
        return <return code of stopping daemon>
}

case "$1" in
    start)
        start
        ;;
    stop)
        stop
        ;;
    status)
        <report the status of the daemons in free-form format,
        perhaps with the status function>
        ;;
    restart)
        stop
        start
        ;;
    reload)
        <cause the service configuration to be reread, either with
        kill -HUP or by restarting the daemons, in a manner similar
        to restart above>
        ;;
    condrestart)
        <Restarts the servce if it is already running. For example:>
        [ -f /var/lock/subsys/<service> ] && restart || :
    probe)
        <optional.  If it exists, then it should determine whether
        or not the service needs to be restarted or reloaded (or
        whatever) in order to activate any changes in the configuration
        scripts.  It should print out a list of commands to give to
        $0; see the description under the probe tag below.>
        ;;
    *)
        echo "Usage: <servicename> {start|stop|status|reload|restart[|probe]"
        exit 1
        ;;
esac
exit $?

#!/bin/bash

# /etc/rc.d/init.d/<servicename>

# <description of the *service*>

# <any general comments about this init script>

# <tags -- see below for tag definitions. *Every line* from the top

# of the file to the end of the tags section must begin with a #

# character. After the tags section, there should be a blank line.

# This keeps normal comments in the rest of the file from being

# mistaken for tags, should they happen to fit the pattern.>

# Source function library.

. /etc/init.d/functions

start() {

echo -n "Starting <servicename>: "

touch /var/lock/subsys/<servicename>

return <return code of starting daemon>

}

stop() {

echo -n "Shutting down <servicename>: "

rm -f /var/lock/subsys/<servicename>

return <return code of stopping daemon>

}

case "$1" in

start)

start

;;

stop)

stop

;;

status)

<report the status of the daemons in free-form format,

perhaps with the status function>

;;

restart)

stop

start

;;

reload)

<cause the service configuration to be reread, either with

kill -HUP or by restarting the daemons, in a manner similar

to restart above>

;;

condrestart)

[ -f /var/lock/subsys/<service> ] && restart || :

probe)

<optional. If it exists, then it should determine whether

or not the service needs to be restarted or reloaded (or

whatever) in order to activate any changes in the configuration

scripts. It should print out a list of commands to give to

$0; see the description under the probe tag below.>

;;

echo "Usage: <servicename> {start|stop|status|reload|restart[|probe]"

exit 1

;;

esac

exit $?

#===================================================================
Other Method , Soon :__)

How to disable directory listing in Apache or HTTPd

Posted on June 20, 2015June 20, 2015 by coder

Where is the config file
>>> httpd.conf

find /etc -name  "httpd.conf"
# This command will find the httpd config file location. 

 >>> Options Includes Indexes FollowSymLinks MultiViews

  >>>>  then remove word Indexes and save the file. The line should look like this one:

!!!!!! REMOVED   >>>>    indexes   <<<<<  !!!!!
 Options Includes FollowSymLinks MultiViews

then restart  apache or httpd

find /etc -name "httpd.conf"

# This command will find the httpd config file location.

>>> Options Includes Indexes FollowSymLinks MultiViews

>>>> then remove word Indexes and save the file. The line should look like this one:

!!!!!! REMOVED >>>> indexes <<<<< !!!!!

Options Includes FollowSymLinks MultiViews

then restart apache or httpd

sendmail bash script to send mail

Posted on June 20, 2015January 9, 2016 by coder

# send mail config

# 1) If sendmail is not installed, do install it:
$apt-get install sendmail
# 2) Configure hosts file correctly: 
$nano /etc/hosts

#And make sure the line looks like this:
127.0.0.1 localhost localhost.localdomain yourhostnamehere

$/etc/init.d/networking stop
$/etc/init.d/networking start

$sendmailconfig

# 1) If sendmail is not installed, do install it:

$apt-get install sendmail

# 2) Configure hosts file correctly:

$nano /etc/hosts

#And make sure the line looks like this:

127.0.0.1 localhost localhost.localdomain yourhostnamehere

$/etc/init.d/networking stop

$/etc/init.d/networking start

$sendmailconfig

$ echo "My test email being sent from sendmail" | /usr/sbin/sendmail me@kutayzorlu.com

1	$ echo "My test email being sent from sendmail" \| /usr/sbin/sendmail me@kutayzorlu.com

#!/bin/bash
# kutayzorlu.com
# some variables
# refactoring the script such that all these values are
# passed from the outside as arguments should be easy

from="sender@kutayzorlu.com"
to="recipient@kutayzorlu.org"
subject="Some fancy title"
boundary="ZZ_/afg6432dfgkl.94531q"
body="This is the body of our email"
declare -a attachments
attachments=( "foo.pdf" "bar.jpg" "archive.zip" )

# Build headers
{

printf '%s\n' "From: $from
To: $to
Subject: $subject
Mime-Version: 1.0
Content-Type: multipart/mixed; boundary=\"$boundary\"

--${boundary}
Content-Type: text/plain; charset=\"US-ASCII\"
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

$body
"
 
# now loop over the attachments, guess the type
# and produce the corresponding part, encoded base64
for file in "${attachments[@]}"; do

  [ ! -f "$file" ] && echo "Warning: attachment $file not found, skipping" >&2 && continue

  mimetype=$(get_mimetype "$file") 
 
  printf '%s\n' "--${boundary}
Content-Type: $mimetype
Content-Transfer-Encoding: base64
Content-Disposition: attachment; filename=\"$file\"
"
 
  base64 "$file"
  echo
done
 
# print last boundary with closing --
printf '%s\n' "--${boundary}--"
 
} | sendmail -t -oi   # one may also use -f here to set the envelope-from

#!/bin/bash

# kutayzorlu.com

# some variables

# refactoring the script such that all these values are

# passed from the outside as arguments should be easy

from="sender@kutayzorlu.com"

to="recipient@kutayzorlu.org"

subject="Some fancy title"

boundary="ZZ_/afg6432dfgkl.94531q"

body="This is the body of our email"

declare -a attachments

attachments=( "foo.pdf" "bar.jpg" "archive.zip" )

# Build headers

{

printf '%s\n' "From: $from

To: $to

Subject: $subject

Mime-Version: 1.0

Content-Type: multipart/mixed; boundary=\"$boundary\"

--${boundary}

Content-Type: text/plain; charset=\"US-ASCII\"

Content-Transfer-Encoding: 7bit

Content-Disposition: inline

$body

# now loop over the attachments, guess the type

# and produce the corresponding part, encoded base64

for file in "${attachments[@]}"; do

[ ! -f "$file" ] && echo "Warning: attachment $file not found, skipping" >&2 && continue

mimetype=$(get_mimetype "$file")

printf '%s\n' "--${boundary}

Content-Type: $mimetype

Content-Transfer-Encoding: base64

Content-Disposition: attachment; filename=\"$file\"

base64 "$file"

echo

done

# print last boundary with closing --

printf '%s\n' "--${boundary}--"

} | sendmail -t -oi # one may also use -f here to set the envelope-from

Page 15 of 27

« 1 … 13 14 15 16 17 … 27 »

Serverside

Client Side

Permanent NFS mounting

Installing hadoop 2.4.0

Installing Java The mentioned tutorials suggest a potentially unsafe procedure in order to install the jdk through apt-get,

Setup SSH

Disable IPV6

Hadoop

Download and install Hadoop

Setup the dedicated user environment Switch to the hadoop-user user and add the following lines at the end of ~/.bashrc:

Configure Hadoop

Formatting the distributed file system

Permissions for both HDFS and local fileSystem paths

1. You need to create a script file inside of

We will add this script to chkconfig

Installing Java
The mentioned tutorials suggest a potentially unsafe procedure in order to install the jdk through `apt-get`,

Setup the dedicated user environment
Switch to the `hadoop-user` user and add the following lines at the end of `~/.bashrc`: