davidlaxer (4) [Avatar] Offline
#1
Hi,

The command 'hdfs namenode -format' doesn't run on my Mac running Maverics 10.9.3.

There's a missing class ...

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/security/RefreshUserToGroupMappingsProtocol

I tried building Hadoop 2.4.0 from sources ...

[WARNING] Failed to create parent directories for tracking file /Users/davidlaxer/.m2/repository/org/apache/maven/plugins/maven-war-plugin/2.1/maven-war-plugin-2.1.pom.lastUpdated
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Apache Hadoop Main ................................ SUCCESS [03:17 min]
[INFO] Apache Hadoop Project POM ......................... SUCCESS [ 34.385 s]
[INFO] Apache Hadoop Annotations ......................... SUCCESS [ 18.529 s]
[INFO] Apache Hadoop Assemblies .......................... SUCCESS [ 0.727 s]
[INFO] Apache Hadoop Project Dist POM .................... SUCCESS [01:30 min]
[INFO] Apache Hadoop Maven Plugins ....................... SUCCESS [ 24.267 s]
[INFO] Apache Hadoop MiniKDC ............................. SUCCESS [01:34 min]
[INFO] Apache Hadoop Auth ................................ SUCCESS [01:18 min]
[INFO] Apache Hadoop Auth Examples ....................... FAILURE [ 0.818 s]
[INFO] Apache Hadoop Common .............................. SKIPPED
[INFO] Apache Hadoop NFS ................................. SKIPPED
[INFO] Apache Hadoop Common Project ...................... SKIPPED
[INFO] Apache Hadoop HDFS ................................ SKIPPED
[INFO] Apache Hadoop HttpFS .............................. SKIPPED
[INFO] Apache Hadoop HDFS BookKeeper Journal ............. SKIPPED
[INFO] Apache Hadoop HDFS-NFS ............................ SKIPPED
[INFO] Apache Hadoop HDFS Project ........................ SKIPPED
[INFO] hadoop-yarn ....................................... SKIPPED
[INFO] hadoop-yarn-api ................................... SKIPPED
[INFO] hadoop-yarn-common ................................ SKIPPED
[INFO] hadoop-yarn-server ................................ SKIPPED
[INFO] hadoop-yarn-server-common ......................... SKIPPED
[INFO] hadoop-yarn-server-nodemanager .................... SKIPPED
[INFO] hadoop-yarn-server-web-proxy ...................... SKIPPED
[INFO] hadoop-yarn-server-applicationhistoryservice ...... SKIPPED
[INFO] hadoop-yarn-server-resourcemanager ................ SKIPPED
[INFO] hadoop-yarn-server-tests .......................... SKIPPED
[INFO] hadoop-yarn-client ................................ SKIPPED
[INFO] hadoop-yarn-applications .......................... SKIPPED
[INFO] hadoop-yarn-applications-distributedshell ......... SKIPPED
[INFO] hadoop-yarn-applications-unmanaged-am-launcher .... SKIPPED
[INFO] hadoop-yarn-site .................................. SKIPPED
[INFO] hadoop-yarn-project ............................... SKIPPED
[INFO] hadoop-mapreduce-client ........................... SKIPPED
[INFO] hadoop-mapreduce-client-core ...................... SKIPPED
[INFO] hadoop-mapreduce-client-common .................... SKIPPED
[INFO] hadoop-mapreduce-client-shuffle ................... SKIPPED
[INFO] hadoop-mapreduce-client-app ....................... SKIPPED
[INFO] hadoop-mapreduce-client-hs ........................ SKIPPED
[INFO] hadoop-mapreduce-client-jobclient ................. SKIPPED
[INFO] hadoop-mapreduce-client-hs-plugins ................ SKIPPED
[INFO] Apache Hadoop MapReduce Examples .................. SKIPPED
[INFO] hadoop-mapreduce .................................. SKIPPED
[INFO] Apache Hadoop MapReduce Streaming ................. SKIPPED
[INFO] Apache Hadoop Distributed Copy .................... SKIPPED
[INFO] Apache Hadoop Archives ............................ SKIPPED
[INFO] Apache Hadoop Rumen ............................... SKIPPED
[INFO] Apache Hadoop Gridmix ............................. SKIPPED
[INFO] Apache Hadoop Data Join ........................... SKIPPED
[INFO] Apache Hadoop Extras .............................. SKIPPED
[INFO] Apache Hadoop Pipes ............................... SKIPPED
[INFO] Apache Hadoop OpenStack support ................... SKIPPED
[INFO] Apache Hadoop Client .............................. SKIPPED
[INFO] Apache Hadoop Mini-Cluster ........................ SKIPPED
[INFO] Apache Hadoop Scheduler Load Simulator ............ SKIPPED
[INFO] Apache Hadoop Tools Dist .......................... SKIPPED
[INFO] Apache Hadoop Tools ............................... SKIPPED
[INFO] Apache Hadoop Distribution ........................ SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 09:27 min
[INFO] Finished at: 2014-05-22T17:04:14-08:00
[INFO] Final Memory: 67M/164M
[INFO] ------------------------------------------------------------------------
[ERROR] Plugin org.apache.maven.plugins:maven-war-plugin:2.1 or one of its dependencies could not be resolved: Failed to read artifact descriptor for org.apache.maven.plugins:maven-war-plugin:jar:2.1: Could not transfer artifact org.apache.maven.plugins:maven-war-plugin:pom:2.1 from/to central (http://repo.maven.apache.org/maven2): Specified destination directory cannot be created: /Users/davidlaxer/.m2/r ...

Any ideas?

Thanks in advance.

-Dave
davidlaxer (4) [Avatar] Offline
#2
Re: HDFS Error
MacBook-Pro:common $ pwd
/usr/local/hadoop/share/hadoop/common

MacBook-Pro:common $ jar tf hadoop-common-2.4.0.jar | grep security.Refresh
org/apache/hadoop/security/RefreshUserMappingsProtocol.class
Alex Holmes (47) [Avatar] Offline
#3
Re: HDFS Error
There are some OSX related installation instructions here, but I've never tried them:

http://wiki.apache.org/hadoop/Running_Hadoop_On_OS_X_10.5_64-bit_(Single-Node_Cluster)

I know this doesn't solve the problem of your local Hadoop setup, but to start out it may be easiest if you use one of the VM's that has Hadoop pre-installed from Cloudera, Hortonworks or one of the other distributions.