Running Hadoop The most common problem encountered in the process is DataNode Can't start normally , All kinds of problems are possible , Let's talk about the two situations I encountered :

(1) The first is Master Your firewall is not closed . This is starting Hadoop When ,Master This node can start normally , however Master After the firewall is turned on , bring Slave Cannot access Master Of 9000 port . This situation , stay Slave Of DataNode It started and then shut down immediately .

(2) The reason for the second situation : Every time NameNode -format Will recreate a NameNode ID, and tmp/dfs/data/current The files under the VERSION It includes the last format Under the ID,NameNode -format It's empty NameNode The data under the , But it's not empty DataNode The data under the , Causes failure at startup . All you have to do is every time fotmat front , Empty tmp Look at all the directories in the table .

After the above two questions , Say something that can't be overemphasized , That is to learn to look at the running log information . Both of the above questions will be shown in the log .

Which node is not started , Just check which node logs Log information in the folder . Take the second question as an example ,DataNode You can see the log information of that node without starting :

-- ::, ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: java.io.IOException: Incompatible namespaceIDs in /home/admin/joe.wangh/hadoop/data/dfs.data.dir: namenode namespaceID = ; datanode namespaceID =
at org.apache.hadoop.hdfs.server.datanode.DataStorage.doTransition(DataStorage.java:)
at org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:)
at org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:)
at org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(DataNode.java:)
at org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:)
at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:)
at org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:)
at org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:)

Error message namespaceIDs atypism .

After finding the cause of the mistake , Only in this way , Find a solution to the problem :

Here are two solutions , I use the second one .

Workaround 1: Start from scratch

I can testify that the following steps solve this error, but the side effects won't make you happy (me neither). The crude workaround I have found is to:

1.     stop the cluster

2.     delete the data directory on the problematic datanode: the directory is specified by dfs.data.dir in conf/hdfs-site.xml; if you followed this tutorial, the relevant directory is /usr/local/hadoop-datastore/hadoop-hadoop/dfs/data

3.     reformat the namenode (NOTE: all HDFS data is lost during this process!)

4.     restart the cluster

When deleting all the HDFS data and starting from scratch does not sound like a good idea (it might be ok during the initial setup/testing), you might give the second approach a try.

Workaround 2: Updating namespaceID of problematic datanodes

Big thanks to Jared Stehler for the following suggestion. I have not tested it myself yet, but feel free to try it out and send me your feedback. This workaround is "minimally invasive" as you only have to edit one file on the problematic datanodes:

1.     stop the datanode

2.     edit the value of namespaceID in <dfs.data.dir>/current/VERSION to match the value of the current namenode

3.     restart the datanode

If you followed the instructions in my tutorials, the full path of the relevant file is /usr/local/hadoop-datastore/hadoop-hadoop/dfs/data/current/VERSION (background: dfs.data.dir is by default set to ${hadoop.tmp.dir}/dfs/data, and we set hadoop.tmp.dir to /usr/local/hadoop-datastore/hadoop-hadoop).

If you wonder how the contents of VERSION look like, here's one of mine:

#contents of <dfs.data.dir>/current/VERSION

namespaceID=393514426

storageID=DS-1706792599-10.10.10.1-50010-1204306713481

cTime=1215607609074

storageType=DATA_NODE

layoutVersion=-13

Hadoop Run the error encountered in debugging more related articles

  1. hadoop Mistakes and solutions in the process

    In this paper, the author sorts out the problems in hadoop All kinds of problems in the process of learning . windows Build the development environment In most cases , We are all in windows Lower development ,hadoop They are generally deployed in linux The server ( Whether it's CDH Still original hadoo ...

  2. php Error debugging is temporarily turned on in the code

    Yes php.ini Setting of parameters in Also available in php Complete in code . call : call ini_set() function // Turn on php.ini Medium display_errors Instructions ini_set('display_errors ...

  3. .net Record of error resolution during operation

    1. Application failed to start , Because the parallel configuration of the application is not correct . For more information , See the application event log , Or use the command line sxstrace.exe Tools . https://jingyan.baidu.com/article ...

  4. ejabberd stay eclipse(erlide) Configuration in 、 debugging 、 function

    I've been tossing about lately ejabberd, take ejabberd Project configuration to eclipse To compile . Commissioning, etc , Now write down the process , I hope I can help those in need . Get ready This time the environment is linux In the middle of , Blogger's linux yes fedora2 ...

  5. hadoop Errors during installation

    This time, let's record that I'm installing Hadoop Errors during installation , The installation process refers to the MOOC linziyu tutorial , The following errors occurred during the attempt : 1. In the installation Ubuntu when , When you create a new virtual computer , Not in the version input box Ubunt ...

  6. gdb Debug a process that is already running

    One . Debug process on server , There is no source code on the server , So you need to upload the source code to the server , To debug and see the source code , Here are the steps : 1. View service progress id:pgrep service name [user@user-MP app]$ pgrep ...

  7. Self study from scratch Hadoop(12):Hadoop In command

    Read the directory order HDFS Commands User Commands Administration Commands Debug Commands quote Series index The copyright of this article belongs to mephisto Share with blog Park ...

  8. 【 turn 】 Section error debugging artifact - Core Dump Detailed explanation

    from:http://www.embeddedlinux.org.cn/html/jishuzixun/201307/08-2594.html Section error debugging artifact - Core Dump Detailed explanation source : interconnection ...

  9. hadoop Run process analysis at source code level

    Preface : I've been analyzing hadoop Operation flow , We looked up a lot of information , Although I have a perceptual understanding of this process, I always feel right about mapreduce There is still no comprehensive understanding of the operation of , So I decided to do it at the source level mapred ...

Random recommendation

  1. Android Translucent effect implementation scheme in application development

    Here's the custom Activity Examples of translucent effects :res/values/styles.xml<resources>  <stylename="Transparent " ...

  2. POJ C++ Programming Programming questions #2 World of Warcraft II : equipment

    Programming questions #2: World of Warcraft II : equipment source : POJ (Coursera Statement : stay POJ Exercises completed on will not be counted in Coursera My final result .) Be careful : Total time limit : 1000ms Memory limit : 65536kB ...

  3. iOS Development —— Conclusion &amp; Common development summary

    One . notice 1. Monitoring notification - (void)addObserver:(id)observer selector:(SEL)aSelector name:(NSString *)aName object:(i ...

  4. 【Daily】 2014-4-28

    KEEP GOING   Express product ideas , Explore product issues , Listen to the question of possibility . Do it in one go , There are results to show , Take the initiative to show Let's establish the graph first , After the establishment of the code . Hold dream, and never let it go ...

  5. vs2016 establish vsto excel A problem with file projects

    New jobs need to be based on Excel Develop a tool ,vs Of vsto It's very functional , Encapsulates the development based office Some of the tools that I opened . But in practice , Creating projects always reports errors , Prompt to open excel File failed . The project is to create a e ...

  6. Study Spark—— Environment building (Mac edition )

    Big data complex Remember the last time I changed jobs , I've talked to a lot of headhunters , One of the headhunters told me , Whole IT Job hopping is quite frequent , But relatively speaking , Compare big data " lazy " some , I'm not willing to move . This view was later confirmed in an article ...

  7. PGM: Template based representation

    http://blog.csdn.net/pipisorry/article/details/52537660 introduction Probability graph model ( No matter Bayesian network or Markov Network ) In a fixed set of random variables X There's a union specified on the table ...

  8. Distributed monitoring system development 【day38】: Host survivability detection program parsing ( 7、 ... and )

    One . Directory structure Two . entrance 1. file MonitorServer.py import os import sys if __name__ == "__main__": os.enviro ...

  9. 10 Minutes to understand Android Of Handler Mechanism

    Handler The mechanism is Android Quite classic asynchronous message mechanism in , stay Android The history of development plays a very important role , Whether it's the application layer we face directly or FrameWork layer , There are still quite a lot of scenes . Analysis of source code ...

  10. Use jdb debugging apk

    jdb It's a support java Code level debugging tools , It is from java jdk Provided , Exist in xxx\Java\jdk1.6.0_21\bin under Use ddms During debugging , The host will open another network port , stay DDMS Check it out , One ...