1 java.io.IOException: java.io.IOException: java.lang.IllegalArgumentException: offset (0) + length (8) exceed the capacity of the array: 4
做簡(jiǎn)單的incr操作時(shí)出現(xiàn)，原因是之前put時(shí)放入的是int 長(zhǎng)度為 vlen=4 ，不適用增加操作，只能改為long型 vlen=8
2 寫數(shù)據(jù)到column時(shí) org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 1 action: NotServingRegionException: 1 time, servers with issues: 10.xx.xx.37:60020, 或是 org.apache.hadoop.hbase.NotServingRegionException: Region is not online: 這兩種出錯(cuò)，master-status中出現(xiàn)Regions in Transition 長(zhǎng)達(dá)十幾分鐘，一直處于PENDING_OPEN狀態(tài)，導(dǎo)致請(qǐng)求阻塞。目前把10.xx.xx.37這臺(tái)機(jī)器下線，運(yùn)行一夜穩(wěn)定，沒(méi)有出現(xiàn)因split造成的阻塞。懷疑是機(jī)器問(wèn)題。Hmaster的日志顯示這臺(tái)region server 不停的open close，不做任何split 或flush
RIT 的全稱是region in transcation. 每次hbase master 對(duì)region 的一個(gè)open 或一個(gè)close 操作都會(huì)向Master 的RIT中插入一條記錄,因?yàn)閙aster 對(duì)region 的操作要保持原子性,region 的 open 和 close 是通過(guò)Hmaster 和 region server 協(xié)助來(lái)完成的. 所以為了滿足這些操作的協(xié)調(diào),回滾,和一致性.Hmaster 采用了 RIT 機(jī)制并結(jié)合Zookeeper 中Node的狀態(tài)來(lái)保證操作的安全和一致性.
OFFLINE, // region is in an offline statePENDING_OPEN, // sent rpc to server to open but has not begunOPENING, // server has begun to open but not yet doneOPEN, // server opened region and updated metaPENDING_CLOSE, // sent rpc to server to close but has not begunCLOSING, // server has begun to close but not yet doneCLOSED, // server closed region and updated metaSPLITTING, // server started split of a regionSPLIT // server completed split of a region
進(jìn)一步發(fā)現(xiàn)是load balance的問(wèn)題 region server不停重復(fù)的被open close，參考http://abloz.com/hbase/book.html#regions.arch.assignment 重啟了region server正常
后來(lái)的代碼運(yùn)行中又出現(xiàn)region not on line ，是NotServingRegionException拋出的，原因是“Thrown by a region server if it is sent a request for a region it is not serving.”
為什么會(huì)不斷請(qǐng)求一個(gè)離線的region？且這種錯(cuò)誤集中在150個(gè)中的3個(gè)region，追蹤服務(wù)器端log，region 會(huì)被CloseRegionHandler關(guān)掉，過(guò)了20分鐘左右才重新打開(kāi)，關(guān)掉后客戶端請(qǐng)求的region仍然是這個(gè)關(guān)閉的region？
3 設(shè)置開(kāi)關(guān)不寫入hbase并不生效
代碼初上線，增加了開(kāi)關(guān)，萬(wàn)一hbase有問(wèn)題則關(guān)閉掉開(kāi)關(guān)。但是出現(xiàn)問(wèn)題了發(fā)現(xiàn)程序卡死，目前認(rèn)為原因是不斷加長(zhǎng)的retry機(jī)制，60秒超時(shí)，1-32秒的10次retry，萬(wàn)一出問(wèn)題，切換開(kāi)關(guān)也沒(méi)有用。
需要配置rpc超時(shí)參數(shù)和retry time解決它
4 flush、split、compact導(dǎo)致stop-the-world
出現(xiàn)長(zhǎng)時(shí)間的flush split操作導(dǎo)致hbase服務(wù)器端無(wú)法響應(yīng)請(qǐng)求。需要調(diào)整region大小，并測(cè)試獲取flush次數(shù)
5 hbase參數(shù)設(shè)置
hbase.regionserver.handler.count
考慮到sas盤的io能力，設(shè)置為50
hbase.hregion.memstore.block.multiplier
當(dāng)memstore的大小為hbase.hregion.memstore.flush.size的multiplier倍數(shù)時(shí)，阻塞讀寫進(jìn)行flush，默認(rèn)為2
6 region server crush
Regionserver crash的原因是因?yàn)镚C時(shí)間過(guò)久導(dǎo)致Regionserver和zookeeper之間的連接timeout。
Zookeeper內(nèi)部的timeout如下：
minSessionTimeout 單位毫秒，默認(rèn)2倍tickTime。
maxSessionTimeout 單位毫秒，默認(rèn)20倍tickTime。
（tickTime也是一個(gè)配置項(xiàng)。是Server內(nèi)部控制時(shí)間邏輯的最小時(shí)間單位）
如果客戶端發(fā)來(lái)的sessionTimeout超過(guò)min-max這個(gè)范圍，server會(huì)自動(dòng)截取為min或max，然后為這個(gè)Client新建一個(gè)Session對(duì)象。
默認(rèn)的tickTime是2s，也就是客戶端最大的timeout為40s，及時(shí)regionserver的zookeeper.session.timeout設(shè)置為60s也沒(méi)用。
改動(dòng)：
7 代碼問(wèn)題導(dǎo)致死鎖
master慢查詢?nèi)罩局幸粋€(gè)查詢達(dá)到了2小時(shí)，最終導(dǎo)致服務(wù)器響應(yīng)變慢，無(wú)法應(yīng)對(duì)大寫入。追究原因是getColumns操作一下取出十幾萬(wàn)的數(shù)據(jù)，沒(méi)有做分頁(yè)；更改程序分頁(yè)500條左右，目前沒(méi)有出現(xiàn)問(wèn)題
8 operation too slow
2012-07-26 05:30:39,141 WARN org.apache.hadoop.ipc.HBaseServer: (operationTooSlow): {"processingtimems":69315,"ts":9223372036854775807,"client":"10.75.0.109:34780","starttimems":1343251769825,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"delete","totalColumns":1,"table":"trackurl_status_list","families":{"sl":[{"timestamp":1343251769825,"qualifier":"zzzn1VlyG","vlen":0}]},"row":""}
刪除一行數(shù)據(jù)用了69315s
而且神奇的是row為""，row無(wú)法設(shè)置null進(jìn)去，但可以增加空串。做了一輪測(cè)試
空row-key 刪除不存在的column 耗時(shí) 700ms
空row-key 刪除存在的column 耗時(shí) 5ms非空row-key 刪除任意的column 耗時(shí) 3ms
不清楚是否是個(gè)bug，也還不知道怎么就傳了個(gè)空row-key進(jìn)去，目前策略為在代碼端避免對(duì)空row-key做操作。
9 responseTooSlow
2012-07-31 17:52:06,619 WARN org.apache.hadoop.ipc.HBaseServer: (responseTooSlow): {"processingtimems":1156438,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@3dbb29e5), rpc version=1, client version=29, methodsFingerPrint=-1508511443","client":"10.75.0.109:35245","starttimems":1343727170177,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"multi"}引用hbase說(shuō)明：The output is tagged with operation e.g.(operationTooSlow)if the call was a client operation, such as a Put, Get, or Delete, which we expose detailed fingerprint information for. If not, it is tagged(responseTooSlow)and still produces parseable JSON output, but with less verbose information solely regarding its duration and size in the RPC itself.
目前做法是取消了對(duì)某個(gè)key多個(gè)column的批量delete操作避免阻塞，沒(méi)有發(fā)現(xiàn)新問(wèn)題
10 output error
2012-07-31 17:52:06,812 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server Responder, call get([B@61574be4, {"timeRange":[0,9223372036854775807],"totalColumns":1,"cacheBlocks":true,"families":{"c":["ALL"]},"maxVersions":1,"row":"zOuu6TK"}), rpc version=1, client version=29, methodsFingerPrint=-1508511443 from 10.75.0.151:52745: output error
11 rollbackMemstore問(wèn)題較頻繁出現(xiàn)這樣的log：
2012-08-07 10:21:49,887 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: rollbackMemstore rolled back 0 keyvalues from start:0 to end:0
方法解釋為：Remove all the keys listed in the map from the memstore. This method is called when a Put has updated memstore but subequently fails to update the wal. This method is then invoked to rollback the memstore.
很奇怪的是開(kāi)始和結(jié)束的index都為0
方法中循環(huán)： for (int i = start; i < end; i++) {
因此是空數(shù)據(jù)，空回滾。需要進(jìn)一步調(diào)查
12 新上線一個(gè)region server 導(dǎo)致region not on line
往錯(cuò)誤的region server服務(wù)器請(qǐng)求region
13 請(qǐng)求不存在的region，重新建立tablepool也不起作用
請(qǐng)求的時(shí)間戳 1342510667
最新region rowkey相關(guān)時(shí)間戳 1344558957
最終發(fā)現(xiàn)維持region location表的屬性是在HConnectionManager中
get Get，delete Delete，incr Increment 是在 ServerCallable類 withRetries處理
情景1 若有出錯(cuò)（SocketTimeoutException ConnectException RetriesExhaustedExcetion），則清理regionServer location
情景2 numRetries 若設(shè)置為1 ，則 循環(huán)只執(zhí)行一次，connect(tries!=0) 為connect(false),即reload=false，不會(huì)進(jìn)行l(wèi)ocation更新，當(dāng)為numRetries>1的時(shí)候才會(huì)重新獲取
get Gets List, put Put或Puts List，delete Deletes List 則調(diào)用HConnectionManager的 processBatch去處理，當(dāng)發(fā)現(xiàn)批量get或者put、delete操作結(jié)果有問(wèn)題，則刷新regionServer location
設(shè)置 numRetries為>1次， 我這里是3次，解決問(wèn)題
14 zookeeper.RecoverableZooKeeper(195): Possibly transient ZooKeeper exception: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/master
這是在我單機(jī)做測(cè)試時(shí)出現(xiàn)的，無(wú)論是從ide或是bin啟動(dòng)hbase，從shell里可以正常連接，從測(cè)試程序中無(wú)法連接，zookeeper端口是2181,客戶端端口應(yīng)該與zookeeper無(wú)關(guān)才對(duì)，
最終更改配置21818端口換為2181 運(yùn)行正常，應(yīng)該是單機(jī)環(huán)境才要做這種更改。
<property> <name>hbase.zookeeper.property.clientPort</name> <value>2181</value> <description>Property from ZooKeeper's config zoo.cfg. The port at which the clients will connect. </description> </property>
聯(lián)系客服
