測(cè)試一個(gè)configserver服務(wù)器掛機(jī)后，集群是否能讀寫數(shù)據(jù)

測(cè)試架構(gòu)：

測(cè)試目的：

測(cè)試一個(gè)config server 服務(wù)器掛機(jī)后，集群是否能讀寫數(shù)據(jù)。

測(cè)試原因：

上周因?yàn)閮?nèi)存吃緊，準(zhǔn)備添加內(nèi)存。在查看服務(wù)器時(shí)，把一臺(tái)服務(wù)器關(guān)機(jī)檢查。
關(guān)機(jī)后，WEB 端應(yīng)用出錯(cuò)，用戶無法登錄，所有都出錯(cuò)。

但在我理解，因?yàn)镃ONFIG是3臺(tái)冗余的，一臺(tái)關(guān)機(jī)，應(yīng)該不會(huì)產(chǎn)生影響才對(duì)，或者說不會(huì)影響到無法
登錄，應(yīng)用不能正常運(yùn)行。只是會(huì)有數(shù)據(jù)不均衡啊。

但我使用mongochef 登錄，確實(shí)登錄數(shù)據(jù)庫不成功。

測(cè)試結(jié)果：

經(jīng)下面的測(cè)試，印證了我的理解，一臺(tái)config 宕機(jī)，只是會(huì)提示連接config 出錯(cuò)，
但能登錄數(shù)據(jù)庫，插入數(shù)據(jù)的代碼（mongo -port 28000 登錄后運(yùn)行的JS 代碼），還能正常運(yùn)行下去。

但為什么當(dāng)時(shí)WEB 端會(huì)出錯(cuò)，我看到他們php 配置文件中如下：
$connect = new MongoClient("mongodb://192.168.2.188:28000,192.168.2.132:28000", array("replicaSet" => "myReplSet"));

卻沒加MongoConnectionException。 容錯(cuò)出錯(cuò)沒有的話，當(dāng)正在操作時(shí)，一個(gè)collection打開出錯(cuò)，不會(huì)再重新連接數(shù)據(jù)庫，
就報(bào)錯(cuò)了。如果這時(shí)能重新連接數(shù)據(jù)庫，應(yīng)該能再找到另外正常的節(jié)點(diǎn)連接的。

測(cè)試環(huán)境為： mongodb3.0.3
2個(gè)shard，3個(gè)配置config，3個(gè)mongos

set host：port stateStr
------------------------------------------------------------
shard1 192.168.2.188:28011 PRIMARY
shard1 192.168.2.132:28011 SECONDARY
shard1 192.168.2.134:28011 SECONDARY

shard2 192.168.2.188:28012 SECONDARY
shard2 192.168.2.132:28012 SECONDARY
shard2 192.168.2.134:28012 PRIMARY

--------------------------------------------------------------------
192.168.2.188:28010 config
192.168.2.188:28000 mongos

192.168.2.132:28010 config
192.168.2.132:28000 mongos

192.168.2.134:28010 config
192.168.2.134:28000 mongos

#備用服務(wù)器，當(dāng)134掛機(jī)后，把134 對(duì)應(yīng)的t3 服務(wù)器名稱修改為 135

192.168.2.135:28010 config

各服務(wù)器的hosts 文件：
[root@localhost bin]# cat /etc/hosts
192.168.2.188 t1
192.168.2.132 t2
192.168.2.134 t3

192.168.2.135 t4

一、配置集群：

初始化replica set shard1
用mongo 連接其中一個(gè)節(jié)點(diǎn)： 比如：shard11 執(zhí)行以下：

> config = {_id: \'shard1\', members: [
{_id: 0, host:\'t1:28011\'}]
}

> rs.initiate(config);

rs.add({_id: 1, host:\'t2:28011\'})
rs.add({_id: 2, host:\'t3:28011\'})

初始化replica set shard2
用mongo 連接其中一個(gè)節(jié)點(diǎn)： 比如：shard11 執(zhí)行以下：

> config = {_id: \'shard2\', members: [
{_id: 0, host:\'t1:28012\'}]
}

> rs.initiate(config);

rs.add({_id: 1, host:\'t2:28012\'})
rs.add({_id: 2, host:\'t3:28012\'})

#遇到如下錯(cuò)誤
shard2:PRIMARY> rs.add({_id: 1, host:\'t2:28012\'})
{
"ok" : 0,
"errmsg" : "Quorum check failed because not enough voting nodes responded; required 2 but only the following 1 voting nodes responded: t1:28012; the following nodes did not respond affirmatively: t2:28012 failed with Failed attempt to connect to t2:28012; couldn\'t connect to server t2:28012 (192.168.2.132), connection attempt failed",
"code" : 74
}
#原因：有防火墻
#關(guān)閉防火墻
[root@t2 ~]# chkconfig iptables off
[root@t2 ~]# service iptables stop
iptables：清除防火墻規(guī)則： [確定]
iptables：將鏈設(shè)置為政策 ACCEPT：nat mangle filter [確定]
iptables：正在卸載模塊： [確定]
[root@t2 ~]#

二、配置分片

1：配置一下時(shí)間一致，不然無法建立分片集群：

*/5 * * * * /usr/sbin/ntpdate -u ntp.sjtu.edu.cn

2.mongos 運(yùn)行時(shí)的參數(shù)如下：
/opt/mongodb3.0.3/bin/mongos --configdb t1:28010,t2:28010,t3:28010 --port 28000 --logpath /opt/mongodb3.0.3/logs/mongos.log --logappend --fork

3.連接到其中一個(gè)mongos進(jìn)程，并切換到admin數(shù)據(jù)庫做以下配置
3.1. 連接到mongs，并切換到admin
./mongo 192.168.20.11:28000/admin
>db
Admin
3.2. 加入shards
命令格式：
replicaSetName/[:port] [,serverhostname2[:port],…]

執(zhí)行如下：
>use admin;
>db.runCommand({addshard:"shard1/t1:28011,t2:28011,t3:28011"});
>db.runCommand({addshard:"shard2/t1:28012,t2:28012,t3:28012"});

添加分片后，把chunksize 改成 1M，以方便測(cè)試
>use config
mongos> db.shards.find()
{ "_id" : "shard1", "host" : "shard1/t1:28011,t2:28011,t3:28011" }
{ "_id" : "shard2", "host" : "shard2/t1:28012,t2:28012,t3:28012" }
mongos> db.settings.find()
{ "_id" : "chunksize", "value" : 64 }
mongos> db.settings.update({_id:\'chunksize\'},{$set:{value:1}})
WriteResult({ "nMatched" : 1, "nUpserted" : 0, "nModified" : 1 })
mongos> db.settings.find()
{ "_id" : "chunksize", "value" : 1 }
mongos>

到此分片集群配置已完成

三、添加數(shù)據(jù)

db.runCommand({"enablesharding":"test"})
db.runCommand({shardcollection:"test.customer",key:{_id:1}});

#表中插入數(shù)據(jù)：
use test;
for (i = 1; i <= 90000; i++){
db.customer.insert({_id:i,name:\' test name test name test name test name test name test name test name test name test name end\',
address:\'address test customer al;dkjfa;lsdfjsal;#^end\',province:\'test customer province\',city:\'test customer city\'});
}

用于測(cè)試時(shí)查看各數(shù)據(jù)所在的分片。

從下面的查詢中，可以看到test.customer數(shù)據(jù)已分片到shard1,shard2

mongos> db.chunks.find()
{ "_id" : "test.customer-_id_MinKey", "lastmod" : Timestamp(2, 1), "lastmodEpoch" : ObjectId("55b99f3ff5ddb9333eda7464"), "ns" : "test.customer", "min" : { "_id" : { "$minKey" : 1 } }, "max" : { "_id" : 2 }, "shard" : "shard2" }
{ "_id" : "test.customer-_id_2.0", "lastmod" : Timestamp(1, 2), "lastmodEpoch" : ObjectId("55b99f3ff5ddb9333eda7464"), "ns" : "test.customer", "min" : { "_id" : 2 }, "max" : { "_id" : 6 }, "shard" : "shard2" }
{ "_id" : "test.customer-_id_6.0", "lastmod" : Timestamp(2, 2), "lastmodEpoch" : ObjectId("55b99f3ff5ddb9333eda7464"), "ns" : "test.customer", "min" : { "_id" : 6 }, "max" : { "_id" : 2120 }, "shard" : "shard1" }
{ "_id" : "test.customer-_id_2120.0", "lastmod" : Timestamp(2, 3), "lastmodEpoch" : ObjectId("55b99f3ff5ddb9333eda7464"), "ns" : "test.customer", "min" : { "_id" : 2120 }, "max" : { "_id" : 4579 }, "shard" : "shard1" }
{ "_id" : "test.customer-_id_4579.0", "lastmod" : Timestamp(2, 5), "lastmodEpoch" : ObjectId("55b99f3ff5ddb9333eda7464"), "ns" : "test.customer", "min" : { "_id" : 4579 }, "max" : { "_id" : 6693 }, "shard" : "shard1" }
{ "_id" : "test.customer-_id_6693.0", "lastmod" : Timestamp(2, 6), "lastmodEpoch" : ObjectId("55b99f3ff5ddb9333eda7464"), "ns" : "test.customer", "min" : { "_id" : 6693 }, "max" : { "_id" : 9371 }, "shard" : "shard1" }
{ "_id" : "test.customer-_id_9371.0", "lastmod" : Timestamp(2, 7), "lastmodEpoch" : ObjectId("55b99f3ff5ddb9333eda7464"), "ns" : "test.customer", "min" : { "_id" : 9371 }, "max" : { "_id" : { "$maxKey" : 1 } }, "shard" : "shard1" }
mongos> db.collections.find()
{ "_id" : "test.customer", "lastmod" : ISODate("2015-07-30T03:51:28.671Z"), "dropped" : false, "key" : { "_id" : 1 }, "unique" : false, "lastmodEpoch" : ObjectId("55b99f3ff5ddb9333eda7464") }
mongos>

四、測(cè)試掛掉一個(gè)config:

4.1

[mongo@localhost bin]$ ll /opt/mongodb3.0.3/config
總用量 0
[mongo@localhost bin]$ ./configd_start.sh
about to fork child process, waiting until server is ready for connections.
forked process: 2638
child process started successfully, parent exiting
[mongo@localhost bin]$ ps -ef|grep mongod
mongo 2455 1 28 10:51 ? 00:04:14 /opt/mongodb3.0.3/bin/mongod --storageEngine wiredTiger -f shard1.conf
mongo 2518 1 7 10:51 ? 00:01:11 /opt/mongodb3.0.3/bin/mongod --storageEngine wiredTiger -f shard2.conf
mongo 2638 1 7 10:53 ? 00:00:59 /opt/mongodb3.0.3/bin/mongod --storageEngine wiredTiger -f configsvr.conf
mongo 2931 2083 8 11:06 pts/0 00:00:00 grep mongod
[mongo@localhost bin]$ kill -9 2638
[mongo@localhost bin]$ ps -ef|grep mongod
mongo 2455 1 27 10:51 ? 00:04:16 /opt/mongodb3.0.3/bin/mongod --storageEngine wiredTiger -f shard1.conf
mongo 2518 1 7 10:51 ? 00:01:13 /opt/mongodb3.0.3/bin/mongod --storageEngine wiredTiger -f shard2.conf
mongo 2941 2083 6 11:06 pts/0 00:00:00 grep mongod
[mongo@localhost bin]$

#t1登錄沒問題
[mongo@t1 bin]$ ./mongo -port 28000
MongoDB shell version: 3.0.3
connecting to: 127.0.0.1:28000/test
mongos> show dbs;
admin (empty)
config (empty)
test 0.002GB
mongos>

#t2登錄 也沒問題
[mongo@t2 bin]$ ./mongo -port 28000
MongoDB shell version: 3.0.3
connecting to: 127.0.0.1:28000/test
mongos>

#同樣，使用mongochef 客戶端登錄，也同樣驗(yàn)證了上面的問題，

4.2.
把分片從hostname修改成ip:

> config = {_id: \'shard1\', members: [
{_id: 0, host:\'192.168.2.188:28011\'}]
}

> rs.initiate(config);

rs.add({_id: 1, host:\'192.168.2.132:28011\'})
rs.add({_id: 2, host:\'192.168.2.134:28011\'})

> config = {_id: \'shard2\', members: [
{_id: 0, host:\'192.168.2.188:28012\'}]
}

> rs.initiate(config);

rs.add({_id: 1, host:\'192.168.2.132:28012\'})
rs.add({_id: 2, host:\'192.168.2.134:28012\'})

#把mongos 的參數(shù)中的hostname也修改成IP

[mongo@t1 bin]$ cat mongos_router.sh
/opt/mongodb3.0.3/bin/mongos --configdb 192.168.2.188:28010,192.168.2.132:28010,192.168.2.134:28010 --port 28000 --logpath /opt/mongodb3.0.3/logs/mongos.log --logappend --fork
[mongo@t1 bin]$

#添加分片，把chunksize 改成 1M，以方便測(cè)試

>use admin;
>db.runCommand({addshard:"shard1/192.168.2.188:28011,192.168.2.132:28011,192.168.2.134:28011"});
>db.runCommand({addshard:"shard2/192.168.2.188:28012,192.168.2.132:28012,192.168.2.134:28012"});

db.settings.update({_id:\'chunksize\'},{$set:{value:1}})

#當(dāng)所有配置設(shè)置成使用IP后，宕掉config server.
#再?gòu)谋緳C(jī)嘗試連接，發(fā)現(xiàn)不成功。

[mongo@localhost bin]$ ps -ef|grep mongod
mongo 4062 1 10 11:48 ? 00:09:56 /opt/mongodb3.0.3/bin/mongod --storageEngine wiredTiger -f shard1.conf
mongo 4082 1 8 11:48 ? 00:08:09 /opt/mongodb3.0.3/bin/mongod --storageEngine wiredTiger -f shard2.conf
mongo 4102 1 11 11:48 ? 00:10:45 /opt/mongodb3.0.3/bin/mongod --storageEngine wiredTiger -f configsvr.conf
mongo 4121 1 5 11:49 ? 00:04:55 /opt/mongodb3.0.3/bin/mongos --configdb 192.168.2.188:28010,192.168.2.132:28010,192.168.2.134:28010 --logpath /opt/mongodb3.0.3/logs/mongos.log --logappend --fork
root 9459 9420 0 13:23 pts/1 00:00:00 tail -330f /opt/mongodb3.0.3/logs/shard1.log
mongo 9471 2083 5 13:23 pts/0 00:00:00 grep mongod
[mongo@localhost bin]$ kill -9 4102
[mongo@localhost bin]$ ps -ef|grep mongod
mongo 4062 1 10 11:48 ? 00:10:20 /opt/mongodb3.0.3/bin/mongod --storageEngine wiredTiger -f shard1.conf
mongo 4082 1 8 11:48 ? 00:08:18 /opt/mongodb3.0.3/bin/mongod --storageEngine wiredTiger -f shard2.conf
mongo 4121 1 5 11:49 ? 00:04:59 /opt/mongodb3.0.3/bin/mongos --configdb 192.168.2.188:28010,192.168.2.132:28010,192.168.2.134:28010 --logpath /opt/mongodb3.0.3/logs/mongos.log --logappend --fork
root 9459 9420 0 13:23 pts/1 00:00:00 tail -330f /opt/mongodb3.0.3/logs/shard1.log
mongo 9488 2083 5 13:24 pts/0 00:00:00 grep mongod

[mongo@localhost bin]$ ./mongo -port 28000
MongoDB shell version: 3.0.3
connecting to: 127.0.0.1:28000/test
2015-07-31T13:25:52.166+0800 W NETWORK Failed to connect to 127.0.0.1:28000, reason: errno:111 Connection refused
2015-07-31T13:25:53.514+0800 E QUERY Error: couldn\'t connect to server 127.0.0.1:28000 (127.0.0.1), connection attempt failed
at connect (src/mongo/shell/mongo.js:181:14)
at (connect):1:6 at src/mongo/shell/mongo.js:181
exception: connect failed
[mongo@localhost bin]$ ./mongo -port 28000
MongoDB shell version: 3.0.3
connecting to: 127.0.0.1:28000/test
2015-07-31T13:26:30.823+0800 W NETWORK Failed to connect to 127.0.0.1:28000, reason: errno:111 Connection refused
2015-07-31T13:26:32.929+0800 E QUERY Error: couldn\'t connect to server 127.0.0.1:28000 (127.0.0.1), connection attempt failed
at connect (src/mongo/shell/mongo.js:181:14)
at (connect):1:6 at src/mongo/shell/mongo.js:181
exception: connect failed

[mongo@localhost bin]$ ps -ef|grep mongod
mongo 4062 1 12 11:48 ? 00:11:48 /opt/mongodb3.0.3/bin/mongod --storageEngine wiredTiger -f shard1.conf
mongo 4082 1 8 11:48 ? 00:08:29 /opt/mongodb3.0.3/bin/mongod --storageEngine wiredTiger -f shard2.conf
mongo 4121 1 5 11:49 ? 00:05:09 /opt/mongodb3.0.3/bin/mongos --configdb 192.168.2.188:28010,192.168.2.132:28010,192.168.2.134:28010 --logpath /opt/mongodb3.0.3/logs/mongos.log --logappend --fork
root 9459 9420 0 13:23 pts/1 00:00:00 tail -330f /opt/mongodb3.0.3/logs/shard1.log
mongo 9754 2083 7 13:26 pts/0 00:00:00 grep mongod
[mongo@localhost bin]$ ./mongo -port 28000
MongoDB shell version: 3.0.3
connecting to: 127.0.0.1:28000/test
2015-07-31T13:28:17.397+0800 W NETWORK Failed to connect to 127.0.0.1:28000, reason: errno:111 Connection refused
2015-07-31T13:28:19.161+0800 E QUERY Error: couldn\'t connect to server 127.0.0.1:28000 (127.0.0.1), connection attempt failed
at connect (src/mongo/shell/mongo.js:181:14)
at (connect):1:6 at src/mongo/shell/mongo.js:181
exception: connect failed
[mongo@localhost bin]$

#而其它節(jié)點(diǎn)（t2,t1）服務(wù)器連接沒有問題。使用mongochef 測(cè)試，也印證了這點(diǎn)。
是否說使用Ip地址，真的是在服務(wù)器集群中，穩(wěn)定性會(huì)差一些呢。
并且我同在宕掉config server. 時(shí)，測(cè)試循環(huán)插入數(shù)據(jù)的代碼沒有中止。后面也一直在插入數(shù)據(jù)。

[mongo@t2 bin]$ ./mongo -port 28000
MongoDB shell version: 3.0.3
connecting to: 127.0.0.1:28000/test
mongos>
connecting to: 127.0.0.1:28000/test
mongos> db.customer.count()
10024
mongos> db.customer.count()
10091
mongos> db.customer.count()
10103
mongos>

可以從下面的日志中看到這點(diǎn)：

2015-07-31T13:34:55.114+0800 I NETWORK [LockPinger] trying reconnect to 192.168.2.134:28010 (192.168.2.134) failed
2015-07-31T13:34:55.328+0800 W NETWORK [LockPinger] Failed to connect to 192.168.2.134:28010, reason: errno:111 Connection refused
2015-07-31T13:34:55.383+0800 I WRITE [conn258] insert test.customer query: { _id: 11119.0, name: " test name test name test name test name test name test name test name test name test name end", address: "address test customer al;dkjfa;lsdfjsal;#^end", province: "test customer province", city: "test customer city" } ninserted:1 keyUpdates:0 writeConflicts:0 numYields:0 locks:{ Global: { acquireCount: { w: 2 } }, Database: { acquireCount: { w: 2 } }, Collection: { acquireCount: { w: 1 } }, oplog: { acquireCount: { w: 1 } } } 115ms
2015-07-31T13:34:55.419+0800 I NETWORK [LockPinger] reconnect 192.168.2.134:28010 (192.168.2.134) failed failed couldn\'t connect to server 192.168.2.134:28010 (192.168.2.134), connection attempt failed
2015-07-31T13:34:55.556+0800 I COMMAND [conn258] command test.$cmd command: insert { insert: "customer", documents: [ { _id: 11119.0, name: " test name test name test name test name test name test name test name test name test name end", address: "address test customer al;dkjfa;lsdfjsal;#^end", province: "test customer province", city: "test customer city" } ], ordered: true, metadata: { shardName: "shard1", shardVersion: [ Timestamp 4000|1, ObjectId(\'55bb0603bb39501b3b7c2934\') ], session: 0 } } ntoreturn:1 keyUpdates:0 writeConflicts:0 numYields:0 reslen:140 locks:{ Global: { acquireCount: { w: 2 } }, Database: { acquireCount: { w: 2 } }, Collection: { acquireCount: { w: 1 } }, oplog: { acquireCount: { w: 1 } } } 312ms
2015-07-31T13:34:55.618+0800 I NETWORK [LockPinger] scoped connection to 192.168.2.188:28010,192.168.2.132:28010,192.168.2.134:28010 not being returned to the pool
2015-07-31T13:34:55.714+0800 W SHARDING [LockPinger] distributed lock pinger \'192.168.2.188:28010,192.168.2.132:28010,192.168.2.134:28010/t3:28011:1438320143:660128477\' detected an exception while pinging. :: caused by :: SyncClusterConnection::update prepare failed: 192.168.2.134:28010 (192.168.2.134) failed:9001 socket exception [CONNECT_ERROR] server [192.168.2.134:28010 (192.168.2.134) failed]
2015-07-31T13:34:55.773+0800 I COMMAND [conn258] command test.$cmd command: insert { insert: "customer", documents: [ { _id: 11120.0, name: " test name test name test name test name test name test name test name test name test name end", address: "address test customer al;dkjfa;lsdfjsal;#^end", province: "test customer province", city: "test customer city" } ], ordered: true, metadata: { shardName: "shard1", shardVersion: [ Timestamp 4000|1, ObjectId(\'55bb0603bb39501b3b7c2934\') ], session: 0 } } ntoreturn:1 keyUpdates:0 writeConflicts:0 numYields:0 reslen:140 locks:{ Global: { acquireCount: { w: 2 } }, Database: { acquireCount: { w: 2 } }, Collection: { acquireCount: { w: 1 } }, oplog: { acquireCount: { w: 1 } } } 105ms
2015-07-31T13:34:56.034+0800 I WRITE [conn258] insert test.customer query: { _id: 11121.0, name: " test name test name test name test name test name test name test name test name test name end", address: "address test customer al;dkjfa;lsdfjsal;#^end", province: "test customer province", city: "test customer city" } ninserted:1 keyUpdates:0 writeConflicts:0 numYields:0 locks:{ Global: { acquireCount: { w: 2 } }, Database: { acquireCount: { w: 2 } }, Collection: { acquireCount: { w: 1 } }, oplog: { acquireCount: { w: 1 } } } 114ms
2015-07-31T13:34:56.161+0800 I COMMAND [conn258] command test.$cmd command: insert { insert: "customer", documents: [ { _id: 11121.0, name: " test name test name test name test name test name test name test name test name test name end", address: "address test customer al;dkjfa;lsdfjsal;#^end", province: "test customer province", city: "test customer city" } ], ordered: true, metadata: { shardName: "shard1", shardVersion: [ Timestamp 4000|1, ObjectId(\'55bb0603bb39501b3b7c2934\') ], session: 0 } } ntoreturn:1 keyUpdates:0 writeConflicts:0 numYields:0 reslen:140 locks:{ Global: { acquireCount: { w: 2 } }, Database: { acquireCount: { w: 2 } }, Collection: { acquireCount: { w: 1 } }, oplog: { acquireCount: { w: 1 } } } 256ms
2015-07-31T13:34:56.367+0800 I COMMAND [conn258] command test.$cmd command: insert { insert: "customer", documents: [ { _id: 11122.0, name: " test name test name test name test name test name test name test name test name test name end", address: "address test customer al;dkjfa;lsdfjsal;#^end", province: "test customer province", city: "test customer city" } ], ordered: true, metadata: { shardName: "shard1", shardVersion: [ Timestamp 4000|1, ObjectId(\'55bb0603bb39501b3b7c2934\') ], session: 0 } } ntoreturn:1 keyUpdates:0 writeConflicts:0 numYields:0 reslen:140 locks:{ Global: { acquireCount: { w: 2 } }, Database: { acquireCount: { w: 2 } }, Collection: { acquireCount: { w: 1 } }, oplog: { acquireCount: { w: 1 } } } 110ms

#重啟config 后，再登錄也出錯(cuò)。
可能是本機(jī)mongos 已記錄沒找到config ，后再重啟mongos ，全部正常。
這有點(diǎn)說不過去。

[mongo@localhost bin]$ ./configd_start.sh
about to fork child process, waiting until server is ready for connections.
forked process: 12102
child process started successfully, parent exiting
[mongo@localhost bin]$ ./mongo -port 28000
MongoDB shell version: 3.0.3
connecting to: 127.0.0.1:28000/test
2015-07-31T13:46:49.324+0800 W NETWORK Failed to connect to 127.0.0.1:28000, reason: errno:111 Connection refused
2015-07-31T13:46:50.747+0800 E QUERY Error: couldn\'t connect to server 127.0.0.1:28000 (127.0.0.1), connection attempt failed
at connect (src/mongo/shell/mongo.js:181:14)
at (connect):1:6 at src/mongo/shell/mongo.js:181
exception: connect failed
[mongo@localhost bin]$ ps -ef|grep mongod
mongo 4062 1 23 11:48 ? 00:27:18 /opt/mongodb3.0.3/bin/mongod --storageEngine wiredTiger -f shard1.conf
mongo 4082 1 8 11:48 ? 00:10:29 /opt/mongodb3.0.3/bin/mongod --storageEngine wiredTiger -f shard2.conf
mongo 4121 1 5 11:49 ? 00:06:56 /opt/mongodb3.0.3/bin/mongos --configdb 192.168.2.188:28010,192.168.2.132:28010,192.168.2.134:28010 --logpath /opt/mongodb3.0.3/logs/mongos.log --logappend --fork
root 11468 9420 0 13:40 pts/1 00:00:01 tail -330f /opt/mongodb3.0.3/logs/shard1.log
mongo 12102 1 12 13:46 ? 00:00:05 /opt/mongodb3.0.3/bin/mongod --storageEngine wiredTiger -f configsvr.conf
mongo 12190 2083 3 13:46 pts/0 00:00:00 grep mongod
[mongo@localhost bin]$ kill -9 4121
[mongo@localhost bin]$ ./mongos_router.sh
about to fork child process, waiting until server is ready for connections.
forked process: 12220
child process started successfully, parent exiting
[mongo@localhost bin]$ ps -ef|grep mongod
mongo 4062 1 23 11:48 ? 00:27:57 /opt/mongodb3.0.3/bin/mongod --storageEngine wiredTiger -f shard1.conf
mongo 4082 1 9 11:48 ? 00:11:09 /opt/mongodb3.0.3/bin/mongod --storageEngine wiredTiger -f shard2.conf
root 11468 9420 0 13:40 pts/1 00:00:01 tail -330f /opt/mongodb3.0.3/logs/shard1.log
mongo 12102 1 19 13:46 ? 00:00:31 /opt/mongodb3.0.3/bin/mongod --storageEngine wiredTiger -f configsvr.conf
mongo 12220 1 5 13:47 ? 00:00:04 /opt/mongodb3.0.3/bin/mongos --configdb 192.168.2.188:28010,192.168.2.132:28010,192.168.2.134:28010 --logpath /opt/mongodb3.0.3/logs/mongos.log --logappend --fork
mongo 12299 2083 3 13:48 pts/0 00:00:00 grep mongod
[mongo@localhost bin]$ ./mongo -port 28000
MongoDB shell version: 3.0.3

cfg = rs.conf()
cfg.members[0].priority = 2;
cfg.members[1].priority = 3;
cfg.members[2].priority = 1;
rs.reconfig(cfg);