@mstump @BrianGallew
The repair command fails on running the script.
[gyadav@server prod]$ LOG_LEVEL="DEBUG" ./range_repair.py -k Keyspace_Demo -s 50
INFO 2014-05-10 08:01:03,922 get_ring_tokens 66 : running nodetool ring, this will take a little bit of time
DEBUG 2014-05-10 08:01:06,009 get_ring_tokens 72 : ring tokens found, creating ring token list...
DEBUG 2014-05-10 08:01:06,863 get_host_tokens 86 : host tokens found, creating host token list...
DEBUG 2014-05-10 08:01:06,864 repair_keyspace 170 : repair over range (042535295865117307932921825928971026432, 042535295865117307932921825928971026433] with 50 steps for keyspace Keyspace_Demo
DEBUG 2014-05-10 08:01:06,864 repair_keyspace 176 : step 0050 repairing range (042535295865117307932921825928971026432, 042535295865117307932921825928971026432] for keyspace Keyspace_Demo ...
ERROR 2014-05-10 08:01:08,361 repair_keyspace 179 : FAILED: /var/apache-cassandra-2.0.6/bin/nodetool repair Keyspace_Demo -local -pr -st 042535295865117307932921825928971026432 -et 042535295865117307932921825928971026432
ERROR 2014-05-10 08:01:08,361 repair_keyspace 180 :
The ring output is as under for the Keyspace Keyspace_Demo
Datacenter: AAA
Address Rack Status State Load Owns Token
53169119831396634916152282411213783040
10.50.41.208 RAC1 Up Normal 119.32 GB 25.00% 74436767763955288882613195375699296256
10.50.41.209 RAC1 Up Normal 122.03 GB 25.00% 85070591730234615865843651857942052864
10.50.41.211 RAC1 Up Normal 152.79 GB 25.00% 106338239662793269832304564822427566080
10.50.41.212 RAC1 Up Normal 119.37 GB 25.00% 116972063629072596815535021304670322688
10.50.41.213 RAC1 Up Normal 121.23 GB 25.00% 127605887595351923798765477786913079296
10.50.41.214 RAC1 Up Normal 119.85 GB 25.00% 138239711561631250781995934269155835904
10.50.41.215 RAC1 Up Normal 127.71 GB 25.00% 148873535527910577765226390751398592512
10.50.41.216 RAC1 Up Normal 140.93 GB 25.00% 159507359494189904748456847233641349120
10.50.41.220 RAC1 Up Normal 119.92 GB 25.00% 95704415696513942849074108340184809472
10.50.41.201 RAC1 Up Normal 126.08 GB 25.00% 0
10.50.41.203 RAC1 Up Normal 120.45 GB 25.00% 21267647932558653966460912964485513216
10.50.41.202 RAC1 Up Normal 127.07 GB 25.00% 10633823966279326983230456482242756608
10.50.41.205 RAC1 Up Normal 122.37 GB 25.00% 42535295865117307932921825928971026432
10.50.41.204 RAC1 Up Normal 119.79 GB 25.00% 31901471898837980949691369446728269824
10.50.41.207 RAC1 Up Normal 120.03 GB 25.00% 63802943797675961899382738893456539648
10.50.41.206 RAC1 Up Normal 120.41 GB 25.00% 53169119831396634916152282411213783040
Datacenter: BBB
Address Rack Status State Load Owns Token
1
10.80.14.124 RAC1 Up Normal 94.37 GB 16.67% 141784319550391032739561396922763706369
10.80.14.123 RAC1 Up Normal 91.3 GB 16.67% 127605887595351923798765477786913079297
10.80.14.148 RAC1 Up Normal 87.22 GB 16.67% 14178431955039101857246194831382806529
10.80.14.122 RAC1 Up Normal 97.31 GB 16.67% 113427455640312814857969558651062452225
10.80.14.149 RAC1 Up Normal 87.18 GB 16.67% 56713727820156407428984779325531226113
10.80.14.150 RAC1 Up Normal 87.31 GB 16.67% 99249023685273724806639570993792679937
10.80.14.121 RAC1 Up Normal 91.93 GB 16.67% 85070591730234615865843651857942052865
10.80.14.151 RAC1 Up Normal 80.16 GB 16.67% 155962751505430122790891384580033478657
10.80.14.120 RAC1 Up Normal 79.78 GB 16.67% 70892159775195516369780698461381853185
10.80.14.119 RAC1 Up Normal 114.02 GB 16.67% 42535295865117307932921825928971026433
10.80.14.118 RAC1 Up Normal 97.02 GB 16.67% 28356863910078203714492389662765613057
10.80.14.117 RAC1 Up Normal 89.26 GB 16.67% 1
Datacenter: CCC
Address Rack Status State Load Owns Token
63802943797675961899382738893456539650
10.28.13.112 RAC1 Up Normal 120.46 GB 25.00% 106338239662793269832304564822427566082
10.28.13.113 RAC1 Up Normal 144.31 GB 25.00% 127605887595351923798765477786913079298
10.28.13.114 RAC1 Up Normal 121.32 GB 25.00% 148873535527910577765226390751398592514
10.28.13.107 RAC1 Up Normal 118.72 GB 25.00% 2
10.28.13.109 RAC1 Up Normal 118.52 GB 25.00% 42535295865117307932921825928971026434
10.28.13.108 RAC1 Up Normal 119.92 GB 25.00% 21267647932558653966460912964485513218
10.28.13.111 RAC1 Up Normal 118.95 GB 25.00% 85070591730234615865843651857942052866
10.28.13.110 RAC1 Up Normal 121.47 GB 25.00% 63802943797675961899382738893456539650
Logs From Cassandra
INFO [RMI TCP Connection(431166)-10.50.41.205] 2014-05-10 07:51:29,518 StorageService.java (line 2472) starting user-requested repair of range (42535295865117307932921825928971026432, 42535295865117307932921825928971026432] for keyspace Keyspace_Demo and column families []
INFO [Thread-156415] 2014-05-10 07:51:29,519 StorageService.java (line 2538) Starting repair command #515, repairing 1 ranges for keyspace Keyspace_Demo
ERROR [Thread-156415] 2014-05-10 07:51:29,521 StorageService.java (line 2559) Repair session failed:
java.lang.IllegalArgumentException: Requested range intersects a local range but is not fully contained in one; this would lead to imprecise repair
at org.apache.cassandra.service.ActiveRepairService.getNeighbors(ActiveRepairService.java:164)
at org.apache.cassandra.repair.RepairSession.(RepairSession.java:128)
at org.apache.cassandra.repair.RepairSession.(RepairSession.java:117)
at org.apache.cassandra.service.ActiveRepairService.submitRepairSession(ActiveRepairService.java:97)
at org.apache.cassandra.service.StorageService.forceKeyspaceRepair(StorageService.java:2619)
at org.apache.cassandra.service.StorageService$5.runMayThrow(StorageService.java:2555)
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.lang.Thread.run(Thread.java:724)
@mstump @BrianGallew
The repair command fails on running the script.
[gyadav@server prod]$ LOG_LEVEL="DEBUG" ./range_repair.py -k Keyspace_Demo -s 50
INFO 2014-05-10 08:01:03,922 get_ring_tokens 66 : running nodetool ring, this will take a little bit of time
DEBUG 2014-05-10 08:01:06,009 get_ring_tokens 72 : ring tokens found, creating ring token list...
DEBUG 2014-05-10 08:01:06,863 get_host_tokens 86 : host tokens found, creating host token list...
DEBUG 2014-05-10 08:01:06,864 repair_keyspace 170 : repair over range (042535295865117307932921825928971026432, 042535295865117307932921825928971026433] with 50 steps for keyspace Keyspace_Demo
DEBUG 2014-05-10 08:01:06,864 repair_keyspace 176 : step 0050 repairing range (042535295865117307932921825928971026432, 042535295865117307932921825928971026432] for keyspace Keyspace_Demo ...
ERROR 2014-05-10 08:01:08,361 repair_keyspace 179 : FAILED: /var/apache-cassandra-2.0.6/bin/nodetool repair Keyspace_Demo -local -pr -st 042535295865117307932921825928971026432 -et 042535295865117307932921825928971026432
ERROR 2014-05-10 08:01:08,361 repair_keyspace 180 :
The ring output is as under for the Keyspace Keyspace_Demo
Datacenter: AAA
Address Rack Status State Load Owns Token
53169119831396634916152282411213783040
10.50.41.208 RAC1 Up Normal 119.32 GB 25.00% 74436767763955288882613195375699296256
10.50.41.209 RAC1 Up Normal 122.03 GB 25.00% 85070591730234615865843651857942052864
10.50.41.211 RAC1 Up Normal 152.79 GB 25.00% 106338239662793269832304564822427566080
10.50.41.212 RAC1 Up Normal 119.37 GB 25.00% 116972063629072596815535021304670322688
10.50.41.213 RAC1 Up Normal 121.23 GB 25.00% 127605887595351923798765477786913079296
10.50.41.214 RAC1 Up Normal 119.85 GB 25.00% 138239711561631250781995934269155835904
10.50.41.215 RAC1 Up Normal 127.71 GB 25.00% 148873535527910577765226390751398592512
10.50.41.216 RAC1 Up Normal 140.93 GB 25.00% 159507359494189904748456847233641349120
10.50.41.220 RAC1 Up Normal 119.92 GB 25.00% 95704415696513942849074108340184809472
10.50.41.201 RAC1 Up Normal 126.08 GB 25.00% 0
10.50.41.203 RAC1 Up Normal 120.45 GB 25.00% 21267647932558653966460912964485513216
10.50.41.202 RAC1 Up Normal 127.07 GB 25.00% 10633823966279326983230456482242756608
10.50.41.205 RAC1 Up Normal 122.37 GB 25.00% 42535295865117307932921825928971026432
10.50.41.204 RAC1 Up Normal 119.79 GB 25.00% 31901471898837980949691369446728269824
10.50.41.207 RAC1 Up Normal 120.03 GB 25.00% 63802943797675961899382738893456539648
10.50.41.206 RAC1 Up Normal 120.41 GB 25.00% 53169119831396634916152282411213783040
Datacenter: BBB
Address Rack Status State Load Owns Token
1
10.80.14.124 RAC1 Up Normal 94.37 GB 16.67% 141784319550391032739561396922763706369
10.80.14.123 RAC1 Up Normal 91.3 GB 16.67% 127605887595351923798765477786913079297
10.80.14.148 RAC1 Up Normal 87.22 GB 16.67% 14178431955039101857246194831382806529
10.80.14.122 RAC1 Up Normal 97.31 GB 16.67% 113427455640312814857969558651062452225
10.80.14.149 RAC1 Up Normal 87.18 GB 16.67% 56713727820156407428984779325531226113
10.80.14.150 RAC1 Up Normal 87.31 GB 16.67% 99249023685273724806639570993792679937
10.80.14.121 RAC1 Up Normal 91.93 GB 16.67% 85070591730234615865843651857942052865
10.80.14.151 RAC1 Up Normal 80.16 GB 16.67% 155962751505430122790891384580033478657
10.80.14.120 RAC1 Up Normal 79.78 GB 16.67% 70892159775195516369780698461381853185
10.80.14.119 RAC1 Up Normal 114.02 GB 16.67% 42535295865117307932921825928971026433
10.80.14.118 RAC1 Up Normal 97.02 GB 16.67% 28356863910078203714492389662765613057
10.80.14.117 RAC1 Up Normal 89.26 GB 16.67% 1
Datacenter: CCC
Address Rack Status State Load Owns Token
63802943797675961899382738893456539650
10.28.13.112 RAC1 Up Normal 120.46 GB 25.00% 106338239662793269832304564822427566082
10.28.13.113 RAC1 Up Normal 144.31 GB 25.00% 127605887595351923798765477786913079298
10.28.13.114 RAC1 Up Normal 121.32 GB 25.00% 148873535527910577765226390751398592514
10.28.13.107 RAC1 Up Normal 118.72 GB 25.00% 2
10.28.13.109 RAC1 Up Normal 118.52 GB 25.00% 42535295865117307932921825928971026434
10.28.13.108 RAC1 Up Normal 119.92 GB 25.00% 21267647932558653966460912964485513218
10.28.13.111 RAC1 Up Normal 118.95 GB 25.00% 85070591730234615865843651857942052866
10.28.13.110 RAC1 Up Normal 121.47 GB 25.00% 63802943797675961899382738893456539650
Logs From Cassandra
INFO [RMI TCP Connection(431166)-10.50.41.205] 2014-05-10 07:51:29,518 StorageService.java (line 2472) starting user-requested repair of range (42535295865117307932921825928971026432, 42535295865117307932921825928971026432] for keyspace Keyspace_Demo and column families []
INFO [Thread-156415] 2014-05-10 07:51:29,519 StorageService.java (line 2538) Starting repair command #515, repairing 1 ranges for keyspace Keyspace_Demo
ERROR [Thread-156415] 2014-05-10 07:51:29,521 StorageService.java (line 2559) Repair session failed:
java.lang.IllegalArgumentException: Requested range intersects a local range but is not fully contained in one; this would lead to imprecise repair
at org.apache.cassandra.service.ActiveRepairService.getNeighbors(ActiveRepairService.java:164)
at org.apache.cassandra.repair.RepairSession.(RepairSession.java:128)
at org.apache.cassandra.repair.RepairSession.(RepairSession.java:117)
at org.apache.cassandra.service.ActiveRepairService.submitRepairSession(ActiveRepairService.java:97)
at org.apache.cassandra.service.StorageService.forceKeyspaceRepair(StorageService.java:2619)
at org.apache.cassandra.service.StorageService$5.runMayThrow(StorageService.java:2555)
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.lang.Thread.run(Thread.java:724)