Riak doesn't start anymore - Waiting for service riak_kv to start (540 seconds) [JIRA: RIAK-2273]

[Zerebokep]

After a server crash riak doesn't seem to start anymore. There are no errors logged.

Here is the console.log:

2015-11-09 16:04:31.030 [info] <0.7.0> Application lager started on node 'riak@127.0.0.1'
2015-11-09 16:04:31.038 [info] <0.7.0> Application sasl started on node 'riak@127.0.0.1'
2015-11-09 16:04:31.039 [info] <0.7.0> Application asn1 started on node 'riak@127.0.0.1'
2015-11-09 16:04:31.040 [info] <0.7.0> Application crypto started on node 'riak@127.0.0.1'
2015-11-09 16:04:31.040 [info] <0.7.0> Application public_key started on node 'riak@127.0.0.1'
2015-11-09 16:04:31.044 [info] <0.7.0> Application ssl started on node 'riak@127.0.0.1'
2015-11-09 16:04:31.046 [info] <0.7.0> Application riak_sysmon started on node 'riak@127.0.0.1'
2015-11-09 16:04:31.051 [info] <0.7.0> Application os_mon started on node 'riak@127.0.0.1'
2015-11-09 16:04:31.054 [info] <0.73.0> alarm_handler: {set,{{disk_almost_full,"/"},[]}}
2015-11-09 16:04:31.059 [info] <0.7.0> Application runtime_tools started on node 'riak@127.0.0.1'
2015-11-09 16:04:31.068 [info] <0.7.0> Application erlang_js started on node 'riak@127.0.0.1'
2015-11-09 16:04:31.068 [info] <0.7.0> Application xmerl started on node 'riak@127.0.0.1'
2015-11-09 16:04:31.082 [info] <0.7.0> Application inets started on node 'riak@127.0.0.1'
2015-11-09 16:04:31.084 [info] <0.7.0> Application mochiweb started on node 'riak@127.0.0.1'
2015-11-09 16:04:31.089 [info] <0.7.0> Application webmachine started on node 'riak@127.0.0.1'
2015-11-09 16:04:31.089 [info] <0.7.0> Application basho_stats started on node 'riak@127.0.0.1'
2015-11-09 16:04:31.093 [info] <0.7.0> Application bitcask started on node 'riak@127.0.0.1'
2015-11-09 16:04:31.101 [info] <0.7.0> Application clique started on node 'riak@127.0.0.1'
2015-11-09 16:04:31.101 [info] <0.7.0> Application eleveldb started on node 'riak@127.0.0.1'
2015-11-09 16:04:31.101 [info] <0.7.0> Application pbkdf2 started on node 'riak@127.0.0.1'
2015-11-09 16:04:31.101 [info] <0.7.0> Application poolboy started on node 'riak@127.0.0.1'
2015-11-09 16:04:31.120 [info] <0.161.0>@exometer_report:do_start_reporters:613 Starting reporters with []
2015-11-09 16:04:31.120 [info] <0.7.0> Application exometer_core started on node 'riak@127.0.0.1'
2015-11-09 16:04:31.254 [info] <0.199.0>@riak_core_capability:process_capability_changes:555 New capability: {riak_core,vnode_routing} = proxy
2015-11-09 16:04:31.261 [info] <0.199.0>@riak_core_capability:process_capability_changes:555 New capability: {riak_core,staged_joins} = true
2015-11-09 16:04:31.269 [info] <0.199.0>@riak_core_capability:process_capability_changes:555 New capability: {riak_core,resizable_ring} = true
2015-11-09 16:04:31.285 [info] <0.199.0>@riak_core_capability:process_capability_changes:555 New capability: {riak_core,fold_req_version} = v2
2015-11-09 16:04:31.301 [info] <0.199.0>@riak_core_capability:process_capability_changes:555 New capability: {riak_core,security} = true
2015-11-09 16:04:31.317 [info] <0.199.0>@riak_core_capability:process_capability_changes:555 New capability: {riak_core,bucket_types} = true
2015-11-09 16:04:31.334 [info] <0.199.0>@riak_core_capability:process_capability_changes:555 New capability: {riak_core,net_ticktime} = true
2015-11-09 16:04:31.426 [info] <0.7.0> Application riak_core started on node 'riak@127.0.0.1'
2015-11-09 16:04:31.430 [info] <0.199.0>@riak_core_capability:process_capability_changes:555 New capability: {riak_pipe,trace_format} = ordsets
2015-11-09 16:04:31.439 [notice] <0.47.0>@lager_app:132 Deprecated lager_file_backend config detected, please consider updating it
2015-11-09 16:04:31.442 [info] <0.7.0> Application riak_pipe started on node 'riak@127.0.0.1'
2015-11-09 16:04:31.444 [info] <0.7.0> Application sidejob started on node 'riak@127.0.0.1'
2015-11-09 16:04:31.445 [info] <0.7.0> Application riak_dt started on node 'riak@127.0.0.1'
2015-11-09 16:04:31.445 [info] <0.7.0> Application protobuffs started on node 'riak@127.0.0.1'
2015-11-09 16:04:31.445 [info] <0.7.0> Application riak_pb started on node 'riak@127.0.0.1'
2015-11-09 16:04:31.450 [warning] <0.305.0>@riak_api_web:get_listeners:40 Setting riak_core/http is deprecated, please use riak_api/http
2015-11-09 16:04:31.465 [info] <0.7.0> Application riak_api started on node 'riak@127.0.0.1'
2015-11-09 16:04:31.494 [info] <0.331.0>@riak_kv_env:doc_env:46 Environment and OS variables:
2015-11-09 16:04:31.539 [info] <0.377.0>@riak_core:wait_for_service:489 Waiting for service riak_kv to start (0 seconds)
2015-11-09 16:04:31.549 [info] <0.385.0>@riak_kv_entropy_manager:set_aae_throttle_limits:790 Setting AAE throttle limits: [{-1,0},{200,10},{500,50},{750,250},{900,1000},{1100,5000}]
2015-11-09 16:04:31.586 [info] <0.394.0>@riak_kv_js_vm:init:73 Spidermonkey VM (thread stack: 16MB, max heap: 8MB, pool: riak_kv_js_map) host starting (<0.394.0>)
2015-11-09 16:04:31.588 [info] <0.395.0>@riak_kv_js_vm:init:73 Spidermonkey VM (thread stack: 16MB, max heap: 8MB, pool: riak_kv_js_map) host starting (<0.395.0>)
2015-11-09 16:04:31.591 [info] <0.396.0>@riak_kv_js_vm:init:73 Spidermonkey VM (thread stack: 16MB, max heap: 8MB, pool: riak_kv_js_map) host starting (<0.396.0>)
2015-11-09 16:04:31.593 [info] <0.397.0>@riak_kv_js_vm:init:73 Spidermonkey VM (thread stack: 16MB, max heap: 8MB, pool: riak_kv_js_map) host starting (<0.397.0>)
2015-11-09 16:04:31.596 [info] <0.398.0>@riak_kv_js_vm:init:73 Spidermonkey VM (thread stack: 16MB, max heap: 8MB, pool: riak_kv_js_map) host starting (<0.398.0>)
2015-11-09 16:04:31.598 [info] <0.399.0>@riak_kv_js_vm:init:73 Spidermonkey VM (thread stack: 16MB, max heap: 8MB, pool: riak_kv_js_map) host starting (<0.399.0>)
2015-11-09 16:04:31.600 [info] <0.400.0>@riak_kv_js_vm:init:73 Spidermonkey VM (thread stack: 16MB, max heap: 8MB, pool: riak_kv_js_map) host starting (<0.400.0>)
2015-11-09 16:04:31.603 [info] <0.401.0>@riak_kv_js_vm:init:73 Spidermonkey VM (thread stack: 16MB, max heap: 8MB, pool: riak_kv_js_map) host starting (<0.401.0>)
2015-11-09 16:04:31.606 [info] <0.403.0>@riak_kv_js_vm:init:73 Spidermonkey VM (thread stack: 16MB, max heap: 8MB, pool: riak_kv_js_reduce) host starting (<0.403.0>)
2015-11-09 16:04:31.609 [info] <0.404.0>@riak_kv_js_vm:init:73 Spidermonkey VM (thread stack: 16MB, max heap: 8MB, pool: riak_kv_js_reduce) host starting (<0.404.0>)
2015-11-09 16:04:31.612 [info] <0.405.0>@riak_kv_js_vm:init:73 Spidermonkey VM (thread stack: 16MB, max heap: 8MB, pool: riak_kv_js_reduce) host starting (<0.405.0>)
2015-11-09 16:04:31.615 [info] <0.406.0>@riak_kv_js_vm:init:73 Spidermonkey VM (thread stack: 16MB, max heap: 8MB, pool: riak_kv_js_reduce) host starting (<0.406.0>)
2015-11-09 16:04:31.618 [info] <0.407.0>@riak_kv_js_vm:init:73 Spidermonkey VM (thread stack: 16MB, max heap: 8MB, pool: riak_kv_js_reduce) host starting (<0.407.0>)
2015-11-09 16:04:31.622 [info] <0.408.0>@riak_kv_js_vm:init:73 Spidermonkey VM (thread stack: 16MB, max heap: 8MB, pool: riak_kv_js_reduce) host starting (<0.408.0>)
2015-11-09 16:04:31.625 [info] <0.410.0>@riak_kv_js_vm:init:73 Spidermonkey VM (thread stack: 16MB, max heap: 8MB, pool: riak_kv_js_hook) host starting (<0.410.0>)
2015-11-09 16:04:31.628 [info] <0.411.0>@riak_kv_js_vm:init:73 Spidermonkey VM (thread stack: 16MB, max heap: 8MB, pool: riak_kv_js_hook) host starting (<0.411.0>)
2015-11-09 16:04:31.632 [info] <0.199.0>@riak_core_capability:process_capability_changes:555 New capability: {riak_kv,vnode_vclocks} = true
2015-11-09 16:04:31.634 [info] <0.199.0>@riak_core_capability:process_capability_changes:555 New capability: {riak_kv,legacy_keylisting} = false
2015-11-09 16:04:31.635 [info] <0.199.0>@riak_core_capability:process_capability_changes:555 New capability: {riak_kv,listkeys_backpressure} = true
2015-11-09 16:04:31.637 [info] <0.199.0>@riak_core_capability:process_capability_changes:555 New capability: {riak_kv,index_backpressure} = true
2015-11-09 16:04:31.639 [info] <0.199.0>@riak_core_capability:process_capability_changes:555 New capability: {riak_kv,mapred_system} = pipe
2015-11-09 16:04:31.646 [info] <0.199.0>@riak_core_capability:process_capability_changes:555 New capability: {riak_kv,mapred_2i_pipe} = true
2015-11-09 16:04:31.648 [info] <0.199.0>@riak_core_capability:process_capability_changes:555 New capability: {riak_kv,anti_entropy} = enabled_v1
2015-11-09 16:04:31.650 [info] <0.199.0>@riak_core_capability:process_capability_changes:555 New capability: {riak_kv,handoff_data_encoding} = encode_raw
2015-11-09 16:04:31.652 [info] <0.199.0>@riak_core_capability:process_capability_changes:555 New capability: {riak_kv,object_format} = v1
2015-11-09 16:04:31.653 [info] <0.199.0>@riak_core_capability:process_capability_changes:555 New capability: {riak_kv,secondary_index_version} = v3
2015-11-09 16:04:31.655 [info] <0.199.0>@riak_core_capability:process_capability_changes:555 New capability: {riak_kv,vclock_data_encoding} = encode_zlib
2015-11-09 16:04:31.657 [info] <0.199.0>@riak_core_capability:process_capability_changes:555 New capability: {riak_kv,crdt} = [pncounter,riak_dt_pncounter,riak_dt_orswot,riak_dt_map]
2015-11-09 16:04:31.659 [info] <0.199.0>@riak_core_capability:process_capability_changes:555 New capability: {riak_kv,crdt_epoch_versions} = [{riak_dt_map,2},{riak_dt_orswot,2},{riak_dt_pncounter,2}]
2015-11-09 16:04:31.661 [info] <0.199.0>@riak_core_capability:process_capability_changes:555 New capability: {riak_kv,put_fsm_ack_execute} = enabled
2015-11-09 16:04:31.705 [info] <0.199.0>@riak_core_capability:process_capability_changes:555 New capability: {riak_control,member_info_version} = v1
2015-11-09 16:05:32.139 [info] <0.377.0>@riak_core:wait_for_service:489 Waiting for service riak_kv to start (60 seconds)
2015-11-09 16:06:32.739 [info] <0.377.0>@riak_core:wait_for_service:489 Waiting for service riak_kv to start (120 seconds)
2015-11-09 16:07:33.339 [info] <0.377.0>@riak_core:wait_for_service:489 Waiting for service riak_kv to start (180 seconds)
2015-11-09 16:08:33.939 [info] <0.377.0>@riak_core:wait_for_service:489 Waiting for service riak_kv to start (240 seconds)
2015-11-09 16:09:34.539 [info] <0.377.0>@riak_core:wait_for_service:489 Waiting for service riak_kv to start (300 seconds)
2015-11-09 16:10:35.139 [info] <0.377.0>@riak_core:wait_for_service:489 Waiting for service riak_kv to start (360 seconds)
2015-11-09 16:11:35.739 [info] <0.377.0>@riak_core:wait_for_service:489 Waiting for service riak_kv to start (420 seconds)
2015-11-09 16:12:36.339 [info] <0.377.0>@riak_core:wait_for_service:489 Waiting for service riak_kv to start (480 seconds)
2015-11-09 16:13:36.939 [info] <0.377.0>@riak_core:wait_for_service:489 Waiting for service riak_kv to start (540 seconds)
2015-11-09 16:14:37.539 [info] <0.377.0>@riak_core:wait_for_service:489 Waiting for service riak_kv to start (600 seconds)
2015-11-09 16:15:38.139 [info] <0.377.0>@riak_core:wait_for_service:489 Waiting for service riak_kv to start (660 seconds)

[JeetKunDoug]

Can you try a few things for me:

1. See if you have anything in crash.log or error.log, which may help us understand what's going on.
2. Run riak as riak console, which will start riak but keep all its output on the console, and see if you get any further information.

Thanks,

Doug

[Zerebokep]

1. The last few errors in the error log are:

2015-11-08 23:20:15.760 [error] <0.229.0> Supervisor riak_pipe_fitting_sup had child undefined started with riak_pipe_fitting:start_link() at <0.5108.3055> exit with reason noproc in context shutdown_error
2015-11-08 23:21:01.390 [error] <0.229.0> Supervisor riak_pipe_fitting_sup had child undefined started with riak_pipe_fitting:start_link() at <0.8717.3055> exit with reason noproc in context shutdown_error
2015-11-08 23:25:46.886 [error] <0.229.0> Supervisor riak_pipe_fitting_sup had child undefined started with riak_pipe_fitting:start_link() at <0.27331.3055> exit with reason noproc in context shutdown_error
2015-11-09 15:57:56.647 [error] <0.10544.7>@riak_kv_console:status:185 Status failed exit:{shutdown,{gen_server,call,[riak_core_vnode_manager,{all_vnodes,riak_kv_vnode},infinity]}}

p.s. The riak_pipe_fitting_sup error exists since we use riak and should be unrelated to this issue.

crash log:

2015-11-08 23:17:35 =SUPERVISOR REPORT====
Supervisor: {local,riak_pipe_fitting_sup}
Context: shutdown_error
Reason: noproc
Offender: [{pid,<0.23412.3054>},{name,undefined},{mfargs,{riak_pipe_fitting,start_link,[]}},{restart_type,temporary},{shutdown,2000},{child_type,worker}]

2015-11-08 23:20:15 =SUPERVISOR REPORT====
Supervisor: {local,riak_pipe_fitting_sup}
Context: shutdown_error
Reason: noproc
Offender: [{pid,<0.5108.3055>},{name,undefined},{mfargs,{riak_pipe_fitting,start_link,[]}},{restart_type,temporary},{shutdown,2000},{child_type,worker}]

2015-11-08 23:21:01 =SUPERVISOR REPORT====
Supervisor: {local,riak_pipe_fitting_sup}
Context: shutdown_error
Reason: noproc
Offender: [{pid,<0.8717.3055>},{name,undefined},{mfargs,{riak_pipe_fitting,start_link,[]}},{restart_type,temporary},{shutdown,2000},{child_type,worker}]

2015-11-08 23:25:46 =SUPERVISOR REPORT====
Supervisor: {local,riak_pipe_fitting_sup}
Context: shutdown_error
Reason: noproc
Offender: [{pid,<0.27331.3055>},{name,undefined},{mfargs,{riak_pipe_fitting,start_link,[]}},{restart_type,temporary},{shutdown,2000},{child_type,worker}]

riak console

config is OK
-config /etc/riak/app.config -args_file /etc/riak/vm.args -vm_args /etc/riak/vm.args
Exec: /usr/lib/riak/erts-5.10.3/bin/erlexec -boot /usr/lib/riak/releases/2.0.6/riak -config /etc/riak/app.config -args_file /etc/riak/vm.args -vm_args /etc/riak/vm.args -pa /usr/lib/riak/lib/basho-patches -- console
Root: /usr/lib/riak
Erlang R16B02_basho8 (erts-5.10.3) [source] [64-bit] [smp:8:8] [async-threads:16] [kernel-poll:true] [frame-pointer]

Eshell V5.10.3 (abort with ^G)
(riak@127.0.0.1)1>

1. I've also found this one in our kern.log:

beam.smp[3150]: segfault at 0 ip 00007feea1acb8ed sp 00007fee972cfbb0 error 4 in eleveldb.so[7feea1a75000+85000]

It's a debian server, running with riak 2.0.6.

[Zerebokep]

I'm not sure if this helps, but here is some output of riak top:

'riak@127.0.0.1' 09:21:44
Load: cpu 0 Memory: total 79619 binary 208
procs 1391 processes 41132 code 13421
runq 0 atom 549 ets 6949

Pid Name or Initial Func Time Reds Memory MsgQ Current Function

<6410.92.0> riak_sysmon_filter '-' 2813 2744 0 gen_server:loop/6
<6410.103.0> erlang:apply/2 '-' 2351 2600 0 cpu_sup:measurement_server_loop/1
<6410.341.0> riak_kv_put_fsm_sj_stats '-' 1112 5720 0 gen_server:loop/6
<6410.365.0> riak_kv_stat_sj_stats '-' 1090 5720 0 gen_server:loop/6
<6410.353.0> riak_kv_get_fsm_sj_stats '-' 1064 5720 0 gen_server:loop/6
<6410.308.0> http://127.0.0.1:8098_moc '-' 936 4528 0 gen_server:loop/6
<6410.349.0> riak_kv_get_fsm_sj_5 '-' 714 3848 0 gen_server:loop/6
<6410.135.0> webmachine_log_event '-' 374 21528 0 gen_event:fetch_msg/5
<6410.372.0> riak_kv_vnode_master '-' 264 2704 0 timer:sleep/1
<6410.2460.0> mochiweb_acceptor:init/3 '-' 109 2776 0 prim_inet:accept0/2

'riak@127.0.0.1' 09:21:49
Load: cpu 0 Memory: total 79644 binary 233
procs 1391 processes 41134 code 13421
runq 0 atom 549 ets 6949

Pid Name or Initial Func Time Reds Memory MsgQ Current Function

<6410.92.0> riak_sysmon_filter '-' 2607 3888 0 gen_server:loop/6
<6410.308.0> http://127.0.0.1:8098_moc '-' 2571 6400 0 gen_server:loop/6
<6410.349.0> riak_kv_get_fsm_sj_5 '-' 1641 3848 0 gen_server:loop/6
<6410.341.0> riak_kv_put_fsm_sj_stats '-' 1379 5720 0 gen_server:loop/6
<6410.365.0> riak_kv_stat_sj_stats '-' 1360 8736 0 gen_server:loop/6
<6410.353.0> riak_kv_get_fsm_sj_stats '-' 1321 5720 0 gen_server:loop/6
<6410.135.0> webmachine_log_event '-' 1122 21528 0 gen_event:fetch_msg/5
<6410.348.0> riak_kv_get_fsm_sj_4 '-' 435 3848 0 gen_server:loop/6
<6410.372.0> riak_kv_vnode_master '-' 272 2704 0 timer:sleep/1
<6410.20.0> net_kernel '-' 156 4048 0 gen_server:loop/6

'riak@127.0.0.1' 09:21:53
Load: cpu 0 Memory: total 79632 binary 232
procs 1391 processes 41102 code 13421
runq 0 atom 549 ets 6949

Pid Name or Initial Func Time Reds Memory MsgQ Current Function

<6410.3.0> erl_prim_loader '-' 15777 142624 0 erl_prim_loader:loop/3
<6410.92.0> riak_sysmon_filter '-' 2813 3888 0 gen_server:loop/6
<6410.103.0> erlang:apply/2 '-' 2351 2600 0 cpu_sup:measurement_server_loop/1
<6410.195.0> riak_core_capability '-' 2312 21528 0 gen_server:loop/6
<6410.197.0> riak_core_claimant '-' 2111 88504 0 gen_server:loop/6
<6410.353.0> riak_kv_get_fsm_sj_stats '-' 1117 5720 0 gen_server:loop/6
<6410.341.0> riak_kv_put_fsm_sj_stats '-' 1112 5720 0 gen_server:loop/6
<6410.365.0> riak_kv_stat_sj_stats '-' 1090 8736 0 gen_server:loop/6
<6410.308.0> http://127.0.0.1:8098_moc '-' 1079 6400 0 gen_server:loop/6
<6410.349.0> riak_kv_get_fsm_sj_5 '-' 844 3848 0 gen_server:loop/6

'riak@127.0.0.1' 09:21:58
Load: cpu 0 Memory: total 79630 binary 232
procs 1391 processes 41120 code 13421
runq 0 atom 549 ets 6949

Pid Name or Initial Func Time Reds Memory MsgQ Current Function

<6410.308.0> http://127.0.0.1:8098_moc '-' 5045 6400 0 gen_server:loop/6
<6410.349.0> riak_kv_get_fsm_sj_5 '-' 3400 2704 0 gen_server:loop/6
<6410.103.0> erlang:apply/2 '-' 2351 2600 0 cpu_sup:measurement_server_loop/1
<6410.135.0> webmachine_log_event '-' 2244 13624 0 gen_event:fetch_msg/5
<6410.341.0> riak_kv_put_fsm_sj_stats '-' 1403 5720 0 gen_server:loop/6
<6410.365.0> riak_kv_stat_sj_stats '-' 1367 5720 0 gen_server:loop/6
<6410.353.0> riak_kv_get_fsm_sj_stats '-' 1346 5720 0 gen_server:loop/6
<6410.92.0> riak_sysmon_filter '-' 806 2744 0 gen_server:loop/6
<6410.348.0> riak_kv_get_fsm_sj_4 '-' 536 2704 0 gen_server:loop/6
<6410.20.0> net_kernel '-' 305 4048 0 gen_server:loop/6

'riak@127.0.0.1' 09:22:02
Load: cpu 0 Memory: total 79634 binary 231
procs 1391 processes 41124 code 13421
runq 0 atom 549 ets 6949

Pid Name or Initial Func Time Reds Memory MsgQ Current Function

<6410.3.0> erl_prim_loader '-' 15207 142624 0 erl_prim_loader:loop/3
<6410.92.0> riak_sysmon_filter '-' 2813 2744 0 gen_server:loop/6
<6410.195.0> riak_core_capability '-' 2494 21528 0 gen_server:loop/6
<6410.103.0> erlang:apply/2 '-' 2351 2600 0 cpu_sup:measurement_server_loop/1
<6410.197.0> riak_core_claimant '-' 2111 88504 0 gen_server:loop/6
<6410.353.0> riak_kv_get_fsm_sj_stats '-' 1117 5720 0 gen_server:loop/6
<6410.341.0> riak_kv_put_fsm_sj_stats '-' 1112 5720 0 gen_server:loop/6
<6410.365.0> riak_kv_stat_sj_stats '-' 1090 5720 0 gen_server:loop/6
<6410.308.0> http://127.0.0.1:8098_moc '-' 430 6400 0 gen_server:loop/6
<6410.349.0> riak_kv_get_fsm_sj_5 '-' 426 2704 0 gen_server:loop/6

[russelldb]

Nope, and that is probably you�re problem, leveldb failing to get a lock, and backing off, and trying again.

On 10 Nov 2015, at 11:28, Zerebokep notifications@github.com wrote:

Another thing, in /var/lib/riak/leveldb there are some files/folders owned by root instead of riak, is this normal?

The files are:

-rw-r--r-- 1 root root 0 Nov 10 10:09 000002.log
-rw-r--r-- 1 root root 16 Nov 10 10:09 CURRENT
-rw-r--r-- 1 root root 0 Nov 10 10:09 LOCK
-rw-r--r-- 1 root root 2.4K Nov 10 10:09 LOG
-rw-r--r-- 1 root root 148 Nov 10 10:09 LOG.old
-rw-r--r-- 1 root root 50 Nov 10 10:09 MANIFEST-000001
drwxr-xr-x 2 root root 4.0K Nov 10 10:09 sst_0
drwxr-xr-x 2 root root 4.0K Nov 10 10:09 sst_1
drwxr-xr-x 2 root root 4.0K Nov 10 10:09 sst_2
drwxr-xr-x 2 root root 4.0K Nov 10 10:09 sst_3
drwxr-xr-x 2 root root 4.0K Nov 10 10:09 sst_4
drwxr-xr-x 2 root root 4.0K Nov 10 10:09 sst_5
drwxr-xr-x 2 root root 4.0K Nov 10 10:09 sst_6

�
Reply to this email directly or view it on GitHub.

[Zerebokep]

I've changed the permissions, but it doesn't seem to solve the problem.

[Zerebokep]

That's weird, even the status commands are telling me that everything is fine.

riak-admin member-status

================================= Membership ==================================

Status Ring Pending Node

valid 100.0% -- 'riak@127.0.0.1'

Valid:1 / Leaving:0 / Exiting:0 / Joining:0 / Down:0

riak-admin ring-status

================================== Claimant ===================================
Claimant: 'riak@127.0.0.1'
Status: up
Ring Ready: true

============================== Ownership Handoff ==============================
No pending changes.

============================== Unreachable Nodes ==============================
All nodes are up and reachable

[hmmr]

To me Segfault at 0 suggests it's a NULL pointer dereference, somewhere in eleveldb C++ code. @mvm?

[Zerebokep]

Okay guys, problem is solved. I've repaired all leveldb partitions and riak started normally. It seems there is some reporting issue with riak when something is wrong with an leveldb partition.

[DeadZen]

Did you do a eleveldb:repair to resolve this?