Yes, I am on a train now. I will do it tonight from home.
If it is the “known mono gc bug”, is there a workaround?
Mirko
I am not a lucky guy today. I get a SIGSEGV but after 1 second, so it seems that my gdb is not working fine.
gdb mono
GNU gdb (Ubuntu/Linaro 7.4-2012.04-0ubuntu2.1) 7.4-2012.04
Copyright © 2012 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type “show copying”
and “show warranty” for details.
This GDB was configured as “x86_64-linux-gnu”.
For bug reporting instructions, please see:
http://bugs.launchpad.net/gdb-linaro/…
Reading symbols from /data/mono/bin/mono…done.
Mono support loaded.
(gdb) handle SIGXCPU SIG33 SIG35 SIGPWR nostop noprint
Signal Stop Print Pass to program Description
SIGXCPU No No Yes CPU time limit exceeded
SIGPWR No No Yes Power fail/restart
SIG33 No No Yes Real-time event 33
SIG35 No No Yes Real-time event 35
(gdb) run --debug EventStore.SingleNode.exe --run-projections=ALL
Starting program: /data/mono/bin/mono --debug EventStore.SingleNode.exe --run-projections=ALL
[Thread debugging using libthread_db enabled]
Using host libthread_db library “/lib/x86_64-linux-gnu/libthread_db.so.1”.
[New Thread 0x7ffff70f5700 (LWP 21642)]
[New Thread 0x7ffff5e91700 (LWP 21643)]
Program received signal SIGSEGV, Segmentation fault.
0x00007ffff723f9d6 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
(gdb)
If you have any other hint on how to provide you a backtrace I am logged into the machine now.
Mirko
a gdb prompt type “backtrace”
Did you set the mono specific signal handlers up in gdb?
Sadly the current workaround is to use .NET! Xamarin have been working to try to find the bugs in the GC, but not sure if or when fixes will hit. Alternatively you need a supervisor to restart it in the case of crashes.
Oh yes 
But it seems to crash too fast. The ES without gdb last about 10 minutes, inside gdb it crashed in a second.
Here is the backtrace:
(gdb) backtrace
#0 0x00007ffff723f9d6 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00000000004bd1fb in emit_line_number_info (w=0xa53040, method=":runtime_invoke_void__this___object ()",
start_symbol=0x0, code=0x40013aa0 "UH\213\354AUAVAWH\203\354\070L\213\357L\213\366H\211U\260L\213\371H\307E\300",
code_size=173, end_symbol=<optimized out>, debug_info=<optimized out>)
at /usr/include/x86_64-linux-gnu/bits/string3.h:52
#2 0x00000000004be7db in emit_all_line_number_info (w=0xa53040) at dwarfwriter.c:886
#3 mono_dwarf_writer_close (w=0xa53040) at dwarfwriter.c:1032
#4 0x00000000004d6d62 in xdebug_end_emit (w=0xa533a0, dw=0xa53040, method=) at xdebug.c:200
#5 0x00000000004d6f3c in mono_xdebug_flush () at xdebug.c:253
#6 0x00000000004d70c5 in mono_save_xdebug_info (cfg=0xc29030) at xdebug.c:283
#7 0x000000000041f585 in mini_method_compile (method=“System.Configuration.ConfigInfoCollection:.ctor ()”, opts=
101804543, domain=0x7ffff7e38cc0, run_cctors=<optimized out>, compile_aot=0, parts=0) at mini.c:5432
#8 0x0000000000420e10 in mono_jit_compile_method_inner (jit_ex=0x7fffffffbb18, opt=101804543, target_domain=
0x7ffff7e38cc0, method="System.Configuration.ConfigInfoCollection:.ctor ()") at mini.c:5735
#9 mono_jit_compile_method_with_opt (method=“System.Configuration.ConfigInfoCollection:.ctor ()”, opt=101804543, ex=
0x7fffffffbb18) at mini.c:5999
#10 0x000000000042192d in mono_jit_compile_method (method=) at mini.c:6027
#11 0x00000000004a8c81 in common_call_trampoline (regs=0x7fffffffbde8, code=
0x4001912a "H\213\f$H\270\300\236\351\365\377\177", m="System.Configuration.ConfigInfoCollection:.ctor ()", vt=
0x0, vtable_slot=<optimized out>, need_rgctx_tramp=0, tramp=<optimized out>) at mini-trampolines.c:568
#12 0x0000000040002186 in ?? ()
#13 0x000000004001912a in ?? ()
#14 0x00007ffff5ea19c0 in ?? ()
#15 0x0000000040013e09 in ?? ()
#16 0x00007fffffffc108 in ?? ()
#17 0x000000000060bf9a in monoeg_g_hash_table_insert_replace (hash=0x7fffffffbf00, key=0xc753f0, value=0x7fffffffc108,
replace=13114920) at ghashtable.c:258
#18 0x0000000000421ce8 in mono_jit_runtime_invoke (method=
<error reading variable: Cannot access memory at address 0x8b4cb05589484934>, obj=0x0, params=0x0, exc=
0x7fffffffc108) at mini.c:6388
#19 0x00000000005a52eb in mono_runtime_invoke (method=“System.Configuration.SectionGroupInfo:.cctor ()”, obj=0x0,
params=0x0, exc=0x7fffffffc108) at object.c:2827
#20 0x00000000005a553d in mono_runtime_class_init_full (vtable=vtable(“System.Configuration.SectionGroupInfo”),
raise_exception=0) at object.c:376
#21 0x0000000000421251 in mono_jit_compile_method_inner (jit_ex=0x7fffffffc228, opt=, target_domain=
0x7ffff7e38cc0, method="System.Configuration.SectionGroupInfo:.ctor ()") at mini.c:5929
#22 mono_jit_compile_method_with_opt (method=“System.Configuration.SectionGroupInfo:.ctor ()”, opt=, ex=
0x7fffffffc228) at mini.c:5999
#23 0x000000000042192d in mono_jit_compile_method (method=) at mini.c:6027
#24 0x00000000004a8c81 in common_call_trampoline (regs=0x7fffffffc4f8, code=
0x40018ee3 "H\213D$ I\211D$8I\213L$ \203\070", m="System.Configuration.SectionGroupInfo:.ctor ()", vt=0x0,
vtable_slot=<optimized out>, need_rgctx_tramp=0, tramp=<optimized out>) at mini-trampolines.c:568
#25 0x0000000040002186 in ?? ()
#26 0x0000000040018ee3 in ?? ()
#27 0x00007fffffffcaf8 in ?? ()
#28 0x0000000000000000 in ?? ()
Are these?
(gdb) handle SIGXCPU SIG33 SIG35 SIGPWR nostop noprint
The problem is that in this case I am getting a crash every 10 minutes after the restart, it seems that it hasn’t finished to read events forward.
That’s an interesting one, we’ve previously only seen it under very heavy write load. Can you try with a newer version of mono? (3.2.8 would be ideal)
Sure, tomorrow I will update mono.
I am only surprised of the quick crash under gdb. Sure it is a crash due to ES in this example?
What do you mean for " very heavy write load", I have to rewrite tons of streams next week.
Mirko
So the places we’ve seen segfaults from mono before are after approx 12-14 millions writes depending on OS. The backtraces indicate that sgen is crashing during nursery collection. Out of interest do you have projections turned on? It’s a lot harder to
make it crash if they’re off.
Yes they are on, I need them.
Now I have tried to turn them completely OFF.
So that each 12-14M of writes it crashed, then you restart it and it is able to catch up and go on for about the same number of writes?
Mirko
I can confirm that with “run-projections=NONE” it is up and running.
Holy crap I am not able to restart my ES with projections enabled.
Damn Mono. Tomorrow I will update it.
Mirko
Hi James, is it normal to have such messages in dmesg?
[1457442.598720] XFS (dm-0): xlog_space_left: head behind tail
[1457442.598720] tail_cycle = 31, tail_bytes = 193352704
[1457442.598721] GH cycle = 31, GH bytes = 193352512
[1457463.210609] XFS (dm-0): xlog_space_left: head behind tail
[1457463.210610] tail_cycle = 31, tail_bytes = 193406976
[1457463.210611] GH cycle = 31, GH bytes = 193406784
[1457463.215614] XFS (dm-0): xlog_space_left: head behind tail
[1457463.215615] tail_cycle = 31, tail_bytes = 193406976
[1457463.215617] GH cycle = 31, GH bytes = 193406784
[1457470.134410] XFS (dm-0): xlog_space_left: head behind tail
[1457470.134412] tail_cycle = 31, tail_bytes = 193432576
[1457470.134412] GH cycle = 31, GH bytes = 193430336
[1457470.139665] XFS (dm-0): xlog_space_left: head behind tail
[1457470.139666] tail_cycle = 31, tail_bytes = 193432576
[1457470.139667] GH cycle = 31, GH bytes = 193430336
[1457470.147451] XFS (dm-0): xlog_space_left: head behind tail
[1457470.147452] tail_cycle = 31, tail_bytes = 193432576
[1457470.147453] GH cycle = 31, GH bytes = 193431872
[1457470.154150] XFS (dm-0): xlog_space_left: head behind tail
[1457470.154151] tail_cycle = 31, tail_bytes = 193432576
[1457470.154152] GH cycle = 31, GH bytes = 193431872
[1457470.161630] XFS (dm-0): xlog_space_left: head behind tail
[1457470.161631] tail_cycle = 31, tail_bytes = 193433600
[1457470.161632] GH cycle = 31, GH bytes = 193432896
[1457470.161975] XFS (dm-0): xlog_space_left: head behind tail
[1457470.161976] tail_cycle = 31, tail_bytes = 193433600
[1457470.161977] GH cycle = 31, GH bytes = 193432896
[1457470.161978] XFS (dm-0): xlog_space_left: head behind tail
[1457470.161979] tail_cycle = 31, tail_bytes = 193433600
[1457470.161980] GH cycle = 31, GH bytes = 193432896
[1457481.317227] XFS (dm-0): xlog_space_left: head behind tail
[1457481.317229] tail_cycle = 31, tail_bytes = 193566720
[1457481.317229] GH cycle = 31, GH bytes = 193566528
[1457481.322646] XFS (dm-0): xlog_space_left: head behind tail
[1457481.322647] tail_cycle = 31, tail_bytes = 193566720
[1457481.322648] GH cycle = 31, GH bytes = 193566528
[1457481.330120] XFS (dm-0): xlog_space_left: head behind tail
[1457481.330121] tail_cycle = 31, tail_bytes = 193567744
[1457481.330122] GH cycle = 31, GH bytes = 193567552
[1457481.336592] XFS (dm-0): xlog_space_left: head behind tail
[1457481.336593] tail_cycle = 31, tail_bytes = 193567744
[1457481.336594] GH cycle = 31, GH bytes = 193567552
[1457501.195399] XFS (dm-0): xlog_space_left: head behind tail
[1457501.195400] tail_cycle = 31, tail_bytes = 193817088
[1457501.195401] GH cycle = 31, GH bytes = 193814848
[1457501.200529] XFS (dm-0): xlog_space_left: head behind tail
[1457501.200530] tail_cycle = 31, tail_bytes = 193817088
[1457501.200531] GH cycle = 31, GH bytes = 193814848
[1457501.211545] XFS (dm-0): xlog_space_left: head behind tail
[1457501.211546] tail_cycle = 31, tail_bytes = 193817088
[1457501.211547] GH cycle = 31, GH bytes = 193815872
[1457501.217956] XFS (dm-0): xlog_space_left: head behind tail
[1457501.217957] tail_cycle = 31, tail_bytes = 193817088
[1457501.217958] GH cycle = 31, GH bytes = 193815872
[1457501.224853] XFS (dm-0): xlog_space_left: head behind tail
[1457501.224854] tail_cycle = 31, tail_bytes = 193818112
[1457501.224854] GH cycle = 31, GH bytes = 193816896
[1457501.225004] XFS (dm-0): xlog_space_left: head behind tail
[1457501.225005] tail_cycle = 31, tail_bytes = 193818112
[1457501.225006] GH cycle = 31, GH bytes = 193816896
[1457501.225009] XFS (dm-0): xlog_space_left: head behind tail
[1457501.225010] tail_cycle = 31, tail_bytes = 193818112
[1457501.225010] GH cycle = 31, GH bytes = 193816896
[1457501.225825] XFS (dm-0): xlog_space_left: head behind tail
[1457501.225826] tail_cycle = 31, tail_bytes = 193819136
[1457501.225827] GH cycle = 31, GH bytes = 193817920
[1457501.225829] XFS (dm-0): xlog_space_left: head behind tail
[1457501.225830] tail_cycle = 31, tail_bytes = 193819136
[1457501.225831] GH cycle = 31, GH bytes = 193817920
[1457501.255808] XFS (dm-0): xlog_space_left: head behind tail
[1457501.255810] tail_cycle = 31, tail_bytes = 193820160
[1457501.255812] GH cycle = 31, GH bytes = 193818944
[1457501.263169] XFS (dm-0): xlog_space_left: head behind tail
[1457501.263170] tail_cycle = 31, tail_bytes = 193820160
[1457501.263170] GH cycle = 31, GH bytes = 193819968
[1457501.269546] XFS (dm-0): xlog_space_left: head behind tail
[1457501.269547] tail_cycle = 31, tail_bytes = 193820160
[1457501.269548] GH cycle = 31, GH bytes = 193819968
[1457501.305213] XFS (dm-0): xlog_space_left: head behind tail
[1457501.305214] tail_cycle = 31, tail_bytes = 193823232
[1457501.305215] GH cycle = 31, GH bytes = 193823040
[1457501.305396] XFS (dm-0): xlog_space_left: head behind tail
[1457501.305397] tail_cycle = 31, tail_bytes = 193823232
[1457501.305398] GH cycle = 31, GH bytes = 193823040
[1457501.305399] XFS (dm-0): xlog_space_left: head behind tail
[1457501.305400] tail_cycle = 31, tail_bytes = 193823232
[1457501.305400] GH cycle = 31, GH bytes = 193823040
[1457501.325640] XFS (dm-0): xlog_space_left: head behind tail
[1457501.325641] tail_cycle = 31, tail_bytes = 193826304
[1457501.325642] GH cycle = 31, GH bytes = 193826112
[1457501.332593] XFS (dm-0): xlog_space_left: head behind tail
[1457501.332595] tail_cycle = 31, tail_bytes = 193826304
[1457501.332595] GH cycle = 31, GH bytes = 193826112
[1457501.364494] XFS (dm-0): xlog_space_left: head behind tail
[1457501.364495] tail_cycle = 31, tail_bytes = 193829376
[1457501.364496] GH cycle = 31, GH bytes = 193829184
[1457501.364698] XFS (dm-0): xlog_space_left: head behind tail
[1457501.364699] tail_cycle = 31, tail_bytes = 193829376
[1457501.364700] GH cycle = 31, GH bytes = 193829184
[1457501.364701] XFS (dm-0): xlog_space_left: head behind tail
[1457501.364702] tail_cycle = 31, tail_bytes = 193829376
[1457501.364703] GH cycle = 31, GH bytes = 193829184
[1457501.385547] XFS (dm-0): xlog_space_left: head behind tail
[1457501.385548] tail_cycle = 31, tail_bytes = 193832448
[1457501.385549] GH cycle = 31, GH bytes = 193832256
[1457501.391853] XFS (dm-0): xlog_space_left: head behind tail
[1457501.391854] tail_cycle = 31, tail_bytes = 193832448
[1457501.391855] GH cycle = 31, GH bytes = 193832256
[1457501.427103] XFS (dm-0): xlog_space_left: head behind tail
[1457501.427104] tail_cycle = 31, tail_bytes = 193835520
[1457501.427105] GH cycle = 31, GH bytes = 193835328
[1457501.433681] XFS (dm-0): xlog_space_left: head behind tail
[1457501.433682] tail_cycle = 31, tail_bytes = 193835520
[1457501.433683] GH cycle = 31, GH bytes = 193835328
[1457501.441604] XFS (dm-0): xlog_space_left: head behind tail
[1457501.441605] tail_cycle = 31, tail_bytes = 193836544
[1457501.441605] GH cycle = 31, GH bytes = 193836352
[1457501.447969] XFS (dm-0): xlog_space_left: head behind tail
[1457501.447970] tail_cycle = 31, tail_bytes = 193836544
[1457501.447971] GH cycle = 31, GH bytes = 193836352
[1457530.351059] XFS (dm-0): xlog_space_left: head behind tail
[1457530.351061] tail_cycle = 31, tail_bytes = 193923584
[1457530.351061] GH cycle = 31, GH bytes = 193921344
[1457530.356045] XFS (dm-0): xlog_space_left: head behind tail
[1457530.356046] tail_cycle = 31, tail_bytes = 193923584
[1457530.356047] GH cycle = 31, GH bytes = 193921344
[1457530.363287] XFS (dm-0): xlog_space_left: head behind tail
[1457530.363287] tail_cycle = 31, tail_bytes = 193923584
[1457530.363288] GH cycle = 31, GH bytes = 193922368
[1457530.369332] XFS (dm-0): xlog_space_left: head behind tail
[1457530.369332] tail_cycle = 31, tail_bytes = 193923584
[1457530.369333] GH cycle = 31, GH bytes = 193922368
[1457530.376705] XFS (dm-0): xlog_space_left: head behind tail
[1457530.376706] tail_cycle = 31, tail_bytes = 193924608
[1457530.376707] GH cycle = 31, GH bytes = 193923392
[1457530.376893] XFS (dm-0): xlog_space_left: head behind tail
[1457530.376894] tail_cycle = 31, tail_bytes = 193924608
[1457530.376894] GH cycle = 31, GH bytes = 193923392
[1457530.376897] XFS (dm-0): xlog_space_left: head behind tail
[1457530.376898] tail_cycle = 31, tail_bytes = 193924608
[1457530.376898] GH cycle = 31, GH bytes = 193923392
[1457530.377731] XFS (dm-0): xlog_space_left: head behind tail
[1457530.377732] tail_cycle = 31, tail_bytes = 193925632
[1457530.377733] GH cycle = 31, GH bytes = 193924416
[1457530.377735] XFS (dm-0): xlog_space_left: head behind tail
[1457530.377736] tail_cycle = 31, tail_bytes = 193925632
[1457530.377737] GH cycle = 31, GH bytes = 193924416
[1457530.406510] XFS (dm-0): xlog_space_left: head behind tail
[1457530.406511] tail_cycle = 31, tail_bytes = 193926656
[1457530.406511] GH cycle = 31, GH bytes = 193925440
[1457530.413453] XFS (dm-0): xlog_space_left: head behind tail
[1457530.413454] tail_cycle = 31, tail_bytes = 193926656
[1457530.413455] GH cycle = 31, GH bytes = 193926464
[1457530.419331] XFS (dm-0): xlog_space_left: head behind tail
[1457530.419332] tail_cycle = 31, tail_bytes = 193926656
[1457530.419332] GH cycle = 31, GH bytes = 193926464
[1457530.460734] XFS (dm-0): xlog_space_left: head behind tail
[1457530.460735] tail_cycle = 31, tail_bytes = 193929728
[1457530.460736] GH cycle = 31, GH bytes = 193929536
[1457530.466885] XFS (dm-0): xlog_space_left: head behind tail
[1457530.466886] tail_cycle = 31, tail_bytes = 193929728
[1457530.466887] GH cycle = 31, GH bytes = 193929536
[1457530.476049] XFS (dm-0): xlog_space_left: head behind tail
[1457530.476049] tail_cycle = 31, tail_bytes = 193931776
[1457530.476050] GH cycle = 31, GH bytes = 193931584
[1457530.482080] XFS (dm-0): xlog_space_left: head behind tail
[1457530.482081] tail_cycle = 31, tail_bytes = 193931776
[1457530.482082] GH cycle = 31, GH bytes = 193931584
[1457530.488843] XFS (dm-0): xlog_space_left: head behind tail
[1457530.488844] tail_cycle = 31, tail_bytes = 193932800
[1457530.488844] GH cycle = 31, GH bytes = 193932608
[1457530.489103] XFS (dm-0): xlog_space_left: head behind tail
[1457530.489104] tail_cycle = 31, tail_bytes = 193932800
[1457530.489104] GH cycle = 31, GH bytes = 193932608
[1457530.489106] XFS (dm-0): xlog_space_left: head behind tail
[1457530.489106] tail_cycle = 31, tail_bytes = 193932800
[1457530.489107] GH cycle = 31, GH bytes = 193932608
[1457530.489912] XFS (dm-0): xlog_space_left: head behind tail
[1457530.489912] tail_cycle = 31, tail_bytes = 193933824
[1457530.489913] GH cycle = 31, GH bytes = 193933632
[1457530.489914] XFS (dm-0): xlog_space_left: head behind tail
[1457530.489915] tail_cycle = 31, tail_bytes = 193933824
[1457530.489915] GH cycle = 31, GH bytes = 193933632
[1457530.520768] XFS (dm-0): xlog_space_left: head behind tail
[1457530.520769] tail_cycle = 31, tail_bytes = 193935872
[1457530.520770] GH cycle = 31, GH bytes = 193935680
[1457530.528092] XFS (dm-0): xlog_space_left: head behind tail
[1457530.528093] tail_cycle = 31, tail_bytes = 193935872
[1457530.528094] GH cycle = 31, GH bytes = 193935680
[1457530.576076] XFS (dm-0): xlog_space_left: head behind tail
[1457530.576077] tail_cycle = 31, tail_bytes = 193938944
[1457530.576078] GH cycle = 31, GH bytes = 193938752
[1457530.577064] XFS (dm-0): xlog_space_left: head behind tail
[1457530.577065] tail_cycle = 31, tail_bytes = 193938944
[1457530.577066] GH cycle = 31, GH bytes = 193938752
[1457530.577068] XFS (dm-0): xlog_space_left: head behind tail
[1457530.577068] tail_cycle = 31, tail_bytes = 193938944
[1457530.577069] GH cycle = 31, GH bytes = 193938752
[1457530.602425] XFS (dm-0): xlog_space_left: head behind tail
[1457530.602426] tail_cycle = 31, tail_bytes = 193944064
[1457530.602426] GH cycle = 31, GH bytes = 193943872
[1457530.602747] XFS (dm-0): xlog_space_left: head behind tail
[1457530.602748] tail_cycle = 31, tail_bytes = 193944064
[1457530.602749] GH cycle = 31, GH bytes = 193943872
[1457530.602750] XFS (dm-0): xlog_space_left: head behind tail
[1457530.602751] tail_cycle = 31, tail_bytes = 193944064
[1457530.602752] GH cycle = 31, GH bytes = 193943872
[1457530.621611] XFS (dm-0): xlog_space_left: head behind tail
[1457530.621612] tail_cycle = 31, tail_bytes = 193945088
[1457530.621612] GH cycle = 31, GH bytes = 193944896
[1457548.748945] XFS (dm-0): xlog_space_left: head behind tail
[1457548.748946] tail_cycle = 31, tail_bytes = 194149376
[1457548.748947] GH cycle = 31, GH bytes = 194149184
[1457548.754420] XFS (dm-0): xlog_space_left: head behind tail
[1457548.754421] tail_cycle = 31, tail_bytes = 194149376
[1457548.754421] GH cycle = 31, GH bytes = 194149184
[1457551.205379] XFS (dm-0): xlog_space_left: head behind tail
[1457551.205380] tail_cycle = 31, tail_bytes = 194192896
[1457551.205381] GH cycle = 31, GH bytes = 194192704
[1457551.210506] XFS (dm-0): xlog_space_left: head behind tail
[1457551.210507] tail_cycle = 31, tail_bytes = 194192896
[1457551.210507] GH cycle = 31, GH bytes = 194192704
[1457551.217583] XFS (dm-0): xlog_space_left: head behind tail
[1457551.217584] tail_cycle = 31, tail_bytes = 194193920
[1457551.217584] GH cycle = 31, GH bytes = 194193728
[1457551.223680] XFS (dm-0): xlog_space_left: head behind tail
[1457551.223681] tail_cycle = 31, tail_bytes = 194193920
[1457551.223682] GH cycle = 31, GH bytes = 194193728
[1457560.258081] XFS (dm-0): xlog_space_left: head behind tail
[1457560.258082] tail_cycle = 31, tail_bytes = 194384384
[1457560.258083] GH cycle = 31, GH bytes = 194382144
[1457560.263061] XFS (dm-0): xlog_space_left: head behind tail
[1457560.263062] tail_cycle = 31, tail_bytes = 194384384
[1457560.263062] GH cycle = 31, GH bytes = 194382144
[1457560.270181] XFS (dm-0): xlog_space_left: head behind tail
[1457560.270181] tail_cycle = 31, tail_bytes = 194384384
[1457560.270182] GH cycle = 31, GH bytes = 194383168
[1457560.276196] XFS (dm-0): xlog_space_left: head behind tail
[1457560.276196] tail_cycle = 31, tail_bytes = 194384384
[1457560.276197] GH cycle = 31, GH bytes = 194383168
[1457560.282803] XFS (dm-0): xlog_space_left: head behind tail
[1457560.282805] tail_cycle = 31, tail_bytes = 194385408
[1457560.282805] GH cycle = 31, GH bytes = 194384192
[1457560.283076] XFS (dm-0): xlog_space_left: head behind tail
[1457560.283077] tail_cycle = 31, tail_bytes = 194385408
[1457560.283078] GH cycle = 31, GH bytes = 194384192
[1457560.283079] XFS (dm-0): xlog_space_left: head behind tail
[1457560.283080] tail_cycle = 31, tail_bytes = 194385408
[1457560.283081] GH cycle = 31, GH bytes = 194384192
[1457560.300510] XFS (dm-0): xlog_space_left: head behind tail
[1457560.300511] tail_cycle = 31, tail_bytes = 194385408
[1457560.300512] GH cycle = 31, GH bytes = 194385216
[1457590.329642] XFS (dm-0): xlog_space_left: head behind tail
[1457590.329643] tail_cycle = 31, tail_bytes = 194442752
[1457590.329644] GH cycle = 31, GH bytes = 194442560
[1457590.335806] XFS (dm-0): xlog_space_left: head behind tail
[1457590.335807] tail_cycle = 31, tail_bytes = 194442752
[1457590.335809] GH cycle = 31, GH bytes = 194442560
[1457592.985923] XFS (dm-0): xlog_space_left: head behind tail
[1457592.985924] tail_cycle = 31, tail_bytes = 194514944
[1457592.985925] GH cycle = 31, GH bytes = 194514752
[1457592.990881] XFS (dm-0): xlog_space_left: head behind tail
[1457592.990882] tail_cycle = 31, tail_bytes = 194514944
[1457592.990883] GH cycle = 31, GH bytes = 194514752
[1457592.998297] XFS (dm-0): xlog_space_left: head behind tail
[1457592.998298] tail_cycle = 31, tail_bytes = 194515968
[1457592.998298] GH cycle = 31, GH bytes = 194515776
[1457592.998546] XFS (dm-0): xlog_space_left: head behind tail
[1457592.998547] tail_cycle = 31, tail_bytes = 194515968
[1457592.998547] GH cycle = 31, GH bytes = 194515776
[1457592.998549] XFS (dm-0): xlog_space_left: head behind tail
[1457592.998549] tail_cycle = 31, tail_bytes = 194515968
[1457592.998550] GH cycle = 31, GH bytes = 194515776
[1457641.592443] XFS (dm-0): xlog_space_left: head behind tail
[1457641.592444] tail_cycle = 31, tail_bytes = 194928640
[1457641.592445] GH cycle = 31, GH bytes = 194926400
[1457641.597521] XFS (dm-0): xlog_space_left: head behind tail
[1457641.597522] tail_cycle = 31, tail_bytes = 194928640
[1457641.597523] GH cycle = 31, GH bytes = 194926400
[1457641.604962] XFS (dm-0): xlog_space_left: head behind tail
[1457641.604963] tail_cycle = 31, tail_bytes = 194928640
[1457641.604964] GH cycle = 31, GH bytes = 194927936
[1457641.610983] XFS (dm-0): xlog_space_left: head behind tail
[1457641.610984] tail_cycle = 31, tail_bytes = 194928640
[1457641.610984] GH cycle = 31, GH bytes = 194927936
[1457641.618036] XFS (dm-0): xlog_space_left: head behind tail
[1457641.618037] tail_cycle = 31, tail_bytes = 194929664
[1457641.618037] GH cycle = 31, GH bytes = 194928960
[1457641.618286] XFS (dm-0): xlog_space_left: head behind tail
[1457641.618287] tail_cycle = 31, tail_bytes = 194929664
[1457641.618287] GH cycle = 31, GH bytes = 194928960
[1457641.618289] XFS (dm-0): xlog_space_left: head behind tail
[1457641.618289] tail_cycle = 31, tail_bytes = 194929664
[1457641.618290] GH cycle = 31, GH bytes = 194928960
[1457650.616706] XFS (dm-0): xlog_space_left: head behind tail
[1457650.616707] tail_cycle = 31, tail_bytes = 194969600
[1457650.616708] GH cycle = 31, GH bytes = 194967360
[1457650.621736] XFS (dm-0): xlog_space_left: head behind tail
[1457650.621738] tail_cycle = 31, tail_bytes = 194969600
[1457650.621738] GH cycle = 31, GH bytes = 194967360
[1457650.628564] XFS (dm-0): xlog_space_left: head behind tail
[1457650.628565] tail_cycle = 31, tail_bytes = 194969600
[1457650.628566] GH cycle = 31, GH bytes = 194968384
[1457650.634470] XFS (dm-0): xlog_space_left: head behind tail
[1457650.634471] tail_cycle = 31, tail_bytes = 194969600
[1457650.634472] GH cycle = 31, GH bytes = 194968384
[1457650.641265] XFS (dm-0): xlog_space_left: head behind tail
[1457650.641266] tail_cycle = 31, tail_bytes = 194970624
[1457650.641266] GH cycle = 31, GH bytes = 194969408
[1457650.647291] XFS (dm-0): xlog_space_left: head behind tail
[1457650.647291] tail_cycle = 31, tail_bytes = 194970624
[1457650.647292] GH cycle = 31, GH bytes = 194969408
[1457650.653133] XFS (dm-0): xlog_space_left: head behind tail
[1457650.653134] tail_cycle = 31, tail_bytes = 194970624
[1457650.653135] GH cycle = 31, GH bytes = 194970432
[1457650.653996] XFS (dm-0): xlog_space_left: head behind tail
[1457650.653997] tail_cycle = 31, tail_bytes = 194971648
[1457650.653997] GH cycle = 31, GH bytes = 194970432
[1457650.653999] XFS (dm-0): xlog_space_left: head behind tail
[1457650.654000] tail_cycle = 31, tail_bytes = 194971648
[1457650.654000] GH cycle = 31, GH bytes = 194970432
[1457650.670979] XFS (dm-0): xlog_space_left: head behind tail
[1457650.670980] tail_cycle = 31, tail_bytes = 194972672
[1457650.670981] GH cycle = 31, GH bytes = 194971456
[1457650.677414] XFS (dm-0): xlog_space_left: head behind tail
[1457650.677415] tail_cycle = 31, tail_bytes = 194972672
[1457650.677416] GH cycle = 31, GH bytes = 194972480
[1457650.683332] XFS (dm-0): xlog_space_left: head behind tail
[1457650.683333] tail_cycle = 31, tail_bytes = 194972672
[1457650.683333] GH cycle = 31, GH bytes = 194972480
[1457650.691056] XFS (dm-0): xlog_space_left: head behind tail
[1457650.691057] tail_cycle = 31, tail_bytes = 194975744
[1457650.691058] GH cycle = 31, GH bytes = 194975552
[1457650.691816] XFS (dm-0): xlog_space_left: head behind tail
[1457650.691817] tail_cycle = 31, tail_bytes = 194975744
[1457650.691818] GH cycle = 31, GH bytes = 194975552
[1457650.691820] XFS (dm-0): xlog_space_left: head behind tail
[1457650.691821] tail_cycle = 31, tail_bytes = 194975744
[1457650.691822] GH cycle = 31, GH bytes = 194975552
[1457650.709362] XFS (dm-0): xlog_space_left: head behind tail
[1457650.709363] tail_cycle = 31, tail_bytes = 194977792
[1457650.709363] GH cycle = 31, GH bytes = 194977600
[1457650.715256] XFS (dm-0): xlog_space_left: head behind tail
[1457650.715257] tail_cycle = 31, tail_bytes = 194977792
[1457650.715258] GH cycle = 31, GH bytes = 194977600
[1457650.722368] XFS (dm-0): xlog_space_left: head behind tail
[1457650.722369] tail_cycle = 31, tail_bytes = 194978816
[1457650.722370] GH cycle = 31, GH bytes = 194978624
[1457650.728326] XFS (dm-0): xlog_space_left: head behind tail
[1457650.728327] tail_cycle = 31, tail_bytes = 194978816
[1457650.728328] GH cycle = 31, GH bytes = 194978624
[1457650.738542] XFS (dm-0): xlog_space_left: head behind tail
[1457650.738543] tail_cycle = 31, tail_bytes = 194979840
[1457650.738544] GH cycle = 31, GH bytes = 194979648
[1457650.744538] XFS (dm-0): xlog_space_left: head behind tail
[1457650.744539] tail_cycle = 31, tail_bytes = 194979840
[1457650.744539] GH cycle = 31, GH bytes = 194979648
[1457671.840677] XFS (dm-0): xlog_space_left: head behind tail
[1457671.840678] tail_cycle = 31, tail_bytes = 195193344
[1457671.840679] GH cycle = 31, GH bytes = 195193152
[1457671.846755] XFS (dm-0): xlog_space_left: head behind tail
[1457671.846756] tail_cycle = 31, tail_bytes = 195193344
[1457671.846757] GH cycle = 31, GH bytes = 195193152
[1457671.853509] XFS (dm-0): xlog_space_left: head behind tail
[1457671.853510] tail_cycle = 31, tail_bytes = 195194368
[1457671.853511] GH cycle = 31, GH bytes = 195194176
[1457671.859463] XFS (dm-0): xlog_space_left: head behind tail
[1457671.859464] tail_cycle = 31, tail_bytes = 195194368
[1457671.859464] GH cycle = 31, GH bytes = 195194176
[1457681.585097] XFS (dm-0): xlog_space_left: head behind tail
[1457681.585098] tail_cycle = 31, tail_bytes = 195431936
[1457681.585099] GH cycle = 31, GH bytes = 195429696
[1457681.590072] XFS (dm-0): xlog_space_left: head behind tail
[1457681.590073] tail_cycle = 31, tail_bytes = 195431936
[1457681.590075] GH cycle = 31, GH bytes = 195429696
[1457681.597612] XFS (dm-0): xlog_space_left: head behind tail
[1457681.597612] tail_cycle = 31, tail_bytes = 195431936
[1457681.597613] GH cycle = 31, GH bytes = 195431232
[1457681.603750] XFS (dm-0): xlog_space_left: head behind tail
[1457681.603750] tail_cycle = 31, tail_bytes = 195431936
[1457681.603751] GH cycle = 31, GH bytes = 195431232
[1457681.611292] XFS (dm-0): xlog_space_left: head behind tail
[1457681.611293] tail_cycle = 31, tail_bytes = 195432960
[1457681.611294] GH cycle = 31, GH bytes = 195432256
[1457681.611580] XFS (dm-0): xlog_space_left: head behind tail
[1457681.611581] tail_cycle = 31, tail_bytes = 195432960
[1457681.611582] GH cycle = 31, GH bytes = 195432256
[1457681.611584] XFS (dm-0): xlog_space_left: head behind tail
[1457681.611585] tail_cycle = 31, tail_bytes = 195432960
[1457681.611586] GH cycle = 31, GH bytes = 195432256
[1457701.662714] XFS (dm-0): xlog_space_left: head behind tail
[1457701.662715] tail_cycle = 31, tail_bytes = 195452928
[1457701.662716] GH cycle = 31, GH bytes = 195452736
[1457701.667806] XFS (dm-0): xlog_space_left: head behind tail
[1457701.667807] tail_cycle = 31, tail_bytes = 195452928
Mirko
Hi Mirko,
Can you try this with the MONO_GC_DEBUG=clear-at-gc set in your environment? We’ve been unable to reproduce the segmentation faults with that set.
Cheers,
James