I checked with ObjectSpace.count_objects and ObjectSpace.memsize_of_all, so mruby is built with rake CONFIG=host-debug.

• ruby code: stackerr.rb

  This code stores the exception object in the variable e in the rescue block, but assigns nil at the end of the block so that there is no reference to the exception object.

  def infinity_loop
    infinity_loop
  end
  puts "## Before a stack error occurs"
  p ObjectSpace.count_objects
  p ObjectSpace.memsize_of_all
  begin
    infinity_loop
  rescue SystemStackError => e
    e.backtrace  # Convert internal data by RData to an array of string
    e.set_backtrace [] if ARGV[0] == "1"
    e = nil
  end
  puts "## After a stack error occurs"
  p ObjectSpace.count_objects
  p ObjectSpace.memsize_of_all

• When a stack error occurs (normal)

  % bin/mruby stackerr.rb 0
  ## Before occurs stack error
  {:TOTAL=>1024, :FREE=>214, :T_OBJECT=>2, :T_CLASS=>73, :T_MODULE=>10, :T_ICLASS=>16, :T_SCLASS=>89, :T_PROC=>522, :T_ARRAY=>1, :T_HASH=>1, :T_STRING=>26, :T_EXCEPTION=>2, :T_ENV=>64, :T_DATA=>3, :T_ISTRUCT=>1}
  43243
  ## After occurs stack error
  {:TOTAL=>3072, :FREE=>1299, :T_OBJECT=>2, :T_CLASS=>73, :T_MODULE=>10, :T_ICLASS=>16, :T_SCLASS=>89, :T_PROC=>522, :T_ARRAY=>2, :T_HASH=>1, :T_STRING=>988, :T_EXCEPTION=>2, :T_ENV=>64, :T_DATA=>3, :T_ISTRUCT=>1}
  221261

  221261 bytes are reported.

• When an empty array is set

  % bin/mruby stackerr.rb 1
  ## Before a stack error occurs
  {:TOTAL=>1024, :FREE=>214, :T_OBJECT=>2, :T_CLASS=>73, :T_MODULE=>10, :T_ICLASS=>16, :T_SCLASS=>89, :T_PROC=>522, :T_ARRAY=>1, :T_HASH=>1, :T_STRING=>26, :T_EXCEPTION=>2, :T_ENV=>64, :T_DATA=>3, :T_ISTRUCT=>1}
  43243
  ## After a stack error occurs
  {:TOTAL=>2048, :FREE=>1237, :T_OBJECT=>2, :T_CLASS=>73, :T_MODULE=>10, :T_ICLASS=>16, :T_SCLASS=>89, :T_PROC=>522, :T_ARRAY=>2, :T_HASH=>1, :T_STRING=>26, :T_EXCEPTION=>2, :T_ENV=>64, :T_DATA=>3, :T_ISTRUCT=>1}
  43291

  43291 bytes are reported.

The reason is that when a stack error occurs, mrb_state::stack_err becomes its instance, from which there is a reference to backtrace information, which is always marked even if GC occurs.
I think the same problem exists with mrb_state::arena_err, which is enabled when MRB_GC_FIXED_ARENA is defined.

The suggested modification I came up with is to not mark mrb_state::stack_err as the root object in the mark phase.
Then, just before the sweep phase, mark only mrb_state::stack_err and initialize mrb_state::stack_err->{iv,mesg,backtrace} if needed at that time.
If there are references to mrb_state::stack_err from other objects, they are marked normally.

I tried to write an actual patch, but it failed due to my lack of understanding of GC.
So I will only report the problem.