creating gist index on ltree column failed

Hello,
I tried to create gist index on a ltree column, first I got error
=====
ERROR: stack depth limit exceeded
HINT: Increase the configuration parameter "max_stack_depth" (currently 32768kB), after ensuring the platform's stack depth limit is adequate.
=====
I increased system memory size with "ulimit -S -s 131072", then run "set max_stack_depth = '64MB';"
this time I got some other errors:
"failed to add item to index page" or "no unpinned buffers available"

sometimes if I rerun it, or run vacuum on the table first, the creating index runs for ever until all the system memory was eaten up, then server was hang, I had to reboot the server.
one thing I notice is, there are some rows with many levels(more than 1k levels) in the ltree, if I delete those rows, creating gist index works

is it a bug in the gist index? does anyone know how to workaround this problem?

thank you

James