[LRUG] debugging memory use

[James Cox]

On 10 Jan 2008, at 15:46, Dan Webb wrote:

> Hello Peeps,
>
> Im coming across something that Ive never had to deal with before with
> ruby's memory usage.  I've got a few scripts that take a while to run
> and imports lots of XML data.  Most of them work fine but one
> particular one (which is nearly identical to the others) tends to use
> lots of memory.
>
> When you run the script its memory usage is fine until just before it
> finishes where it quickly takes up about 600MB and spends lots of time
> chewing around even though its finished the task in hand.  So....
>
> Firstly, has anyone seen this behaviour before?  Any clues as to
> what's going on? Not after specifics...just general pointers would be
> excellent.

I've done this a few times (importing large iTunes library xml for  
processing is a great data set to trigger this). I've found that  
typically each line of XML is processed, but that once the whole tree  
has been parsed, it has to figure out how to nest it properly. that's  
the bit at the end - it's constructing the final object to hand back.

I didn't dig too deep - but that's what it seemed to be doing - you  
basically double your object alloc space, and it's not always freed  
properly at the end (blame bad GC on Ruby's part).

Better would be finding a way to chunk it, or try and discard attribs  
etc you don't need on the way....

-james