Jun 19, 2013; 13:33
Rob Brooks
Large Datasets
Hello, I have a script that fetches around 100MB from the database for a
report. It then parses the data and writes it to a file. Eventually if run
multiple times, this script will exhaust Lasso memory and will fail. It
appears that Lasso restarts itself when this happens. After this, other
scripts will run but the report script will always fail until a manual
restart of Lasso is done.
I understand that grabbing this much data is expensive. However, there's
plenty of memory on the machine and the script can run fine for a few times
before this happens. What I don't understand is why Lasso is not releasing
memory after script completion. Watching Lasso memory usage follows a
pattern of increasing significantly the first time the script is run (around
1GB) and then increasing slowly the next few times run but always staying
well below machine memory ceiling. Using 'top' or 'htop' for instance never
shows Lasso exhausting the machine's memory. Sometimes 'top' will show
Lasso stop increasing in memory usage when the script is run multiple times
but eventually the script will fail and not run again until manual restart
of Lasso.
e.g.
Lasso restarted manaully
Lasso showing ~50MB memory usage in 'top'
run script
Lasso mem usage goes up to 1.3G (never drops after script completes)
run script
Lasso mem usage goes up to 1.7G (never drops after script completes)
run script
Lasso mem usage stays at 1.7G
(repeat last 2 steps a few times)
script fails
The script follows this basic flow
1. mysql inline to get the data.
2. records loop where data is parsed and put in a local data structure.
3. data written to file
I set all variables that hold data to the empty string when done just to be
safe and to try to force Lasso to release this memory. Is there a way to
force Lasso to release the memory stored for the dataset itself as returned?
( I would think this would happen at script completion)
This is on CentOS 5.9
Lasso 8.6.1
apache 2.2.3
mysql 5.0.95
Thanks,
Rob
#############################################################
Attend the Lasso Developer Conference 2013!
Sept 12-14, 2013 in Niagara Falls, Canada
http://www.lassosoft.com/LDC-niagara-falls-2013
#############################################################
This message is sent to you because you are subscribed to
the mailing list Lasso
Lasso@lists.lassosoft.com
To unsubscribe, E-mail to: <Lasso-unsubscribe@lists.lassosoft.com>
Send administrative queries to <Lasso-request@lists.lassosoft.com>