Date: 2009-04-28 06:56
Sender: Guillaume Smet
Thanks for the feedback.
Yes, it's version 1.1, you can use this one.
Do you confirm that it analyzes your multi-lines query correctly now and that you don't need to preprocess the log anymore? |
Date: 2009-04-28 06:08
Sender: Dmitri Koulikov
Hello. Now it works 2048 MG turned out to be enough.
Thank you.
I've taken the version from CSV. Is it version 1.1? |
Date: 2009-04-26 11:05
Sender: Dmitri Koulikov
Yes, I will try and inform you later. |
Date: 2009-04-26 11:04
Sender: Guillaume Smet
FYI, I just changed the default memory limit from 128 to 512 MB and you can set it even higher by using -memorylimit 1024 for example.
What can also be interesting is to run pgFouine with -debug option so that you can see how the memory usage grows.
Thanks for your feedback. |
Date: 2009-04-26 10:53
Sender: Guillaume Smet
OK, I added the normalization of IN() clauses in CVS HEAD so you won't have to preprocess this part from now on.
I also fixed yesterday a bug in stderr analysis which can explain that multilines statements were not correctly analyzed.
Could you try to analyze a vanilla log (ie without any pre processing) with CVS HEAD of pgFouine and tell me if it's OK?
If it's not, please provide it to me (send me an email in private to guillaume.smet at gmail dot com with an URL if it's big) and I'll see what I can do.
I plan to release the new version this afternoon so I'm really interested in your input.
Ping me if you need any help, I can provide you a tarball if you can't get CVS HEAD by yourself.
Thanks in advance.
--
Guillaume |
Date: 2009-04-26 10:32
Sender: Dmitri Koulikov
And I forget to mention that our pre-processing script now
substitutes expressions like /[[:digit:]]+(,[[:digit:]]+)+/
with the string '124'. And this does not help |
Date: 2009-04-26 10:29
Sender: Dmitri Koulikov
Yes, we do pre-process the logs as pgfouine often does not
recognize the sql statements divided on lines correctly, so
we join the lines. I've just checked the pre-processing
script and saw that the lines of WARNING/HINT/ERROR kind now
do not get into the log being given to pgfouine.
Nevertheless usually only about 300000 lines are processed
before the fail. |
Date: 2009-04-26 10:22
Sender: Guillaume Smet
OK, so at least for the undefined method bug, it's a log file bug. I think you manipulate your log file before analyzing it as I can't reproduce your problem.
For the memory limit log, you have the same problem with WARNING/HINT/ERROR with no new lines.
And I think your main problem is that pgFouine doesn't normalize IN(list) statement. It's what causes your memory problem. I check if I can fix that. |
Date: 2009-04-26 09:40
Sender: Guillaume Smet
There is something weird in your log files. If you check the undefined method one, 2 lines are together on the same line whereas they should be on 2 separate lines:
2009-03-25 17:53:09 UTC [13531]: [527-1] LOG: duration: 902.038 ms statement: SELECT DISTINCT platform_id FROM links WHERE project_id = 13340 AND NOT status_id IN (8,9)2009-03-25 17:53:09 UTC [13531]: [528-1] WARNING: nonstandard use of \\ in a string literal at character 802009-03-25 17:53:09 UTC [13531]: [529-1] HINT: Use the escape string syntax for backslashes, e.g., E'\\'.
As you can see the WARNING is on the same line and this is wrong.
Did you manipulate your log files in a way that can cause this problem?
That said, it's not the only problem I have in analyzing your log file so I'm still digging. |
Date: 2009-04-25 21:44
Sender: Dmitri Koulikov
Sorry, I was sure the file was attached. |
Date: 2009-04-25 21:40
Sender: Dmitri Koulikov
sorry, i |
Date: 2009-04-25 19:14
Sender: Guillaume Smet
By the way, I'm also interested in test files for the other problem you reported: http://pgfoundry.org/tracker/index.php?func=detail&aid=1010210&group_id=1000152&atid=618 so that I can fix it too.
Thanks |
Date: 2009-04-25 19:12
Sender: Guillaume Smet
Hi Dmitri,
For your first problem, you need to use the -memorylimit option of pgFouine. Note that if you don't use a connection pooling solution, the version I'll release tomorrow is going to improve the situation a lot.
For your second problem, I'm interested in the log file or at least the sequence of lines causing the problem.
Thanks in advance. |
Date: 2009-04-03 08:38
Sender: Dmitri Koulikov
I've to add that changing memory_limit to ANY other value in
php.ini does not change the vale 134217728 in the first error. |
