SCM

[#1010799] Invalid memory alloc request size 1073741824

View Trackers | Bugs | Download .csv | Monitor

Date:
2010-04-29 16:11
Priority:
3
State:
Open
Submitted by:
Krzysztof Nienartowicz (krzychu)
Assigned to:
Nobody (None)
Category:
Group:
Resolution:
Accepted
 
Summary:
Invalid memory alloc request size 1073741824

Detailed description
Hello,
First, thanks for the tool - I'm to handle multi-terabyte database and it is a huge helper.
I have a problem however: my table has some bytea columns that store double arrays and I am converting textual array form that pg_bulkload accepts to bytea using filter function (written in pljava).
pg_bulkload is launched for sets of 10-50K rows fetched via pipe with three float8 arrays - it consistently fails after processing same number of rows (as you see not many - 25) for specific records ( I loaded few million before without a problem). when the same record is loaded in separation it works! So it's not a problem of a record itself,rather in a sequence.
It seems there is some buffer overflow but I do not have any experience how to debug contrib modules. Are there any limits on thefilter function input parameters?
No relevant information is printed in the log (with debug1 level on the server)

I tried to play with INITIAL_BUF_LEN in lib/parser_csv.c by incrfeasing it - but it actually exposed error even earlier - when multiplied by 8 failure is constant after 17 records...

Would be grateful for any hints, this is rather urgent.

Also, support for partitioned tables would be a huge win!
Best Regards,
Krzysztof

pg_bulkload 3.0alpha2 on 2010-04-29 10:58:57.614783+02

INFILE = STDIN
PARSE_BADFILE = /home/nienarto/logs/lm0376l.tar.gz_pg_bad_records.log
DUPLICATE_BADFILE = /home/nienarto/logs/lm0376l.tar.gz_pg_duplicates.log
LOGFILE = /home/nienarto/logs/lm0376l.tar.gz_pg_bulk.log
TABLE = oglehip.ts_part_lm0276m00001_lm0584k00000
TRUNCATE = NO
LIMIT = INFINITE
PARSE_ERRORS = 0
DUPLICATE_ERRORS = 0
ON_DUPLICATE_KEEP = NEW
VERBOSE = NO
ENCODING = LATIN9
CHECK_CONSTRAINTS = NO
TYPE = CSV
SKIP = 0
DELIMITER = ,
QUOTE = '
ESCAPE = "\""
NULL =
FILTER = oglehip.tsConvert( varchar, integer, integer, float8[], float8[], float8[])
WRITER = DIRECT

Parse error Record 1: Input Record 26: Rejected - column 6. invalid memory alloc request size 1073741824
Maximum parse error count exceeded - 1 error(s) found in input file

0 Rows skipped.
25 Rows successfully loaded.
1 Rows not loaded due to parse errors.
0 Rows not loaded due to duplicate errors.
0 Rows deleted due to duplicate errors.

Run began on 2010-04-29 10:58:57.614783+02
Run ended on 2010-04-29 10:59:05.617247+02

CPU 0.76s/1.89u sec elapsed 8.00 sec



Failed record is cut in the report file, not sure if this indicates anything:

lm0376l10039,34,403,'{332.82646,342.87585,376.80293,379.84642,383.77552,392.85761,399.79420,402.74150,403.65860,404.76630,410.80479,429.67517,431.78877,433.71307,435.74636,440.78766,444.76476,446.81966,451.67375,455.64235,469.63384,476.75954,486.71784,491.68734,495.67554,498.70574,508.68584,512.70444,517.64905,536.59416,569.54889,686.92076,706.82935,750.86792,751.80582,756.79741,761.75830,763.83420,774.81779,781.85038,784.75348,787.78828,789.80998,796.65577,801.74256,806.73106,810.83086,814.82875,820.81805,823.69585,827.71385,836.77654,839.71644,843.81054,845.79864,860.76744,865.69284,872.63674,877.62464,880.64085,893.67435,897.58326,903.63156,909.60686,941.54960,941.55930,947.49270,1032.91196,1035.88376,1069.85315,1090.84584,1115.85672,1116.69042,1122.69601,1145.73399,1155.74027,1167.60566,1182.84505,1183.79645,1184.76905,1185.78365,1197.71164,1198.72824,1209.69464,1218.66044,1228.65834,1228.70414,1229.63674,1238.61194,1243.67254,1393.82426,1400.89596,1412.83426,1421.91986,1431.79696,1433.84295,1434.87135,1435.79415,1439.81465,1443.79465,1444.79345,1451.84164,1454.89004,1458.78174,1460.76694,1462.80613,1464.78763,1467.82143,1469.74313,1475.81792,1476.79372,1479.85072,1481.77002,1483.74541,1484.80601,1491.80191,1493.81260,1495.77130,1497.81140,1500.76650,1502.75469,1504.76349,1509.76269,1510.75899,1522.73487,1524.68807,1525.79477,1527.75917,1529.75467,1531.74926,1533.77896,1538.72586,1546.63525,1548.77305,1550.79045,1552.76795,1554.77465,1556.81605,1558.75255,1563.72734,1566.70864,1568.70314,1570.70394,1572.82094,1574.73024,1576.77004,1578.70984,1580.72794,1582.73134,1586.72284,1590.67784,1592.68734,1594.73594,1597.70504,1602.61134,1606.64764,1609.62044,1612.62285,1616.61405,1618.63865,1620.67945,1625.66665,1627.62155,1630.57396,1635.52786,1641.55387,1645.61537,1654.57898,1657.55338,1660.55188,1695.50002,1728.93405,1745.90586,1751.87826,1756.89846,1760.83986,1764.91586,1768.89636,1773.86766,1776.89766,1782.84026,1784.83376,1786.83366,1790.90956,1791.89306,1804.88495,1810.79975,1812.82065,1814.80074,1816.81754,1822.79684,1824.79414,1826.83254,1828.82753,1829.68003,1837.79563,1839.78392,1840.83142,1843.84512,1846.67292,1850.71141,1854.75231,1858.67840,1860.70610,1861.78840,1864.70730,1866.71080,1868.70829,1870.73909,1872.76679,1874.71329,1878.72468,1880.71898,1882.72398,1884.72958,1886.72917,1887.74477,1888.75577,1889.72647,1890.75617,1892.69877,1893.81337,1895.77667,1895.85507,1897.78786,1901.63196,1903.62586,1905.63006,1908.61556,1910.83885,1912.81155,1914.86615,1918.81405,1919.77655,1921.69495,1921.70895,1924.73745,1926.74824,1928.76564,1929.76864,1931.73754,1933.81154,1935.75984,1937.76614,1938.75234,1939.78274,1940.78184,1941.79054,1950.68154,1952.68464,1954.68524,1956.69924,1958.74944,1960.75704,1963.67474,1965.66304,1967.66384,1969.72354,1972.64184,1974.65054,1976.63065,1980.63255,1982.62525,1987.62295,1993.61196,1995.59386,1997.60546,2000.60606,2003.58706,2008.59537,2011.59487,2016.58597,2026.55178,2029.54089,2032.55049,2046.53190,2051.51101,2057.50772,2074.45503,2095.92945,2105.89635,2123.91866,2128.87626,2131.82646,2134.81726,2140.85956,2144.83826,2160.78716,2165.83235,2168.82435,2171.81795,2173.81555,2178.87485,2186.82044,2187.82774,2190.80924,2194.71633,2198.70093,2199.85503,2200.81793,2202.70963,2204.68662,2211.78072,2214.64871,2223.65790,2225.72040,2226.61760,2228.70060,2230.66500,2231.65360,2233.65389,2236.64009,2238.66709,2240.67729,2242.64328,2248.63538,2250.64778,2254.63847,2255.70817,2257.78707,2259.67237,2262.67676,2262.73526,2264.66886,2266.74586,2296.77074,2298.81224,2300.74284,2301.78814,2309.78514,2312.78144,2316.76964,2318.74594,2320.72974,2321.73334,2322.72804,2324.69244,2326.66374,2328.67164,2330.69574,2337.68934,2344.71115,2345.63645,2347.64515,2349.69765,2351.64785,2354.64675,2356.68665,2381.58947,2388.52918,2390.58828,2395.59009,2411.54410,2464.92865,2554.75874,2556.77724,2558.82683,2561.76433,2564.77373,2565.70753,2573.78872,2574.77742,2576.78662,2578.79111,2579.79171,2580.78451,2582.77831,2595.83690,2598.81099,2605.61359,2605.80579,2609.65438,2615.61598,2615.81498,2619.58867,2619.74487,2621.64107,2628.84066,2631.63386,2631.81016,2634.68516,2634.85076,2637.63966,2637.81396,2640.63015,2640.77065,2643.65885,2643.77965,2646.61185,2646.78595,2649.63815,2649.81215,2652.58175,2659.65314,2662.61244,2662.71864,2674.62854,2674.79424,2681.62284,2684.66634,2695.57124,2695.73134,2698.63574}','{19.995,20.091,20.008,19.979,19.818,19.997,20.071,20.037,20.019,20.056,20.109,20.032,19.955,20.119,20.224,19.969,20.129,20.058,20.047,20.051,20.184,20.074,20.178,20.105,20.112,20.093,20.183,20.137,20.072,20.020,20.138,19.912,19.932,19.960,20.141,20.228,19.888,20.104,19.975,19.988,20.063,19.971,20.043,20.259,20.005,19.952,20.029,19.935,20.017,20.040,20.004,20.095,20.051,20.060,20.099,19.871,20.118,20.000,19.776,20.005,19.920,20.036,19.942,19.845,20.070,19.578,19.991,19.927,19.908,20.125,20.068,20.079,19.959,19.760,20.106,19.909,20.196,20.034,19.926,19.827,19.859,20.061,20.027,19.924,20.083,20.082,20.058,19.942,20.125,19.892,19.940,20.211,19.942,20.066,20.045,20.122,20.063,20.029,19.991,19.773,20.040,19.900,19.973,20.090,19.798,19.985,19.934,20.084,19.958,19.936,19.837,19.990,19.964,20.081,19.872,19.989,20.009,19.984,20.122,19.904,20.168,19.990,19.929,19.953,19.897,20.003,20.054,20.050,20.075,20.154,20.103,20.058,20.141,20.066,19.980,19.974,19.902,20.050,20.137,19.898,20.039,19.706,20.023,20.164,19.924,20.125,19.887,20.075,19.985,19.910,19.875,20.337,20.043,19.987,19.999,20.041,19.986,19.922,20.068,19.742,19.983,20.404,19.981,19.909,19.990,20.221,19.987,19.774,20.172,19.870,19.896,19.968,20.351,20.154,19.831,19.848,20.005,19.875,19.678,19.720,20.021,19.804,19.914,20.086,20.217,19.779,19.943,19.989,20.088,20.100,19.970,19.914,19.990,19.864,19.720,19.927,20.087,20.071,20.086,20.069,20.071,20.079,19.698,19.914,19.797,20.214,19.969,20.077,19.978,19.942,20.029,20.065,20.002,20.048,19.939,19.945,20.327,19.640,20.118,19.885,19.996,20.070,20.092,19.948,20.031,20.048,19.942,19.978,19.939,19.956,19.816,19.975,20.292,20.051,19.539,19.730,19.843,20.039,20.035,19.942,19.898,20.213,20.112,20.109,20.053,19.920,19.972,20.043,20.200,19.849,20.235,20.016,20.095,20.096,20.003,20.081,20.038,19.970,19.913,19.876,19.789,20.066,20.235,20.037,19.615,20.061,19.903,19.968,19.738,20.767,19.929,19.889,19.852,19.839,20.267,20.073,19.854,20.015,20.356,20.381,20.013,20.334,19.813,20.341,20.124,20.081,20.058,20.179,19.866,19.954,19.995,20.014,19.995,19.501,19.995,19.778,19.960,19.989,19.994,19.949,20.103,19.870,19.994,20.044,19.944,20.002,19.940,19.987,20.182,20.065,20.293,20.096,19.801,19.897,19.867,20.054,20.094,20.153,19.903,20.038,19.839,20.074,19.921,19.796,19.957,19.970,19.829,20.082,19.854,20.103,20.004,20.001,20.036,20.050,20.094,19.889,19.967,19.946,20.016,19.882,19.527,19.715,20.188,19.881,19.739,19.931,20.108,19.843,20.063,20.550,19.743,19.931,19.828,19.993,20.199,19.902,19.957,19.702,19.765,19.921,19.646,19.766,20.605,20.030,20.044,20.348,19.782,19.800,19.872,20.811,19.829,19.952,19.891,19.719,19.800,19.913,19.951,19.955,20.142,20.109,19.724,19.593,19.757,20.005,20.125,19.945,20.021,19.940,20.019,19.838,20.038,20.099,19.586,19.939,19.913,19.721,20.039,19.798,20.087,20.218,20.060,20.121,19.867}','{0.071,0.096,0.074,0.079,0.088,0.091,0.083,0.071,0.081,0.070,0.116,0.080,0.073,0.075,0.105,0.120,0.144,0.139,0.100,0.089,0.126,0.196,0.113,0.080,0.139,0.129,0.950,0.099,0.090,0.103,0.100,0.081,0.221,0.126,0.127,0.236,0.072,0.117,0.073,0.096,0.070,0.074,0.083,0.332,0.121,0.069,0.079,0.096,0.084,0.111,0.132,0.089,0.090,0.104,0.089,0.119,0.245,0.110,0.111,0.118,0.103,0.100,0.125,0.196,0.222,0.134,0.148,0.095,0.109,0.124,0.117,0.087,0.101,0.189,0.118,0.079,0.223,0.127,0.097,0.081,0.077,0.074,0.225,0.117,0.071,0.102,0.096,0.107,0.213,0.127,0.161,0.214,0.149,0.300,0.109,0.164,0.161,0.168,0.121,0.277,0.589,0.204,0.171,0.194,0.229,0.174,0.132,0.141,0.107,0.173,0.174,0.120,0.120,0.202,0.102,0.107,0.103,0.109,0.129,0.080,0.191,0.138,0.140,0.091,0.117,0.094,0.079,0.081,0.096,0.112,0.145,0.182,0.120,0.101,0.082,0.080,0.070,0.080,0.091,0.204,0.142,0.122,0.233,0.172,0.100,0.143,0.083,0.200,0.091,0.087,0.135,0.265,0.263,0.168,0.105,0.103,0.100,0.108,0.129,0.127,0.198,0.412,0.



the record itself looks like this:

lm0376l10020,34,394,'{332.82646,342.87585,376.80293,379.84642,383.77552,392.85761,399.79420,402.74150,403.65860,404.76630,410.80479,429.67517,431.78877,433.71307,435.74636,440.78766,444.76476,446.81966,451.67375,455.64235,469.63384,476.75954,486.71784,491.68734,495.67554,498.70574,508.68584,512.70444,517.64905,536.59416,569.54889,686.92076,706.82935,750.86792,751.80582,756.79741,761.75830,763.83420,774.81779,781.85038,784.75348,787.78828,789.80998,796.65577,801.74256,806.73106,810.83086,814.82875,820.81805,823.69585,827.71385,836.77654,839.71644,843.81054,845.79864,860.76744,865.69284,872.63674,877.62464,880.64085,893.67435,897.58326,903.63156,909.60686,941.55930,947.49270,1032.91196,1035.88376,1069.85315,1090.84584,1115.85672,1116.69042,1122.69601,1145.73399,1155.74027,1167.60566,1182.84505,1183.79645,1184.76905,1185.78365,1197.71164,1198.72824,1209.69464,1218.66044,1228.65834,1228.70414,1229.63674,1238.61194,1243.67254,1393.82426,1400.89596,1412.83426,1421.91986,1431.79696,1433.84295,1434.87135,1435.79415,1439.81465,1443.79465,1444.79345,1451.84164,1454.89004,1458.78174,1462.80613,1464.78763,1467.82143,1469.74313,1475.81792,1476.79372,1479.85072,1481.77002,1483.74541,1484.80601,1491.80191,1493.81260,1495.77130,1497.81140,1500.76650,1502.75469,1504.76349,1509.76269,1510.75899,1522.73487,1524.68807,1525.79477,1527.75917,1529.75467,1531.74926,1533.77896,1538.72586,1546.63525,1548.77305,1550.79045,1552.76795,1554.77465,1556.81605,1558.75255,1563.72734,1566.70864,1568.70314,1570.70394,1572.82094,1574.73024,1576.77004,1578.70984,1580.72794,1582.73134,1586.72284,1590.67784,1594.73594,1597.70504,1602.61134,1606.64764,1609.62044,1612.62285,1616.61405,1618.63865,1620.67945,1625.66665,1627.62155,1630.57396,1635.52786,1641.55387,1645.61537,1654.57898,1657.55338,1660.55188,1695.50002,1728.93405,1745.90586,1751.87826,1756.89846,1764.91586,1768.89636,1773.86766,1776.89766,1782.84026,1784.83376,1786.83366,1790.90956,1791.89306,1804.88495,1810.79975,1812.82065,1814.80074,1816.81754,1822.79684,1824.79414,1826.83254,1828.82753,1829.68003,1837.79563,1839.78392,1840.83142,1843.84512,1846.67292,1850.71141,1854.75231,1858.67840,1860.70610,1861.78840,1864.70730,1866.71080,1868.70829,1870.73909,1872.76679,1874.71329,1878.72468,1880.71898,1882.72398,1884.72958,1886.72917,1887.74477,1888.75577,1889.72647,1890.75617,1892.69877,1893.81337,1895.77667,1895.85507,1897.78786,1901.63196,1903.62586,1905.63006,1908.61556,1910.83885,1912.81155,1914.86615,1918.81405,1919.77655,1921.69495,1921.70895,1924.73745,1926.74824,1928.76564,1929.76864,1931.73754,1933.81154,1935.75984,1937.76614,1938.75234,1939.78274,1940.78184,1941.79054,1950.68154,1952.68464,1954.68524,1956.69924,1958.74944,1960.75704,1963.67474,1965.66304,1967.66384,1969.72354,1972.64184,1974.65054,1976.63065,1978.64155,1980.63255,1982.62525,1987.62295,1993.61196,1995.59386,1997.60546,2000.60606,2003.58706,2008.59537,2016.58597,2026.55178,2029.54089,2032.55049,2046.53190,2051.51101,2057.50772,2074.45503,2095.92945,2105.89635,2123.91866,2128.87626,2131.82646,2134.81726,2140.85956,2144.83826,2160.78716,2165.83235,2168.82435,2171.81795,2173.81555,2178.87485,2187.82774,2190.80924,2194.71633,2198.70093,2199.85503,2200.81793,2202.70963,2204.68662,2211.78072,2214.64871,2223.65790,2225.72040,2226.61760,2228.70060,2230.66500,2231.65360,2233.65389,2236.64009,2238.66709,2240.67729,2248.63538,2250.64778,2254.63847,2255.70817,2257.78707,2259.67237,2262.67676,2262.73526,2264.66886,2266.74586,2296.77074,2298.81224,2300.74284,2301.78814,2309.78514,2312.78144,2316.76964,2318.74594,2320.72974,2321.73334,2322.72804,2324.69244,2326.66374,2328.67164,2337.68934,2344.71115,2345.63645,2347.64515,2349.69765,2351.64785,2354.64675,2356.68665,2381.58947,2388.52918,2395.59009,2411.54410,2464.92865,2554.75874,2556.77724,2558.82683,2561.76433,2564.77373,2565.70753,2573.78872,2574.77742,2576.78662,2578.79111,2579.79171,2580.78451,2582.77831,2595.83690,2605.61359,2605.80579,2609.65438,2615.61598,2615.81498,2619.58867,2619.74487,2621.64107,2628.84066,2631.63386,2631.81016,2634.68516,2634.85076,2637.63966,2637.81396,2640.63015,2640.77065,2643.65885,2643.77965,2646.61185,2646.78595,2649.63815,2649.81215,2652.58175,2659.65314,2662.61244,2662.71864,2674.62854,2674.79424,2681.62284,2684.66634,2695.57124,2695.73134,2698.63574}','{21.268,21.741,21.957,21.469,21.244,21.824,21.162,21.406,21.531,21.806,21.359,21.699,21.685,21.677,22.090,21.109,20.845,21.544,22.152,21.671,21.907,21.220,22.050,21.728,21.156,21.672,21.259,21.871,21.782,21.943,21.569,21.142,22.155,21.459,21.441,21.273,21.824,21.626,21.358,21.601,21.462,21.670,21.529,21.625,21.779,21.562,21.582,21.514,21.678,21.488,20.784,21.446,21.843,22.197,21.890,21.126,22.136,21.657,21.020,20.963,21.158,21.321,21.329,21.033,22.906,21.259,21.350,20.966,21.791,21.321,21.519,21.372,21.841,21.902,21.540,21.897,21.213,21.788,21.119,21.404,21.539,21.295,21.910,21.720,21.593,21.825,21.935,22.405,21.712,21.872,21.629,23.140,21.635,21.830,21.191,21.617,21.593,21.595,26.123,20.985,23.508,21.035,21.569,22.170,21.487,21.567,21.306,21.004,20.843,21.860,22.039,21.639,21.590,21.171,21.186,21.673,21.758,21.625,21.325,21.541,21.772,21.754,21.512,21.359,21.651,21.573,21.343,21.273,21.372,20.660,22.055,21.674,22.055,21.987,21.294,21.785,21.413,20.760,21.368,21.157,21.124,21.809,21.713,21.236,21.545,21.330,21.587,22.238,21.209,21.778,20.917,21.474,22.215,21.385,21.382,21.261,21.335,21.178,20.582,21.453,20.969,21.824,21.331,22.186,21.340,21.335,21.537,21.448,21.187,20.462,21.123,21.039,21.522,20.963,20.466,21.286,21.529,21.192,21.456,22.168,23.030,21.948,21.413,21.269,21.417,21.262,21.236,21.369,20.944,20.841,20.908,21.583,21.794,21.406,21.579,21.755,21.243,21.231,21.026,21.305,20.344,21.859,22.108,21.713,21.306,21.999,21.264,21.903,21.525,21.373,21.547,20.716,21.477,21.287,21.195,23.715,21.015,21.096,21.400,21.079,21.414,21.133,21.445,21.776,21.433,21.430,20.899,20.864,20.585,21.915,21.279,20.923,20.685,21.588,21.431,22.470,21.380,21.512,21.212,22.066,21.778,22.453,21.028,21.343,21.193,21.430,23.287,21.310,21.519,22.166,22.242,21.391,21.259,21.265,21.644,21.163,20.631,20.340,21.370,21.554,21.844,21.677,21.033,21.305,21.230,20.859,21.032,21.961,20.850,23.591,21.101,21.065,21.288,22.020,20.591,21.580,20.850,21.034,21.664,22.258,21.802,21.979,21.524,20.810,21.378,20.982,20.716,21.931,21.026,20.977,21.169,21.018,21.404,21.821,21.565,21.294,20.968,20.937,21.105,21.112,21.149,21.453,20.852,23.471,22.975,21.159,21.545,21.817,20.662,20.873,22.221,22.344,21.337,21.198,21.109,21.664,21.513,20.380,21.456,21.344,20.809,20.483,21.440,21.637,22.718,21.235,21.066,21.062,22.209,21.838,21.815,22.377,20.878,20.697,21.280,21.218,21.304,22.879,21.119,21.488,21.473,21.616,20.702,21.056,20.869,21.222,21.511,21.162,21.250,20.760,20.926,21.178,20.605,21.176,20.966,21.082,21.604,22.475,21.537,21.182,21.477,20.549,21.651,20.938,20.765,21.633,21.078,20.583,20.501,21.300,21.432,21.715,20.445,22.367,21.276,21.037,21.619,21.721,20.701,21.267,20.858,20.946,21.249,20.524,20.954,21.855,22.712,21.666,22.717,22.303,20.430,22.107,20.890,21.270}','{0.200,0.586,0.565,0.310,0.474,0.802,0.220,0.241,0.343,0.375,0.461,0.394,0.332,0.270,0.602,0.424,0.296,0.945,0.546,0.274,2.831,1.409,0.469,0.376,0.518,1.541,2.032,0.710,0.535,1.874,0.427,0.292,1.099,1.318,0.622,1.426,0.462,0.734,0.274,0.605,0.243,0.396,0.391,3.299,3.747,0.323,0.364,0.443,0.499,0.848,0.333,0.320,0.720,1.493,0.554,0.575,1.278,1.062,0.525,0.354,0.396,0.424,0.672,1.354,2.385,0.791,0.502,0.364,1.214,0.554,0.368,0.860,2.274,1.702,0.520,5.891,0.564,1.569,0.331,0.385,0.267,1.113,2.983,0.305,0.647,0.837,2.183,1.299,3.720,4.815,1.758,3.480,2.043,1.113,0.677,3.473,4.355,0.895,54.287,4.352,4.882,0.866,5.116,0.954,1.367,1.713,0.547,0.760,0.703,3.218,3.206,3.299,1.384,0.429,0.398,1.114,1.570,0.610,1.171,1.625,3.232,0.912,1.058,0.381,0.428,0.344,0.394,0.434,0.856,0.390,1.328,0.821,1.315,0.922,0.279,0.457,0.346,0.690,0.974,1.007,0.774,3.425,1.608,0.589,0.643,1.259,0.580,3.838,1.016,4.093,0.584,0.769,3.846,0.532,0.786,0.668,1.873,1.253,0.513,2.447,0.400,3.427,4.699,3.563,2.419,1.019,3.849,0.908,0.643,0.611,1.239,1.259,0.673,0.999,0.497,3.575,0.609,0.516,1.006,3.215,3.145,1.377,0.344,0.489,0.647,0.372,0.860,0.383,0.473,0.606,1.530,1.182,1.573,0.646,0.823,1.084,0.765,0.452,1.290,3.356,0.295,3.266,2.203,4.332,0.634,3.247,0.297,1.701,0.442,1.400,1.117,0.405,1.978,1.291,1.099,5.907,0.526,0.566,0.613,0.559,0.517,0.367,0.422,0.752,0.930,0.852,0.410,0.498,0.544,3.597,1.553,0.757,0.293,1.814,0.460,2.000,0.518,0.493,0.388,1.994,1.171,4.296,0.453,0.474,1.212,3.078,3.983,0.677,3.121,1.899,2.152,1.285,0.438,0.921,2.910,0.990,0.659,17.062,3.161,1.813,2.019,2.637,0.570,1.015,1.486,2.142,2.701,4.209,0.483,5.271,3.967,0.895,1.462,4.728,0.898,3.920,0.459,0.614,3.092,2.092,4.135,2.691,0.953,0.907,2.606,0.718,0.341,3.497,0.685,0.850,0.688,0.476,0.935,1.340,1.897,0.649,0.341,1.422,3.271,0.415,0.389,3.637,0.326,4.721,2.988,0.441,3.893,3.345,0.711,0.327,3.220,0.987,0.627,0.755,0.509,3.766,0.842,0.420,1.391,1.045,0.683,1.365,3.836,3.943,1.849,1.141,1.119,0.601,2.471,3.926,1.387,1.465,0.856,1.095,1.092,0.591,0.942,2.146,0.800,0.966,1.382,4.012,2.213,2.975,1.701,0.574,3.568,0.715,1.688,0.518,1.098,2.601,0.621,1.023,0.944,1.147,3.282,1.886,3.286,3.372,1.425,0.274,3.358,0.580,0.812,3.332,0.367,0.312,0.654,3.349,3.504,3.210,0.356,2.057,0.888,0.552,1.598,2.123,0.336,0.584,0.351,0.306,0.538,0.311,0.928,2.959,2.345,1.011,1.654,2.108,0.465,3.152,0.360,0.614}'


Followup

Message
Date: 2010-05-10 09:07
Sender: Takahiro Itagaki

Sorry for not replying sooner.
(This web app has no PUSH notifications...)

I agree prealloc() in L.463 should be fixed, but it is
not the best fix to remove the code completely because
one CSV row could be larger than the initial buffer size.
I'll continue to research the bug.
Date: 2010-04-30 00:47
Sender: Krzysztof Nienartowicz

at the second attempt I simply commented out this reallocating block and
everything seems to be ok now..
cheers
Krzysztof
Date: 2010-04-29 21:17
Sender: Krzysztof Nienartowicz

Actually rec_buf is expanded every few records, not at every record. How to
avoid this?
Date: 2010-04-29 19:25
Sender: Krzysztof Nienartowicz

Ok, I decorated the code a little and clearly buffer expansion is called with
every record. probably because the field is quoted.

as a quick hack i change to increase the size of the buffer by 1024 bytes
instead of doubling (line numbering might have slightly changed.
lib/parser_csv.c:

456 self->cur = self->rec_buf;
457 }
458 else
459 {
460 int j;
461 char *old_buf = self->field_buf;
462
463 self->buf_len += 1024;
464 self->field_buf = repalloc(self->field_buf, self-
>buf_len);
465 /*

I have no time to understand how to fix this bug fully still, would be grateful
for a fix as reallocating kills performance...
Best Regards
Krzysztof

Attached Files:

Changes:

Field Old Value Date By
ResolutionNone2010-05-10 09:07itagaki
Powered By FusionForge