[ python-Bugs-1000420 ] Implement message streaming

[noreply at pgfoundry.org]

Bugs item #1000420, was opened at 2005-11-16 13:56
You can respond by visiting: 
http://pgfoundry.org/tracker/?func=detail&atid=442&aid=1000420&group_id=1000094

Category: PQ/PQueue
Group: To do
>Status: Closed
>Resolution: Rejected
Priority: 3
Submitted By: James Pye (jwp)
Assigned to: James Pye (jwp)
Summary: Implement message streaming

Initial Comment:
Currently, PQueue's buffer implementations require all of the message data to be received before returning the message. The problem shows it's ugly face when Very Large Messages(tm) are received by the client. This is quite likely to occur if the database stores large bytea columns.

My current vision is to alter the passive buffer's semantics(or perhaps extend them in a subclass thereof) to take the reader upon instantiation so that it creates raw message interfaces that takes data directly from the reader, if the message size were to justify streaming.
This idea would require some alterations to the message class parsers so that references to past accessed data is not done, and string operations are only done after a Python string is resolved from the raw message interface(The current parsing methods, more or less, assume a Python string interface).

This solution will only improve the performance of the transformations from raw messages to structured messages.

Further improvements to extend the streaming queue may be provided by the structured message parsing methods.

----------------------------------------------------------------------

>Comment By: James Pye (jwp)
Date: 2007-02-25 14:07

Message:
I'm not so sure this is actually worth the effort anymore. ISTM, that if this were actually done, it would merely pass the problem onto local tuple instantiation--which has it's own issues as well(protocol transaction completion depends on everything being read...). While, I do believe this is ideal, I don't feel that jumping through hoops to get the laziness is really worth it.

I think it might be best to force the user to code more defensively against such potential problems.

----------------------------------------------------------------------

You can respond by visiting: 
http://pgfoundry.org/tracker/?func=detail&atid=442&aid=1000420&group_id=1000094