From nyamada at millburncorp.com  Sat Mar  8 15:33:50 2008
From: nyamada at millburncorp.com (Norman Yamada)
Date: Sat, 8 Mar 2008 10:33:50 -0500
Subject: [Dbi-link-general] dbi-link and limit queries on remote servers --
Message-ID: <B2CE72FB-4367-43B6-AAB1-0F79DD629384@millburncorp.com>

Dear all:

Having set up a schema for a remote postgres database, I tried the  
following query:

select * from signals limit 1;

The query was taking forever to come back, so I looked on the remote  
box and saw that the query sent in was:

select * from signals;

Since signals is a _very_ large table, I tried to kill the query from  
my psql prompt with a Ctrl-C. The local database server hung, and then  
eventually crashed and recovered.

Augh!

1) Are limit queries broken in dbi-link for postgres=>postgres tables?
2) How do I kill a runaway dbi-link query?

Thanks,

Norman Yamada
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 2431 bytes
Desc: not available
Url : http://pgfoundry.org/pipermail/dbi-link-general/attachments/20080308/1a4b20c6/attachment.bin 

From david at fetter.org  Sat Mar  8 16:26:13 2008
From: david at fetter.org (David Fetter)
Date: Sat, 8 Mar 2008 08:26:13 -0800
Subject: [Dbi-link-general] dbi-link and limit queries on remote	servers
	--
In-Reply-To: <B2CE72FB-4367-43B6-AAB1-0F79DD629384@millburncorp.com>
References: <B2CE72FB-4367-43B6-AAB1-0F79DD629384@millburncorp.com>
Message-ID: <20080308162613.GC23803@fetter.org>

On Sat, Mar 08, 2008 at 10:33:50AM -0500, Norman Yamada wrote:
> Dear all:
>
> Having set up a schema for a remote postgres database, I tried the 
> following query:
>
> select * from signals limit 1;
>
> The query was taking forever to come back, so I looked on the remote box 
> and saw that the query sent in was:
>
> select * from signals;

That's a limitation inherent in current versions of DBI-Link, which in
turn derives from the invisibility of qualifiers to set-returning
functions.  Neil Conway has written a preliminary patch, a descendent
of which I hope to get into PostgreSQL 8.4, which makes these
qualifiers available to applications like DBI-Link.

> Since signals is a _very_ large table, I tried to kill the query
> from my psql prompt with a Ctrl-C. The local database server hung,
> and then eventually crashed and recovered.
>
> Augh!

I'm sorry about that.

> 1) Are limit queries broken in dbi-link for postgres=>postgres tables?

The workaround I currently recommend is to use remote_select()
directly, including your qualifiers, and to cast the output using the
AS () syntax.  I understand that this is not optimal, but until
PostgreSQL supports something better, we are stuck with it or
something very much like it.

> 2) How do I kill a runaway dbi-link query?

I thought control-C would do it, but your situation makes me unsure.
Would you be willing to look into how best to get perl to know about
this?

Cheers,
David.
-- 
David Fetter <david at fetter.org> http://fetter.org/
Phone: +1 415 235 3778  AIM: dfetter666  Yahoo!: dfetter
Skype: davidfetter      XMPP: david.fetter at gmail.com

Remember to vote!
Consider donating to Postgres: http://www.postgresql.org/about/donate

From nyamada at millburncorp.com  Sat Mar  8 19:39:53 2008
From: nyamada at millburncorp.com (Norman Yamada)
Date: Sat, 8 Mar 2008 14:39:53 -0500
Subject: [Dbi-link-general] dbi-link and limit queries on remote servers
 --
In-Reply-To: <20080308162613.GC23803@fetter.org>
References: <B2CE72FB-4367-43B6-AAB1-0F79DD629384@millburncorp.com>
	<20080308162613.GC23803@fetter.org>
Message-ID: <49FDBB8D-00D8-4CDD-A8A9-CD8C731E3B3B@millburncorp.com>

Hm. Being a newbie to dbi-link, doesn't this mean that dbi-link at the  
moment is useful only if your tables are small enough to fit  
comfortably in working memory?

What would the advantage of using remote_select be over using dblink?  
(which I've done in the past -- but, yes, the casting syntax is  
painful).

When I can set up a database server that I can safely crash at will,  
I'll see if I can dig into the ctrl-c problem with dbi-link. It's a  
little scary: I tried the same query scenario again (start a long- 
running remote query; hit ctrl-C); but this time I manually stopped  
the server before it crashed.  The load average on the remote database  
server had started to climb steadily, even though there were no other  
queries running on the box.

On Mar 8, 2008, at 11:26 AM, David Fetter wrote:

>>
>> 1) Are limit queries broken in dbi-link for postgres=>postgres  
>> tables?
>
> The workaround I currently recommend is to use remote_select()
> directly, including your qualifiers, and to cast the output using the
> AS () syntax.  I understand that this is not optimal, but until
> PostgreSQL supports something better, we are stuck with it or
> something very much like it.
>
>> 2) How do I kill a runaway dbi-link query?
>
> I thought control-C would do it, but your situation makes me unsure.
> Would you be willing to look into how best to get perl to know about
> this?
>
> Cheers,
> David.
> -- 

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 2431 bytes
Desc: not available
Url : http://pgfoundry.org/pipermail/dbi-link-general/attachments/20080308/fc074665/attachment.bin 

From david at fetter.org  Sat Mar  8 20:16:36 2008
From: david at fetter.org (David Fetter)
Date: Sat, 8 Mar 2008 12:16:36 -0800
Subject: [Dbi-link-general] dbi-link and limit queries on remote	servers
	--
In-Reply-To: <49FDBB8D-00D8-4CDD-A8A9-CD8C731E3B3B@millburncorp.com>
References: <B2CE72FB-4367-43B6-AAB1-0F79DD629384@millburncorp.com>
	<20080308162613.GC23803@fetter.org>
	<49FDBB8D-00D8-4CDD-A8A9-CD8C731E3B3B@millburncorp.com>
Message-ID: <20080308201636.GH23803@fetter.org>

On Sat, Mar 08, 2008 at 02:39:53PM -0500, Norman Yamada wrote:
> On Mar 8, 2008, at 11:26 AM, David Fetter wrote:
>>> 1) Are limit queries broken in dbi-link for postgres=>postgres
>>> tables?
>>
>> The workaround I currently recommend is to use remote_select()
>> directly, including your qualifiers, and to cast the output using
>> the AS () syntax.  I understand that this is not optimal, but until
>> PostgreSQL supports something better, we are stuck with it or
>> something very much like it.
>>
>>> 2) How do I kill a runaway dbi-link query?
>>
>> I thought control-C would do it, but your situation makes me
>> unsure.  Would you be willing to look into how best to get perl to
>> know about this?
> 
> Hm. Being a newbie to dbi-link, doesn't this mean that dbi-link at
> the moment is useful only if your tables are small enough to fit
> comfortably in working memory?

No, because there is a way to do it with remote_select().  DBI-Link
goes with the Perl philosophy of making easy things easy and hard
things possible.

> What would the advantage of using remote_select be over using dblink? 

Heterogeneous data stores.  DBI-Link can talk to anything DBI can talk
to, most easily things which have the DBI::table_info(),
and DBI::column_info() methods.

> (which I've done in the past -- but, yes, the casting syntax is
> painful).
>
> When I can set up a database server that I can safely crash at will,
> I'll see if I can dig into the ctrl-c problem with dbi-link.  It's a
> little scary: I tried the same query scenario again (start a
> long-running remote query; hit ctrl-C); but this time I manually
> stopped the server before it crashed.  The load average on the
> remote database server had started to climb steadily, even though
> there were no other queries running on the box.

I'm guessing I'll have to put in some signal-handling magic in the
PL/PerlU code on the DBI-Link side.

Thanks for letting me know about this :)

Cheers,
David.

P.S.  I rearranged your response so it's kind of in-line.  In future,
please respond in-line rather than top-posting as you did, as
responding in-line makes threads easy to follow.
-- 
David Fetter <david at fetter.org> http://fetter.org/
Phone: +1 415 235 3778  AIM: dfetter666  Yahoo!: dfetter
Skype: davidfetter      XMPP: david.fetter at gmail.com

Remember to vote!
Consider donating to Postgres: http://www.postgresql.org/about/donate