Re: bytea memory improvement

Kris Jurka wrote:
>
>
> On Wed, 28 Jun 2006, Luis Vilar Flores wrote:
>
>> Can any JDBC developer give some feedback about this patch ?
>> Memory usage is a real issue for a few applications ...
>>
>
> It looks like a reasonable thing to do, but could you give us some
> more details on the cost/benefits?
My sugestion is to trade some CPU time for memory in large files.
The benefits are: memory saving (one byte[] of final size at least),
less garbage collecting, these buffers are always created on a bytea read,
we also save the CPU time of one System.arraycopy, but this is a native
method and should be very fast, so it's not a big deal.
The only cost is one more pass through the original buffer, this pass
is just to count the final array size.

In simple terms, the original method did:
incoming buffer size N
allocate temp buffer size N
for through incoming buffer, translate data from incoming to temp
allocate final buffer, size M (M is between N and N/4)
copy temp buffer to final buffer
return final buffer

The new method does:
incoming buffer size N
for through incoming buffer, calculate final buffer size
allocate final buffer, size M (M is between N and N/4)
for through incoming buffer, translate data from incoming to final
return final buffer

> Your original email said it cost an extra 30ms for 5MB of data on
> your machine. What's percentage
> of the original cost is this?
This cost is to read and count the escaped bytes of a 5MB byte array in
a for cycle on a CeleronM 1.6, but keep in mind that we save the last
System.arraycopy (last if in the method).
Originally this extra passage didn't existed.
I will make more detailed timing in the full method body (old and new),
and send results tomorrow.
> Also you could be more clear on what percentage of memory this saves.
> For the worst case scenario your going to get four bytes of escaped
> data for every real byte so the total size of the original method
> would be 4 + 4 + 1 and for the new method 4 + 1, so a savings of 44%?
> Is that what you've calculated?
Yeap, that's for files larger than 1MB on the incoming buffer (the
threshold could be defined by some property) - with the extra pass on
the incoming array I calculate the final size, so I can skip the temp
buffer, so we save the 44% (on worst case), the minimum savings is 16.7%
for no escaped data.
>
> Finally you haven't actually submitted a patch, you've just sent a
> modified copy of a whole file. Since it's a small file that changes
> infrequently it's not a big deal, but we prefer context diffs if you can.
>
> Kris Jurka
>
> ---------------------------(end of broadcast)---------------------------
> TIP 1: if posting/reading through Usenet, please send an appropriate
> subscribe-nomail command to majordomo(at)postgresql(dot)org so that your
> message can get through to the mailing list cleanly
>
I will send the diff too.

Thanks for the comments ...

--

Luis Flores

Analista de Sistemas

*Evolute* - Consultoria Informática

<http://www.evolute.pt> Email: lflores(at)evolute(dot)pt
<mailto:lflores(at)evolute(dot)pt>

Tel: (+351) 212949689

AVISO DE CONFIDENCIALIDADE
Esta mensagem de correio electrónico e eventuais ficheiros anexos são
confidenciais e destinados apenas à(s) pessoa(s) ou entidade(s) acima
referida(s), podendo conter informação privilegiada e confidencial, a
qual não poderá ser divulgada, copiada, gravada ou distribuída nos
termos da lei vigente. Caso não seja o destinatário da mensagem, ou se
ela lhe foi enviada por engano, agradecemos que não faça uso ou
divulgação da mesma. A distribuição ou utilização da informação nela
contida é interdita. Se recebeu esta mensagem por engano, por favor
notifique o remetente e apague este e-mail do seu sistema. Obrigado.

CONFIDENTIALITY NOTICE
This e-mail transmission and eventual attached files are intended only
for the use of the individual(s) or entity(ies) named above and may
contain information that is both privileged and confidential and is
exempt from disclosure under applicable law. If you are not the intended
recipient, you are hereby notified that any disclosure, copying,
distribution or use of any of the information contained in this
transmission is strictly restricted. If by any means you have received
this transmission in error, please immediately notify the sender and
delete this e-mail from your system. Thank you.