Re: string port slow output on big string

[Alan Grover]

Exponential growth set off a warning bell for me, but you probably have
other problems by the time it bites you (consider what you are doing to
the page-cache when you copy to the new block).

After about the 6th allocation, things converge such that 1/6 of that
total allocation is unused on average, i.e. there's a reserve on average
of 1/3 the string's size (1/3 of the string was just allocated). E.g. A
1mb string implies an estimated 300kb reserve (actual: ~148k, which is
~1/8).

A 1mb string takes 22 allocations/moves (~15000 under the previous
code), 1gb requires 39 allocations/moves (about ~15,000,000 under the
previous code).

Kevin Ryde wrote:
> I made the change below, it leaves the code alone, just grows the
> buffer more each time, by a factor 1.5x so copying time is no longer
> quadratic in the output size.
> 
> I think I'll do this in the 1.6 branch too.  Backtraces there have
> been slow to the point of unusable for me in some parsing stuff I've
> been doing with say 50k or so strings in various parameters.  (The
> backtrace goes via an output string so that it can truncate big args
> like that.)
> 
> 
> 
> 
> ------------------------------------------------------------------------
> 
> Index: strports.c
> ===================================================================
> RCS file: /cvsroot/guile/guile/guile-core/libguile/strports.c,v
> retrieving revision 1.108
> diff -u -u -r1.108 strports.c
> --- strports.c        23 May 2005 19:57:21 -0000      1.108
> +++ strports.c        1 Aug 2005 23:47:06 -0000
> @@ -65,7 +65,30 @@
>     has been written to, but this is only updated after a flush.
>     read_pos and write_pos in principle should be equal, but this is only true
>     when rw_active is SCM_PORT_NEITHER.
> -*/
> +
> +   ENHANCE-ME - output blocks:
> +
> +   The current code keeps an output string as a single block.  That means
> +   when the size is increased the entire old contents must be copied.  It'd
> +   be more efficient to begin a new block when the old one is full, so
> +   there's no re-copying of previous data.
> +
> +   To make seeking efficient, keeping the pieces in a vector might be best,
> +   though appending is probably the most common operation.  The size of each
> +   block could be progressively increased, so the bigger the string the
> +   bigger the blocks.
> +
> +   When `get-output-string' is called the blocks have to be coalesced into a
> +   string, the result could be kept as a single big block.  If blocks were
> +   strings then `get-output-string' could notice when there's just one and
> +   return that with a copy-on-write (though repeated calls to
> +   `get-output-string' are probably unlikely).
> +
> +   Another possibility would be to extend the port mechanism to let SCM
> +   strings come through directly from `display' and friends.  That way if a
> +   big string is written it can be kept as a copy-on-write, saving time
> +   copying and maybe saving some space.  */
> +
>  
>  scm_t_bits scm_tc16_strport;
>  
> @@ -117,7 +140,14 @@
>  #define SCM_WRITE_BLOCK 80
>  
>  /* ensure that write_pos < write_end by enlarging the buffer when
> -   necessary.  update read_buf to account for written chars.  */
> +   necessary.  update read_buf to account for written chars.
> +
> +   The buffer is enlarged by 1.5 times, plus SCM_WRITE_BLOCK.  Adding just a
> +   fixed amount is no good, because there's a block copy for each increment,
> +   and that copying would take quadratic time.  In the past it was found to
> +   be very slow just adding 80 bytes each time (eg. about 10 seconds for
> +   writing a 100kbyte string).  */
> +
>  static void
>  st_flush (SCM port)
>  {
> @@ -125,7 +155,7 @@
>  
>    if (pt->write_pos == pt->write_end)
>      {
> -      st_resize_port (pt, pt->write_buf_size + SCM_WRITE_BLOCK);
> +      st_resize_port (pt, pt->write_buf_size * 3 / 2 + SCM_WRITE_BLOCK);
>      }
>    pt->read_pos = pt->write_pos;
>    if (pt->read_pos > pt->read_end)
> 
> 
> ------------------------------------------------------------------------
> 
> _______________________________________________
> Guile-devel mailing list
> address@hidden
> http://lists.gnu.org/mailman/listinfo/guile-devel

-- 
Alan Grover
address@hidden
+1.734.476.0969

signature.asc
Description: OpenPGP digital signature