[CALUG] More I/O buffering foolishness

[Jason C. Miller]

Per suggestion (1):
  Haven't tried that one yet.  Sounds interesting.

Per suggestion (2):
  Good try, but.....   :)
   It's not the output of my script that's being buffered, it's the 
input (which is the output of the command before  
   it).  The 'tar' command that comes with solaris (which is where the 
original output is coming from), unlike GNU
   tar, doesn't have an option to unbuffer output when STDOUT is not 
attached to a TTY.  Therefore, if the input
   to your PERL script is buffered, then how can its output not be 
buffered as well?

Per suggestion(3):
   As mentioned in (2), the non-GNU tar that comes with Solaris (which 
is the command on the left of the pipe)
   doesn't have such an option. 


                                                    -jason



Mordechai T. Abzug wrote:

>On Wed, Mar 01, 2006 at 09:39:22PM -0500, Jason C. Miller wrote:
>  
>
>>Arg.  Alright....some more details.  :)
>>
>>1. It is a Solaris 8 machine
>>2. My code is not as simple as a single perl print() statement.  That 
>>was simply there to provide a simple example that doesn't work on my 
>>system.  It works fine at the command prompt but not from a script 
>>(which, after more reading, I understand a lot more about how programs 
>>see what devices STDOUT is attached to and how they act accordingly).
>>3. As mentioned, it doesn't matter what command (perl, awk, sed, grep, 
>>etc) comes after that next pipe.  All the STDIN for those programs are 
>>block-buffered.
>>    
>>
>
>[Was going through old email, and saw this.  Apparently, the problem is
>not fixed, so I'm responding even though it's an old post.]
>
>This issue has bitten me in the past.  As someone else said, the
>underlying problem is that *nix libc and other *nix I/O
>implementations tend to autodetect if their stdout is a TTY or a
>file/pipe, and adjust buffering appropriately.  The easiest way to
>demo this is to run a program with slow output, and then run the same
>program while piping to cat; if you see different results, you're
>seeing buffering.  Examples:
>
>perl -le 'while (1) {print ++$a; sleep 1}'
>perl -le 'while (1) {print ++$a; sleep 1}'|cat
>
>There are various fixes:
>
>(1) If you have expect, there is a script called "unbuffer" that comes
>    with it that acts as a wrapper for any program you run,
>    redirecting output through a pty.  The upshot of this is that the
>    program switches to line-buffered output.  So instead of "command1
>    | command2 | command3", you can use "unbuffer command1 | unbuffer
>    command2 | command3".  For the sample case:
>    unbuffer perl -le 'while (1) {print ++$a; sleep 1}'|cat
>
>    This solution is my preferred solution for this problem in the
>    general case, since it is portable and requires no source code
>    modifications.
>
>    The only issues I have had with it are: (1) some Linux distros
>    don't put "unbuffer" in the default path even after you install
>    expect; and (2) older versions of unbuffer took no arguments,
>    while newer versions require -p for unbuffer to work with stdin.
>
>(2) If you can modify the script/program in question, you can usually
>    have it specify no buffering or line buffering.  Perl has a "$|"
>    variable that can disable the current (ie. stdout) buffering
>    completely, or if you use FileHandle/IO::Handle, you can use
>    setvbuf() to set line buffering, or you can manually flush() after
>    each line.  C has a setvbuf(3) call, or you can manually fflush()
>    after each line.  For the sample case:
>
>    perl -le '$|=1; while (1) {print ++$a; sleep 1}'|cat
>
>    This is portable, but only works for scripts/programs for which
>    you have the source code and are willing to modify them.
>
>    In general, it's better to setup line buffering than no buffering.
>    You can have performance problems disabling buffering altogether.
>
>(3) Some canned commands have special options to disable buffering or
>    enable line buffering.  tcpdump's "-l" leaps to mind.  This is not
>    a general case, but is worth keeping in mind.
>
>- Morty
>  
>


-- 
***************************************************
My blog: http://millersplace.blogspot.com/ 
***************************************************