MacOSX 10.6.7 cuts off stdin at 1024 chars

I've written a little perl script that I invoke as follows:

perl -pe'...' <a

I wanted to copy the contents of the input file 'a' from a web browser (a whole wordpress blog entry, to be exact). The copy part works perfectly, the full contents show up in ClipMenu. But when I try to paste it into a file using cat and redirect:

cat >a

It emits exactly 1024 characters, then stops accepting input - only CTRL-C can help. I've tried pasting only 1023 bytes/chars, and that works, after pressing enter, I could enter it into the file without any problems. But if I entered 1024 chars on a single line, there was no way to do anything else other than CTRL-C.

I've tried iTerm and Terminal.app, both show the exact same behavior. I've tried bash from macports and /bin, with the exact same results. I've even tried dd:

dd bs=1M of=a

To the exact same result. Nothing seems to help.

I'm clueless - why does this happen? And how could I copy-paste into a file on Mac OS X easily, and in a way that I can be sure the contents are not altered in any way?

Agoston Horvath

Posted 2011-03-30T10:31:17.913

Reputation: 243

That last part is a completely separate question, right? – Daniel Beck – 2011-03-30T10:52:24.050

Well spotted, Daniel. It is sort of an explanation of the job I want to do, one way or another, that puts the question into perspective. – Agoston Horvath – 2011-03-30T10:57:34.373

2Separate questions should go into separate, well, questions on this site. A single "I have the following problems:" topic doesn't help anyone but the original poster (granted, your second question is harmless in comparison) -- you should probably focus on your 1024 character issue in this topic. I'm curious, why is it you want to paste exactly what the source contains? In the realm of text, you might end up with multiple incompatible character sets in the same file. – Daniel Beck – 2011-03-30T11:01:30.800

@Daniel: I was trying to fix links in the HTML version of a wordpress blog entry that have been vandalised by wordpress' buggy visual editor. Wordpress, firefox, iTerm and macos console all support unicode, so charsets are no problem. To do this, the most simplistic approach (without using any brainz) was to select the full blog entry, copy it to a file, execute perl script, and then copy it back. And then I've hit this weird issue that I've spent an hour trying to explain and google for, without any results. – Agoston Horvath – 2011-04-06T10:16:06.487

Answers

% pbpaste | perl -e '...'

You're hitting the maximum line length for the tty. If you switch the tty out of line mode, it'll work.

But pbpaste is really what you want.

ZILjr

Posted 2011-03-30T10:31:17.913

Reputation: 331

1I'm interested in this 'switch tty out of line mode' - how can one do that? Can you give some pointers? – Agoston Horvath – 2011-04-06T10:18:25.690

Check the man page for "stty" to learn more. It's not for the faint of heart, and basically will break everything you come to expect about your terminal. But it can be done temporarily, and the end result will be... identical to using pbpaste as I described above. – ZILjr – 2011-08-17T06:23:14.010

Every unix I've ever used has a limit on the length of cooked-mode input, though it probably differs from OS to OS. That's why you can only type so much in one line when you do a cat>a. The OS buffers input from the TTY so that you can do editing (e.g., backspace over text and change it) before sending it to the applciation (cat, in this case). The OS allocates a fixed-size buffer for this, and you're running into its limit. Your best option is to use an interactive program to write the file. For example, you could run vi and enter the

:set paste

command to turn off auto-indenting. Press "i" to enter insert mode, paste your code in, hit escape, and run:

:w a
:q

To save the file and quit.

George

Posted 2011-03-30T10:31:17.913

Reputation: 1 496

1024 seems a bit low as limit though. Any way to increase it? – Daniel Beck – 2011-03-30T20:36:05.200

I've never hit any such limit with linux. Then again, I think Debian/Ubuntu has remote echo enabled since a looong time. Macos seems to lag behind (as usual). – Agoston Horvath – 2011-04-06T10:20:49.260