91
39
I am using the bash shell and would like to pipe the out of the command openssl rand -base64 1000
to the command dd
such as dd if={output of openssl} of="sample.txt bs=1G count=1
.
I think I can use variables but I am however unsure how best to do so. The reason I would like to create the file is because I would like a 1GB file with random text.
3Note if it says
dd: warning: partial read (33554431 bytes); suggest iflag=fullblock
it will create a truncated file so add theiflag=fullblock
flag, then it works. – rogerdpack – 2018-09-27T20:21:42.443Thanks. A few questions, does using the command
openssl rand -base64 $(( 2**30 * 3/4 )) > sample.txt
give you a true text file? Secondly I don't quite follow the use ofbs=64M count=16
. Can you elaborate further? – PeanutsMonkey – 2012-09-06T19:06:05.5132
I posted a question regarding compressing large files at http://superuser.com/questions/467697/why-does-a-zip-file-appear-larger-than-the-source-file-especially-when-it-is-tex and was advised that using
– PeanutsMonkey – 2012-09-06T19:10:03.823/dev/urandom
generates a binary file and not a true text file.@PeanutsMonkey: What do you mean by a "true text file"? A file that only contains printable characters, I'm guessing? Then yes, the
-base64
option tells OpenSSL to output a "text" file. – user1686 – 2012-09-06T19:23:21.150@PeanutsMonkey: But beware that random data does not compress well, regardless of whether it is "binary" or "true text". – user1686 – 2012-09-06T19:23:52.223
2@PeanutsMonkey: Right; you would need something like
dd if=/dev/urandom bs=750M count=1 | uuencode my_sample > sample.txt
. – Scott – 2012-09-06T19:33:41.873@Scott - Can you elaborate what that does exactly as well as why you are using a byte size of 750M and a count of 1? – PeanutsMonkey – 2012-09-06T19:52:58.103
@grawity - Well people keep bouncing the term "true text file" and based on my previous post it was suggested that
/dev/urandom
generates binary files. My understanding is that a text file is one with printable characters although am unsure whether ASCII characters would count. I thought-base64
is used to convert binary data to text? – PeanutsMonkey – 2012-09-06T19:56:09.253@grawity - If random data does not compress well, how can I create a file that mimics real world scenarios? – PeanutsMonkey – 2012-09-06T19:56:46.023
3
@PeanutsMonkey: There's no single "real world scenario", some scenarios might be dealing with gigabytes of text, others – with gigabytes of JPEGs, or gigabytes of compiled software... If you want a lot of text, download a Wikipedia dump for example.
– user1686 – 2012-09-06T20:06:27.3932@PeanutsMonkey: The
dd
reads 750,000,000 bytes from/dev/urandom
and pipes them intouuencode
.uuencode
encodes its input into a form of base64 encoding (not necessarily consistent with other programs). In other words, this converts binary data to text. I used 750M because I trusted grawity's statement that base64 encoding expands data by 33⅓%, so you need to ask for ¾ as much binary data as you want in your text file. – Scott – 2012-09-06T20:07:33.557@Scott: Pure Base64 always encodes 3 bytes to 4 (33.(3)%). OpenSSL's encoder splits output into 64-character lines (so about 35.4% overhead; I forgot to account for this – would be
*48/65
). UUencode uses even shorter lines and adds length prefixes, header & footer, resulting in ~40% overhead. – user1686 – 2012-09-06T20:15:09.953@Scott - That makes sense although am curious to understand why limit the count to 1? – PeanutsMonkey – 2012-09-06T20:41:56.887
@grawity - I am astounded by the depth of knowledge. Where are you learning all of this stuff? – PeanutsMonkey – 2012-09-06T20:42:24.870
1@leighmcc: FYI: using
>
redirection does not make the writes pass through bash – it is equivalent to having the program open the file directly. – user1686 – 2013-05-10T14:03:14.433