How can I use a "here document" in the middle of a pipe?

Question

I am wanting to generate some content using a heredoc as a template:

passphrase=$(<passphrase) envsubst <<EOF
apiVersion: v1
kind: Secret
metadata:
  name: openshift-passphrase
stringData:
  passphrase: ${passphrase}
EOF

and pipe it to oc create -f -.

If I add a pipe after the EOF, it doesn't work.

How can I pipe a heredoc with variable substitution to something that consumes it?

Answer 1

First you need to quote any part of EOF just after <<. The most natural way is <<"EOF", but <<E"OF" or even <<""EOF will do. Without this envsubst will get the string with ${passphrase} already expanded. Since envsubst operates on strings with literal $foo or ${foo} substrings, expanding them beforehand means envsubst has nothing to do. Additionally in your case the shell will most probably expand ${passphrase} to an empty string, because the variable definition in your code only affects envsubst, not the shell itself; unless the variable with the same name is (accidentally?) set in the shell beforehand.

Now we get to your explicit question. You can pipe the result to whatever command you wish, but you still need to keep the final EOF in a separate line. One way to do it is like this:

passphrase=$(<passphrase) envsubst <<"EOF" | oc create -f -
apiVersion: v1
kind: Secret
metadata:
  name: openshift-passphrase
stringData:
  passphrase: ${passphrase}
EOF

Or you can run the code you have in a subshell:

( passphrase=$(<passphrase) envsubst <<"EOF"
apiVersion: v1
kind: Secret
metadata:
  name: openshift-passphrase
stringData:
  passphrase: ${passphrase}
EOF
) | oc create -f -

Note Bash Reference Manual says

Each command in a pipeline is executed in its own subshell

so even in the first solution, when we managed to construct our pipe without ( ), its first part (before |) runs in a subshell anyway. The second solution makes this subshell explicit. After we use explicit (, the shell waits for explicit ). This allows us to place something after the terminating EOF.

Surprisingly even with the first solution you can use more than one here document (<<) in a single compound command. Such redirections make little sense in a pipe, but they may be useful with && and ||.

command1 <<EOF && command2 <<EOF || command3 <<EOF
content1
EOF
content2
EOF
content3
EOF

The same rearranged, with explicit subshells:

( command1 <<EOF
content1
EOF
) && ( command2 <<EOF
content2
EOF
) || command3 <<EOF
content3
EOF

Depending on situation you may prefer one notation over the other.

Back to your specific example. With a subshell you don't even need envsubst:

( passphrase=$(<passphrase); oc create -f - <<EOF
apiVersion: v1
kind: Secret
metadata:
  name: openshift-passphrase
stringData:
  passphrase: ${passphrase}
EOF
)

There are interesting differences between this way and the former two:

• This time the subshell itself should expand ${passphrase}, hence <<EOF, not <<"EOF".
• For this to work, the variable must be known to the subshell, not only to oc; this means … passphrase=$(<passphrase) oc create -f - <<… (note the lack of semicolon) wouldn't work.
• Technically the same code not in a subshell (i.e. without ( )) would also work, but then the variable would remain in the main shell. Running the code in a subshell makes the variable die with it. In your original code the variable is not set for the main shell, so I guess this is what you want.