1
We are parsing some large EDI files that do not contain CR/LF. However, they do have ~
(tilde) as a segment delimiter.
I am trying to extract the control record for the file and the last bytes of my 120 MB file look something like this:
~REF*1L*0711882~SE*62300*39093~GE*1*500001242~IEA*1*500001241~
There is only one control record in the file and it always starts with ~SE
.
So, is there an easy way using standard Unix cut, awk, grep, etc. tools to cut this file to get the SE*62300*39093 segment, other than converting the ~
to CRLF and tailing the last three lines of the file?
Disclaimer:
I am not a Unix guru, so the answer may be obvious to an experienced user. Also, I have no control over the file format.
What's wrong with converting the
~
to newlines and tailing the last 3 lines of the file. If the file is known to not already contain newlines then this does not introduce any ambiguity into the format, and frankly it's the best way to massage the file into a format that makes it easy for all those line-based tools to work with. – Celada – 2013-01-31T20:11:04.483@Celada: I'm not a unix person, but converting hundreds of megabytes to extract the last 100 or so characters just seems like overkill; Some of these files can be very large, and I'm looking for the easiest way to do this. – Noah – 2013-01-31T20:15:49.023
You can filter down to the last few lines of a file using
tail
. No need to parse it all. Something liketail edi_file | grep ~SE | cut -d'~' -f 3
(where edi_file is the name of your large file) (Disclaimer: Example only works if the required field is in field #3 (delimited by ~'s as by -d~
. That might need adjusting. Can we get a larger size example of the input file? – Hennes – 2013-01-31T20:16:57.143120MB is not so big. Nobody ever worried about squeezing every last bit of performance out of a shell script. If you want that, use C :-) So Michael Kohne's answer is pretty much what I would do. Or if the file really is too big for you to want to read the whole thing, pre-filter it with something like
tail --bytes=5000 ding
... and then you hope that the last 5000 bytes are enough to encompass the 3 lines that you need. – Celada – 2013-01-31T20:33:19.277For a once off thing. I agree. Let it run. For something used daily I like only to only parse the tail. Both because it is not wasteful and because it just feels wrong to needlessly waste. (Not that trying to come up with an answer for 20 minutes is not wasteful. No --bytes option in BSD find though). – Hennes – 2013-01-31T20:36:59.457
@Hennes: You may have missed it in the question, there is only 1 line in the file. – Noah – 2013-01-31T20:40:43.887
Aye. Missed that 5 minutes into solving it. Tried to come up with a way to only read the last few KB of a file, but no option for that in BSD's find. Ended with
tr "~" "\n" < edi_file | tail -20 | grep ^SE
which already matches the answer from Michael. – Hennes – 2013-01-31T20:42:59.990