How to split large file on Windows?

Is there a tool to split large text file (9Gb) into smaller files so that I can open it and look through?

Anything usable from command line that comes with Windows (XP)?

Or what's the best way to split it? Can I use 7z to create separate volumes and then unzip one of them separately? Will it be readable or does it need all the other parts to unzip into the big file again?

Update

I put together quick 48 lines python script that split the large file into 0.5GB files which are easy to open even in vim. I've just needed to look through data towards the last part of the log (yes it is a log file). Each record is split across multiple lines so grep would not do.

stefanB

Posted 2010-01-11T01:01:09.763

Reputation: 923

I notice you solved your problem, If you're still around, could you post the solution so others might benefit? – Journeyman Geek – 2012-10-19T03:30:02.183

This has been discussed in much detail at SO

– Rishi Dua – 2013-06-26T06:33:00.453

I see you edited to mention grep. Do you have cygwin or unxutils installed? You could have used grep -n with head and tail to see chunks of the file. Example, grep -n "something" file.txt returns 95625: something. You want to see that line and 9 lines below it for a total of 10 lines: head -n 95635 file.txt | tail -n 10. – John T – 2010-01-11T04:10:04.923

Answers

There is a freeware Windows file splitter called HJSplit

Available here. The website claims it can split files of any type and size, but 9GB is a big file.

pavium

Posted 2010-01-11T01:01:09.763

Reputation: 5 956

HJSplit works so ugly... It does not work with 5Gb+ files. The biggest part of file just losing at all. Chunck peases counter nofuses after 999, and something else failes if use bigger peases – user2602807 – 2016-09-28T12:08:23.983

On 64bit systems it only works with a GUI in Windows. If you want to automate things, this is not a solution – 576i – 2017-11-30T10:20:16.157

hjsplit is a dead URL – Serge Pavlov – 2019-11-14T17:57:49.753

1According to the publisher, it supports splitting of files "over 100 Gigabytes" – yosh m – 2013-04-11T07:57:43.560

8I was able to use this program to split a large file into smaller pieces, but only by size (kB or MB). It does not respect lines, so it's not very useful for splitting log files. It's also very slow. – nullability – 2014-03-19T17:46:49.117

The GNU Core Utils package (available here for windows) includes the Split utility. It works well for me :-)

The --help documentation is as follows:

Usage: split [OPTION] [INPUT [PREFIX]]
Output fixed-size pieces of INPUT to PREFIXaa, PREFIXab, ...; default
size is 1000 lines, and default PREFIX is `x'.  With no INPUT, or when INPUT
is -, read standard input.

Mandatory arguments to long options are mandatory for short options too.
  -a, --suffix-length=N   use suffixes of length N (default 2)
  -b, --bytes=SIZE        put SIZE bytes per output file
  -C, --line-bytes=SIZE   put at most SIZE bytes of lines per output file
  -d, --numeric-suffixes  use numeric suffixes instead of alphabetic
  -l, --lines=NUMBER      put NUMBER lines per output file
      --verbose           print a diagnostic to standard error just
                            before each output file is opened
      --help     display this help and exit
      --version  output version information and exit

SIZE may have a multiplier suffix: b for 512, k for 1K, m for 1 Meg.

For example, to split input.txt into 100Mb chunks, only splitting at the ends of lines,

split input.txt -C 100m

will give you output files named xaa, xab, xac, etc.

Flyto

Posted 2010-01-11T01:01:09.763

Reputation: 516

Cmder is amazing for this – Umber Ferrule – 2016-03-20T00:36:42.347

2for example git msys download includes the utility as well. – eis – 2014-05-20T06:55:04.473

Another is GSplit - according to their site it can split very large files (larger than 4Gb <-- since they crossed the 4Gb limit, I guess they can do 9 Gb as well).

But, another thing - you say you want to split it into smaller parts so you can open it up and look at it. That sounds like a very big perhaps log file.

In any case, for opening large text files, may I recommend EmEditor - they claim themselves it can open very large files (up to cca. 250 Gb), and I've used it in the past for files up to 2 Gb. But in any case, I think it may be a better solution than splitting.

Rook

Posted 2010-01-11T01:01:09.763

Reputation: 21 622

I've just tried out GSplit. It works, albeit through a confusing interface, but it doesn't offer any way to only split at a linebreak - for splitting ASCII data files, therefore, it's not too useful as the split will be half way through a line. – Flyto – 2013-05-10T15:09:55.393

1@Flyto It actually is possible to split by line if you choose "Split After Occurrence Number" under the Pieces > Type and Size menu. You then choose the number of lines and the line delimiter in hex. By default it has Windows format CR+LF (0x0D0x0A). – nullability – 2014-03-19T17:56:06.280

@nullability thanks - I wouldn't have thought of that. But, I found a much simpler solution through the GNU "split" utility - see my new answer :-) – Flyto – 2014-03-19T22:23:12.570

One can use 7-Zip to create segments of text file in certain size (e.g. 100 MB segments out of 1.5 GB log file).

The Key options are - Use "Store" as opposed to "Compress" - Use "Split to volumes"

You should be able to see text in .001 (.nnn) files.

enter image description here

Mehul

Posted 2010-01-11T01:01:09.763

Reputation: 91

The attached image is perfect! – Erik – 2018-11-05T13:04:11.150

Check out Large Text File Viewer, it's great for things like this. Most archivers and splitters will separate the file into pieces which cannot be used to read each piece of data independently and properly, you need to extract them all to get the file back.

alt text

Large Text File Viewer is free and portable.

John T

Posted 2010-01-11T01:01:09.763

Reputation: 149 037

You can download it from the internet archive: http://web.archive.org/web/20151019115332/http://swiftgear.com/

– quip – 2017-04-13T23:28:38.953

The link seems to be broken, this works as of now: http://www.softpedia.com/get/Office-tools/Other-Office-Tools/Large-Text-File-Viewer.shtml

– huseyint – 2012-07-11T20:24:14.527

The original (or edited?) link is working now; I'd take that link over softpedia any day!

Also, just tried LTFViewer, and it's awesome, opened an 818MB file that even Notepad++ complained about opening. Not much in the way of editing features, but once I'd viewed the logs in LTFViewer, echo. > myfile.txt resolved the 818MB logfile issue :D – Doktor J – 2013-07-01T19:16:02.523

LTFViewer is realy great ( i'm using it to ), but there's an limit. I tryed to open an 3GB SQL file with it, but it freezes. Mayby i didn't have enough patience or something... – Mathlight – 2013-08-09T06:37:57.290

LTFViewer worked very well for me. I used it on a (relatively) smaller file, only 750MB, but it opened it in under 5 seconds. – Mike_OBrien – 2014-03-17T14:52:26.307

You can use 7zip itself to split the files. (You can save as a .zip or .7z format.) When you go to create the archive there is an option called "Split volume, bytes". Just select how large you want the chunks.

And yes, you can unzip them individually if you wish.

Felix

Posted 2010-01-11T01:01:09.763

Reputation: 151

There's an online tool that splits text files if anyone is looking to split files quickly. http://www.textfilesplitter.com.

Works great for me. And splits files respecting lines which is what I was looking for. It also says it's all HTML5 client side so it's safe to use. I'm not sure how big it can go but I think it depends on your machine's ram.

Joe One

Posted 2010-01-11T01:01:09.763

Reputation: 31

4Are you really suggesting to upload a 9 GB file to have it splitted in your browser?? Really?? – spaghettidba – 2016-03-01T09:43:30.917

@spaghettidba: He says it works client side using HTML5. If that's the case, than it does not need to be uploaded. I admit, I had the same reaction as you when I first read the answer, though. – mwolfe02 – 2016-04-14T15:32:17.020

Yep, I split 100+ MB in just a couple of seconds. No way it could have uploaded worked and downloaded that fast. – Karl Henselin – 2019-11-01T14:48:46.920

The idea of seeing part of the file before deciding what to do with it is for me the best option.

The Large Text Viewer App can be installed on Windows through the Microsoft Store and it offers an option to cut the file in chunks of size. It may well be that it uses the same editor previously mentioned (behind the scenes), but the option to install it from a known source is better IMHO than the alternative links offered. It worked great for me.

The only issue if splitting by size is that it not necessarily breaks the file at a good place, so you may have to edit it to capture the exact content you want.

Leo

Posted 2010-01-11T01:01:09.763

Reputation: 1

1Please [edit] your answer to include the name of the app, so it can be indexed and found by search. – Scott – 2019-07-20T17:17:22.033