5
1
I just tried to import it into excel, but it can only accept a little over a million rows of data. It's 3 columns of data, and all I want to do is graph column 1 against 2 and 3 for two graphs.
I am thinking of making a grid preprocessor which divides the 2D landscape up into cells and marks each cell as having or not having an element within it. There will be a fiddle factor for making the cells small enough to discern information from the graph while large enough to be under 1Million filled cells to fit in excel.
While I do on that or something else, anyone know how to graph all of the data easily?
1Why not use a database like SQL or shudder Microsoft Access? – James Mertz – 2013-05-28T23:25:53.727
Lack of knowledge of their existence. I'll try it out. – SwimBikeRun – 2013-05-28T23:27:51.453
@SwimBikeRun what format are the data in right now? – nhinkle – 2013-05-28T23:40:51.610
tab separated floating point values – SwimBikeRun – 2013-05-28T23:44:58.190
1In Excel you can handle more than 1M rows of data in PivotTable and PivotCharts. Go through the import data dialogs - but instead of storing in the Excel sheet in the last step, store it as PivotChart. From here, you can create your chart... Haven't tried with 6M rows, but can imagine that this should work! Good luck! – Peter Albert – 2013-05-29T10:29:24.523
2You should consider methods for aggregating your data. I don't think you'll be able to visually discern 6M rows (or even many less than that). Consider the resolution at which you'll be viewing the data (screen or paper) and that medium's resolution and consider that you can't discern more than one data point to unit of resolution (e.g. dpi or pixel). For example at 1200 dpi, you'd need 5000 inches/417 feet to display 6M datapoints. – dav – 2013-05-29T12:11:36.253