0
I seek to use my 500GB NVMe SSD as pseudo-RAM for deep learning by increasing pagefile size. My RAM's 28GB, I seek 64GB+. From the limited info I could find on DDR4, this SSD appears to have 10% its read speed - excellent, and sufficient for my application.
Reading up on others doing the same, advice has been shifting from "SSD funeral" to "no longer bad" - yet I remain uncertain, especially for my application. One train epoch reads 64GB of data, but writes ~zero. The linked SSD boasts 1.2 Petabyte writes, but no specs on reads; I presume it's much greater, if at all relevant? Assuming equal, the SSD can handle 18,000+ epochs - which makes degradation a non-issue.
This said, what's the status on this matter now, in 2019?
(As a side question, is its 2,280 Gb/s data transfer rate spec a likely typo?)
1Pagefile does not give you extra RAM. It's used to page out programs that don't need RAM at the moment. If you need more RAM in an instance of time than you physically have, pagefile won't help you. – gronostaj – 2019-09-05T11:09:41.973
1@gronostaj I didn't claim it gives extra RAM - I said "pseudo-RAM" - and frankly, for my application, it is extra RAM; the SSD's read speed is 10% of my RAM's - more than enough for unnoticeable data load time – OverLordGoldDragon – 2019-09-05T14:30:14.330
"One train epoch reads 64GB of data" so if I understand correctly, you need 64 GB used all at once. Pagefile won't help you with this. – gronostaj – 2019-09-06T06:33:24.723
@gronostaj Only 480MB at a time, not all at once - I've done the benchmarks with Numpy memory mapping: a memmapped array loads in 0.3 secs, vs 5 secs unmapped, and that's the figure on my HDD snail - i.e. the SDD should pull < 0.01 secs. (But yes, I do intend to map all 64GB onto pagefile - it shouldn't be a problem, correct?) – OverLordGoldDragon – 2019-09-06T13:41:30.463
I'm not sure, but I'd first give it a try on a HDD before you spend any money on this. At this rate filling your 28 GB should take about a minute. – gronostaj – 2019-09-06T14:15:07.110
@K7AAY Any input on this? Also, correction - I have a hybrid drive, and pagefile's defaulted to C-drive with the "SSD portion" - as for testing, it works for up to 5GB of memmapping, though didn't try increasing pagefile size as disk space's limited - but can free if needed – OverLordGoldDragon – 2019-09-06T14:21:25.720