.AUR to .VDB converter

[Morbid Angel]

Oh, it should be multithreaded already, but it's possible that the reading and writing from the disk are slowing it down. How does it look in Task Manager?

When I said multithreaded I meant it should load multiple vdb's at once. The network is not an issue we are on 10G network. The io during conversion is small. I think the most time is spent on actual conversion. I can write 5 batch scripts each doing a chunk and its way faster, I think this can be implemented into the tool.

[Joelaff]

A 10gig network is about 7 times slower than modern local NVME drives.

Interesting that the multi batch runs faster. I suppose you could parallelize it with something like Deadline, and also get dependencies, etc.

[Morbid Angel]

A 10gig network is about 7 times slower than modern local NVME drives.

Interesting that the multi batch runs faster. I suppose you could parallelize it with something like Deadline, and also get dependencies, etc.

Why would you compare network to nvme? if course its faster. But we just output 14Tb of caches over 24 hours, you can't expect nvme to handle that volume right? There has to be a better solution.

[Joelaff]

My point was that others (possibly Chaos) may be testing locally rather than on a network. We do everything over the network as well, except for the initial setup and tweaking which is local nvme.

It was also in reference to my earlier post about doing the processing locally on the file server (or a Vm thereof). Your drives are likely faster than the network. So that was another option to achieve (some of) the speed up you desire.

I believe there are options to handle that kind of data with nvme if you really wanted (PCI cards). But, yes, impractical at this point.

Any performance increase is always welcomed!

[Svetlin.Nikolov]

Ah yes, writing and reading caches over the network is the worst case scenario performance wise. If anything can happen locally and then be copied over to the network, do it this way. One thing that is in the middle is something some colleagues do - using google drive for caches, and the simulator reads and writes caches in a directory inside the G: drive created by google drive file stream under Windows - this way the writing happens instantly and the files are synced to google drive at some later point, but it's automatic and you don't have to manually start the process. However, when reading caches in a new machine, sometimes weird things happen since it's trying to download the files on demand and make it look like it's just like working with local files, but it's not 100% reliable yet. I've also heard some colleagues complaining about google drive file stream being buggy, and also not really sure if it even works under other OSes.