ply2vrmesh performance

[dboogert]

Not strictly an SDK question but wasn't sure if the was a better forum for this question....

For a test render I'm converting a 123 million face .obj to .vrmesh and it takes 1h 40m. Is this to be expected? The log says the majority of the time is in the writing but there must be other processing happening for it to take so long right?

Thanks
- Don

Log:

Converter of .ply .obj .bin .geo .abc and .vrscene files to .vrmesh files
Copyright (C) 2003-2014 by Chaos Group Ltd. All rights reserved.
Build from Sep 11 2014


[0h 0m 0.0s] [0 MB] Converting file "/my.obj"
[0h 0m 0.0s] [0 MB] Output file is  "/my.vrscene"
[0h 0m 0.0s] [0 MB] Reading .obj file
[0h 0m 48.7s] [269 MB] Reading .obj file... 10.0 % (Memory usage Rss: 270MB Pss: 269MB Uss: 269MB)
[0h 1m 37.8s] [528 MB] Reading .obj file... 20.0 % (Memory usage Rss: 529MB Pss: 528MB Uss: 528MB)
[0h 2m 27.6s] [831 MB] Reading .obj file... 30.0 % (Memory usage Rss: 832MB Pss: 831MB Uss: 831MB)
[0h 3m 21.6s] [1270 MB] Reading .obj file... 40.0 % (Memory usage Rss: 1271MB Pss: 1270MB Uss: 1269MB)
[0h 3m 59.4s] [1770 MB] Reading .obj file... 50.0 % (Memory usage Rss: 1771MB Pss: 1770MB Uss: 1770MB)
[0h 4m 27.2s] [2282 MB] Reading .obj file... 60.0 % (Memory usage Rss: 2283MB Pss: 2282MB Uss: 2282MB)
[0h 4m 54.6s] [2784 MB] Reading .obj file... 70.0 % (Memory usage Rss: 2785MB Pss: 2784MB Uss: 2784MB)
[0h 5m 22.0s] [3285 MB] Reading .obj file... 80.0 % (Memory usage Rss: 3286MB Pss: 3285MB Uss: 3285MB)
[0h 5m 49.5s] [3787 MB] Reading .obj file... 90.0 % (Memory usage Rss: 3788MB Pss: 3787MB Uss: 3787MB)
[0h 6m 16.8s] [4286 MB] Reading .obj file... done                                                            
[0h 6m 18.7s] [4287 MB]   Number of faces: 123636736
[0h 6m 18.8s] [4287 MB]   Number of vertices: 61831585
[0h 6m 18.8s] [4287 MB]   Number of texture vertices: 61906993
[0h 6m 18.8s] [4287 MB]   Number of vertex normals: 0
[0h 6m 20.0s] [4287 MB] Subdividing mesh to file "/my.vrscene"
[0h 6m 20.0s] [4287 MB] Subdividing frame 1 of 1
[0h 6m 20.0s] [4287 MB] Processing voxel 1 of 2
[0h 6m 20.0s] [4287 MB] Reading voxel
[0h 6m 20.0s] [4287 MB] Reading voxel... done                                                            
[0h 6m 22.6s] [8533 MB] Subdividing mesh voxel
[0h 6m 35.4s] [9008 MB] Subdividing mesh voxel... 10.0 % (Memory usage Rss: 9009MB Pss: 9008MB Uss: 9008MB)
[0h 6m 40.6s] [9015 MB] Subdividing mesh voxel... 20.0 % (Memory usage Rss: 9016MB Pss: 9015MB Uss: 9015MB)
[0h 6m 45.0s] [9028 MB] Subdividing mesh voxel... 30.0 % (Memory usage Rss: 9029MB Pss: 9028MB Uss: 9028MB)
[0h 6m 53.2s] [9046 MB] Subdividing mesh voxel... 40.0 % (Memory usage Rss: 9047MB Pss: 9046MB Uss: 9045MB)
[0h 7m 3.2s] [9068 MB] Subdividing mesh voxel... 50.0 % (Memory usage Rss: 9069MB Pss: 9068MB Uss: 9068MB)
[0h 7m 7.6s] [9094 MB] Subdividing mesh voxel... 60.0 % (Memory usage Rss: 9095MB Pss: 9094MB Uss: 9094MB)
[0h 7m 12.4s] [9125 MB] Subdividing mesh voxel... 70.0 % (Memory usage Rss: 9126MB Pss: 9125MB Uss: 9125MB)
[0h 7m 20.7s] [9160 MB] Subdividing mesh voxel... 80.0 % (Memory usage Rss: 9161MB Pss: 9160MB Uss: 9160MB)
[0h 7m 26.0s] [9201 MB] Subdividing mesh voxel... 90.0 % (Memory usage Rss: 9203MB Pss: 9201MB Uss: 9201MB)
[0h 7m 30.0s] [9248 MB] Subdividing mesh voxel... done                                                            
[0h 7m 30.1s] [9248 MB] Saving mesh voxel
[0h 15m 7.9s] [9248 MB] Saving mesh voxel... 10.0 % (Memory usage Rss: 9249MB Pss: 9248MB Uss: 9248MB)
[0h 22m 45.2s] [9248 MB] Saving mesh voxel... 20.0 % (Memory usage Rss: 9249MB Pss: 9248MB Uss: 9248MB)
[0h 30m 11.0s] [9248 MB] Saving mesh voxel... 30.0 % (Memory usage Rss: 9249MB Pss: 9248MB Uss: 9248MB)
[0h 37m 51.2s] [9248 MB] Saving mesh voxel... 40.0 % (Memory usage Rss: 9249MB Pss: 9248MB Uss: 9248MB)
[0h 45m 26.1s] [9248 MB] Saving mesh voxel... 50.0 % (Memory usage Rss: 9249MB Pss: 9248MB Uss: 9248MB)
[0h 52m 57.1s] [9248 MB] Saving mesh voxel... 60.0 % (Memory usage Rss: 9249MB Pss: 9248MB Uss: 9248MB)
[1h 11m 41.8s] [9248 MB] Saving mesh voxel... 70.0 % (Memory usage Rss: 9249MB Pss: 9248MB Uss: 9248MB)
[1h 24m 44.1s] [9248 MB] Saving mesh voxel... 80.0 % (Memory usage Rss: 9249MB Pss: 9248MB Uss: 9248MB)
[1h 32m 5.5s] [9248 MB] Saving mesh voxel... 90.0 % (Memory usage Rss: 9249MB Pss: 9248MB Uss: 9248MB)
[1h 39m 35.0s] [8776 MB] Saving mesh voxel... done                                                            
[1h 39m 35.1s] [8776 MB] Processing voxel 2 of 2
[1h 39m 35.1s] [8776 MB] Reading voxel
[1h 39m 35.1s] [8776 MB] Reading voxel... done                                                            
[1h 39m 35.2s] [8776 MB] Mesh successfully subdivided to file "/my.vrscene"

[vlado]

I think it really is only compressing and writing out the file. There might be a way to make this faster (f.e. not flushing the OS file caches right away or something), but in general it is an effect that we experience with Phoenix too where writing the simulation data to disk can sometimes take way more time than the actual simulation.

Best regards,
Vlado

[dboogert]

On both windows & linux?

In another test I wrote to a local SSD rather than a network share but experienced similar performance. Would it be possible to get a linux build with symbols or source which we can execute locally under vTune?

Cheers
- Don

[vlado]

Yes, sure - email me to vlado@chaosgroup.com and I'll get you the sources.

Best regards,
Vlado

[t.petrov]

Another option is to send us the file (we can provide a ftp, where to upload it) and then we can run some tests locally to see what is slow.