ply2vrmesh limits ?

[raxto]

Hi,

is there some limitations of polycount, i trying to convert 618 milions polygons to vrmesh file, but output vrmesh file is 1kb big.

thx for reply
Rasto

[t.petrov]

The only limitation should be available RAM, as far as I know.
Probably some limitations could be caused by the 32bit integers we're using here and there, but you're not that close to this limit (2 billion polygons).

Have you tried to convert smaller mesh exported by the same applications to see if it works fine.
Also have you check what happens with the used memory on the machine. Is it swapping, etc.

[raxto]

i have available 32 GB RAM, its never go over or swapping.

with 1 billion do you mean 10⁹ or 10¹² ?

yes i tried smaller meshes, it's working up to about 60 millions with binary ply.
converting ascii files to vrmesh with about 60 millions of polygons or more are freezing or it's extremly time consuming.

now i am trying convert ascii ply with 618 milions polygons and it's processing: [2h 21m 56.8s] Processing voxel 5 of 14
our plans are render 5 times more polygons than this one.

[t.petrov]

I mean 2³², which is roughly 2.10⁹. This limitation is fixable, because we can start using int64, which is 2⁶⁴.

Your problem seems different...
Also what version of ply2vrmesh are you using? What OS? Do you use network filesystems?
Is there any chance you can send us this whole dataset, so we can do some tests in here?
Can tell us which source format are you using for the data and what application has been used to generate the files?

[raxto]

when it's fixable is nice to hear it because i mean we will hit the limit soon.

ply2vrmesh: Build from Nov 20 2013
OS: Windows 7 Pro SP1 x64
i use only local filesystems - 1TB SSD.

i am not sure with data but i must check it, because of NDA and size. it's our custom application with standard export to ply file.

[t.petrov]

ply2vrmesh: Build from Nov 20 2013

What build of V-Ray is this coming from?

i am not sure with data but i must check it, because of NDA and size. it's our custom application with standard export to ply file.

We can sign a NDA and we can provide FTP access, where large files can be uploaded.

[raxto]

===============================================
V-Ray Standalone, version 2.45.01 for x64
Copyright (C) 2000-2012 Chaos Group Ltd. All rights reserved.
Use -credits option for additional third-party copyright notices.

Build 24487 from Apr 28 2014, 06:06:12
Compiled with Intel C++ compiler, version 14
Operating system is Microsoft(tm) Windows(tm), version 6.1, Service Pack 1

V-Ray core version is 2.00.02
===============================================

Can you send me email to provide more details ?

[t.petrov]

Can you send me email to provide more details ?

Please send an email to support @ chaosgroup.com (don't forget to mention this topic) and they'll make sure to help you.

[asen.atanasov]

Thanks for the file! We have found bug in our code. If the last character of the file is a digit, the program hangs. This will be fixed in tomorrow's nightly builds both for V-Ray 2/3. You can work around this problem in your current version if you just add "new line" character in the end of the file. On my machine with 12 GB of ram the conversion took 9h 6m. I recommend that you consider of using the binary ply format for such huge data sets. The file will be a couple of times smaller (the resulting vrmesh is 6.71 gb) and the conversion will be a couple of times faster. We should be able to handle 5 times larger files and more, but if you encounter any problems please let us know.

Concerning the rendering looks like we have problem with the multithreading. My first guess is that the default "facePerVoxel=20000" is too small for such a huge mesh, so you can try with 200 000 or 500 000. I will run some test as well this night and then I can tell you more.

Regards,
Asen

[asen.atanasov]

We couldn't make it for today's nightly builds, but we have it for tomorrow's. I have good news. With the option -facesPerVoxel 200000 and -facesPerVoxel 500000 the conversion time drops to 1h (with you ssd should be even faster) and the multithreading for rendering is improved.

Regards,
Asen

[raxto]

thx for info Asen, it's fixed in vray_adv_24501_maya2013_x64_24740 ?

i have done some tests with still old version and here are my results:
now i use 12.3GB binary, instead of 44GB ascii PLY.

ply2vrmesh.exe - Build from Jul 3 2014

SCENE1.ply
HP Z620 (DUAL XEON E5-2630 v2, 64GB RAM, Win7 Pro SP1 x64)
[19h 7m 32.3s] facesPerVoxel not defined
[1h 1m 50.2s] facesPerVoxel=500k

custom made PC (i7-3770K@4.6GHz, 32GB, Win7 Pro SP1 x64)
[5h 45m 44.7s] facesPerVoxel not defined
[0h 26m 33.9s] facesPerVoxel=500k

SCENE2.ply
[1h 42m 11.5s] facesPerVoxel=500k - HP Z620
[0h 27m 57.6s] facesPerVoxel=500k - custom made PC

you can see big slow down difference 4 vs 12 cores system. 4 is winner !!! why ?
can we generate vrmesh directly ? is there some DOCs to vrmesh format ?

regards,
Rasto

[Dariusz Makowski (Dadal)]

It may be because ur Custom build PC is a lot faster on single threaded jobs there fore its faster than ur monster multi core

[vlado]

you can see big slow down difference 4 vs 12 cores system. 4 is winner !!! why ?

If you mean during rendering, then you can try increasing the "dynamic memory limit" option in V-Ray; otherwise the different threads all fight for the same memory pool, which slows things down. If you mean the conversion process, I don't think this is multithreaded currently.

can we generate vrmesh directly ? is there some DOCs to vrmesh format ?

You certainly can, although generating the files on your own is maybe a bit too tedious. The format is described in the docs/vrmesh_format.html file in the V-Ray SDK, but the preferred approach is to write a C++ class derived from MeshInterface and then let the "subdivideMeshToFile" function do the actual .vrmesh creation.

Best regards,
Vlado

[asen.atanasov]

Hi Rasto,

first thing to mention is that ply2vrmesh currently works entirely single threaded. Dadal is right. And in this particular case the speed of the drive plays very important role.

Yes, the problem is fixed in vray_adv_24501_maya2013_x64_24740.

As we can see it's very important to increase face per voxel for big meshes. You can conduct some more conversion-rendering tests to figure out what is the optimal voxel size for your case. What is interesting to me is also what is the difference in conversion time between binary and ascii?

The only way to write vrmesh files yourself is to use V-Ray SDK.

Best regards,
Asen

[asen.atanasov]

Sorry, I was wrong that V-Ray SDK is the only way. But it is certainly the easier.