Octo core I7 processor and memory allocation errors

[hbeh]

So, i'm getting Memory Allocation Errors with black squares

"Memory allocation failure; 1120000 bytes requested; 1366745340 bytes already in heap; 2893608 bytes free heap space."

and then followed by bugsplat on a test render at 640x480 resolution - have read many of the forums on this very topic. However, i've noticed this error on my new dual quad (octo) core I7 system running windows 7 with 16 gigs of ram! (total ram usage when rendering is going is around 3.91 GB)

I rendered the same scene on my older system - dual core with 4 gigs of ram and running Vista 64bit and no errors (total ram usage when rendering is going on this system is 2.5 GB).... any idea why this is happening? Do i have to adjust certain settings to let v-ray properly render with 8 cores? Is the fact that they are more cores causing it to use more memory and sketchup is hitting it's limits early with 8 cores?

Thanks!

[dkendig]

Re: Octo core I7 processor and memory allocation errors

cores don't effect ram usage, and 16gb of ram won't do you a lick of good. V-Ray for SketchUp is a 32-bit process, so it can't use more than 4gb of ram on a 64bit operating system, or 2gb on a 32-bit operating system. This has been discussed many times on the forum, and is also mentioned on our FAQ page.

Question:
I'm going to be upgrading my OS, should I get 32-bit or 64-bit?
Answer:
The reason that most people are interested in 64-bit Operating Systems (OS) over 32-Bit systems is the added memory usage that they allow. Memory limits, as well as a number of aspects, should considered in regards to making the choice between 32-bit and 64-bit operating systems. One of the first should be whether your hardware and/or peripheral manufacturers make drivers for the 64-bit OS that you are researching. If a manufacturer does not offer a driver for 64-bit OS, then that may be enough to decide on a 32-bit operating system.

In regards to memory usage, it is important to understand the current memory limits with 32-bit systems and the limits for 32-bit applications on 64-bit systems. With 32-bit systems, the maximum amount of memory that can be use by default is 2 gbs per process. There is an option, referred to as a 3gb Switch, that will allow for your computer to use more memory, equating to 3 gbs of memory available per process. However, the application must be compiled with the Large Address Aware tag in order to utilize the extra gig of memory that the 3gb Switch allows.

With 64-bit systems, you will only programs that are native 64-bit programs will allow for full usage of all of the memory that you have installed on your system. 32-bit programs that are not compiled with the Large Address Aware tag will still only be allowed to use 2 gbs of memory. 32-bit programs that do have the Large Address Aware tag will be able to use up to 4 gbs on a 64-bit operating system.

Currently SketchUp is only available as a 32-bit application and is not a Large Address Aware application which means that the memory usage under 32-bit and 64-bit operating systems will be the same. However, current builds of the DR Spawner are compiled with the Large Address Aware tag and there will most likely be 64-bit versions of the DR Spawner with the next full release of V-Ray for SketchUp.

Currently Rhino 4 is also only available as a 32-bit application, but it is a Large Address Aware application. This will allow for up to 4 gbs of memory usage. The next version of Rhino will have a native 64-bit version available which will allow for usage of all the memory that you have installed on your system. At this point, McNeel is very early in the beta process and there is no release date set for the next version of Rhino.

Memory References
http://msdn.microsoft.com/en-us/libr...px#Differences
3Gb Switch
http://www.microsoft.com/whdc/system...AE/PAEmem.mspx
http://en.wiki.mcneel.com/default.as...ressAware.html

Here's a post by Micha that gives some good tips on optimizing the scene (even though it's for V-Ray for Rhino)

http://forum.asgvis.com/index.php?topic=5887.0

[hbeh]

Re: Octo core I7 processor and memory allocation errors

Yeah, the errors start happening whenever total RAM usage goes over 4GB. My model is actually a very simple 3 story building and the file is around 10MB, the confusing thing is that when i bring the model over to my older system (dual core, 4GB ram) the ram usage does not go over 4GB and so no crash...but that same model being rendered using the same settings on this Octo Cored machine spikes the ram usage over 4GB and it crashes - thats why i was wondering if running 8 cores at a time spikes RAM usage and with sketchup being a 32 bit application if i would actually be better working on a machine with less cores???

[hbeh]

Re: Octo core I7 processor and memory allocation errors

Yeah, the errors start happening whenever total RAM usage goes over 4GB. My model is actually a very simple 3 story building and the file is around 10MB, the confusing thing is that when i bring the model over to my older system (dual core, 4GB ram) the ram usage does not go over 4GB and so no crash...but that same model being rendered using the same settings on this Octo Cored machine spikes the ram usage over 4GB and it crashes - thats why i was wondering if running 8 cores at a time spikes RAM usage and with sketchup being a 32 bit application if i would actually be better working on a machine with less cores???

I'm also working on setting up rendering space on a cloud computing network where i can have 32 cores on a single virtual machine....but if i'm only limited to 4GB of ram then wouldnt you think 32 cores will use more ram than 2? So that means that having a virtual cloud machine with a couple of dozen cores will actually cause more problems??

[hbeh]

Re: Octo core I7 processor and memory allocation errors

Here's another observation, same file, same scene, same v-ray settings on two computers:

Computer 1: Windows 7 64bit running sketchup 7.1 with V-ray 1.05....16gb ram, I7 Q840 Octo core

- Ram usage continues to climb as image renders until it hits 3.5GB and then i get the black squares with "memory allocation error..etc" and bugsplat

Computer 2: Windows Vista 64bit sp2 running sketchup 7.1 with v-ray 1.05.... 8gb ram, Centrino dual core

- Ram usage climbs maybe 200-500 megs to around 2.7gb and then remains stable and image renders succesfully. Why is ram usage on this system stable while the other system continues to consume ram during the render??

I'm at my witts end trying to figure out why my new system is undeperforming and unable to render while the old system is crash free. I've tried upping the virtual memory, running sketchup in vista compatibility mode..... i guess i can uninstall and re-install everything to see if it helps... any ideas? well besides trashing my new system and sticking with the old Thanks!

[dkendig]

Re: Octo core I7 processor and memory allocation errors

There's some RAM overhead associated with each core that gets used, and currently our plugin is a 32-bit plugin, so the ram is limited as explained before. So I guess the scene you were trying to render was just at the limit of RAM usage on the old machine, but on the new machine, that extra overhead pushes it over the limit and you run out of memory. This will be fixed when we figure out a way to make our plugin 64-bit, but that will be a while. As far as cloud computing, virtual machines, etc. We didn't develop our product to run in those types of environments specifically, so if you experience any issues with a setup like that, we won't really know where to begin troubleshooting.

[hbeh]

Re: Octo core I7 processor and memory allocation errors

Yup, that's pretty much the conclusion i've arrived at.

I guess there's no way to set v-ray to only use a certain number of cores (it always uses 100% of every available CPU) - would that be a viable feature to have or is that not the way the software and the OS handels resouce management? I'm just thinking aloud here, since it would be nice to be able to multi-task on my machines while lets say a large rendering is going on in the background since i do have all that "extra" CPU and RAM! I guess I should look closer at windows virtual machines and see how they allocate ram and core usage because i could have v-ray installed on one of those virtual sessions and just allocate 4 gigs of ram to that virtual machine...etc

Thanks for your feedback and thoughts.

[jon2]

Re: Octo core I7 processor and memory allocation errors

Yup, that's pretty much the conclusion i've arrived at.

I guess there's no way to set v-ray to only use a certain number of cores (it always uses 100% of every available CPU) - would that be a viable feature to have or is that not the way the software and the OS handels resouce management? I'm just thinking aloud here, since it would be nice to be able to multi-task on my machines while lets say a large rendering is going on in the background since i do have all that "extra" CPU and RAM! I guess I should look closer at windows virtual machines and see how they allocate ram and core usage because i could have v-ray installed on one of those virtual sessions and just allocate 4 gigs of ram to that virtual machine...etc

Thanks for your feedback and thoughts.

I did a quick test or two using the old affinity processor trick to see if Vray would only use one core if I set Sketchup to only use one core. (Since you can't easily see Vray as a separate process) I also wanted to watch the ram usage and any difference there may be.

First thing I noticed with a really simple couple of boxes with one material applied was that sketchup was using 200,000 K already. Crazy being this model was a couple of boxes and one circle. Anyway, I hit Render and the Ram usage went up a tiny amount. When finished I hit render again. It keeps going up. Slowly, but surely. No changes to the model, the Ram usage just keeps going up after each render. Maybe a good reason to save your model, then restart sketchup before attempting a new render iteration.

I'm running a quad core zeon with hyper threading enabled. So the machine sees 8 cores. I went into the task manager and right clicked on Sketchup/ then chose Set Affinity.

You see all 8 cores selected:


I then set the affinity of Sketchup to only one core.


After doing this, the ram dropped about 2,000 K, I'm not sure why. As subsequent renders still continued to climb in ram usage. Albeit at a much smaller pace. I did notice that this simple render of simple objects all of a sudden took much longer to finish rendering. I have no idea why since sketchup supposedly only will use one processor. Or does one processor with 4 or 8 cores still count as one processor? It's odd, because the render time seemed to double or triple once I set the affinity to one core. I'm clueless as to why this is though.

Anyway, as for Vray rendering, you can still see Vray using all 8 cores, even with sketchup's affinity set to 1 core.



Same sketchup file, nothing changed. Still looks like Vray is using all 8 cores for at least part of this render process. But with Sketchup set to one core, the render takes way longer. And ram usage didn't seem to be impacted much at all by turning the other cores off. Maybe it's ram usage just didn't climb as fast. After 8 to 10 renders with a mix of affinity to all and affinity to one, sketchup was showing a total of 228,000 K usage. Changing the affinity back to all cores in sketchup and rendering again showed that the quick rendering time was back.

This was a tiny model, so I'm not sure if this all changes with a more realistic scene render with multiple materials and objects or not. Just thought I'd give it a try.

[dkendig]

Re: Octo core I7 processor and memory allocation errors

the ram usage shouldn't exceed the peak amount that the first render reaches in any subsequent renders. Any differences in the amount of ram while not rendering shouldn't effect the peak ram usage. That's what we saw in our tests anyway. Not 100% what is being cached or held on to between renders that makes it jump about like that though.

[jon2]

Re: Octo core I7 processor and memory allocation errors

Yeah that is the one thing I noticed that seemed odd. It did go up in ram usage, ever so slightly, with each render. That increased ram usage would then continue to hold. Much like a web browser builds the total ram its using as you continue to fill the cache. Something is being retained after the render. Not sure if it is sketchup or Vray though. The only time I saw it drop a little, or fluctuate downwards at all was when I locked sketchup out of the other cores. But even though it looked like Vray was still using the cores, the render time suffered by doing that. So I'm going to stick with the plan of saving/closing/reopening and then rendering just in case it is somehow locking up more ram each time.

I'm using 1.48.89 on a win7 x64 with Sketchup 7.1.6860, 3.2 quad core zeon.

[dkendig]

Re: Octo core I7 processor and memory allocation errors

so the peak ram usage was going up? or the usage after the render was going up? or both? I know the usage after the render can fluctuate, but the peak usage during a render should be consistent.

[jon2]

Re: Octo core I7 processor and memory allocation errors

so the peak ram usage was going up? or the usage after the render was going up? or both? I know the usage after the render can fluctuate, but the peak usage during a render should be consistent.

At the beginning, the file started at 200,000 K usage. Hitting render, the ram usage started to climb. During the render it fluctuated upwards, and a little downwards, but always ended up higher than where we started at before the render button was selected. That increase would continue to hold at that level after the render. Hitting render subsequent times produced more of the same, and after 8 to 10 renders the application was now using an extra 28,000 K for a total of 228,000 K. So I would say both, but both were climbing at a small amount. The model is pretty simple and only one material was added so I don't know if this climb in usage is more apparent in larger models with more textures/mats or not.

The part that concerned me a bit was that it was now holding onto that higher amount, even though nothing in the model had changed, and no settings in Vray had changed. Given that Sketchup is restricted to a limited amount of ram, (2gb or whatever), it seems that even doing small region renders a number of times would continue to get your system closer to that maximum ram usage, even if you weren't adding to polygon totals or anything. So you might get to a point where a model was rendering just fine, until you did more than one render while tuning some settings, and now your sketchup app is using more ram than it will allow and it might then crash. So then you are stuck with a model you can't render, even if you finally have the settings setup the way you want them. At least not without saving, and restarting the program to get it to release some of that ram it was withholding for whatever reason.

Sketchup's ram limitations are ruining all our fun. I'm guessing what I'm noticing here is probably just another issue with Sketchup'/ram usage.

Are there any ram issues at all right now with 3Dmax and Vray? I might have to give up wasting my time with sketchup soon and move to something else.

[dkendig]

Re: Octo core I7 processor and memory allocation errors

I don't know of any issues with SketchUp holding on to ram itself. It's probably something that has to do with our plugin. Caching data somewhere along the line and not releasing it. We did have a huge memory issue a few versions back, but that has been resolved. We will continue to try to hunt down any lingering memory issues. Thank you for your detailed report!