question to NAS users, how to use it for 3d work best ?

[lukx]

I bought Synology 712+ and now I'm thinking what would be the best way to use it for 3d work
Options I have on mind are (maybe there are others so you can give me a hint):

1) Have it on RAID 1 for safety reasons and keep all textures, models, vrayproxies there.

2) RAID 1 and keep projects (but I'm afraid that loading scene form it might take much longer than from my workstation drive)

3) No RAID and just use it for backup for all stuff I have on my workstation.

4) RAID 1 and keep both projects and textures, models, vrayproxies.

or something else... ?

I not to keen on using RAID 0 cause if it dies, everything dies
What do you think?

[Morne]

I would go RAID5 (not sure if you have at least 3 drives in there or not)
This will give you speed and redundancy

I would use it for ALL fileserver stuff. Projects as well as textures. That way dr slaves can also pick it up. AND you can easily switch to a DR slave as your main pc, in case your workstation dies, without worrying about file locations
I would get ANOTHER external drive (just a single drive) to backup everything from the NAS. I would sync it once a week or once a month

[Dariusz Makowski (Dadal)]

Can your NAS actually reach 100/200mb/s transfers? If you cant keep up with 200mb read/write with files ranging 1mb - 1000 mb then I would not even touch it with 50m long stick. Use it as backup for ur work.

[lukx]

no RAID 5 , 712+ is only two drive bay. around 100 are the transfers: http://www.smallnetbuilder.com/nas/n...owall=&start=1

[Morne]

in that case go raid 0, and use another drive to backup this one

(or as DADAL suggest, skip NAS and get something internal). In that case I would go RAID5 and have an external to backup the RAID5

[lukx]

in this case I will use it strictly just for backup

[windrockwater]

I quit using my RAIDED NAS for regular file storage. Only because it simply was not as fast when saving and opening large files. So i use it for backups and deep storage and use a regular network hard disk as my primary working storage area.

[Dariusz Makowski (Dadal)]

I think that if some1 want effective storage system you need:

A: Home Fiber Optics to all devices 100gb+
B: Build a micro pc(micro motherboard etc etc) for HDD storage - cheaper than most pro class nas and you can hock up 10hdds in raid 1/10/5 or so.
C: You might need good smart switch - managed one but thats high-end.

[theedge]

Loading files from the NAS will of course be slower than from your workstation. But that should not be a problem unless your files are multi-gigabyte heavy. Also, keeping everything on the NAS is a much better solution : that makes switching workstation, adding render nodes, etc. so much simpler.
But first thing to do is at minimum to plug an external hard drive to the NAS for backup, because whatever RAID configuration you chose it will be useless if the NAS hardware itself fails (psu, motherboard, ...)

[Dariusz Makowski (Dadal)]

Loading files from the NAS will of course be slower than from your workstation. But that should not be a problem unless your files are multi-gigabyte heavy. Also, keeping everything on the NAS is a much better solution : that makes switching workstation, adding render nodes, etc. so much simpler.
But first thing to do is at minimum to plug an external hard drive to the NAS for backup, because whatever RAID configuration you chose it will be useless if the NAS hardware itself fails (psu, motherboard, ...)

Humh... I dont really agreee..

10x render nodes rendering from nas require at least 10-20gbi/s data data transfers.... 1/2 gb/s in simple terms. That is 2x vertex 3 500mb read/write in raid 0 to hold that transfers... Only there there would not be bottleneck...

[JohnVK]

From experience I do agree with Theedge.

I have two NAS boxes where I store all of my (3ds Max) files, data and resources. I'm using one NAS box actively while the other NAS is used for backup only. It is running a backup every night: differential copy of the primary NAS box. Also both NAS boxes have RAID5. So my files are pretty safe. Rendering with one main workstation and 7 render nodes is no problem at all. I have no noticeable slow downs or other issues whatsoever and I am always using high res HDRI's, large texture files, XRef's, proxies, etc.

I have been using this type of setup for over three years now and never run into any NAS related problems. Like Theedge said, it makes it SO much easier to switch and add workstations / render nodes since all files are always centrally stored in the same location for all machines.

The NAS boxes are not super duper heavy duty pro boxes either. They are simply QNAP TS439 Pro boxes with 4 2TB disks in RAID5 with double 1Gb network adapters set up in a team in load balancing mode. But one thing to keep in mind is that the backbone of your network is very important as well, i.e., the switches connecting your equipment. The processors in switches need to also be able to handle all the network traffic and therefor it is important to get yourself some quality switches! Cheap ones just won't be able to cope with the bandwidth and WILL cause slowdowns / drop outs.

Just my 2 cents.

[Dariusz Makowski (Dadal)]

Heya

JohnVK can you download this little tool and run a test for me of 10/50/100/500/1000/2000 mb files ? and let us know ur results? /screenshots?

http://www.totusoft.com/downloads.html

at the bottom there is free download link.

This should tell us how fast ur nas really is.

[JohnVK]

Hi Dadal,

Sure, no problem. I'll do that when I'm back in my office.

But no matter what the outcome is, in practice this situation is working absolutely fine for me. And it has been for the last few years. So don't put too much weight on this (somewhat theoretical) data

It all comes down to real world performance, which is obviously not only dependent on NAS performance! The bottleneck often is not the speed of the NAS but could be many, many other things. Like actual performance of the NIC's in the computers or speed of the processor of the computer / switch / NAS (or basic global throughput inside the machine), amount of RAM in the computer (which, if there isn't enough of it in the workstation to cope with the scene, will cause MASSIVE slowdowns on render node performance!), the rate at which render nodes crunch the data and request new data sets, etc. etc. etc.

It could very well be that the NAS is outperforming a number of other components in the network, thereby giving it enough breathing room to cope with the data transfers. A chain is only as strong as its weakest link, right!? And if you don't buy premium hardware you are never going to get premium performance. Simple as that. I get what I have paid for, I am very happy with it and I will never go back to any other (localised) solution!

From the top of my head the upload/write speed (to the NAS) is around 23MB/s and the download/read speed around 90MB/s to 100MB/s on average (Mb = Megabits, MB = Megabytes). But this obviously is when a single workstation is accessing the NAS and when all anti-virus is turned off (real-time scanning is a performance hog as well!). And you cannot simply divide this speed by 8 (because I have 8 stations in total) because the demand is not 100% all of the time from all of the workstations. A render node only downloads what it needs at that particular moment, except for the initial download of the scene file. And I'm pretty sure they aren't all doing everything at the same time all of the time. So I would have to do analysis on each switch & computer NIC's traffic/loads to get some actual real-world performance values. Which I really am not going to do because I simply don't care about the numbers. I only care about actual performance as it happens and it works perfectly fine for me

Anyway, I'm not trying to convince anyone, I'm just sharing my experiences I'll back later (unfortunately probably not today... I'm sorry) with the results of your tool.

cheers!
John

[Dariusz Makowski (Dadal)]

Heya

Just want to state that I don't want to sound like - u bad thats bad and so on - I'm just bad at expressing myself! The truth is that the more I learn about hardware/software/coding the more I see how big companies rip off us little people selling us PRO hardware that in reality is junk.

The thing is NAS is a fancy name for Micro PC workstation that has all software build in and is plug and play based.

The thing is for the price of NAS that like you state can do 23write - 90 read you can have a micro system PC that can do 500/500 read write...

In any case I'm waiting for your results.

I think in your case could work if your project is 50-100mb then yea.. It should be quite fast...

But on my end if I render stuff and all textures are loaded pretty much at the same time then... 10x 500mb texture + 100mb file + more... you know that takes a lot of time to render. And the pro NAS to handle it would cost us £2000+ where building micro PC would cost 500£ or so and would be a lot more easy to maintain and customize

Anyway see you around ! Looking forward to your reply.

[JohnVK]

No worries, Dadal. I wasn't taking offense or anything like that You're a good guy. I know from reading many of your posts! Text is just bad for communicating sometimes, since you cannot see facial expressions or intent. I understand! I often have the same problem


So, to make a long story longer....

Yes, you are right! In essence, a NAS is nothing more than a small computer (in the QNAP case, running linux). And there are many options to choose from. In my particular case I chose based on functionality that is built-in into the NAS, like FTP server, HTTP server, backup tools, USB port sharing, eSATA connections, power consumption, hot swapping drives, ability to expand disk volumes online, size of the device and more. Like I mentioned earlier, this is not a pro device. It's a home use or at best small office level device. And for my purposes this is more than sufficient. I think the price is about right for a device that offers this type of functionality but if you go for pure raw performance then yeah, definitely take the route you are describing! it would be better. I always tell people to first inform themselves properly and buy the equipment that will work for them now and in the near future, so as to not waste money on equipment that will not work for them in the long run. Always think ahead.

But even if your machine can do 500/500 you would still need a network infrastructure to support that kind of network traffic volume as well. If it doesn't than every effort to make machines faster is kind of futile.

I am not using 500MB textures (ever). Maximum size of my textures is usually around 300MB for HDRI and 50MB for regular textures. But those are very high res. I make it my business to optimize each and every scene and its contents (I am from the era where you needed to insert actual chips in sockets on the mainboard to upgrade your 16KB memory to 64KB memory LOL). So most of the time my textures and other secondary files are much smaller. I really don't need 15000px or larger textures when I'm rendering at 5000px max. If someone does then they obviously need to invest in a solid infrastructure that can move such immense payloads. I have no need for that and I think we can safely assume that most of us don't need that kind of high end equipment & performance.

So I most definitely agree with you that one should research and buy the equipment that best suites their needs. Which can often, like you said, make the difference between paying 2000GBP or 500GBP. Absolutely! It's totally dependent on individual needs.

Speed doesn't come cheap unfortunately. And even though prices for such pro level equipment are pretty steep I don't think they are overpriced. Usually it's not just the device you're buying but also the accompanying (pro level) support on the equipment plus guaranteed availability of components over years to come. Always nice to not have your long term investments go to Hell simply because a device is no longer manufactured. Whether or not you like the price I think is totally depending on available budget. I cannot pay for 3000 euro costing switches, or rather, don't see the need to in such a small network infrastructure as mine. But they most definitely are worth it in a high end professional environment. This type of devices have excellent ROI and TCO. In a professional high end environment there's much, much more at play than simply looking at the initial cost of purchase.

Most people are getting ripped off or simply buy the wrong equipment because they don't know what they are doing. It's kind of odd... many try to educate themselves to know even the smallest detail about the applications they are using and demand high end performance but they fail to show any interest at all in the infrastructure that must carry the data... which is at least equally important! Hence they blame the software manufacturer for issues caused by their own lack of knowledge. And also, many buy into expensive softwares and computers and then go and buy a $30 switch to connect the bunch together :/ doh! ... LOL


Anyway, we could talk about this for days in a row if we wanted ;p I have attached screen shots of the results of the tool you linked to:

I have tested with four different files sizes as that usually gives a better view of what's going on. Those are 5MB, 10MB, 50MB and 100MB, which best reflect the files sizes I'm using on a regular basis.

As you can see speed fluctuates. For some reason, it also seems to address only ONE ethernet port on the NAS, hence the maximum download speed reduced to 50% of what I'm used to seeing. No idea why it is doing that exactly but I think it has something to do with the load balancing feature I have enabled on my NAS and switches (which is IEEE 802.3ad and only works if the switches support this and are properly configured to support it!). The tool seems to occupy a single path in the teamed network interfaces, hence using only one NIC of the NAS.

The upload speed is what I said it was and this is because the NIC controller + Processor + disk controller in the NAS simply can't process more data. It's at its maximum capacity.

One thing worth noting with this tool is that it bypasses any caching methods in place within the applications, computers and NAS. So this value isn't really reflecting a real-world situation. Also, lots of smaller files may or may not affect transfer speed negatively. Larger files could make the NAS go into streaming mode and have the switches use QOS to add extra resources to the transfer of those payloads, where it speeds up the transfer process. There are way too many factors at play in a live network to be able to draw any meaningful conclusions from these simple speed tests. But admittedly, they give you a general idea


So yeah, in conclusion, things could be better but they could also be a lot worse. But however we run the numbers and twist and turn to make them fit (or not), in the end I'm still very happy with my setup!


Cheers mate!
John