Replacing NAS with Cloud Storage Ideas ?

[theedge]

Hello,
Looking for some feedback/advice about working directly on a cloud storage with 3ds max and V-Ray.

I have been using a Netgear Readynas Pro NAS for 10 years now for projects and assets. It is still working wonderfully. It's built like a tank and super reliable. But it is no longer being supported and the associated backup service that I use will stop working in october. Also, 1Gb connection is becoming seriously limiting for large scenes.
Instead of investing in a new NAS devices I was thinking maybe a Cloud Storage Plan with enough space (several TBs needed) could replace a physical storage device. It would be used by 2/3 workstations and 4 render nodes.

I am currently doing some early tests with Dropbox and Google Drive.
- Google Drive seems to work well so far. When opening a scene in 3ds max the File Streamer kicks in and downloads all linked assets and xrefs. Same for render nodes : the client instantly streams the necessary files and everything runs smooth. I still have to make more tests for more complex scenarios but so far it is promising.
- Dropbox is more problematic. The Smart Sync feature does not "detect" all of the linked assets. Lots of files don't get downloaded. Even trying to retarget them to the dropbox folder doesn't work. 3ds max desperately flags them as missing. Anyone ran into this problem too ?

It seems that Onedrive also does Smart Sync but haven't tried it yet

Did you have any success using of these two, or other Cloud Services ?


PS : I aware that Autodesk does not recommend using Cloud Storage Services but hey, they also don't support ANY NAS or SAN either, so ...

[Joelaff]

Why not just get another NAS, or build one if you like playing with linux or unix?

If 1Gigabit is too slow don't you think your internet connection is going to be too slow? Or do you like in South Korea or someplace where you have 10Gigabit Ethernet for your internet?

Seriously, I can't stand cloud ANYTHING. You put your reliance on a third party for the service, then again on your ISP, and their ISP and every router in between. You have to pay every month or lose your data. If you live in the US, due to a outdated legal precedent known as the Third Party Doctrine you give everything you give to a third party to our government WITHOUT a warrant or probable cause (the third party can demand the government provide one, but they have no requirement to do so, and many just give in to these requests with no oversight).

Of course according to your profile you live in France so this would not apply to you. Though it would apply to any of your data stored on a US server, or possibly with a US company. I don not know privacy laws of France.

So I have no desire to get worse performance, add failure modes, and give up all rights to my data.

Synology makes decent NAS devices as does Drobo.

Good luck.

[theedge]

Hi Joelaff, All valid points there tank you.
- Getting another NAS is of course still high on my list, just exploring options here.
- Concerning performance, we now have 5Gb internet available here. But even considering a slower connection, the advantage of the cloud client is that it will download files only once and cache them locally. So with good SSDs on each machine scenes load even faster than on the network.
- Money : I have yet to compare the total cost over several years. That will be a determining factor of course.

[Joelaff]

Nice to have 5Gb internet!

Also take into account latency, however if things are being cached by the cloud client that is a big advantage. Fusion (compositor) uses local caching, and this is wonderful.

Note that you could also do local caching by a number of methods (Windows Offline Files comes to mind, or auto-syncing with GoodSync).

If you or someone you know can build you a NAS it is pretty dirt cheap. You can use a lower end processor (most commercial NAS even use ATOM chips, which are really slow... too slow for my tastes). Drives are cheap, and now you could use SSDs. Linux RAID is not terribly hard to setup. The advantage of a homebrew solution is you don't have to worry about some company deciding when your device is EOL, trying to get more money out of you. Most people use open source software which is constantly updated, and you can use much better hardware for the price.

Not to mention 10Gb Ethernet locally makes a big difference with large files. That would be a good upgrade, for the NAS and switch if nothing else.

I don't like third parties in my data stream. Too many points of failure with zero liability. Internet dies? You miss a deadline and lose 1K, 10K, 100K worth of work, or worse you get sued for millions for non-completion. No thanks.

Good luck.

[theedge]

Windows Sync sounded interesting, I never tried it before.
But after a few hours of testing and fiddling, unless I'm missing something, it does not do "smart" on-the-flying caching as files are being accessed from remote shares. My only option is to manually set a folder as availabe offline.Which is fine for a project folder, but unconcievable for the assets libraries.

[flino2004]

I changed my business email to google and I got unlimited storage included... if you have the business account you could have "Drive file stream" ..it allows you to setup Gdrive as a local drive and you can select what folders or files have offline access.

[rusteberg]

I have google business and was recently quoted $100usd/mo. for 10TB storage..... I'm with Joelaff. I don't trust any of those fuckers because it all comes down to $$$

[theedge]

Yep, I've been actively testing Google Drive file stream for several days now and I have to say that I am very impressed.

[theedge]

I have google business and was recently quoted $100usd/mo. for 10TB storage..... I'm with Joelaff. I don't trust any of those fuckers because it all comes down to $$$

You actually get unlimited storage if you have more than 5 users and that costs 60$/mo

[leo.surrealismo]

that unlimited storage has a maxium file count of 300.000 files if I remember correctly.

[theedge]

To perform my tests I uploaded a portion of my assets that represents 4Tb and a little more than 500.000 files with rclone and didn't have any problems.
Maybe the 300.000 files limit is per folder ?

[leo.surrealismo]

well, maybe they modified the limitation? that`s really good news.

[rusteberg]

You must have found the unicorn connection into google then.... still don't trust them but I'll continue listening to your sales pitch. may want to check your temperature before you decide to go full cloud in case you got the covid and its making you hallucinate : )

[theedge]

Haha
That number of files limit intrigued me so I searched a bit and I found that there is indeed a limit of 400.000 files but only on Shared Drives. As for "regular" drives I couldn't find any mention about limits.
Also, although this thing works much better than I anticipated I seriously doubt that I'll go full cloud. My main concern being the sheer amount of data to handle. To ensure business continuity I'd have to keep a synced offline replica available at any time, which means having a NAS anyway.