Licensnsing services outage (07 March 2019)

[swords]

We are investigating an issue with the online licensing services. We will keep you updated.
For now, please use offline licenses. (https://docs.chaosgroup.com/display/...or+Offline+Use)

Update: [02:25 UTC]The system has been restored to its functioning state.

Update 2: You can see the status of the licensing services at https://status.chaosgroup.com/ns/ols

A detailed report on what happened will be posted later today (01 March 16 UTC)

Update 3:

Timeline

01:08 UTC	Packet forwarding tables on the virtual machines where Chaos Group Online Licensing backends reside are exhausted.
01:09 UTC	New connections cannot not be open from and to the Licensing service. Operations engineers are alerted to the problem.
02:10 UTC	The cause of the problem is identified and VM instances are recreated as starting new Kubernetes pods does not alleviate the situation. A topology change providing a service fallback in case of disruption, which had been scheduled for deployment for Saturday, March 2, is also rolled out.
02:23 UTC	Re-creation of services and re-deployment is started.
02:25 UTC	Service status returns to normal.

We are still investigating the root cause of the incident

[bobacom]

is there a 'how to' on using offline licenses?

[glorybound]

I am not sure what 'offline licenses' means.

[pietervanstee]

please use offline licenses.

Great, where do you suggest we get this offline license? I've switched to online licensing.
I'm having a meeting in 6hrs with 95 renders to go on the farm.

Get this fixed!!!

[swords]

You can use your online licenses offline for a period of time. You can find more info here: https://docs.chaosgroup.com/display/...or+Offline+Use

[gearhead7]

It's back up for me. I'm interested too! I think I noticed something about offline licensing (I use Rhino) so I think it was under Vray --> Help --> License --> License Server and there's an option there iirc. I was going to try it right now but the server's up and I got work to render so I didn't bother to see what it does. HTH.

[squintnic]

Can you just build in a 24hr grace period instead? This is introducing a whole extra layer of unnecessary problems. For those of us that work in large organisations and cant do this easily it can take ages!
Would be good if someone could just crack NEXT and we could use that as the license instead.

[pietervanstee]

Ok, but don't you need a connection to the license server at the moment you're borrowing the licenses?
If they're down I guess this doesn't work, or does it?

[swords]

Ok, but don't you need a connection to the license server at the moment you're borrowing the licenses?
If they're down I guess this doesn't work, or does it?

The back-end infrastructure is split in a way that allows for borrowing of licenses for offline use even if the license-granting service is unavailable.

[pietervanstee]

Can you just build in a 24hr grace period instead? This is introducing a whole extra layer of unnecessary problems. For those of us that work in large organisations and cant do this easily it can take ages!
Would be good if someone could just crack NEXT and we could use that as the license instead.

yes, such a period woud be very welcome, even if the outage is only 5min reloading my large scenefiles is time I can't spend right now.

[nick_martin]

You can use your online licenses offline for a period of time. You can find more info here: https://docs.chaosgroup.com/display/...or+Offline+Use

Why does this not happen automatically when YOUR servers go down?

[swords]

Can you just build in a 24hr grace period instead? This is introducing a whole extra layer of unnecessary problems. For those of us that work in large organisations and cant do this easily it can take ages!
Would be good if someone could just crack NEXT and we could use that as the license instead.

We are actively working on a solution that will mitigate such problems.

[squintnic]

Great. Looking forward to it

[ruffstuff]

Up to how long can you borrow those licenses?

[swords]

You can borrow them for up to two weeks. If you don't need them on the given machine anymore you can release them back to your online pool.