Wow...never seen a company do this: What happened last night

[Heden]

https://forums.station.sony.com/eqnl...st-night.6414/

Here's a wrap-up. (Remember when we said we'd tell you everything? Well...here goes.)

1) When we tested Landmark internally, we only had a single world. So...we made a few mistakes, one of which was severe. Essentially, our data structure didn't preserve the relationships between the various isles and the world that the isles were on.

2) We also had a few memory leaks. That's pretty normal stuff that we expect to squash during testing, but in this case, it eventually caused different islands to drop and reboot. And that's where the problems started.

3) Because the isles didn't necessarily stay with the world they should have stayed with. Worse, that island would get orphaned out into the nothingness (not attached to a world) and when the world it was *supposed* to be attached to detected its absence, well...it made a new island from scratch instead. (And yes, that's why peoples' claims were disappearing.)

4) Once we realized that, we started figuring out a way to transition the current linkages to a new system that would remember the world it was on.

5) But then about half the islands came down at once (due to memory leakage). And we realized that things were going to get VERY messy if we let them come up again. So we outaged everything at once, and the game came off-line for the first time.

6) Then we started trying to splice together the data we had with where it was supposed to go. And yes, this was exactly the spaghetti mess that it sounds like. In fact, it was so bad that we're now giving up on it (after trying to make it work for six hours, so you *know* we wanted it to work), because no matter what we do there will be some data loss, and it's going to be a real issue to try to tell which players are bugged, which are missing data, and which are in good shape. No way for you to know from your end except that your stuff might be gone...or it might not.

*sigh*

Okay...so what are we doing about it? Well, we're going to reset the voxel data so that everyone has a clean slate in that regard.

Does that mean that you're going to lose your stuff? Are you going to lose progress? The answer is yes and no. Here's details:

a) All voxel data will be reset. That means that the world is going to be reset back to original server conditions. Thus, if you put up a building, or dug a hole, those things are all going to be reset.

b) Things that you crafted (e.g., saw tables, stone forges, etc) WILL STILL BE WHERE YOU LEFT THEM. But that means that if you dug a hole into the side of a hill and put your forge in there, then your forge is now going to be buried inside that same hill. You'll need to dig it out.

c) You will not lose any resources or inventory items.

d) You will not lose your claim. It will still be wherever it was previously.

So, we apologize for any building creativity that you lose from this first day's wipe. This is not something we intended. We hope to not wipe data again anytime soon. (But you might try to save templates of the things you build in the near future, just in case. That way you can restore it if we find it necessary to wipe again.)

Make sense? If not, please ask questions!

-- Dave.

[RJO]

They should stop trying to make it a live service and just wipe imo

[Heden]

I tend to agree...but I must say I am impressed by their level of communication.

[Gawd]

Love the new SOE attitude of openness, i remember in vanguard all we would get is knowledge that the servers were down and a guesstimate on how long they would be down, then get told to go sort the W's from our M&M's and let the smart people do their thing

[Nith]

From an update post by SOE's smokejumper late last night:

In the spirit of "open kimono", here's what happened. We wanted to get claim flags out to everyone that was missing one. We thought we had a clever way of ensuring that only the people that were missing a flag would get one (because everyone having two flags right now would be really bad).

So the coders started ripping into it, got about halfway through it, and then found a logic flaw.

We almost immediately figured out how to solve it...but it was going to be a LOT more work.

But we decided that people not having flags all day Sunday was just not cool, so we're putting the work in to make it happen. And...well...it's taking a while.

I guess that means they don't care who sees their junk. Let it all out!