After the educational reform, several citizens discovered that all their personal stats had dissappeared. This caused quite some disturbance with many people. Today, Davz[323969] took the role of explaining the situation:

Hi Everyone,

This statementis aimed to provide you with a explanation of the problems (mostly, but not exclusively, with personal stats) over the last few days. (If you haven't noticed a problem, great, but I know that many users have!)

Reading the posts on the forums its clear that many users are upset, and realistically its unlikely that this post will make everyone happy, but I'm going to do my best to explain what has happened and what we are doing about it in the future.

Over the last day or so personal stats have mostly "reset" to a value from a short while ago.

The values that they have been reset to was a disaster recovery backup, not intended to be used except in case of disaster. This was a system that is not used for any other data on the site, was developed in a extreme hurry when stats were launched (at the time, we had no idea they would become such a popular feature). At the time stats launched, lag was a massive problem and the priority was to ensure that stats did not make lag worse - rather than to look after the data in stats. For all these reasons we went with a system that did not guarantee 100% data recovery in the event of a disaster (I reiterate that this is the only system on the site that this is the case for, this is specifically NOT the case for the other data)

For a as yet unexplained reason the existing personal stat system failed earlier in the week.

Therefore, a new system was quickly written using recent infrastructure[1] and most stats imported. Some users have noticed some stats have not reset for which I am sorry; the stats as they are now represent the most recent data that we have.

Following feedback in the forum and to improve the survivability of stats in future, as a urgent improvement we will be showing historical stats on the personal stats page which will allow you to see historical stats (probably on a daily granularity), and graph your progress with other users.

These historical stats will be fully backed up (we are unable to backup the "live" stats due to the sheer number of changes that are made to them).

At the same time there have been other reported problems with some other things resetting, in particular levels. It seems that this has the same root cause as the problem that broke stats - namely a primitive caching system which was brought in over a year ago in a desperate measure to control lag. This system is no longer required due to improvements elsewhere ,and it has been disabled. Unfortunately, it seems that there is a small amount of information that was "reset" during this process.

If anyone has any suggestions please do post here. The small team who manage the code and systems that run the site are working extremely hard, and we do listen to all of your feedback even if sometimes it is not possible to achieve everything you guys want.

-Davz

[1] For the techies, the system is based on http://memcachedb.org/. The previous system used memcached (no persistence) to store the stats, and at regular intervals moved changed stats into a key-value DB which we have used to restore from.