During the event on Thursday you could see some trouble around our event management system that made it impossible to automatize the audition and some other functions. Here you can read a short description of the situation and the root causes.
Our system – besides all the phones and tablets – has a server collecting and sharing all information about the event. The server is used for many different functions, not just by Lindy Shock but by other companies. So it has other programs running for example a mail server.
That mail server had been hacked some week before Lindy Shock started. Initially it did not cause any issues. As a result of the hacking the mail server started to send spam. Step-by-step more and more e-mails. On Thursday it reached such an output that the response time got really slow – for all services running on that server.
What we perceived that our mobile equipments connected too slow to the server. Our application can work without continuous data connection. Slowly we started all tablets and audition could started ~20 minute late. At that moment we believed that we have too many mobile equipments connected and that is why the system is slow to answer.
Audition went well for a while and all votings were stored fine. However many people arrives without previous check-in and they needed wristbands for the audition. Also some participants had issue finding the right audition. To handle all these requests we needed to access the system that was almost impossible. As part of the participants had no wristband teachers could not vote on them. This was the moment when we had to stop the audition process and find a plan B.
The server was busy sending out spam that even restarting the server was challenging. Around late afternoon we have cleaned the server and could upload all result of the auditions we received on paper. Also we could evaluate the results we had on the tablets. So things went back to normal.
Sorry about the issue. We had plans in case something goes wrong – however not for this situation.