In short:
telegram in secure chats (http://core.telegram.org/api/end-to-end) was using modified version of Diffie-Hellman algorithm:
key = (pow(g_b, a) mod dh_prime) xor nonce
(original: key = pow(g_b, a) mod dh_prime)
That custom 'nonce' is derived from server and in theory server can send a specially formed nonce which will lead to known client keys ("bookmark"). It means that server as MITM can read all needed chats by request..
Authors of Telegram agreed that it is a big hole and their algorithm needs modification. User, who found the issue will get a prize (not 200,000$, but good enough).
Yes. The contest is not about actually exposing flaws in their cryptosystem, which is why the rules are rigged up in a way that would allow even a terribly insecure protocol (like Telegram's, or Moxie's counter-challenge protocol) to pass as "secure".
Telegram's contest itself is meaningless regarding the security of its protocol (as others explained in details). Finding bugs such as this deserves 200k more than anything else
Offering the contest was shady and stupid enough. Not paying just proves they're chiselers that never intended to pay in the first place. This wins the runner-up award for second most botched PR disaster ever. The consolation prize is a lump of coal.
The contest was stupid and shady, but they did explicitly state the rules. Its not really fair to accuse them of wrongdoing if they don't give this guy $200,000. If I were in charge of telegram, I'd pay the guy the whole amount, but that would just be out of the kindness of my heart. Morally, they aren't obligated to pay just because the rules of their contest are ridiculous.
Yes, outside.
Although the underground of this contest was to prove security and inability to read messages by 3rd parties
contest was about decrypting Durov's chat. Chat is not decrypted and as many say - it's nearly impossible (http://thoughtcrime.org/blog/telegram-crypto-challenge/).
But "bookmark" is found and it means that all the PR was shit.
Telegram already removed 'xor nonce' from their documentation and added a new layer to their api but who knows where they will put a next bookmark. Probably it will be hidden in a deeper place. But trust is already lost.
It appears that either their secure protocol was containing the bookmark specially or that protocol is getting written by non-specialists which cant guarantee any security.
