The query examples looked like Clojure code; is that correct?
The technology of Facebook and Google (where I worked as a contractor) impresses! That said, I like a simpler Internet with smaller scale services like Gnu Social, and simply using email to stay in touch with family and friends. I have this preference both for privacy reasons and also prefer more one on one communication.
>That second form to filter should be a function, but in this case it is an expression that returns a boolean and is not itself a function.
I'm pretty sure this is Clojure or something very close to it. ->> is the thread-last macro. What it does is take the first expression, ($alice) and inserts it as the last expr in the next form so that the second step (assoc $friends) would eval to (assoc $friends $alice). It continues this by inserting this expr into the last part of the next form. So the example would reduce to:
(filter (> age 20) (assoc $friends (assoc $friends $alice)))
But as I mentioned, the second argument to Clojure's filter is a function, and (> age 20) is not a function; therefore, this is not normal Clojure (or Clojure or at all). You'd have to replace the standard library's filter function to get the syntax you see in this article, which I doubt they are doing. therefore: not Clojure.
You could also walk the ast and do something special with unknown symbols (and functions that contain them).
i.e. while walking, we notice a function (> age 20) and age isn't visible in scope. We thus rewrite that into something like (fn [age] (> age 20)) or (fn [x] (> (:age x) 20)).
I've taken this approach before with DSLs. It makes for extremely fast to write business logic where business users don't have to care as much about where their data comes from (they just need field names).
In general code, that would be very hard to read, not knowing if a form represented an actual value or a function. Clojure being dynamically typed makes things hard enough; changing the language syntax as you describe sounds pretty risky to me.
The purpose of such a DSL isn't to let you write generic clojure faster. The DSL isn't even targetted at traditional programmers. The purpose is to enable training people who have no idea what Clojure or Lisp to get things done that are complicated (or expensive) to design graphical UX for. These people might have some excel experience or maybe a programming class in college, but are mostly reliant on examples and documentation to get things done. Why teach them things like how to create a function when they don't need to know or care? There is a huge usability difference for these people between (fn [age] (> age 18)) and (> age 18).
It's Lisp code, more particularly Lisp syntax, and I mean that in the most general sense in which Clojure is Lisp syntax. So are Common Lisp, Scheme, Racket and Emacs Lisp.
For distributed relational or graph databases, seems like the key trick for making queries efficient is to get related data on the same host, whenever possible.
So would be cool to dig into the specifics of the algorithms they are using, to see exactly how they are optimizing where to store the data. With a graph database, it's impossible to guarantee having all related data together (eventually, friend of a friend of a friend...will be on a different host). So needs to be heuristics based.
For tree shaped data, on the other hand, it is possible to have the root of each tree and all of its related data on the same host (assuming each tree is "reasonable" size). Google's F1 project took this approach.
Slight tangent here but is it me or has the experience in social networks taken a hit? Feels like it was better with low complexity hack technology than with technology solutions focused on scaling the experience. I sometimes wonder if we are changing experience too much for technology optimization.
- LinkedIn has been terrible for about 2-3 years since things stopped updating real time and the timeline started getting random
- Facebook randomly suggesting things from your graph based on what it thinks you like
- Twitter I heard soon will be no longer in chronological order?!?
I think there is something to be said for low complexity MySQL and memcached.
yeah I hear that and to be clear I'm not blaming all their experience issues with the stack.
But there are definitely stack complexity trade offs when you get to the scale of these guys.
Also could be more correlation not causation that when you get to a certain scale and more stack complexity you also start hiring specialized staff that end up diluting the culture and spirit of a startup.
I don't think any of those things are caused by optimisations or needing to scale.
Saying that Facebook is recommending things simultaneously at random, based on a reccomendation algorithm, and also as an optimisation which wouldn't be needed if they used MySQL, is obviously nonsense.
@chrisseaton - big fan of your work, eagerly awaiting Truffle JRuby production ready release.
I'm not saying MySQL > some specialized solution for scale... I'm guess I'm drawing the correlation that when companies have to make investments like this they are also trading off experience, or maybe to your point screwing up the experience using recommenders and algorithms instead of letting my feed flow.
Yeah I see your clarification in your other comment where you say that when they start to scale it opens the door for tons of other complexity. That makes more sense.
The technology of Facebook and Google (where I worked as a contractor) impresses! That said, I like a simpler Internet with smaller scale services like Gnu Social, and simply using email to stay in touch with family and friends. I have this preference both for privacy reasons and also prefer more one on one communication.