Database Implementation

[ccorcos]

I'm trying to figure out where the database is within this codebase and it appears to be tightly integrated with the DSL... Is that true? Can someone point me to the right place and maybe give a little explainer about how the Eve database works?

P.S. I've been enjoying reading through the level-fact-base. Curious about similarities.

[thenikso]

Every once in a while I come back to this repo to try to understand how Eve works.

If I grok it – I'm thinking – I could re-implement it where I see it fit; maybe make it great as it should have been.

I wander in the code trying to find a way in; maybe I should just read it. Wait, what do those Proposal fields mean? Perhaps I'll follow a simple program execution with the debugger. Oh my this Transaction is insane. I should learn Rust and read those sources instead. &mut 'a <&Some o_0 wait, who am I again? It's 2 in the night?! What am I even doing and why?? I've gotta go to work tomorrow!

Well maybe there is a way to make it performant enough. Maybe with the right UI people would love it. Maybe I could... if only I could understand it. Or if only I could get rid of this itch.

Oh but this is nonsense! It took them years to get to this point, what do I think I could do in just my spare time? I should be thinking about the code I'll have to write tomorrow, not how Constraints and Nodes are generated. Stop it! It's 3am and now you stop it for good. Close all the editors. In fact, uninstall Rust! That's it, now stop thinking about it. You didn't even wrote Eve programs basically at all. I probably would't even use it myself. In fact I don't like it. Now sleep.

Maybe next time.

[ccorcos]

amazing! 🤣

[ccorcos]

@thenikso please let me know when you figure it out!

[joshuafcole]

Depending on your use case, it's worth mentioning that we eventually made the choice to move away from Eve-as-a-DB, because even a blazing fast database like this one is still many times slower than an AOT or even JIT compiled language. If you're looking to make interactive applications, a transpilation process into JS* is a better bet. We built a very promising prototype of that near the very end which had significantly better performance than 0.4 did.

• The team working on V8 is scary good. For handwritten functions, runtime evaluated JS expressions perform within 2-8x of the speed of a similarly handwritten release-optimized rust implementation.

For folks still interested in the database, I strongly recommend first familiarizing yourself with Frank Mcsherry's Timely Dataflow and WCOJ. Honestly, you might also consider just using Timely if you can -- we only built the DB because we had to.

Frank's work was a major source of inspiration for our incremental implementation, and Generic Join is the backbone of our query evaluator. You might consider reading the Generic Join implementation from Eve 0.2 first -- It's the same algorithm as we use in Eve 0.4, but without all of the craziness involved in making it incremental.

Good luck! :)

[thenikso]

I heard about that speed concern (in the "Against the Current" talk), and yet couldn't Eve have been compared to, say, spreadsheets or datomic rather than compiled programming languages?

The algorithms behind the database are very intriguing anyway, yet I am most interested in playing around with Eve itself. The idea of transpiling crossed my mind, when I grok the whole thing I might give it a shot; especially now that I know it was your next experiment!

The work you've all done is astonishing to say the least. Congrats @joshuafcole !