YARN 
MapReduce has undergone an entire overhaul in hadoop-0.23 and we now have, what we name, MapReduce 2.0 (MRv2) or YARN.
The elemental concept of MRv2 is to separate up the 2 main functionalities of the JobTracker, useful resource administration and job scheduling/monitoring, into separate daemons. The concept is to have a world ResourceManager (RM) and per-application ApplicationMaster (AM). An utility is both a single job within the classical sense of Map-Scale back jobs or a DAG of jobs.
The ResourceManager and per-node slave, the NodeManager (NM), type the data-computation framework. The ResourceManager is the last word authority that arbitrates assets amongst all of the functions within the system.
The per-application ApplicationMaster is, in impact, a framework particular library and is tasked with negotiating assets from the ResourceManager and dealing with the NodeManager(s) to execute and monitor the duties. …
bolt
Bringing multi-dimensional arrays to distributed settings via a unified Python interface. Bolt is an open supply library offering a Python interface to ndarrays backed by native or distributed implementations (presently focusing on Spark). We wish to make working with massive array knowledge in Python as straightforward and seamless as in native settings, whereas exploiting the velocity of confirmed distributed engines. …
Graph Normalizing Flow
We introduce graph normalizing flows: a brand new, reversible graph neural community mannequin for prediction and era. On supervised duties, graph normalizing flows carry out equally to message passing neural networks, however at a considerably lowered reminiscence footprint, permitting them to scale to bigger graphs. Within the unsupervised case, we mix graph normalizing flows with a novel graph auto-encoder to create a generative mannequin of graph buildings. Our mannequin is permutation-invariant, producing whole graphs with a single feed-forward go, and achieves aggressive outcomes with the state-of-the artwork auto-regressive fashions, whereas being higher suited to parallel computing architectures. …
Operational Intelligence (OI)
Operational intelligence (OI) is a class of real-time dynamic, enterprise analytics that delivers visibility and perception into knowledge, streaming occasions and enterprise operations. Operational Intelligence options run queries towards streaming knowledge feeds and occasion knowledge to ship real-time analytic outcomes as operational directions. Operational Intelligence gives organizations the flexibility to make selections and instantly act on these analytic insights, via handbook or automated actions. …
