Business Intelligence & Data Warehousing Thought Leaders

Business Intelligence & Data Warehousing Thought Leaders

Have something to say? Join LinkedIn for free to participate in the conversation. When you join, you can comment and post your own discussions.

Ramesh B.

Ramesh

The big data world is full of small, scrappy startups using their ingenuity to build complex systems out of open source software, but the Walt Disney Company is not one of them. Here's what goes into building a big data...

  • Flag as Inappropriate
  • Comment (8)
  • September 17, 2012
  • Close viewer

Comments

  • Curtis P.

    Curtis

    Curtis P.

    Business Intelligence Consultant

    "In some cases, the team decided to hide the platform’s complexity from users; not to facilitate its use, but to keep loose-cannon developers from doing something crazy that could take down the whole cluster. It could show them all the controls and knobs in a NoSQL database, but “they tend to shoot each other,” Jacob said. “First they shoot themselves, then they shoot each other.”"

    I wonder if this is true or a presumption based on this person's experiences?

    Could it be a case of IT controlling too much?

    If a power user is able to develop in NoSQL, then they would have some pretty advanced skills, so wouldn't they be on development side? Do they have to limit what other IT staff do?

    So many questions ...

  • Arun J.

    Arun

    Arun J.

    Director, Data Solutions at The Walt Disney Company

    Hi Curtis. This quote, among many, was taken out of context. I'm a developer with 20+ years experience, and have developed solutions using Hadoop, Cassandra, Mongo, and Solr/Lucene. I wanted to have this kind of platform to simplify my use of a persistence layer.

    The main thing missing from the new NoSQL databases is clear separation of admin roles from user roles. In other words, users that store and retrieve data can also create databases. This is fine in a non multi tenant situation, but when we are building out a multi tenant cluster, the lack of role separation allowed developers who had stood these platforms up on their laptops to do things like run unit tests that created new database or database like partitions. When they migrated that code to a shared environment, then ran continuous build processes on the client code, the database creation rate resulted in 10s of thousands of databases being created, which compromised the integrity of the entire database to the entire user set.

    Power users are finding that they would rather consume services than write tightly coupled code...in certain circumstances. When persistence is part of a higher level business logic flow, the last thing that most developers want to think about is the relative complexity of persisting data to a distributed cluster. This article provides a great overview of some of the complex issues: http://highlyscalable.wordpress.com/2012/09/18/distributed-algorithms-in-nosql-databases/

    Just like we stopped writing assembler code when we realized that we could do so much more in C, people want to build applications on top of NoSQL applications and not have to operate and maintain clusters. In exchange for service level access, the user gives up some power. It's a convenience/usability tradeoff. Our job is to provide the most convenience and the most rational usability.

  • Ramesh B.

    Ramesh

    Ramesh B.

    Practice Director - Analytics & Information Management (Banking)

    @Curtis Interesting but an intriguing poser.

    @Arun It's a pity that good technical articles and presentations when quoted in the mainstream media gets chopped / edited in the guise of brevity and readability. Thanks Arun for the perspective. Much appreciated.

    Systems and applications are built keeping the end user in the mind irrespective of where the user comes from viz. IT / business. Approach to limit the end user's ability to utilize a development platform to it's fullest potential during it's infancy especially in fast emerging Hadoop related technologies can't be construed as a denial to use - at least IMHO.

  • Curtis P.

    Curtis

    Curtis P.

    Business Intelligence Consultant

    Arun thanks for clarifying. It was the colorful adjectives and references that prompted my questions eg "loose-cannon", "crazy", "shoot each other".

    I wouldn't call my users "loose cannons" if I gave them admin rights and they "went crazy" and "shoot each other" by using these admin rights.

    I would do what you did, which is entirely appropriate, and not give them admin rights and limit transactions to those appropriate to business requirements.

    This is nothing new to application development and methodology. We don't allow users to delete entire tables in SQL databases either, but instead control and log transactions appropriate to business requirements and audit.

  • Arun J.

    Arun

    Arun J.

    Director, Data Solutions at The Walt Disney Company

    Just to clarify, because again the quote about people shooting each other was taken out of context (and I don't believe I ever said 'loose cannon' developers :) : with raw access to NoSQL platforms people have compromised the entire cluster ("shot each other" -- I did say that) because they've done admin-like things that they were used to doing on their laptops, and because these platforms don't separate admin functionality from user functionality. These platforms will mature and provide this separation, but they are only starting to do this.

    Something else I said: it's easy to separate admin from user functionality in a small startup. You talk about it. But in a large organization, hallway conversations don't scale. An API that partitions admin and user access, on the other hand...

  • Curtis P.

    Curtis

    Curtis P.

    Business Intelligence Consultant

    Arun, after seeing your comments here, and rereading the Derrick Harris article, I am with Ramesh, that Harris missed some very interesting aspects of your work and didn't do a very good job of presenting the challenges of working with NoSQL databases. Thanks again for providing the additional information here. Perhaps Harris can do a followup and focus on these challenges.

  • Derrick H.

    Derrick

    Derrick H.

    Senior writer at GigaOM

    Arun, I'm glad to chat about this if in the name of clarifying the point about admin control.

Have something to say? Join LinkedIn for free to participate in the conversation. When you join, you can comment and post your own discussions.

Feedback