Stephan Paukner :: syslog

A Strategic Mistake With Big Data — International Institute for Analytics

tags: bigdata opinion

• Bill Franks
• That mistake is the development of a siloed, distinct big data strategy
• strategy for big data is a new facet of their overall enterprise data and analytic strategy
• the mess that many multi-channel retailers got themselves into through their entry into e-commerce
• retailers launched distinct e-commerce divisions. Some were even separate legal entities. As opposed to viewing e-commerce as a new facet of an overall retail strategy
• distinct processes and distinct infrastructure was created
• provide a consistent experience for customers across channels
• go into a store and grab a product and then find that same product on the retailer’s website.  Guess what? They have no way to match those products
• separate, non-integrated strategies for big data will likely end up with systems and processes that are very difficult to integrate
• integrate big data into the overall infrastructure and current and future analytic processes

Machine learning system can ID cities via pics | Cutting Edge - CNET News

tags: machinelearning computervision datamining research technology

• The system automatically picks out relevant architectural details from photos
• the details woven into the urban fabric that form a pattern
• computers are learning to ID your city just by looking at random photos
• Google Street View images of Paris, London, New York, and Barcelona
• features like the street signs, balconies, and lampposts of Paris to be distinct
• lack of stylistic coherence in American cities
• presented at Siggraph 2012
• emerging field of visual data mining, which is more complex than looking for patterns in text or numbers
• Alexei Efros
• we wish to automatically build a digital visual atlas of not only architectural but also natural geo-informative features for the entire planet

Interviewing data scientist candidates? Ask these questions

tags: datascience career opinion

• “It’s way over-hyped,” said Franks
• chief analytics officer for the data warehouse appliance vendor Teradata
• “softer skills.”
• “data artist.”
• data scientists are those that are able to understand the business problem; they’re able to apply creativity and present the results well
• intuition, which is hard to teach
• five core areas I look at
• Commitment
  • 1.
• Creativity
  • 2.
• thought process they went through deciding what to do. Someone who’s not creative is going to give me a list of the steps they went through one-by-one more from a technical perspective
• Business savvy
  • 3.
• what I want to hear is not just some technical reasons
• Knowing how much information to give to the non-technical people
• Presentation
  • 4.
• specific presentation as part of the interview process
• intuition
  • 5.
• art or music or some other type of creative area

Linux lessons for Hadoop doubters • The Register

Interesting comparison of Hadoop today with the Linux story from the past. This could mean Hadoop/MapReduce as state of the art around 2020.

tags: bigdata cloud technology linux

• While Hadoop is all the rage in the technology media today, it has barely scratched the surface of enterprise adoption
• Hadoop seems set to win despite its many shortcomings
• still in the transition from zero per cent adoption to one per cent adoption
• IBM points to a few specific deficiencies
• lack of performance and scalability, inflexible resource management, and a limitation to a single distributed file system
• IBM, of course, promises to resolve these issues with its proprietary complements to Hadoop
• Hadoop is batch oriented in a world increasingly run in real-time
• customers are buying big into Hadoop
• it’s still possible that other alternatives, like Percolator, will claim the Hadoop crown
• Back in 2000 IBM announced that it was going to invest $1bn in advancing the Linux operating system. This was big news
• it came roughly 10 years after Linus Torvalds released the first Linux source code, and it took another 10 years before Linux really came to dominate the industry
• The same seems true of Hadoop today
• we’re just starting the marathon

Can Big Data cut through your growing resume pile? - Fortune Management

Three data-mangling job sites, all only for the US

tags: bigdata career

• Bright, one of several new companies
• another new job site, Path.to
• Gild, a third major player

How ‘Big Data’ Is Different

tags: bigdata datascience technology opinion

• Thomas H. Davenport, Paul Barth and Randy Bean
• how do the potential insights from big data differ from what managers generate from traditional analytics?
• 1. Paying attention to flows as opposed to stocks
• the data is not the “stock” in a data warehouse but a continuous flow
• Streaming analytics
• organizations will need to develop continuous processes
• data extraction, preparation and analysis took weeks to prepare — and weeks more to execute
• conventional, high-certitude approaches to decision-making are often not appropriate
• new data is often available that renders the decision obsolete
• 2. Relying on data scientists and product and process developers as opposed to data analysts
• the people who work with big data need substantial and creative IT skills
• programming, mathematical and statistical skills, as well as business acumen and the ability to communicate effectively
• EMC Corporation
• started an educational offering for data scientists
• 3. Moving analytics from IT into core business and operational functions
• new products designed to deal with big data
• Hadoop
• Relational databases have also been transformed
• Statistical analysis packages
• the cloud
• “virtual data marts” allow data scientists to share existing data without replicating it
• traditional role of IT— automating business processes — imposes precise requirements
• Analytics has been more of an afterthought for monitoring processes
• business and IT capabilities used to be stability and scale, the new advantages are based on discovery and agility
• discovery and analysis as the first order of business
• IT processes and systems need to be designed for insight, not just automation

PayPal’s Mok Oh On What Is A Data Scientist? - Forbes

The title is misleading: It’s not about what DS is. It’s rather a vision of the ideal solution.

tags: datascience technology opinion

• the old state and the ideal future state, which he calls “Analyst 1.0” and “Analyst 2.0,”
• Analyst 1.0 as the state of maturity achieved by using the last generation of business intelligence tools
• Analyst 1.0 has some coding skills, and perhaps writes an SQL query here and there
• inflexibility of data warehouses and relational databases
• Our current state of affairs, which we’ll call Analyst 1.5, finds us in limbo
• two primary limitations: the immense size and variety of the data, and the complexity of the tools needed
• Hadoop
• to get value from big data, business analysts cannot simply be presented with a programming language
• Analyst 1.5 is characterized by a disconnect between data scientists and the tools and systems in the more complex camp of programmers and computer scientists
• caused data to be totally fragmented
• Analyst 2.0 will have arrived when vendors and IT make analysis easy enough that a typical business user can conduct analysis entirely by themselves
• Tools such as self-learning recommendations engines
• demands new skills, such as a more precise focus on aberrant or statistically significant data in a stream, as well as better tools
• somehow at some point you have to get your analytical inspection down to the equivalent of code level
• what we’re trying to model is every person’s brain–at least the part of the brain that decides how to shop, when to shop, and what you want
• we need to continue to mine for behavioral data, such as what people looked at before and after they made transactions
• among the top pitfalls is the tendency to focus on a very small piece of data without occasionally stepping back
• tendency to over-focus on technology
• organizations are tempted to put the most technology-savvy person on the job, rather than the most business-savvy
• computer scientists are not trained to ask the right business questions