Projects - 10

(Note: This is a follow-up rant of my pessimistic conclusion about what data science means from two years ago.)

Let me come straight to the point: Data science is synonymous for marketing. Period. Do not let yourself be misguided by online data science/machine learning/statistics lectures which only cover topics from the mathematical or programming area. These will only take up 5% of your work! The rest is 60% hot-air blowing, 50% managing/organizing and 40% delegating to juniors or externs, making that a total workload of 155% (60+ h/week), with expectations on you to generate projects and to build Big Data strategies. (After all, you’re smart, right? Otherwise you wouldn’t know that much math and AI. You’re smart, thus you can land big projects, right?) And the area of application is advertising, the data is customers, the outcome is customers—more people buying more stuff.

No, thanks for the offers, but I won’t work in advertising.

Who actually is making Big Data a hot topic? It’s those who create these solutions in the first place and sell them to companies who use these in their marketing processes.

“Data is the new oil.” Really? So, how do you power your cars with data? BS! We’re still living in the oil interval.

It’s so sad that AI is not solving any problems. It’s used for making game characters act smarter, it’s used for more effective advertising, it’s used for making call centers obsolete (by executing voice commands), it’s used for shopping agents (by executing voice commands), it’s used for autonomous driving of cars, vacuums and lawnmowers. It’s entirely used for making us (fat first-world people) even more lazy (fat), even more consumeristic (fat) and even more entertained (fat). These are solutions looking for a problem, but they can’t find one. Seriously, I see much more potential in the blockchain, which deserves its own rant soon.

(Note: This is a rant, so what I’m trying to say is possibly written between the lines.)

2012, three years ago, I was working in the context of computer vision, teaching computers to see. While this is still an exciting field, yielding exciting technology, no one is really making money there so far, because these are solutions looking for a problem—there is no itch to be scratched. Our department was selling tunnel surveillance systems to the traffic industry, which was quite a niche and didn’t contribute to getting our company out of financial trouble. I started a learning phase, trying to get deeper into that machine learning thing, seeing myself as a technical expert in a few years, being known for bringing complex theoretical concepts to life in successful solutions—at a place where such skills throw off money.

During generic research I collected more and more knowledge about the new hot field called data science, a magical mixture of statistical modeling and modern computer technology with its application in business. Since media mentioned IBM as player in the first row, I got in touch with their local office. And really, they hired me! However, I found myself placed onto the wrong track: I was expected to ensure that others do the work I was interested in doing, to generate projects, to devise proposals from zero to signings, to tell bank reps that they had to understand their customers as individuals to compete in today’s market. I was definitely not needed as a mathematician with a knowledge of data mining algorithms there. They needed business economists, marketers and sellers with an understanding of industries. The actual work that I was interested in doing—hacking fancy predictive models—would be delivered by folks who work at external business partners. How could that have happened? Both sides seemed to have had different expectations and interpretations. So, I was immediately job hunting again, and data science disappeared from my career radar during my way back to the software engineering world.

At my current employer, I’m somewhat known as the guy who knows about big data (although I haven’t ever tried Hadoop) and data mining (although some of my coworkers are “real” statisticians). But during the recent months I concluded that all this data science is just one good old thing: marketing. The big part that actually defines data science is totally not explained by its name: It’s definitely and exclusively solving business problems. Data mining, on the other hand, has different interpretations. I, too, was blinded by what tech people see when hit with this buzzword: Hadoop, MapReduce, statistical algorithms, other fancy formula-heavy or technological stuff, applied to data of manifold origin. The business folks however have that marketing interpretation:

Data mining is finding more people to sell stuff to.

Data mining is market basket analysis (what stuff people buy), upselling (more expensive stuff), cross-selling (additional other stuff), understanding a company’s customers (people who buy stuff) to prepare marketing campaigns (telling people to buy stuff). Hey, business analyst, find more people to sell our stuff to! Oh, you’re a data scientist? Well, what difference does it make? Find more people—they might be customers already, possibly thinking about leaving us, or they aren’t our customers just yet. Or, possibly create a new product. Data mining is also about creating more stuff to sell to more people.

So, be careful not to mistake data science with data mining. As a data scientist, you won’t just practice R programming, cleaning data, data analysis, statistical inference, or creating data products. If someone wants to hire a data scientist, they are looking for a business professional who, pointing at data in a spreadsheet, tells CEOs how they should transform their company. See, sometimes, someone tries to headhunt me for “[…] acting as a partner for marketing executives and collaborating with colleagues in management accounting […] Developing procedures to measure marketing campaigns on a global level together with managers and executives in marketing and sales […] identify new business opportunities […] Demonstrate business acumen […]”.

Only rarely it goes like “[…] work with complex, varied, high-volume data sets that have real meaning for our customers’ health and wellbeing […] Identify patterns and correlations of a user’s fitness data […] Good statistical, mathematical and predictive modelling skills to build the algorithms […]”—Wait, what, Runtastic are Austrian!? (Or rather: Runtastic are awesome although they are Austrian!?)

Maybe that topic comes back to me once that pile of sensor data has become higher and the internet of things takes off. But I’m not in my twenties anymore, so the doors and clefts to slip through have become narrower.

• A Strategic Mistake With Big Data — International Institute for Analytics

  tags: bigdata opinion

  • Bill Franks
  • That mistake is the development of a siloed, distinct big data strategy
  • strategy for big data is a new facet of their overall enterprise data and analytic strategy
  • the mess that many multi-channel retailers got themselves into through their entry into e-commerce
  • retailers launched distinct e-commerce divisions. Some were even separate legal entities. As opposed to viewing e-commerce as a new facet of an overall retail strategy
  • distinct processes and distinct infrastructure was created
  • provide a consistent experience for customers across channels
  • go into a store and grab a product and then find that same product on the retailer’s website.  Guess what? They have no way to match those products
  • separate, non-integrated strategies for big data will likely end up with systems and processes that are very difficult to integrate
  • integrate big data into the overall infrastructure and current and future analytic processes

• Machine learning system can ID cities via pics | Cutting Edge - CNET News

  tags: machinelearning computervision datamining research technology

  • The system automatically picks out relevant architectural details from photos
  • the details woven into the urban fabric that form a pattern
  • computers are learning to ID your city just by looking at random photos
  • Google Street View images of Paris, London, New York, and Barcelona
  • features like the street signs, balconies, and lampposts of Paris to be distinct
  • lack of stylistic coherence in American cities
  • presented at Siggraph 2012
  • emerging field of visual data mining, which is more complex than looking for patterns in text or numbers
  • Alexei Efros
  • we wish to automatically build a digital visual atlas of not only architectural but also natural geo-informative features for the entire planet

• Interviewing data scientist candidates? Ask these questions

  tags: datascience career opinion

  • “It’s way over-hyped,” said Franks
  • chief analytics officer for the data warehouse appliance vendor Teradata
  • “softer skills.”
  • “data artist.”
  • data scientists are those that are able to understand the business problem; they’re able to apply creativity and present the results well
  • intuition, which is hard to teach
  • five core areas I look at
  • Commitment
    • 1.
  • Creativity
    • 2.
  • thought process they went through deciding what to do. Someone who’s not creative is going to give me a list of the steps they went through one-by-one more from a technical perspective
  • Business savvy
    • 3.
  • what I want to hear is not just some technical reasons
  • Knowing how much information to give to the non-technical people
  • Presentation
    • 4.
  • specific presentation as part of the interview process
  • intuition
    • 5.
  • art or music or some other type of creative area

• Linux lessons for Hadoop doubters • The Register

  Interesting comparison of Hadoop today with the Linux story from the past. This could mean Hadoop/MapReduce as state of the art around 2020.

  tags: bigdata cloud technology linux

  • While Hadoop is all the rage in the technology media today, it has barely scratched the surface of enterprise adoption
  • Hadoop seems set to win despite its many shortcomings
  • still in the transition from zero per cent adoption to one per cent adoption
  • IBM points to a few specific deficiencies
  • lack of performance and scalability, inflexible resource management, and a limitation to a single distributed file system
  • IBM, of course, promises to resolve these issues with its proprietary complements to Hadoop
  • Hadoop is batch oriented in a world increasingly run in real-time
  • customers are buying big into Hadoop
  • it’s still possible that other alternatives, like Percolator, will claim the Hadoop crown
  • Back in 2000 IBM announced that it was going to invest $1bn in advancing the Linux operating system. This was big news
  • it came roughly 10 years after Linus Torvalds released the first Linux source code, and it took another 10 years before Linux really came to dominate the industry
  • The same seems true of Hadoop today
  • we’re just starting the marathon

• Can Big Data cut through your growing resume pile? - Fortune Management

  Three data-mangling job sites, all only for the US

  tags: bigdata career

  • Bright, one of several new companies
  • another new job site, Path.to
  • Gild, a third major player

• How ‘Big Data’ Is Different

  tags: bigdata datascience technology opinion

  • Thomas H. Davenport, Paul Barth and Randy Bean
  • how do the potential insights from big data differ from what managers generate from traditional analytics?
  • 1. Paying attention to flows as opposed to stocks
  • the data is not the “stock” in a data warehouse but a continuous flow
  • Streaming analytics
  • organizations will need to develop continuous processes
  • data extraction, preparation and analysis took weeks to prepare — and weeks more to execute
  • conventional, high-certitude approaches to decision-making are often not appropriate
  • new data is often available that renders the decision obsolete
  • 2. Relying on data scientists and product and process developers as opposed to data analysts
  • the people who work with big data need substantial and creative IT skills
  • programming, mathematical and statistical skills, as well as business acumen and the ability to communicate effectively
  • EMC Corporation
  • started an educational offering for data scientists
  • 3. Moving analytics from IT into core business and operational functions
  • new products designed to deal with big data
  • Hadoop
  • Relational databases have also been transformed
  • Statistical analysis packages
  • the cloud
  • “virtual data marts” allow data scientists to share existing data without replicating it
  • traditional role of IT— automating business processes — imposes precise requirements
  • Analytics has been more of an afterthought for monitoring processes
  • business and IT capabilities used to be stability and scale, the new advantages are based on discovery and agility
  • discovery and analysis as the first order of business
  • IT processes and systems need to be designed for insight, not just automation

• PayPal’s Mok Oh On What Is A Data Scientist? - Forbes

  The title is misleading: It’s not about what DS is. It’s rather a vision of the ideal solution.

  tags: datascience technology opinion

  • the old state and the ideal future state, which he calls “Analyst 1.0” and “Analyst 2.0,”
  • Analyst 1.0 as the state of maturity achieved by using the last generation of business intelligence tools
  • Analyst 1.0 has some coding skills, and perhaps writes an SQL query here and there
  • inflexibility of data warehouses and relational databases
  • Our current state of affairs, which we’ll call Analyst 1.5, finds us in limbo
  • two primary limitations: the immense size and variety of the data, and the complexity of the tools needed
  • Hadoop
  • to get value from big data, business analysts cannot simply be presented with a programming language
  • Analyst 1.5 is characterized by a disconnect between data scientists and the tools and systems in the more complex camp of programmers and computer scientists
  • caused data to be totally fragmented
  • Analyst 2.0 will have arrived when vendors and IT make analysis easy enough that a typical business user can conduct analysis entirely by themselves
  • Tools such as self-learning recommendations engines
  • demands new skills, such as a more precise focus on aberrant or statistically significant data in a stream, as well as better tools
  • somehow at some point you have to get your analytical inspection down to the equivalent of code level
  • what we’re trying to model is every person’s brain–at least the part of the brain that decides how to shop, when to shop, and what you want
  • we need to continue to mine for behavioral data, such as what people looked at before and after they made transactions
  • among the top pitfalls is the tendency to focus on a very small piece of data without occasionally stepping back
  • tendency to over-focus on technology
  • organizations are tempted to put the most technology-savvy person on the job, rather than the most business-savvy
  • computer scientists are not trained to ask the right business questions

• David MacKay: Information Theory, Pattern Recognition and Neural Networks: The Book

  tags: machinelearning books

  • Information Theory, Inference, and Learning Algorithms
  • 640 pages, Published September 2003
  • PDF (A4) pdf (9M) (fourth printing, March 2005)

• Cookbook for R » Cookbook for R

  HTML

  tags: r statistics programming

  • formerly named R Cookbook
  • It is not related to Paul Teetor’s excellent R Cookbook

• Big Data -- Why the 3Vs Just Don’t Make Sense -- TDWI -The Data Warehousing Institute

  When are we done defining big data? p.1

  tags: datascience bigdata opinion

  • settled on 3 Vs -- volume, variety, and velocity
  • another V: value
  • if big data is understood solely on the basis of these trends, it isn’t clear that it’s at all hype-worthy
  • if “big data” simply describes the volume, variety, and velocity of the information that constitutes it, our existing data management practices are still arguably up to the task
  • big data is hyped on the basis of its real or imagined outputs
  • a lot more interesting when you bring in ‘V’ for value

• No, Really, Some of My Best Friends Are Data Scientists

  When are we done defining data science?

  tags: datascience statistics opinion

  • the skills of a “data scientist” are those of a modern statistician
  • know how to move data around and manipulate data with some programming language
  • know how to draw informative pictures of data
  • Knowledge of stats, errorbars, confidence intervals
  • try to get people from different backgrounds
  • Great communication skills
  • a lot of what we teach The Kids now looks a lot more like machine learning than statistics as it was taught circa 1970, or even circa 1980
  • Everything I know about statistics I’ve learned without formal instruction
  • is not, in my experience, intrinsically hard for anyone who already has a decent grounding in some other mathematical science
  • mastering them really does mean trying to do things and failing
  • potentially hazardous. This is the idea that all that really matters is being “smart”
  • counter-productive for students to attribute their success or failure in learning about something to an innate talent

• Taming The Big Data Tidal Wave: Finding Opportunities in Huge Data Streams with Advanced Analytics (Wiley & SAS Business): Bill Franks

  Bill Franks is Chief Analytics Officer at Teradata

  tags: datascience bigdata books

  • 27. April 2012
  • 336 Seiten

• The Big Question In Big Data Is...What’s The Question? | SmartData Collective

  tags: datascience opinion bigdata

  • Some folks like to confuse Hadoop with big data
  • Focus On the Questions To Ask, Not The Answers
  • The failure of data warehouses to provide real-time data led to the creation of data marts
  • Data marts failed to provide complete and updated and comprehensive views
  • existing solutions still don’t solve the problem. Why? The market and business environment have changed
  • Data moves from structured to unstructured. Sources exponentially proliferate. Data quality is paramount.
  • Real-time is irrelevant because speed does not trump fidelity. Quantity does not trump quality
  • Business questions remained unanswered despite the massive number of reports and views and charts
  • The big shift is about moving from data to decisions

• Information Theory, Pattern Recognition, and Neural Networks

  tags: machinelearning video lectures

  • Draft videos (editing incomplete)
  • Entropy and Data Compression
  • Shannon’s Source Coding Theorem
  • Inference and Information Measures for Noisy Channels
  • Introduction to Bayesian Inference
  • Approximating Probability Distributions
  • Clustering
  • Monte Carlo Methods
  • Gibbs sampling
  • Neural Networks
  • Other course materials - free online text book [Information Theory, Inference, and Learning Algorithms]

Posted from Diigo. The rest of my favorite links are here.

It took me quite a long time to discover that my favorite knowledge management tool, Diigo, provides a feature to post one’s bookmarks to a blog. As I often had the desire to repost certain links I stumbled upon, I will do that occasionally from now on, mainly about everything from the topic pool of data mining (and related buzzwords), with flavors ranging from theory to applications, from technology to business. (I can’t really do that to social media sites, as it’s almost impossible to explicitly consume posts topic-wise. So, blogs aren’t really obsolete—yet.)

Btw, Diigo is really awesome: You can highlight text on webpages and add annotations to help understanding an article and create a summary on the fly, right while going through it. In this sense: If you want to be briefed, read at least this. (And don’t worry, the next episodes will contain less content; this one ranges back a few weeks.)

• Your Laptop Can Now Analyze Big Data - Technology Review

  tags: computerscience datascience technology software

  • GraphChi, exploits the capacious hard drives
  • a Mac Mini running GraphChi can analyze Twitter’s social graph from 2010—which contains 40 million users and 1.2 billion connections—in 59 minutes
  • The previous published result on this problem took 400 minutes using a cluster of about 1,000 computers
  • graph computation is becoming more and more relevant
  • GraphChi is capable of effectively handling many large-scale graph-computing problems without resorting to cloud-based solutions or supercomputers

• Large-scale Incremental Processing Using Distributed Transactions and Notifications

  Google’s Percolator paper

  tags: google datascience bigdata technology

  • Publication Year 2010
  • MapReduce and other batch-processing systems cannot process small updates  individually
  • Percolator, a system for incrementally processing updates to a large  data set

• Big Data in Deutschland – der Status Quo | silicon.de

  tags: bigdata datascience technology opinion

  • das Thema Big Data noch in einem frühen Stadium
  • noch in der Analyse- und Planungsphase
  • Verfügbarkeit neuer Analyse- und Datenbanktechnologien
  • dynamische Zunahme des unternehmensinternen Datenverkehrs
  • Big Data vielfach ‘durch die Hintertür’ ins Unternehmen
  • Datenwachstum von 42 Prozent bis zum Ende des Jahres 2014
  • auf Seiten der Storage-Infrastruktur eine Menge Arbeit
  • mittelständischen (500-999 Mitarbeiter) und
     den Großunternehmen (ab 1.000 Mitarbeiter)
  • Über ein Drittel erwarten sich Kosteneinsparungen. Fast die Hälfte verspricht sich bessere Einsichten in das Informations- und Konsumverhalten der Kunden
  • hohen Erwartungen, die an Dienstleister und Lösungsanbieter gestellt werden

• Why the days are numbered for Hadoop as we know it — Cloud Computing News

  tags: datascience bigdata cloud technology opinion

  • it has become synonymous with big data
  • de facto standard
  • Is the enterprise buying into a technology whose best day has already passed?
  • Hadoop’s inspiration – Google’s MapReduce
  • Google File System (GFS) and Google MapReduce (GMR)
  • make big data processing approachable to Google’s typical user/developer
  • Hadoop Distributed File System and Hadoop MapReduce — was born in the image of GFS and GMR
  • Your code is turned into map and reduce jobs, and Hadoop runs those jobs for you
  • Google evolved. Can Hadoop catch up?
  • GMR no longer holds such prominence in the Google stack
  • Here are technologies that I hope will ultimately seed the post-Hadoop era
  • it will require new, non-MapReduce-based architectures that leverage the Hadoop core (HDFS and Zookeeper) to truly compete with Google
  • Percolator for incremental indexing and analysis of frequently changing datasets
  • each time you want to analyze the data (say after adding, modifying or deleting data) you have to stream over the entire dataset
  • displacing GMR in favor of an incremental processing engine called Percolator
  • dealing only with new, modified, or deleted documents
  • Dremel for ad hoc analytics
  • SQL-like familiarity
  • many interface layers have been built
  • purpose-built for organized data processing (jobs). It is baked from the core for workflows, not ad hoc exploration
  • BI/analytics queries are fundamentally ad hoc, interactive, low-latency
  • Google invented Dremel (now exposed as the BigQuery product)
  • I’m not aware of any compelling open source alternatives to Dremel
  • Pregel for analyzing graph data
  • certain core assumptions of MapReduce are at fundamental odds with analyzing networks of people, telecommunications equipment, documents and other
  • petabyte -scale graph processing on distributed commodity machines
  • Hadoop, which often causes exponential data amplification in graph processing
  • execute graph algorithms such as SSSP or PageRank in dramatically shorter time
  • near linear scaling of execution time with graph size
  • the only viable option in the open source world is Giraph
  • if you’re trying to process dynamic data sets, ad-hoc analytics or graph data structures, Google’s own actions clearly demonstrate better alternatives to the MapReduce paradigm
  • Percolator, Dremel and Pregel make an impressive trio and comprise the new canon of big data
  • similar impact on IT as Google’s original big three of GFS, GMR, and BigTable

Heute haben wir den Vertrag für unsere neue Wohnung unterschrieben. Sie befindet sich deutlich außerhalb von Wien.

Nachdem ich vor einigen Jahren eine 35m² Wohnung in Wien gemietet hatte, die sozusagen als meine Studentenbude fungierte, hat es sich ergeben, dass wir sie schon seit geraumer Zeit zu zweit bewohnen, was sowohl an kapazitive als auch an die Grenzen der Behaglichkeit stößt. Nach dem Studienabschluss im Vorjahr und dem neuen Job beginnt für mich nun ein weiterer Abschnitt im „richtigen Leben“. Persönliche Highlights der neuen Wohnung sind für mich, dass es sich um einen Neubau mit kontrollierter Wohnraumlüftung handelt, große 95m² Wohnfläche + unglaubliche 24m² Terrasse + Kellerabteil + Garagenplatz = Flächenfaktor 3,5(!), 2 Kinderzimmer, Badewanne (endlich!), sowie – Trommelwirbel! – ein Geschirrspüler!!

Dass ich Kinderzimmer erwähne, führt auch schon zum Hauptgrund, warum wir nicht (mehr) in Wien leben wollen: Da wir selbst Landeier sind und mit dem tristen Flair des 11. Wiener Gemeindebezirks nichts anfangen konnten, können wir uns nicht vorstellen, unsere Kinder – so sie eines Tages kommen – zwischen Beton, Asphalt, Hundehaufen und Kopftücheln aufzuziehen. Viel wichtiger ist uns, innerhalb einer Minute in der Natur sein zu können. Der Stadtrand von Wien ist für uns aber unleistbar, womit der Radius weiter zu ziehen war.

Die Ortschaft, in der wir nun leben werden, ist relativ beschaulich, aber groß genug, um die wichtigste Infrastruktur (Schulen, Ärzte, etc.) zu beherbergen. Vielmehr liegt die Entscheidung dafür aber in der strategischen geografischen Lage: Es handelt sich um einen Bahnknotenpunkt, der Weg zur Arbeit nach Wien dauert genauso(!) lang wie der bisherige quer durch die Großstadt, und zwei größere Städte sind – sogar mit der Bahn – nur einen Katzensprung entfernt.

Ein Hausbau kam für uns nicht in Frage, da dann viel weniger finanzieller Spielraum bliebe für einen guten Lebensstil. Allerdings wird der Bau erst im Laufe des Sommers fertiggestellt und im Herbst übergeben. Am Freitag haben wir den Parkettboden ausgesucht. Wir können’s kaum erwarten!

While being employed 40 hours/week I started to repeat basics in functional analysis in January 2006. In April I started to do some general reading on the subject of time-frequency analysis. I wanted to have the topic of my Master’s thesis set until May, but it took me until August to file it with the working title “Gabor Analysis for Image Processing”. The finish of my thesis had originally been targeted for Christmas 2006, but it soon was clear that it would also take the whole spring of 2007.

With May 2006 I reduced my working times to 26 hours/week, and I quit my occupation in January 2007 as I was granted a 6-month scholarship. Now I had time to do numerical experiments and to actually write my thesis. The scholarship ended by July 2007, and I hoped to have my thesis finished by September. The final work, entitled Foundations of Gabor Analysis for Image Processing, was printed in the mid of November and graded an A. I had my master exam on December 18th and finished my studies with distinction.

Some statistics: It took me 8 months to read up on time-frequency analysis (while being employed). I was in the official status of a graduand for 16 months. I authored my thesis within 9 months.