Entries from Stephan Paukner

It took me quite a long time to discover that my favorite knowledge management tool, Diigo, provides a feature to post one’s bookmarks to a blog. As I often had the desire to repost certain links I stumbled upon, I will do that occasionally from now on, mainly about everything from the topic pool of data mining (and related buzzwords), with flavors ranging from theory to applications, from technology to business. (I can’t really do that to social media sites, as it’s almost impossible to explicitly consume posts topic-wise. So, blogs aren’t really obsolete—yet.)

Btw, Diigo is really awesome: You can highlight text on webpages and add annotations to help understanding an article and create a summary on the fly, right while going through it. In this sense: If you want to be briefed, read at least this. (And don’t worry, the next episodes will contain less content; this one ranges back a few weeks.)

• Your Laptop Can Now Analyze Big Data - Technology Review

  tags: computerscience datascience technology software

  • GraphChi, exploits the capacious hard drives
  • a Mac Mini running GraphChi can analyze Twitter’s social graph from 2010—which contains 40 million users and 1.2 billion connections—in 59 minutes
  • The previous published result on this problem took 400 minutes using a cluster of about 1,000 computers
  • graph computation is becoming more and more relevant
  • GraphChi is capable of effectively handling many large-scale graph-computing problems without resorting to cloud-based solutions or supercomputers

• Large-scale Incremental Processing Using Distributed Transactions and Notifications

  Google’s Percolator paper

  tags: google datascience bigdata technology

  • Publication Year 2010
  • MapReduce and other batch-processing systems cannot process small updates  individually
  • Percolator, a system for incrementally processing updates to a large  data set

• Big Data in Deutschland – der Status Quo | silicon.de

  tags: bigdata datascience technology opinion

  • das Thema Big Data noch in einem frühen Stadium
  • noch in der Analyse- und Planungsphase
  • Verfügbarkeit neuer Analyse- und Datenbanktechnologien
  • dynamische Zunahme des unternehmensinternen Datenverkehrs
  • Big Data vielfach ‘durch die Hintertür’ ins Unternehmen
  • Datenwachstum von 42 Prozent bis zum Ende des Jahres 2014
  • auf Seiten der Storage-Infrastruktur eine Menge Arbeit
  • mittelständischen (500-999 Mitarbeiter) und
     den Großunternehmen (ab 1.000 Mitarbeiter)
  • Über ein Drittel erwarten sich Kosteneinsparungen. Fast die Hälfte verspricht sich bessere Einsichten in das Informations- und Konsumverhalten der Kunden
  • hohen Erwartungen, die an Dienstleister und Lösungsanbieter gestellt werden

• Why the days are numbered for Hadoop as we know it — Cloud Computing News

  tags: datascience bigdata cloud technology opinion

  • it has become synonymous with big data
  • de facto standard
  • Is the enterprise buying into a technology whose best day has already passed?
  • Hadoop’s inspiration – Google’s MapReduce
  • Google File System (GFS) and Google MapReduce (GMR)
  • make big data processing approachable to Google’s typical user/developer
  • Hadoop Distributed File System and Hadoop MapReduce — was born in the image of GFS and GMR
  • Your code is turned into map and reduce jobs, and Hadoop runs those jobs for you
  • Google evolved. Can Hadoop catch up?
  • GMR no longer holds such prominence in the Google stack
  • Here are technologies that I hope will ultimately seed the post-Hadoop era
  • it will require new, non-MapReduce-based architectures that leverage the Hadoop core (HDFS and Zookeeper) to truly compete with Google
  • Percolator for incremental indexing and analysis of frequently changing datasets
  • each time you want to analyze the data (say after adding, modifying or deleting data) you have to stream over the entire dataset
  • displacing GMR in favor of an incremental processing engine called Percolator
  • dealing only with new, modified, or deleted documents
  • Dremel for ad hoc analytics
  • SQL-like familiarity
  • many interface layers have been built
  • purpose-built for organized data processing (jobs). It is baked from the core for workflows, not ad hoc exploration
  • BI/analytics queries are fundamentally ad hoc, interactive, low-latency
  • Google invented Dremel (now exposed as the BigQuery product)
  • I’m not aware of any compelling open source alternatives to Dremel
  • Pregel for analyzing graph data
  • certain core assumptions of MapReduce are at fundamental odds with analyzing networks of people, telecommunications equipment, documents and other
  • petabyte -scale graph processing on distributed commodity machines
  • Hadoop, which often causes exponential data amplification in graph processing
  • execute graph algorithms such as SSSP or PageRank in dramatically shorter time
  • near linear scaling of execution time with graph size
  • the only viable option in the open source world is Giraph
  • if you’re trying to process dynamic data sets, ad-hoc analytics or graph data structures, Google’s own actions clearly demonstrate better alternatives to the MapReduce paradigm
  • Percolator, Dremel and Pregel make an impressive trio and comprise the new canon of big data
  • similar impact on IT as Google’s original big three of GFS, GMR, and BigTable

Changes during the recent months:

• Deleted accounts
• Gowalla (meanwhile shut down by FB)
• Foursquare
• FootFeed (that combined the above two)
• FB (after 8 months of deactivation)
• FriendFeed
• Flickr and Yahoo—I had already abandoned Flickr in 2010 and moved to SmugMug last year, but now I also deleted ...
• ... SmugMug, haven’t really dived into it, using Google+ as photo showroom
• PicPlz (only possible via mail to support), using Google+ for random pics
• Last.fm (never used)
• Blip.fm (rarely used)
• Posterous (never used)
• Flattr, had abandoned it in 2010, was too expensive
• Not (yet) deleted
• Brightkite, service was deactivated for weeks, site meanwhile unreachable
• Tupalo, maybe give another chance
• PayPal, closing didn’t work for days due to a “temporary communication problem”—at least I removed critical data
• eBay, not sure if I really don’t need it anymore
• Soup.io, another platform for devotedly wasting one’s time
• Other considerations
• Might set Twitter to private soon, only using it passively

... or, actually, to Perl’s Finance::Quote, under Ubuntu 11.10:

1. Clone one of the source modules in /usr/share/perl5/Finance/Quote (came with libfinance-quote-perl), say, Morningstar.pm, and place it under a new name into /usr/local/lib/perl/5.12.4/Finance/Quote, say, as Morningstar/AT.pm. (That’s what I did, I’ll probably make that module available.)
2. Modify this module into a uniquely new entity, providing appropriate IDs, like morningstar_at.
3. `perl -e ‘use Finance::Quote::Morningstar::AT’` should not display anything—especially no error.
4. Copy /usr/share/perl5/Finance/Quote.pm to /usr/local/lib/perl/5.12.4/Finance and add Morningstar::AT to @modules.
5. Execute `gnc-fq-dump | grep --color=auto morningstar`, you should see both morningstar and morningstar_at.

Your new quote source should now be available to GnuCash:

6. Go to Tools → Security Editor → Add (or Edit) and enter the appropriate “Symbol/abbreviation” and “ISIN, CUSIP or other code”—note that the symbol is not the ISIN in the case of Morningstar.at, it’s the ID in the URL! Check Get Online Quotes, switch to Unknown and select morningstar_at.
7. Go to Tools → Price Editor and click Get Quotes. After a few seconds you should see new entries with the current prices.

Since I plan to back up my worthy RAW pictures into the cloud (Why?), I collected some reference values of various cloud storage providers with the following important conditions in mind:

1. Support for GNU/Linux
2. Support for Android
3. Available for Europeans

The following graph lists the various providers I found and plots their available volume packages against their prices:

All prices are per year, with a fixed exchange ratio of 1 EUR = ¹⁰∕₇ USD. Lower is better. Dashed lines indicate a violation of the proposed conditions—see below.

In the following I provide a short summary for each provider, and a conclusion of my personal considerations.

A look back at the Motorola Milestone

I’m continuing the history of my [cell] phones by replacing my two-year-old Milestone. It literally has been a milestone: It was my first smartphone, and I could do everything with it (which, of course, wasn’t specific for that device, but for Android in general): Contacts and calendar were magically in sync with Google’s web apps. I could browse the web fully, even start embedded Flash videos—a zombie technology, considered dead since years. I used the GPS to do local exploration, e.g. with Google Maps/Places, used location-based games like Brightkite (meanwhile dead), Foursquare or Gowalla (which I lost interest in soon), or recorded my bike rides. It’s so “living in the future” to pan through Google Street View on a mobile device. I was root on a Linux system. GTD task managers and note apps are in sync with their respective web apps. I access important files in my DropBox. I receive audio streams from Google Music. I hold the phone up to a speaker and it freaking tells me what song it’s playing. Apps with AI (e.g. text predicting keyboard apps) are popping up, just as those implementing computer vision and augmented reality.

However, since several months my most used app is Running Services (which I called ruining services), followed by Android’s internal Task Manager: I had to cope with the phone’s limited RAM of only 256 MB every day. I could hardly install additional apps, although I had already applied a mem hack. It was a regular task to copy a Google Maps upgrade from /data/app to /system/app using Root Explorer (and do a hot reboot followed by deleting the old cache file). The phone also had a memory leak (since that infamous Android 2.2 upgrade that every European Milestone user was whining for for months), occasionally killing the alarm clock app during the night, making a precautionary reboot necessary every other day. However, I didn’t have the nerve to flash one of those very experimental alternative unofficial ROMs—Argh, the locked bootloader!—as they often introduced heavily disturbing and way too serious bugs, which I consider out of the question for a productive device that simply ought to work. The actual problem of course is that developers are constantly bloating their apps, keeping track with the hardware specs of the most recent phones.

My Milestone even got two hardware upgrades: A new and stronger battery, and a new LCD, which I had smashed accidentally.

Phone vs. tablet

So, I’m getting a new phone, but I didn’t really want to: Actually, I’m leering at an Android tablet since more than a year, and my intention was to use that device primarily and reduce the smartphone to a simple phone. I want to use a tablet as a kind of e-reader that supports handwritten input—I want to write formulas and draw freaking arrows!—, replacing my non-electronic (cardboard) tablet that holds printed sheets of paper and a pencil. So far, my workflow is to print research papers and read and annotate them with pencil on paper. There are also computer science e-books with hundreds of pages involved, printed incrementally, where I can only carry the currently read sections with me. Sadly, it seems that such a device is still months away. One of the main issues for me is that all of those 10.1" tablets currently only have a pixel count of at most 1280 along the wide edge, resulting in ~140 ppi, what I consider way too low compared to the densities of ~250–300 ppi of current phones. Another thing is precise stylus input using an actively powered stylus, allowing effective palm rejection. Slowly, that technology evolves, e.g. with Samsung’s Galaxy Note. Also, although Android 3.x had been optimized for the tablet form factor, it appeared having been rushed to market. I expect an incarnation of a tablet that meets my expectations within the next months, with an NVIDIA Tegra 3 quad-core CPU, Android 4.x, and e.g. Samsung’s S Pen. But I’m not going to wait any longer.

Coming up: Samsung Galaxy Nexus

This device is at the bleeding edge. But one of the important advantages is, just like with Google’s previous two Nexus phones, that its Android software is a “pure Google experience”, without any adaptation by a specific hardware manufacturer, what is one of the issues people have to deal with on other phones. This ensures that updates or upgrades come early and for a longer time. Of course, the hardware specs are a total win, giving me a dual-core CPU with 1 GB RAM. New to me will be the NFC chip, what will probably be of no use for me initially (as a European, but we’re getting there), and a front-facing camera for video chats. Let’s see if I’ll miss the Milestone’s hardware keyboard. Given that the Galaxy Nexus is also the first official Android 4.x device, it introduces new software features, but those won’t be specific to that device.

As I use to say: With that phone, I won’t need something else for years again.

SCOTTY mobil, den mobilen Reiseplaner der ÖBB, gibt’s jetzt auch offiziell für Android!

Endlich! Nach dem, dem, dem und dem wurde das auch Zeit! Und die App funktioniert sogar sehr gut, hat ein interessantes Zeitauswahl-UI und macht Echtzeitdaten abrufbar.

• I abandoned my Skype account very quickly. I’m not with MS.
• I deactivated my FB account last week. Regarding FB comment embedding into webpages: I like! So I only have to block a single domain to focus on content. FB is a dedicated platform for wasting one’s time. FB keeps crαp out of the usable web. FB is the people you wished you never went to school with. FB is a hideaway for web illiterates; I had started unfriending fellows who got clickjacked. I do not belong there. I’m with Google.
• Of course, it was a great way to keep in touch with non-geeks or relatives or freakin’ everyone I’ve ever met. Well, actually, no, it wasn’t. Without FB Purity, FB was entirely unusable. OTOH, I could so easily show off what nerdy stuff I’m stumbling upon.
• I unfollowed ~60 Twitter accounts during the recent time; my own script involving python-twitter (because UnladenFollow.com was down) was of great use to get stats about who spammed my timeline most. But I still follow ~½ of those accounts in a different way: Via Twitter lists or even in an individual TweetDeck column.
• I actually abandoned my Identi.ca account, although mainly geeks seem to roam there. However, the flow in Identi.ca groups appears to me like spam.
• One can mute Google Buzzers’ imported Twitter stream to avoid duplicity. Though, this can only be done via Gmail, not on their Google profiles.
• I unsubscribed from some high-volume news sites’ RSS. I subscribed to some specific low-volume research feeds (about machine learning & computer vision). I’m using Google Reader, btw.
• 2011: The year the check-in died. Although I still believe that smartphones become a ubiquitous interface for local information, I don’t gain advantages by looking at fellows’ check-ins. The GPS in my smartphone gets restricted to navigational tasks or simple place search.
• Hey, Google, forget “social” and build the next generation web—and hire a bunch of good designers!
• I abandoned Yahoo’s services already several months ago. Instead of Flickr I’m now showing off my photos on SmugMug. I won’t transfer my Delicious bookmarks to their new owner AVOS; btw, Diigo.com is the best tool I’ve ever used: It unites highlighting, annotating and bookmarking. I’m using it for 2½ years now.