Data Scientist Skill Set – Data Science Central

1         Background

Data science is first and foremost a talent-based discipline and capability. Platforms, tools and IT infrastructure play an important but secondary role. Nevertheless, software and technology companies around the globe spend significant amounts of money talking business managers into buying or licensing their products which often times results in unsatisfying outcomes that do not come close to realizing the full potential of data science.

Talent is key – but unfortunately very rare and hard to identify. If you are trying to hire a data scientist these days you are facing the serious risk of recruiting someone with the wrong or an insufficient skill set. On top of things, talent is even more crucial for small or medium-sized companies whose data science teams are likely to stay relatively small. Wasting one or two head counts on wrong profiles might render an entire team inefficient.

The demand for data scientists has risen dramatically in recent years [1, 2, 3, 4, 5]:

  • New technologies significantly improved our ability to manage and process data; including new data types of data as well as large quantities of data.
  • shift in mind set in business environments took place [6] regarding the utilization of data: from data as a reporting and business analytics necessity towards a valuable resource to enable smart decision making.
  • Last but not least exciting new intellectual developments
  • Last but not least exciting new intellectual developments have taken place in relevant related academic disciplines like machine learning [7, 8] or natural language processing.

Due to high demand, the term ‘data scientist’ developed into a recruiting buzz word which is broadly being abused these days. Experienced lead data scientists share a painful experience when trying to fill a vacant position: Out of a hundred applicants, typically only a handful matches the requirements to qualify for an interview. Some candidates feel already qualified to call themselves ‘data scientist’ after finishing a six-week online course on a statistical computing language. Unqualified individuals often times end up being hired by managers who themselves lack data science experience – leading to disappointments, frustration and an erosion of the term ‘data science’.

2         Who is a Data Scientist?

The data scientist skill set described in the following is based on the idea that it fundamentally rests on three pillars, each representing a skill set mostly orthogonal to the remaining two.

Following this idea, a solid data scientist needs to have the following three well-established skill sets:

  1. Technical skills,
  2. Analytical skills and
  3. Business skills.

Although technical skills are often times the focus of data science role descriptions, they represent only the basis of a data scientist’s skill set. Analytical skills are much harder to acquire (and to test) but represent the crucial core of a data scientist’s ability to solve business problems utilizing scientific approaches. Business skills enable a data scientist to thrive in corporate environments.

2.1        Technical skills | Basis

Technical skills are the basis of a data scientist’s skill set. They include coding skills in languages such as R or Python, the ability to handle various computational architectures, including different types of data bases and operating systems but also other skills such as parallel computing or high performance computing.

The ability to handle data is a necessity for data scientists. It includes data management, data consolidation, data cleansing and data modelling amongst others. As there is often times a high demand for these skills in corporate environments, it comes with the risk of focusing data scientists on data management tasks – thus distracting them from their actual work.

Almost more important than a candidate’s current technical skill set is their mind set. A key factor is intellectual agility providing candidates with the ability to adapt to new computational environments in a short amount of time. This includes learning new coding languages, dealing with new types of data bases or data structures or keeping up with current technological developments like moving from relational databases to object-analytical approaches.
A data scientist with a static technical skill set will not thrive for long as the discipline requires constant adaption and learning. Strong candidates show a healthy appetite for developing their technical skills. When a candidate focusses on a tool discussion during an interview it can be an indication of a narrow technical comfort zone with firm constraints.

Unfortunately, data science job profiles are often times narrowly focused on technical skills; caused by a) the misperception that a successful data scientist’s secret lies exclusively in the ability to handle a specific set of tools and b) a lack of knowledge on the hiring manager’s end as to what the right skill set looks like in the first place. Focusing on technical skills when evaluating candidates renders a significant risk.

2.2        Analytical skills | Core

Scientific problem solving is an essential part of data science. Analytical skills represent the ability to succceed at this complex and highly non-linear discipline. Establishing throrough analytical skills requires a high amount of commitment and dedication (which is a limiting factor contributing to the global shortage of data scientists).

Analytical skills include expertise in academic disciplines like computer science, machine learning, advanced statistics, probability theory, causal inference, artificial intelligence, feature extraction and others (including strong mathematical skills). The list can be extended almost infinetely [9, 10, 11] and has been subject to many debates.
Covering all potentially usefull analytical disciplines is a life-time achievement for any data scientist and not a requirement for a successful candidate. Rather, a data scientist needs to have a healthy mix of analytical skills to succeed. For instance, an expert on Markov chains and an expert on Bayesian networks might both be able to develop a solution for the very same business problem although utilizing their respective strengths and thus fundamentally different methods.

Analytical skills are typically beeing developed through pursuing excellence in a highly quantitative academic field such as computer science, theoretical physics, computational math or bioinformatics. These skills are trained in academic institutions through exposure to hard, unsolved research problems that require a high level of intellectual curiosity and dedication to tackle and eventually solve. This is typically done over the course of a PhD.

Mastering a quantitative research question that nobody else has solved before is a non-linear process inadvertedly accompanied by failing over and over again. However, this process of scientitic problem solving shapes the analytical mind and builds the expertise to later succeed in data science. It typically consists of iterative cycles of

  1. implementing and adapting an analytical approach
  2. applying it and observing it fail, then
  3. investigating the problems and
  4. building an understanding why it failed and where the limitations of the approach lie
  5. to come up with a better more refined approach.

These iterations are acompanied with key learnings and represent small steps towards the project goal thus effectively zig-zagging towards the final solution.

A key requirement for analytical excellence is the right mind set: A data scientist needs to have an intrinsic, high level of curiosity and a strong appetite for intellectual challenges. Data scientists need to be able to pick up new methods and mathematical techniques in a short amount of time to then apply them to a problem at hand – often times within the limited time frame of an ongoing project.

A good way to test analytical skills during an interview process is to provide potential candidates with a business problem and real data to then ask them to spend a few of hours working on it remotely. Discussing the code they wrote, the approach they chose, the solution they built and the insights they generated is a great way to evaluate their potential and at the same time provide the candidates with a first feeling for their potential new tasks.

2.3        Business Skills | Enablement

Business skills enable data scientists to thrive in a corporate environment.

It is important for data scientists to communicate effectively with business users utilizing business lingua and at the same time avoiding a shift towards a conversation that is too technical. Healthy data science projects start and end with the discussion of a business problem supported by a valid business case.

Data scientists need to have a good understanding of business processes as it will be required to make sure the solution they build can be integrated and ultimately consumed by the respective business users. Careful and smart change management almost always plays a role in data science projects as well. A solid portion of entrepreneurship and out-of-the-box thinking helps data scientists to consider business problems from new angles utilizing analytical methods that their business partners do not know about. Last but not least, many big and successful data science projects that ultimately lead to significant impact were achieved through ‘connecting the dots’ by data scientists who built up internal knowledge by working on different projects across departments and functions.

Candidates who come with strong technical and analytical skills are often times highly intelligent individuals looking for intellectual challenges. Even if they have no experience in an industry or in navigating a corporate environment, they can pick up required business skills in a short amount of time – given that they have a healthy appetite for solving business cases. Building strong analytical or technical skills takes orders of magnitude longer.

When trying to determine whether a candidate has an intrinsic interest in business questions or whether he or she would rather prefer to work in an academic setting, it can help to ask yourself the following questions:

  • How well can the candidate explain data science methods like deep learning to business users?
  • When discussing a business problem can the candidate communicate effectively in business terms while thinking about potential mathematical or technical approaches?
  • Will the business users collaborate with the data scientist in the future respect him or her as a partner at eye-level?
  • Would you feel comfortable sending the candidate on their own to present to your manager?
  • Do you think the candidate will succeed in your business environment?

3         Recruiting

Data science requires a mix of different skills. In the end, this mix needs to be adapted to the requirements and the situation at hand, and the business problems that represent the biggest potential value for your company. Big data for instance, is a strong buzz word but in many companies data is under-utilized to a degree that a data science team can focus on low hanging fruit for one or two years in the form of small and structured data sets and at the same time already have a strong business impact.

A key characteristic of candidates that has not been mentioned so far and which can be hard to evaluate is attitude. Hiring data scientists for business consultant positions will require a different mindset and attitude than hiring for integration into an analytics unit or even to supplement a business team.

4         References

[1] NY Times, Data Science: The Numbers of Our Lives by Claire Cain Miller
[2] TechCrunch: How To Stem The Global Shortage Of Data Scientists
[3] Bloomberg: Help Wanted: Black Belts in Data
[4] McKinsey on US opportunities for growth
[5] McKinsey on big data and data science
[6] Big Data at Work: Dispelling the Myths, Uncovering the Opportunities; Thomas H. Davenport; Harvard Business Review Press (2014)
[7] Andrew Ng on Deep Learning
[8] Andrew Ng on Deep Learning Applications
[9] Data scientist Venn diagram by Drew Conway
[10] Swami Chandrasekaran’s data scientist skill map:
[11] Forbes: The best machine learning engineers have these 9 traits in common.


소스: Data Scientist Skill Set – Data Science Central

전문가들이 말하는 2017년 빅데이터·분석 전망 15선 – CIO Korea


빅데이터와 분석 관련 기술은 소셜, 모바일, 클라우드와 더불어 디지털 시대의 변혁을 이끄는 주역으로 알려져 있다. 2016년 시장의 주인공이 BI 강화를 주도한 빅데이터 기술이었다면 2017년은 데이터, 분석 분야의 혁신에 주목해야 할 한 해가 될 것이다. 전문가들이 바라본 올해 빅데이터와 분석 관련 전망을 15가지로 정리했다. 맵알테크놀로지스(MapR Technologies)의 설립자이자 수석 경영자로 재직 중인 존 슐로이더 2017

소스: 전문가들이 말하는 2017년 빅데이터·분석 전망 15선 – CIO Korea

R이 왜 최고의 데이터 과학 언어일까요?


최근 R은 데이터 분석을 하는 많은 사람들에게 ​​매우 인기가 높습니다. 통계자료를 보면 지난 10 년간 가장 빠르게 성장하고있는 프로그래밍 언어 중 하나입니다. 사실, 데이터 과학을 시작한다면 여전히 권장하는 언어이자 매우 인기 있고 동급 최강의 데이터 언어입니다. 왜 R이 최근에 가장 좋은 데이터 과학 언어일까요?

R consistently ranks among the best languages
One thing I want you to understand is that right now, R is one of the most highly regarded, highly ranked, and fastest growing languages in existence.

In many ways, R is the data language. In data science, it’s the language to beat (with only 1 or 2 serious contenders).

To understand why this is true, let’s look at the results of several important surveys and programming language rankings to see where R shakes out.

IEEE: R ranks #5

The world’s “largest association of technical professionals,” the IEEE, has created a ranking of programming languages for several years.

This IEEE ranking system uses a set of 12 metrics, including things like Google search volume, Google trends, Twitter hits, Github repositories, Hacker News posts, and more.

Using this methodology, they rank several dozen programming languages and place them into several categories.

In their review of the “Top Programming Languages” of 2016, R climbed to #5.

The IEEE methodology is quite comprehensive, so this is a strong indicator of R’s strength compared to other languages, and the relative value of learning R.

TIOBE: R ranks high with consistent upward trend

Another ranking system, the TIOBE index, creates a similar score and rank for various programming languages.

If we look at R’s performance on the TIOBE index, we can see a solid upward trend for almost a decade.

Keep in mind that the TIOBE index is structured to be “an indicator of the popularity of programming languages. The index is updated once a month. The ratings are based on the number of skilled engineers world-wide, courses and third party vendors. Popular search engines such as Google, Bing, Yahoo!, Wikipedia, Amazon, YouTube and Baidu are used to calculate the ratings.”

For December 2016, R has an overall rank of 17 (among all programming languages). Its maximum rank was #12 in May of 2015.

This suggests that currently, learning R is still an excellent option if you want to learn data science. It may arguably be the best option. (To be clear, Python ranks higher on the TIOBE index, but it’s harder to separate out web and software dev uses of Python from the strictly data-related uses of Python, so it may not be an apples to apples comparison.)

Redmonk: R is #12

Another frequently sited language ranking system is the Redmonk Programming Language Rankings, which are derived from popularity on GitHub (lines of code) and popularity on Stack Overflow (number of tags).

As of November 2016, R ranks number 13 among all programming languages.

Moreover, R has shown a consistent upward trend for several years:

Out of all the back half of the Top 20 languages, R has shown the most consistent upwards movement over time. From its position of 17 back in 2012, it has made steady gains over time, but had seemed to stall at 13 having stuck there for three consecutive quarters. This time around, however, R took over #12 from Perl which in turn dropped to #13. There’s still an enormous amount of Perl in circulation, but the fact that the more specialized R has unseated the language once considered the glue of the web says as much about Perl as it does about R. Which is irrelevant to R advocates, of course. Whatever the cause, R’s relatively unique Top 20 path is one for fans of the language to cheer.

– RedMonk Programming Language Rankings: June 2016
(emphasis mine)

O’Reilly: R is arguably the most common data programming language

Finally, O’Reilly media has conducted a data science survey for the last several years, and they use the survey data to analyze data science trends. Among other things, they analyzed tool usage to identify which tools are most commonly used by data scientists.

In the 2016 survey report, R was the most common programming language (if we exclude SQL, which isn’t a programming language in the sense that I’m using it here). 57% of all respondents used R (compared to 54% using Python).

(As a side note, fully 70% of respondents used SQL. If you’re looking for another tool to learn after R, I’d suggest SQL.)

They also surveyed people to identify data visualization tools. They found that ggplot2 was the most common visualization tool. I’ll explain why I love ggplot2 in an upcoming blog post, but if we’re only tracking popularity, the O’Reilly survey suggests that ggplot2 is highly used (if not best in class).

R is excellent for learning data science
Beyond popularity, another reason that R is an excellent data science programming language is that it is excellent for learning data science.

R is a true “data language”

Part of the reason for this, is the nature of the language itself.

R was ultimately created with statistics and data in mind. The R-Project describes R as a “[programming] language and environment for statistical computing” (emphasis mine).

R is a language that has statistics and data built into its DNA, so to speak.

In this sense, R is nearly unique among programming languages. It is a language that has been built for statistics. It’s been designed for data.

This has advantages when you’re learning data science, because almost any statistical test or technique can be found somewhere within base R or one of its packages.

The best books and resources use R

Related to the fact that R is a “statistical computing” language is the fact that many of the best books and learning materials have adopted R as the language of choice.

This is important. If you’re a beginner, and you’re just getting started in data science, you’ll have a lot to learn. To truly master data science, you’ll need to learn several sub-areas like probability, statistics, data visualization, data manipulation, and machine learning. All of these skill areas have theoretical foundations (which you’ll need to learn) but also practical techniques that you’ll need to execute by writing code.

That means that:

You need a language that has strong capabilities in each of these areas (visualization, manipulation, machine learning (AKA statistical learning), etc)
You need a language for which there are high quality training materials in these skill areas.
While there are many data-related books and courses out there, but many of the best ones are centered on the R programming language.

Learn Probability with R

For example, two excellent books on probability use R for their “hands on” programming examples.

The first is Probability with Applications and R. This book is very approachable, readable, and well organized.

The second is Introduction to Probability which was developed from highly regarded statistics lectures at Harvard.

These are just two examples. If you dig deeper, you’ll find that among probability books that use a programming language, many (if not most) of them use R.

Learn frequentist statistics with R

The same can be said for statistics books.

Because R has statistics “built into its DNA,” many statistics textbooks use R as a learning tool.

For an introductory look at frequentist statistics, here’s one excellent book:

Statistics: an Introduction using R
Again, if you do a quick search on Amazon, and look at many intro stats books, you’ll find that if they use any programming language as a teaching tool, they are more likely to use R than almost any other language.

Learn Bayesian statistics with R

This becomes even more pronounced if you want a hands-on book for learning Bayesian statistics.

If you want to learn Bayesian stats and Bayesian analysis, nearly all of the books use R. There are some exceptions, like a few books that teach Bayesian analysis in C or Python, but overwhelmingly the best books that teach Bayesian statistics use R.

If you’re interested in Bayesian stats, check out these:

Introduction to Bayesian Statistics
Statistical Rethinking
Doing Bayesian Data Analysis
If you’re interested in Bayesian methods, these books are “best in class,” and they all use R.

Learn Data Visualization in R

When you’re learning data visualization, there’s a slightly larger range of programming languages to choose from, but I still maintain that most of the best learning materials use R.

If you’re learning data visualization, I highly recommend the work of Nathan Yau. His blog,, frequently has data visualization tutorials for the R programming language. (I also recommend his book Data Points as a companion, though it teaches principles as opposed to programing language syntax.)

I also highly recommend several books by Hadley Wickham. First, if you’re interested in data visualization in R, you need to own the book ggplot2. It not only teaches you the syntax of this critical R data visualization library, but it will also reshape how you think about visualizing your data.

I also recommend R for Data Science. This book provides a great introduction to data visualization, but additionally teaches you a broad set of data tools in R. It’s excellent, and a “must own” R book.

Learn machine learning with R

Finally, if you want to get started with machine learning, many of best machine learning books use R.

Although I will acknowledge that there’s more diversity among ML books with regard to their programming language, I still maintain that many of the best ones use R.

Here are two excellent introductions to machine learning that teach ML using the R programming language.

An Introduction to Statistical Learning
Applied Predictive Modeling
These books are both rigorous while still being approachable. They will teach you a little bit of theory (but not overwhelm you with math) while also showing you practical techniques.

Without question, these are the two books that I recommend most often for a beginner who wants to learn machine learning, and they both use R.

If you want to learn data science, R is excellent

Ultimately, the point here is that R is an excellent language for learning data science, because many of the best books (and other training materials) use R as the programming language of choice.

So if you’re a beginner in data science, I think that R is the best language – in large part – because of the quantity and quality of data-science learning materials.

A quick note on Python
There are other options, but the only one I’ll address here is Python.

As far as data science programming languages go, Python is the only serious alternative to R right now. (Other alternatives lack a well-developed package ecosystem or are not free/open source.)

I won’t explain my full thoughts on Python here, but I will say that it’s an excellent language. I love Python.

Having said that, for data science beginners, I still think that R is a slightly better choice, largely for the reasons I outlined above.

Again, I think that many of the best textbooks and training materials for foundational data science concepts (probability, statistics, Bayesian statistics, machine learning) are R-based books. That’s not to say that there aren’t excellent data science books that use Python, but I still think that there is a higher average quality among the R-based texts.

The other issue with Python is that many students get caught up in software development. That is, instead of learning statistics, data visualization, data manipulation, probability, etc, they end up spending their time learning about data structures, loops, flow-control, object oriented programming, and web frameworks. These skill areas can complement the core data science toolkit, but they are not data science topics in the sense that I’m using the term here. In fact, I recommend that most beginners learn software development contepts after learning basic data science subjects like data manipulation, visualization, analysis, etc.

Even though most beginners should learn software development principles later, many beginners who start with Python get sidetracked into these software development and web development areas. I think this happens, because in many ways, Python is geared towards these subjects. Most books on Python are not really data science books per se, but instead books on programming, development, etc. So a beginning data science student opens up a Python book intending to learn data science, but they end up going down the software/web development rabbit hole, and don’t come out for a few months (or years).

As much as I love Python, I think this is a risk for beginners. I think it’s better to start with R as it has statistics and data science more “built into its DNA.” With R, it’s easier to learn the foundations, and harder to get sidetracked.

Recap: Learn R if you want to learn data science
What you should take away is that for learning data science, R is arguably the best option. In terms of popularity, R is very highly ranked, and on an upward trajectory. Moreover, many of the best data science books and training materials use R.

If you want to get started learning data science, I recommend the following:

Learn R
Specifically, learn ggplot2, dplyr, tidyr, lubridate, and other Hadleyverse tidyverse tools for data visualization and manipulation
Learn to use these tools together to analyze data
Once you have some background in these essential R packages, bulk up on probability, stats, and machine learning (I recommend the texts that I talk about in this blog post)
Discover how to master R
Do you want to rapidly master R?

Sign up for the email list at Sharp Sight.

Our posts are devoted to helping you rapidly master R, one of the best programming languages in the world, and possibly the best data science language you can learn.

In last week’s blog, I explained why you should Master R (even if it may eventually become obsolete). I wrote that article to address people who claim mastering R is a bit of a waste of time (because it will eventually become obsolete). But when I suggested that R may eventually become obsolete, this seemed […]

소스: Why R is the best data science language to learn today – SHARP SIGHT LABS

클라우드 리포팅 도구 Google DataStudio 소개


구글 클라우드의 리포팅 도구인 DataStudio가 무료화 되었습니다.

빅쿼리, MySQL, YouTube등 다양한 데이타를 시각화할 수 있는 리포팅 도구 입니다. 아래 링크를 참조해 보시지요


소스: Easily Build Custom Reports and Dashboards – Google Data Studio – Google