David Asboth - Data Solutions & Consultancy

2025 Year in Review

2025-12-31T15:00:00+00:00

How long is a year?

My sense of time is variable at best. When it comes to planning or teaching a lesson/workshop, I can pretty accurately judge how long something is going to take. On the other hand, I'm always surprised by how long ago something was. Calendar years are somewhat arbitrary, but I thought I'd find it helpful to recap everything that's happened this year to remind myself that although a year can feel like it's whizzed by, there's actually a lot you can fit into it. Also, my impostor syndrome probably needs to hear all the things I've done.

Book

First of all, my book The Well-Grounded Data Analyst was published!

I still can't believe there is an actual physical book out in the world that I've written. It took me two years to write, only an entire year longer than I initially predicted, and I'm really proud of it. It feels great to have followed through on something of this magnitude and to get my vision out there. I'm particularly happy that I don't feel I had to compromise - this is pretty much the book I set out to write. I'm way overdue writing a retrospective about it - on my list for 2026.

If you like solving data problems and want to get better by practising on true-to-life problems, I think you'll like it too!

If you want to know more, start here: https://davidasboth.com/#book.

If you want to hear me talking more about it, my friend and podcast cohost Shaun McGirr also interviewed me about it on our podcast: https://halfstackdatascience.com/s3-finale-the-well-grounded-data-analyst-with-david-asboth

That brings me on to the podcast.

Podcast

Perhaps the reason you're even reading this is that you've heard our podcast Half Stack Data Science. It's also a project that is exactly as I envisioned it back in the day. We post sporadically partly because we only want to bother with a new episode if it's something we believe in. You may not get weekly episodes from us, but you'll always get us.

This year saw us record five episodes but boy were they something. I got to chat to amazing people like Simon Willison, a fantastic presence in the tech community, or Kirsten Hill who's empowered by AI technology to build bisque, The Thing of her dreams. And best of all, junior developer David back over 10 years ago wouldn't have believed it, but we got to sit down with actual Scott Hanselman. They say "never meet your heroes" to which I say "pick better heroes", because Scott was funny, engaging, and generous with his time. In early episodes of the podcast we talked about some of his work, and to have him sit down with us for an hour was an honour.

We have some ideas for 2026 so expect to hear from us again at some point.

Out and about in the data community

I've been around in the online data community, for better or worse (more on that shortly), but the highlights from my "social calendar" for the year were hanging out with people on Discord and giving a couple of talks.

Talks

I got to give a talk at PyCon UK in Manchester and it was a fantastic experience. Hopefully next time I'll be able to spend more than a single day at the conference. You can read my write-up of it here: https://davidasboth.com/pycon-uk-2025-roundup
I also gave a talk at Randy Au's DataBS Conf, which was a really fun one-day conference that reminded me of the legendary NormConf. The data community might not be as vibrant and easy to find as at the height of Data Twitter, but it's there and the people are still awesome.

You can see both of my talks in my YouTube playlist.

Streaming

I've also tentatively started my YouTube channel and have streamed a bit more on Twitch. I like the idea of "working out loud" so expect to see me streaming more, but still sporadically, in 2026.

Discord

Earlier this year, I started a Discord server for fellow data project enthusiasts. If you like solving data problems, come and hang out with us!

I also posted a "data challenge", a small project you can attempt yourself, and livestreamed my own attempt at a solution. I still need to write that up - something for me to do early 2026.

Client work

In the background, I've been designing and delivering Python and data workshops and courses. One goal for 2026 is to talk more about my work (my brain immediately went "ugh, marketing") but one thing I do want to say now is that I'm starting to see the shift towards AI already. People still want courses in things like Python or Tableau, but more and more clients are asking for AI courses for anyone from software developers to senior execs.

This finds me at something of a crossroads...

Personal

This year I've had waves of feeling like the entire tech industry is whizzing past me. I need to remind myself that AI hasn't taken over the world, ChatGPT hasn't replaced all the data professionals, and that I should spend less time on LinkedIn. That's only partly a joke, I'm really starting to feel like spending time on these platforms is a net negative.

Social media

I said earlier I was active in the data community "for better or worse". The "worse" part is the FOMO I just mentioned.

In the new year, I will try to keep places like LinkedIn and Bluesky as "output only". I'll be there, probably check in most days, but won't spend too much time scrolling but instead signpost people to things I'm doing elsewhere. Blogs I write, videos I post, that sort of thing.

You can find me on Discord though. I'm fairly active on Randy's Discord as well as my own.

For me, 2026 will be about figuring out where I want to fit in in this brave new world of AI. It'll probably be something about doubling down on being human.

Play

This year I'm really rediscovering the value of play. I'm happiest when I have a fun puzzle to solve or a strategic board game to dig into. At work, the most fun things to work on and teach are projects. I suppose ideally everything at work is part of a "project", but what I mean is when you're dropped into a situation and need to come up with a solution given the constraints. And crucially something that doesn't drag on for months. That's always been my preferred type of work as well as my preferred way to teach data skills so I'll be continuing down that path.

Fun, playful projects are coming to a YouTube channel/Twitch stream/Discord server near you.

Reading

I rediscovered a love of reading as an adult and I've probably read more this year than ever before. It's almost impossible to even pick favourites, but here are a few highlights.

Fiction

My go to genre is usually sci-fi, but I read a lot of random stuff this year including:

The Square of Sevens by Laura Shepherd-Robinson: I randomly found this in a charity shop and was vaguely intrigued by the blurb but it was one of my favourite books of 2025. I don't like reading too much about a book before I read it, so I'll say no more, but it's worth your time.
The Dark Forest by Cixin Liu: the second, and in my opinion best, book in the Remembrance of Earth's Past trilogy. I was pleased the TV adaptation for the Three-Body Problem turned out great.
Sleeping Giants by Sylvain Neuvel: a really easy to read, exciting sci-fi trilogy. The storytelling mechanism is clever and really adds to the pacing of all the books.
Children of Time by Adrian Tchaikovsky: this was something of a mind-bender in a good way. The sequel was good too, but I loved this first one. I hope you like spiders.
Sea of Tranquility by Emily St. John Mandel: hard to describe other than it's a beautifully human sci-fi story. Can't wait to read more of her work.
Piranesi by Susanna Clarke: I added this to my "to read" list, borrowed it from a friend months later having forgotten what it's even about, and absolutely couldn't put it down. One of my favourites of the year.

I'm at risk of listing all the great fiction I read this year so I'll stop there.

Nonfiction

I actually find reading nonfiction quite hard, possibly because most nonfiction books are ~200 pages too long. Having said that, here are some good ones I read this year:

The Game Changers by Tim Clare: a history of board games, what's not to love? Both this and Tim's other book "Coward" are well-researched, engaging, and come from a suitably random mind that they keep me interested. Bonus fun point that I went to school with one of the experts interviewed in "Coward".
Abroad in Japan by Chris Broad: I wasn't aware of the book or Chris's YouTube channel, but I've been to, and loved, Japan and this was a really fun read about someone's experience of moving there.
Why Can't I Just Enjoy Things? A Comedian's Guide to Autism by Pierre Novellie: probably my favourite nonfiction book of the year. It's funny, well-researched, and very personal.

I already have a pile of books I'll be digging into in the new year. Also, recommend me things!

What's next for 2026

I'll finish by saying that writing this has been therapeutic for me. So much has happened this year, lots of it good, and here is another timely reminder to myself, and perhaps to you, that you can fit a lot into 365 days.

Next year I plan to follow my interests more aggressively, listen to myself about what sort of things I want to do more and less of, and exponentially increase the amount of true human connection in my life. That means less time mindlessly spent scrolling algorithmic feeds and more time connecting with people. People like you!

Here's to a fun and lovely 2026!

PyCon UK 2025 Roundup

2025-09-26T21:00:00+01:00

I recently attended my first dedicated Python conference - PyCon UK in Manchester. I'd been to, and given talks at, data-themed conferences before but this was my first Python conference. I had a talk accepted and I was excited about the opportunity to go. I thought I'd give a short summary of my experience in case you've been thinking of attending something similar and aren't sure what to expect.

On the day

I could only stay for the first day unfortunately but I crammed in a bunch of talks, gave my own, and met many interesting people. The first day started at 11am to allow for people to arrive to the city but subsequent days started earlier.

The day started with a keynote by Hynek Schlawack - a whistle stop tour of Python's growing pains and its superpowers. The part where Hynek described the enduring appeal of Python's rapid feedback loop really resonated with me. Out of all the programming languages I've tried, I find Python to be the most fun language to code in.

Then there was a well-organised and varied lunch followed by a couple of blocks of talks spread across different rooms. Mine was the penultimate talk on the main stage. Every day ends with an hour of 5-minute lightning talks, which you could pitch on the day by filling in index cards from which the organisers chose the schedule. It was a fun, rapid-fire series of different talks with which to end the day.

A conference for everyone

The conference felt like it went smoothly throughout and I was particularly impressed with how thoughtful the organisers were. There were stickers to indicate whether you're OK with people coming up to you to chat and whether this is your first PyCon, so people could be extra welcoming. This was just one of the aspects that made the event really inclusive. There were toilets for all genders, detailed dietary information, and a quiet room if you got overwhelmed. Every talk was captioned live by a dedicated team and they did their best to gather speakers' presentations ahead of time to better prepare. They even had a free(!!!) crèche for parents who couldn't otherwise organise childcare, which I thought was an amazing touch.

This is clearly a friendly and welcoming community. I didn't know anyone but whoever I spoke to made me feel like I'd always been part of it. Every day there were "official" social activities you could attend, such as a play or a board game evening.

Having an active community on Discord really brought the conference to life. People organised social activities ranging from walking tours of Manchester to breakfast or dinner together.

What if you're a speaker?

There was a lot of information to help speakers prepare. We could ask detailed logistical questions on Discord even on the day. There was a really good variety of talks so whatever your idea, I'd recommend pitching it! We heard from a range of speakers from maintainers of popular Python libraries to people getting over long COVID.

My talk was about Python's exec function that lets you run Python code dynamically at runtime. Amazingly, they uploaded the talk to YouTube within hours. You can watch it here:

Should you attend?

Definitely! I couldn't recommend it enough and I'm really looking forward to the next one.

The gardener's guide to data analysis

2024-04-28T17:00:00+01:00

In her book The Gardener and the Carpenter, Alison Gopnik talks about how parents should think of raising children as gardening, not carpentry. The difference is that in carpentry you are aiming for a specific outcome from the start. Every decision you make takes you towards shaping your child into a version of themselves that you preconceive. In contrast, in the gardening model, your role is to provide a safe and loving environment and let children flourish and grow in their own way. There are a lot of parallels to how to continue learning analyst skills.

Gardening vs. carpentry

I love the gardening metaphor for improving as an analyst.

The continued learning process makes more sense as activities to cultivate your skills rather than aiming for a targeted outcome, especially in an industry of constant change. The technologies you thought you needed five years ago are not the same ones you need now. Instead of spending your learning time on the specifics of any technology, most of it should be spent honing the lasting skills.

Gardening in this context means that once you have done some foundational training in data analysis and have the necessary skills to work on real problems, you should immediately start working on real problems. Rather than narrowly focusing on learning more technical skills, you should adopt the gardening model and have faith that by working on realistic scenarios, you will learn the skills you actually need.

There is a time and place for the carpentry model; sometimes, you just need to learn something, in which case your focus should be dedicated to that thing. In general, though, the gardening model is what optimises your learning towards a marketable skill set.

How to tend to your data garden

How do you actually adopt a gardening model of learning? The key, as I have alluded to, is working on real projects. Stop learning libraries, algorithms, and syntax for the sake of it and switch to a problem-first approach. This means solving problems that people actually have, even if that's just yourself, and trusting that you can and will pick up the necessary skills along the way.

It also means ditching toy examples and tutorials for finding projects to work on. Solving another version of "predicting the survivors of the Titanic" will not get you closer to your goals. A crucial part of an analyst's skill set is defining a problem, so that's the first skill to work on. Find a question that can be answered or a problem that can be solved with data. It does not have to be "business relevant" either, and the solution does not need to be complicated. Sometimes questions can be usefully answered with a single bar chart. As an example, here are some questions I have asked in my daily life, all of which can be (and have been) answered with data:

Which key signatures should I practise if I want to get better at learning jazz piano but also optimise for key signatures that actually occur in jazz? Based on my performance playing online chess on chess.com, which openings should I review to target my specific weaknesses? I wonder which pub name is the most common in the UK?

As you can see, the problems that you work on do not have to be huge in scope. Each of these small projects taught me something beyond the answer to the question: a new piece of syntax, a new Python library, or data sources I didn't know existed.

Gardening is a long-term project

Just like gardening, improving as an analyst means constantly turning up and working on cultivating the foundations. It is a long-term process that requires consistency and trust. You will never be "done", but you are always improving. One side effect of doing lots of gardening beyond what grows in the garden is you get better at gardening itself. The side effect of solving lots of problems beyond finding solutions to the problems themselves is that you get better at solving problems. You are also learning how to learn things on the fly to solve a particular problem. Problem-solving and learning how to learn are the skills that will outlast any technology.

Need more real-world inspiration? My book, The Well-Grounded Data Analyst, is a collection of projects for you to hone your skills. They are specifically designed to help you learn industry-relevant topics, such as data modelling or working with categorical data, which are not typically part of foundational training.

Learn data science like you would learn jazz

2023-06-17T19:00:00+01:00

Aspiring analysts want to know how data analysis is done. Beyond the technical skills required to be an analyst, they assume that, surely, learning the discipline of data analysis includes some sort of repeatable process; a step-by-step method to apply to any analytical problem. Such a method doesn't exist. How seasoned analysts do their work is a mixture of technical and interpersonal skills sprinkled with intuition built up through years of just analysing data. This kind of intuition is hard to put down on paper, which for a student is an unsatisfying thing to hear.

I'm a professional data analyst and educator, and an amateur jazz enthusiast. Hear me out: as it turns out, there are similarities between learning how to analyse data and learning how to play music, especially jazz.

Improvisation

Jazz has multiple components that make it distinct from other musical genres. What makes jazz unique is that it includes improvisation, where musicians can put their own creative stamp on a piece. Because of this, when you hear a performance of a jazz "standard", a popular song that's part of the jazz canon, you never hear the same performance twice. The individual notes aren't as important as the general structure. Jazz musicians don't get given an exact score like classical musicians. In a classical piece, every note is mapped out exactly as the composer intended, and musicians tend to follow this as closely as possible. In jazz, you often just get what's called a lead sheet, which contains a song's main melody, its structure, and its chords. However, even this loose structure is fungible, and jazz musicians take many liberties to make a piece their own. This has parallels for data analysis.

Two analysts attempting the same problem will arrive at different results. The problem, the "analytical lead sheet" if you will, is the same, but because their methods, assumptions, and choices will be different, they end up playing a different tune. There is enough uncertainty in data analysis that this doesn't mean either one is wrong. In fact, the two analysts would likely learn something by comparing notes afterwards.

Just like jazz musicians don't look back and check whether every note they played was perfect, aspiring analysts shouldn't get bogged down with whether every individual piece of their analysis was the perfect choice.

Chances are there were other, equally acceptable, options at multiple points in the analysis.

How does one get to a point where they can confidently improvise? Music and data education are both historically very method-driven. They are focused on drilling the fundamentals before moving on to practical applications. However, because jazz is so different from other musical genres, the best way to learn it is also different and the same should be true of data analysis.

Songs before scales

Rigorous music theory lessons will no doubt make you a better jazz musician. Knowing how notes, scales, and chords interrelate builds a foundation you can use to improve. However, knowing theory doesn't make you better at the practice, in and of itself. You need two things: to apply your knowledge by playing lots of jazz and to learn from the greats. The first is obviously accomplished by sitting down at your instrument and practising. There are no shortcuts here, but one important point is that you want to practise playing songs. Anyone who's ever had lessons in an instrument has had to practice scales and arpeggios. Music lessons typically include these technical practice challenges to hone your skill at the instrument. These are clearly important, but they should not override the purpose of the lessons, which is to make music. If you focus on songs first, learning scales makes more sense since you have the right context for them.

Just like songs should be the first-class citizen of jazz practice, projects should be the first-class citizen of learning data analysis. Once you have the basics under your belt, that is you know how to load, clean, explore, and visualise a dataset, you shouldn't go and learn 10 more tools or algorithms for the sake of it.

More technical training will not give you the best return on investment on your learning time.

It is much better to start solving problems as soon as possible, even if you don't have all the necessary skills yet. Attempting an actual data analysis problem will force you to learn exactly what you need, and you will have the right context for those technical components, which you can always learn when you need them.

There is an important caveat here. You shouldn't learn songs or complete analytical projects in isolation. The way to improve is to do the work, get feedback, and see how the experts do it.

Learning from experts

To figure out how you can get closer to the greatest jazz artists is by listening to and analysing how they play. Rather than guessing which notes you could have played differently to make your solo sound a bit better, it's more effective to listen to a recording of Miles Davis, John Coltrane, or Duke Ellington, and hear what they did. One piece of advice given to aspiring jazz musicians is to try to transcribe your favourite solos. The point of attempting to figure out every note that was played isn't to replicate the solo exactly but to see a master at work and pick up some new tricks. Why couldn't we do the same for data analysis?

When teaching programming, a great tool for the educator is the code-along. Students get a lot of value from an expert instructor talking through every line of code not just to explain the syntax, but to talk about the wider context. On the surface level, the instructor explains the exact command used to remove missing values, but on a deeper level, they reveal the justification for doing so in the first place. This sort of narrated, granular deep dive is incredibly powerful at filling in the gaps in students' knowledge.

Learning from experts by actually seeing them work is underutilised in tech education.

To put my money where my mouth is, I have in the past streamed my attempts at an analytical problem on Twitch, and plan to start doing more of this again. I find that narrating my own thought process helps me learn a lot, too.

Jazz up your data analysis learning

If you want to get better at analysing data, think like a jazz musician. Don't spend long hours reading dry textbooks of yet another analytical tool or algorithm. Play songs instead. Improvise to put your own stamp on a problem, observe the experts, and, most importantly, just make some sweet, sweet data jazz.

How generalists learn

2022-11-22T13:54:00+00:00

One of the most common questions I get from students near the end of a data science course is "what next?". The subtext is often "what is the next algorithm I should study that we didn't cover in this course?". Even a short data science course will reveal the tremendous breadth of topics that fall under the data science umbrella. Students, understandably, think all those topics should be ticked off before they're ready to be data scientists. I've written about this feeling before; it's impostor syndrome in disguise. In "How to use your impostor syndrome to learn anything" I detail the most common piece of advice I give to students:

[you] don't need to learn everything up front, [you] just need to be confident enough that [you] can learn anything when [you] need to

My advice has always been that once you have a foundation, an understanding of how data science and machine learning work, you have enough context to go and learn anything else, and the best way to learn is by doing. This ticks two boxes that I think are important:

Machine learning should be taught intuition first, i.e. process before equations
Learning data science/machine learning should be realistic, i.e. mimic the real world as closely as possible (real data rather than toy data)

Build, build, build

Vicki Boykis talks about how she keeps up to date with her field in "How I learn machine learning" and she shares one of my conclusions:

I cannot emphasize how important it has been for me to build things when I'm learning.

I've never really understood why I struggled to follow a textbook from start to finish. What I now realise is I lacked a goal. Learning algorithm X or method Y for its own sake isn't an interesting enough goal. Solving a problem, answering a question (however trivial) - those are.

Ergest Xheblati in his Data Patterns newsletter explicitly recommends this approach to maximise your value as a data professional, via what he calls "unstructured learning". In "How to build a lucrative career in data" he says:

Experience is a more direct way to demonstrate that you can create value. One way you can get that experience is through unstructured learning by solving real world problems.

Breadth-first

Another key piece of advice Vicki gives is "make sure to learn broadly". You often hear about data science being T-shaped, the idea being that you have a breadth of understanding about lots of topics, and only go deep in one or two.

As these disparate pieces came together in my mind, I realised all this advice is tailored to my generalist mindset. This is not just how I learn, or how I think students can learn effectively; this is how generalists should learn.

As a generalist, going deep in a topic for the sake of it has less value than solving a bunch of problems and learning the required skills on the fly. Doing the latter actually strengthens the skill that is your greatest asset. One of the unique selling points of a generalist is that they know a little bit about everything, and can use that knowledge to go deeper as required. I've heard this be called "just in time" learning, rather than "just in case" and it's the generalist's secret weapon.

Assuming Twitter still exists when you read this, and the thread is still up, read what Ergest has to say about building a "career moat"; a collection of good-enough skills that, when combined, provide you with a unique skill set: the Twitter thread. In case that link stops working, here's what I mean:

My mission and purpose is to help data professionals advance their careers by building a career moat and the best way to do that is by stacking skills. A career moat (as defined by @ejames_c) is a set of rare & valuable skills that allow you to remain competitive in any field.

Specialist roles ("we need a forecasting expert") require deep specialist knowledge, but if you apply for a job that requires a problem solver, breadth beats depth every time.

In his article "Staying Sharp in Data Science" Randy Au advocates for even more radically broad learning:

I am saying with a straight face, to go out and learn anything that strikes your interest that you can even tangentially relate to your work and interests and then try to apply it to your day job

Within reason, to a generalist all knowledge is useful knowledge.

The generalist's formula

What I want you to take away from this is that if, like me, you're a generalist it's helpful to realise that you probably learn differently from other people, and if you want to lean into your generalism, you should ditch the textbooks and:

Read lots of stuff, even it only seems vaguely relevant. Knowing what there is to know is valuable enough without going too deep too soon.
Build stuff, hone your building skills, and learn by doing. This "stuff" could be something with actual business value, or just a pet project. Personally, the former motivates me less than the latter. I even occasionally livecode some silly idea for an analysis on my Twitch channel

If you free yourself from a more structured form of learning, you will strengthen your generalist skills, and hopefully have a lot more fun along the way.

Why I now call myself a "data generalist"

2022-11-21T13:54:00+00:00

I'm at an interesting crossroads in my career.

If anyone asks me what I do, I say I'm a data scientist. That term is too broad, however; a few minutes on any job site will reveal that "data scientist" can mean someone with a PhD in molecular biology just as much as someone who builds robust data pipelines, all the way to someone who can use Excel quite well, and everything in between. We're fast approaching peak data science where everyone wants to be a data scientist.

This presents a problem.

If I keep saying I'm a data scientist, looking for work is a long and painful slog reading every single job description hoping that something suits me. As time goes on, I'm finding myself less and less excited about advertised data science roles. Something's been lost in the name. If I explicitly move away from calling myself that though, I miss out on a huge chunk of the market.

I'm sure I'm not alone in this. Data science attracts people from incredibly diverse backgrounds. There's no medical school for data science; we all have our own skills that we bring to the job. It shouldn't be a surprise if "data scientist" is therefore too broad to summarise us all.

Time to work backwards. Start with my skill set and what I actually enjoy doing, settle on a specific job title later. Here's what I've done in my career to date:

I've been paid to write code for over a decade. I can't imagine a future where I don't spend at least some percentage of my time writing code, but I also know I don't want to be a developer. I enjoy working with data, and people, and thinking about the big picture too much (not that developers don't do these, but it's not as typical).
I've got over 5 years experience analysing, visualising, and presenting data, and building machine learning models. I love the former, ambivalent about the latter. I'm not an expert in statistics and I know I wouldn't want to specialise as a Machine Learning Engineer for example.
I've also spent the last few years focused on designing and delivering data science/programming courses. Education is one of my real passions, maybe what I'm best at, but I am yearning to support this with technical work to keep my skills fresh and my life interesting.
I like building things. Specifically, quick proofs of concept and prototypes to help people decide whether something is a good idea or not, before they go too far down a path that delivers no value. This has been my favourite part of being a data scientist so far, even though it's not officially in the remit. I've "won" multiple company hackathons doing this, and I'd love to explore this skill further in my career.

This is my personal list, but all data scientists will have something similar. Maybe you have a specific bullet point for data modelling, which you've done and enjoyed, maybe it's the project management aspect that appeals to you. Perhaps you enjoy being the "translator" between technical teams. What we all have in common is a variety of past roles and technical/professional skills, a lot of which we learned on the job, and a desire to work with people and help them achieve their goals. We don't want to be fully technical and lock ourselves away from people, nor do we want to be in a purely non-technical role. Whatever axes you place jobs on, we're somewhere in the middle, and "data scientist" doesn't quite cut it.

We're generalists. Maybe that's it; we're data generalists.

As Alan Hylands points out in this great post "Generalists Are The Real Data Science Unicorns":

The further you go in your career, the likelihood will be that you will eventually gravitate more towards [becoming a specialist]. There will be an inevitable attraction (and aptitude) towards one aspect of the profession over the others.

The prevailing wisdom is that of specialisation. Job adverts expect to find specialists, bootcamps train specialists, and most discussion in the datasphere will be filed under "data science" or "data engineering" or "machine learning engineering" etc. Specialists are important. Often you want someone with deep expertise in one field. However, as we have talked about on the Half Stack Data Science podcast, data science in most organisations is messy and undefined, and that arena calls for generalists. Someone who can use their wide experience to solve problems on the fly, without necessarily being an expert in the tool they're using to solve the problem.

After years of struggling to "find my niche" I'm now fully embracing the idea of being a T-shaped problem solver. A generalist. If what I've written here resonates with you, maybe you're a generalist too, and I'm here to tell you you're not alone, and you have a tribe. I have more to say on the practical side of being a generalist, but in the meantime here are some resources to get started:

The marvellous and aptly named Majoring in Everything podcast by Andrea Jones-Rooy. "A show for people who don't know what they want to do with their lives to explore the variety of kinds of lives out there with cool, smart people." - a generalist's dream.
The post "Generalists Are The Real Data Science Unicorns": https://alanhylands.com/generalists-real-data-science-unicorns
The Counting Stuff newsletter by self-proclaimed "die-hard generalist" Randy Au

PS: if "data generalist" takes off, remember where you read it first :-)

Citizen Data Science

2022-11-08T13:54:00+00:00

Citizen Data Science

This was a popular phrase a few years back. At the peak of the data science hype cycle we somehow managed to go beyond "everyone needs data science" and start convincing ourselves that, in fact, everyone needs to be a data scientist. There was also a lot of pushback on this. After all, why should Debbie from Finance learn Python? Why should our salesmen be able to tell the difference between supervised and unsupervised learning? Clearly this was one step too far, or so we thought. At the time something bugged me about this particular debate, but I wasn't sure what. I agreed that data science should largely be restricted to its own function, not "democratised" just because it was becoming popular. My motivation wasn't gatekeeping, but a separation of concern.

What I've since realised though is that some of the skills necessary for data science, namely the ones that make a data scientist a good analyst, are actually useful across the board. Much like how computer literacy is a necessary part of life in the 21st century, knowing a few data science fundamentals will only become more important. Specifically, there are two areas where I believe widespread education would be helpful: data literacy and an automation mindset.

Data literacy

What is data literacy exactly? There need not be a societal push to teach everyone confidence intervals and probability theory. You can be data literate without linear algebra. What most people need is an intuition to always be sceptical about data-driven findings, question every result, and put it in the wider context. Does this chart show what it claims to show? Do these reported numbers make sense, and if they seem plausible, is the process that generated them trustworthy and repeatable? Are there truly fewer Covid-19 cases over the weekend, or is it just that numbers aren't updated and reported until Monday?

All we need to do is let people flex this particular muscle. Discuss case studies, ones that contain realistic ambiguity. Most business decisions are made under uncertainty, regardless of the quality and quantity of data analysis. We need to better understand and embrace this. Data will not tell us what to do, it can only give us suggestions, and we need to be equipped to make the most of these. I argue this isn't a skill that only the biggest decision makers, the C-level execs need, this is something with which we should all get comfortable.

So that's fine, our "citizens" don't need mathematical training, just an intuition for how to use data in the right way. But what about technical skills? How much programming for example do people need? Clearly not everyone needs to know how to train and deploy a machine learning model (although exposure to this process can't hurt, if only to help demystify it) but we also can't ignore the need for some technical training that people can genuinely benefit from. The balance is struck, in my opinion, when we focus on teaching programming for automation.

Automation

Automation is a superpower. If you have any sort of repetitive task that is part of your day-to-day, business as usual work, imagine how great it would be if the computer could do it for you. No, it wouldn't result in the computers taking away your job. I know people genuinely fear that, I get it. The right way to think about it is that automating the boring stuff (see the book/online course of the same name) frees you, the human, up to do the things that computers can't. Build human relationships. Consider nuance. See the bigger picture.

Students of mine have already heard me on my soapbox about this. Learn just enough programming (Python or otherwise) to automate the boring stuff. The more practice you have with this skill, the more you will see it everywhere, and the smaller your tolerance will be for manually doing repetitive tasks. You should get annoyed that a stakeholder demands a particular report emailed to them every Monday. Find a way to take yourself out of the loop. Got a bunch of spreadsheets to download from different web pages and collate once a week? Write a script for it.

So what do we call this thing?

Let's not call it "citizen data science", that sounds patronising to me. It probably doesn't even need a name (but then how will it ever go viral, David??). After the somewhat misguided idea of citizen data science came and went we're left with some key takeaways. There is a reason to take elements of data science beyond the data science team. We don't want any gatekeeping ("only data scientists should analyse data") and everyone benefits from data literacy and some basic coding knowledge.

How to use your impostor syndrome to learn anything

2022-11-08T13:54:00+00:00

How to use your impostor syndrome to learn anything

I have a theory that data science is a field where impostor syndrome is particularly rife, more so than other fields. There is so much to know under the wide umbrella of "data science" that no single human can know it all, yet we all go around calling ourselves data scientists, so the implication is that we should know it all. The more you learn about what's out there, the easier it is to be overwhelmed that you couldn't possibly ever learn everything, and from there it's not a huge leap to "I will never be good enough". You might see a tweet or read a paper from a fellow data scientist working on a different part of the field and think it might as well be in a foreign language and your heart might sink that there's yet another thing you don't know yet. The truth is that it's perfectly normal and fine for people to specialise and therefore have different strengths, but we can't all see this clearly all the time, hence impostor syndrome proliferating among us.

There are two ways to tackle impostor syndrome: either learn everything and genuinely don't be an impostor, or embrace the fact that you can't know everything at once, and instead get good at being able to learn anything. Clearly only one of those things works, and it's something of a slow-burning epiphany I've had over the years; I don't need to learn everything up front, I just need to be confident enough that I can learn anything when I need to.

How to learn anything

Alongside tackling impostor syndrome I've got my own reasons to want to know how to learn anything. As an educator I need to know how people learn, so I can teach them. As a data scientist, a career that itself requires a lifelong learning journey, I need to be able to figure out new concepts and algorithms pretty fast. As a consultant I also need to be able to teach myself new skills as they come up. When I was a software developer, I never wanted to pigeonhole myself as a "technology X developer", I'd have happily changed programming languages between jobs. Now as a data scientist, I don't even have a choice; every project is just so different. By this point I hopefully don't need to convince you that being able to learn something is a useful skill, but students often ask me how to "learn data science". That question itself is fraught with many rabbit holes, so rather than answering "well it depends on what you mean by data science" I usually give one piece of advice: learn by doing.

"David, that's so obvious, we hear that all the time" I hear you say. At least I hope some of you hear that advice every now and again. Obviously it makes sense to actually do the thing you're learning. Science even backs me up there - we retain more information if we apply our knowledge to a real life scenario. Not to digress here, but a big problem with the education system in many countries has always been too much rote memorisation, not enough application. This is also true of a lot of technical education, and especially counterproductive in data science, which is all about the application. Theories and equations will help you understand only a part of what makes data science a useful discipline. It took me years to learn this and even more years to apply it to how I approach the field. By now I've recognised in myself that the best way I learn something is to have a problem to focus on.

I used to hoard ebooks and links to free online courses with the (decidedly false) hope that one day I'd "get around to them". No points for guessing how many times I've sat down at my desk and thought "I should read one of those random ebooks or take one of those courses". The problem wasn't that I wasn't interested in the material; after all I'd saved the ebook for later. The problem was it wasn't immediately relevant, and that's where learning goes to die. If I don't have a clear example in my mind of where I'd use it, I won't be motivated to sit down and learn it.

I needed a new approach.

Learning by doing means the problem should come first. I'm motivated to solve a particular problem, and only then do I ask myself what I need to know to solve it. Maybe I already know how to do it, maybe I need to read up on something I used to know about, or maybe I need to learn something from scratch. Either way, because there is a problem in front of me, I can be laser focused on learning just enough to get a satisfying answer. This means you learn a fraction of the thing you're using to solve the problem, and this used to terrify me. This sort of learning is a recipe for serious impostor syndrome. Yes, I've just used algorithm X to get to my answer, but don't ask me to derive the underlying equations or remember any of the commands I had to type to get there.

The revelation I had is that this is fine.

It's fine to only know a small percentage of tool X, or algorithm Y. Just because you used it once, doesn't mean you should hold yourself to an expert standard. Clearly you're not an expert at it yet, just someone who managed to work out how to use it for your problem, but the problem was the goal and you got there!

Unsurprisingly to "learn by doing" you need to do lots of that "doing" thing. My advice to students is always to work on little pet projects. They'll help hone your skills, learn new ones, and make a portfolio to show prospective employers. You'd be surprised how few people take me up on that advice (myself included, I'll admit). Maybe because it's only one part of the equation. It's fine to "do stuff" but what if the feeling of impostor syndrome won't go away?

Maybe it doesn't need to.

Channel your inner impostor

Impostor syndrome is a reminder that you don't know everything, but it could also be a reminder that you can't know everything. In fact, being an impostor is not a bad mindset for a data scientist trying to make sense of the business around them.

One mindset shift I recommend for data science is that you should always be the stupidest person in the room, or on a Zoom call I guess. When someone talks about basic concepts, company metrics like conversion, churn, or even "customer", you should always ask for a definition. You might feel it risks singling you out as someone who "doesn't know", but you'd be surprised how hard it is to get a consistent definition for concepts that people talk about all the time. We shouldn't be afraid to question basic assumptions; in fact, progress often lies there. Think of it like a reverse impostor syndrome: by freeing yourself from the weight of knowing everything, you boldly go into a meeting as someone who is ignorant but eager to learn.

Obviously this goes for non-technical topics. You should probably always be the one in the room who knows the most about, say, linear regression, but if you've just been hired at an agricultural firm as a data scientist and know nothing about agriculture, don't be afraid of that fact. Use it to your advantage. The same way anxiety can be a good thing before an important speech or meeting, because it shows you care enough about doing well to be anxious, impostor syndrome can also be a useful companion. Free yourself and ask all the questions.

So let's hear it for impostor syndrome. We only ever talk about it as if it's a disease plaguing our industry, but it can also be our secret weapon in getting things done. Don't worry about learning it all up front. You'll almost always be measured on what you achieve, not what you know. I like the framing "just in time" learning rather than "just in case". In the meantime, just go and do stuff. Stuff that interests you, without worrying about what you'll learn from it or whether you'll learn anything. Chances are you'll learn along the way and be better equipped next time. And think of your self-imposed "impostor" label not as a judgement, but an opportunity to always learn something new.

Machine Learning Haikus

2017-04-17T13:48:00+01:00

You know how there are lots of blog posts out there about machine learning algorithms "in plain English"? They're popular because many people want to learn about machine learning, but don't want to be bombarded with heavy maths the moment they start. I agree, I think machine learning should be taught intuition first. I also recognise that people are busy and don't want to spend hours reading up on algorithms to understand them.

So forget "machine learning in plain English". Instead, I present some of the most popular algorithms in haiku form. Consider it "machine learning for the busy".

Logistic Regression

Inputs weighted, summed.
Passed through logistic function.
Shall we output 1?

Random Forest

Oh, decision trees!
Alone you're not so useful.
How 'bout a forest?

Support Vector Machines

Binary outcomes.
Maximum separation,
Is what is needed.

Linear Regression

Weighted X is summed.
Perhaps regularised, too.
Gives me real numbers.

Neural Networks

Backpropagation:
Works well, is a mystery.
But Geoff Hinton knows.

K Nearest Neighbours

Nothing to learn here.
Find the nearest data points,
And use the average.

K-means Clustering

Find similar things
By grouping based on distance
There's no "right" answer.

Markov Chains

Mr Markov, your
Use of probabilities
Gave us funny text.

Naive Bayes

We're gonna pretend
Features are independent,
Naive as that is.

Analysis: Is Alan Davies Getting Better at QI?

2017-02-15T12:06:00+00:00

This post originally appeared on my blog in 2017

I'm a big fan of the quiz show QI. I was watching a later series of it recently on Netflix and I couldn't help but notice that Alan Davies was winning quite a few episodes. This felt odd, because I was sure that when I first watched the show he routinely finished last, and certainly wasn't winning any shows.

That prompted me to ask the question: is Alan Davies getting better at QI?

To sate my curiosity I tried to answer that question the best way I know how: with data.

The Data

The data comes from The British Comedy Guide, which has an exhaustive list of everything related to British comedy, including every episode of QI. There was a lot of laborious data cleaning involved, which you can see for yourself in the associated Jupyter notebook.

The final dataset includes every episode with its title, broadcast date and each contestant and their scores.

The Analysis

As with many data science projects, once the dataset was nice and clean the question was straightforward to answer. All it needed was a plot to show Alan's win ratio over time. Here's the accompanying Jupyter notebook.

Alan's more or less consistently winning a quarter of shows now

So to answer the question: yes, Alan does appear to be getting better at QI, certainly since the first few series. It does however remain to be seen whether his win ratio will plateau at 25%, I guess we'll see in a few years' time.

Visualising the Worldwide Win Percentage of the Hungarian National Football Team

2017-01-20T11:36:00+00:00

This post originally appeared on my blog in 2017

I've often read, and agreed with, the advice that side projects should be solving problems or answering questions that you yourself are interested in.

To that end, I've always wanted to know how well the Hungarian national team have done against various countries worldwide. Hungary has a glorious football past, although the last 30 years have been possibly the worst ever decades in Hungarian football.

To explore this question, I scraped a dataset of all matches played by the Hungarian national team since 1907, aggregated it and made an interactive world map.

I'll start with the final map, and you can read about some of the details beneath.

The worldwide win percentage of the national team

There is an interactive version where you can zoom and hover over each country to find out more. I encourage you to look at it if you want to dive into the data and explore.

The Details

The Data

I scraped the data from the wonderful http://www.magyarfutball.hu which is a great resource for those with the very niche interest in Hungarian football. The data was unsurprisingly in Hungarian so I also had to translate it!

There is a Jupyter notebook for the scraping code and the final dataset is also available.

The Map

My last experiment making a choropleth map was missing some features I'd have liked to add, such as more interactivity. The library I used, folium, is still under development so for this project I tried something new.

Introducing plot.ly!

It's a great visualisation toolkit with a Python wrapper. It is much easier to make interactive choropleths with it, I specifically used this set of tutorials. The ability to customise the hover label meant I could make a much more useful visualisation, which you can explore.

There is a Jupyter notebook where I aggregated the data, dealt with countries that don't exist anymore (the USSR, Yugoslavia, etc.) and made the final map

The World Map of the 2016 FIFA Awards

2017-01-13T15:06:00+00:00

This post originally appeared on my blog in 2017

The "Best FIFA Football Awards" took place recently and while the final outcome was not surprising, I've always wanted to know which countries vote for which players.

The way the voting works is that the captain and coach of every national team in FIFA, as well as a member of the media for each country, gets to vote for their top 3 players, giving them 5, 3, and 1 point(s) respectively. Like I said, the result was not surprising:

The top 5 highest scoring players in the FIFA awards

FIFA routinely release all the votes, and I wanted to explore the data further. Specifically, I thought there might be some interesting geographic patterns (e.g. all of South America voting for South American players and so on), so this was a map plot waiting to happen.

The Data

FIFA weren't going to make this easy; the dataset is a PDF with tables in it. While there appear to be a few ways to extract tables in Python, none of them worked for extracting these tables, and neither did highlighting it and trying to paste it into Excel.

Tabula to the rescue! Tabula is a great open source tool for automatically finding tables in PDFs and it worked perfectly.

To save anyone else this trouble, here's the final csv file.

Not all countries had all three vote types (captain, coach, media) and I wanted to use a single vote for each country, so I used the priority order of coach -> captain -> media. This is why for example the vote for Argentina isn't for Messi, because the coach vote was missing and Messi himself is the captain who couldn't vote for himself.

The Tools

I wanted this to be a chance to explore folium, a great Python package for making maps.

Folium supports choropleths, but you have to provide a JSON file with topological information. I used data from this gist but had to match each country to the countries in the FIFA tables, because they didn't always match and there are a few countries that weren't present in both data sources.

One modification I had to make is that it's not currently possible to add an ad hoc legend into a folium map, so I built an HTML legend and injected it myself. I knew that HTML knowledge would come in handy for data science!

The code (hopefully) speaks for itself, you can view the Jupyter notebook on GitHub.

The Map

Here's a little preview of the final map, although the HTML version is better for exploring different regions.

That Griezmann sure is popular in Mongolia.

Wrap-up

Unsurprisingly, a lot of countries vote for their own players where they have players good enough. I'm looking at Sweden, Germany, or Poland here. I'm unsure if this is out of national pride or whether they believe their players are genuinely better than Messi or Ronaldo, probably the former. Brazil interestingly bit the bullet and voted for someone else despite having Neymar as a reasonable option.

Folium doesn't currently support hover actions, but a nice addition would be to be able to hover over a country to see who they voted for, especially for the "Other" category. It might be in a future version of folium, as it seems to be actively under development. Realistically, a tool like Tableau would be better suited to a visualisation like this, but the ability to embed maps in Jupyter notebooks makes folium a really good addition to my data science toolkit.

Method Chaining in Pandas

2016-11-30T18:14:00+00:00

When you work with pandas, you'll often perform multiple operations on a DataFrame. Some data cleaning and basic plotting for example, something like this:

import pandas as pd

df = pd.read_csv("my_data.csv")
df = df.drop("column_1", axis=1)
df = df.rename(columns={"column_2":"name", "column_3": "address"})
by_address = df.groupby("address")
by_address["name"].plot(kind="bar")

That works fine, you're incrementally changing the DataFrame until you're ready for aggregation and plotting.

There is an alternative, which you might find more readable.

import pandas as pd

df = pd.read_csv("my_data.csv")
df.drop("column_1", axis=1)  \  
 .rename(columns={"column_2":"name", "column_3": "address"})  \  
 .groupby("address")["name"]  \  
 .plot(kind="bar")

Why can we keep chaining methods together like this?

In pandas, each of those functions returns a copy of the modified DataFrame, which is why we were setting it back to the df variable each time. By chaining methods together we're just calling the next method on the modified DataFrame until we're done.

Notice we have to break lines with a backslash character to allow the chain to go over multiple lines. You can avoid that by putting the entire chain in brackets:

(
    df.drop("column_1", axis=1)
        .rename(columns={"column_2":"name", "column_3": "address"})
        .groupby("address")["name"]
        .plot(kind="bar")
)

Logically and computationally these examples are equivalent, so this is mostly just a stylistic consideration.

Benefits

Readability

I'd argue the second method looks better, it actually reads left-to-right (or up-to-down I suppose) and you can understand logically what you're doing each time.

No Intermediate Variables

In the first example we had to either save back the modified DataFrame to the original df variable, or create a new one each time. This means you have to think about whether you want to store the DataFrame in each of its states and come up with descriptive names for them, and as we all know...

There are two hard things in computer science: cache invalidation, naming things, and off-by-one errors.
— Jeff Atwood (@codinghorror) August 31, 2014

Avoids "inplace" Confusion

In my experience the fact that most DataFrame methods return a copy of the DataFrame is actually confusingly counterintuitive for pandas beginners. You either have to keep saving your DataFrame back to the same variable, or use the "inplace" keyword. Using method chaining means you only have to consider this problem once, i.e. set the final result to a variable, without ever accidentally throwing away any of your changes.

Downsides

No Intermediate Variables

Not having access to the intermediate states of the method can also be a downside. If you want to reuse any of the intermediate steps in the process, you need to keep a copy of it so you might not want to use method chaining all the time.

There is a workaround. You could use the .pipe() method to call a custom function on your DataFrame, including creating a new variable. First, create a custom function:

def save_intermediate_df(df, variable_name):
    globals()[variable_name] = df
    return df

This function takes in a DataFrame and a variable name and creates a new global variable with the requested name. The variable becomes the current state of your DataFrame at that point in the chain.

You then use .pipe() to add your function to the chain:

(
    df.drop("column_1", axis=1)
        .rename(columns={"column_2":"name", "column_3": "address"})
        .pipe(save_intermediate_df, "df_2")
        .groupby("address")["name"]
        .plot(kind="bar")
)

Now you have a global variable df_2, which represents the state of your DataFrame at that point in the chain, and which you can inspect. This is a slightly hacky way to add intermediate variables to a chain, but it gets the job done and skirts around this potential downside.

Debugging is Hard(er)

Debugging a problem in a long method chain is hard. If your chain consists of many intermediate steps and the final output is wrong, or you get an error message, it can be hard to retrace your steps to see what went wrong. If you had each command line by line, like in the first example, you could step through the code with a debugger or simply run the commands one at a time until you find the problem.

However, creating intermediate variables as shown above would greatly help the debugging process.

You Can Get Carried Away

You can take method chaining to extremes...

import pandas as pd

df = pd.read_csv("my_time_series.csv")

# take a deep breath...
(
    df
        .drop("column_1", axis=1)
        .rename(columns={"column_2":"date", "column_3":"price"})
        .dropna(subset="date")
        .fillna(0)
        .loc[df["price"] > 0, ["date", "price"]]
        .set_index("date")
        .resample("M")
        .mean()
        .plot()
)

You could argue that's a bit much, but I'd also argue it's actually self-documenting analytical code.

Best of Both Worlds

There is a time and a place for method chaining.

If you don't care about intermediate steps, and just want a basic plot for example, it's a good option.

If you want to do complex operations that might need serious debugging, maybe it should be avoided (at least initially).

What I tend to do is avoid using it while I'm still exploring my data and the code isn't final, and refactor to use method chaining when I'm confident the code works. The idea is that it helps future readability, so I can better understand my code if I look back on it later.

This post was mostly inspired by the great Modern Pandas series. and the excellent Effective pandas book.

Visualising Decision Trees in Python

2016-11-28T22:03:00+00:00

Interpretability is often an important concern with a machine learning algorithm (despite spellcheck telling me it's not even a word). Having an accurate predictive model may be enough in itself, but in some cases the only way to turn it into a business decision is if you can understand why it's getting the results it's getting.

An obvious candidate for an interpretable classifier is a decision tree.

I won't go into the specifics of decision trees, Machine Learning Mastery has a good tutorial on the subject, but I'll just go over the intuition.

A decision tree is a series of if-then rules that decide what class a data point should belong to (in the case of a classification tree), or what value one of its properties should have (in the case of a regression tree).

If you've ever seen a flowchart, you can imagine a decision tree the same way.

A model might learn a decision tree that can be interpreted as something like "if the petal length is less than 2, classify the flower as setosa, otherwise if the petal width is greater than 1.5 classify it as virginica, otherwise classify it as versicolor".

You can train decision trees with Python using scikit-learn.

from sklearn.datasets import load_iris
from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score

iris = load_iris()

X = iris.data
y = iris.target

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, stratify=y)

dt = DecisionTreeClassifier(min_samples_leaf=5, max_depth=3)
dt.fit(X_train, y_train)

I've set the maximum depth to 3, meaning it won't be grown beyond 3 levels, in this case purely for easier visualisation.

Once you're happy with a model, how do you visualise your tree?

Scikit-learn has a built-in function called export_graphviz which lets you export it to a file, in a specific format.

from sklearn.tree import export_graphviz

export_graphviz(dt, 'iris_tree.dot', feature_names=iris.feature_names)

You can then open the file in Notepad (or any text editor) and view its output online by pasting its contents into the textbox at http://webgraphviz.com

Our iris decision tree looks something like this:

Decision tree trained on the iris dataset

By providing the feature names we can label each decision point so it is obvious what's happening at each step.

The "value" part of each leaf node shows how the examples that make it to that node are split between the different classes.

I wasn't far off with my example - a petal length of 2.45cm separates the setosa class nicely, then a further separation using petal width, length and sepal length is enough to give us over 90% accuracy.

Once you get to deeper trees, this visualisation becomes ungainly, but if you want to keep the tree interpretable you probably want to limit its depth.

For further visualisation options you can follow the instructions on the official scikit-learn page.

SQL For Data Scientists

2016-11-26T20:08:00+00:00

From what I can tell, the biggest difference between data science curricula and data science job postings is usually knowledge of SQL. I assume most businesses want a data scientist who knows SQL because a lot of corporate data is stored in some sort of relational database. For some reason though, data science courses don't always tend to teach it explicitly.

I wanted to collect some of the concepts which I think are useful for aspiring data scientists to learn about databases and SQL. I'll also link to appropriate parts of the w3schools SQL tutorials along the way.

Query Syntax

Obviously the first step is to understand how to write a SQL query.

SQL is a declarative language. All that means is that when you write a SQL query, you're expressing what the result should look like, rather than how to achieve it.

Let's look at a basic SQL query and see how that's the case.

SELECT
    name, age, height
FROM
    people
WHERE
    job = "data scientist"

SQL is not case sensitive, but I capitalised the keywords (which is a typical thing to do anyway).

If you know that you have a table of data, called people, you can pretty much work out what this query will do. The declarative syntax means you can specify the data source (the people table), what you want to extract (name, age and height) and any filters you want to apply (only get the data for people who are data scientists).

There is a lot going on under the hood in terms of the computer deciding how best to store and index the data, but when you write queries you don't want to have to care about that, you just want your results.

The main keywords you need to know are:

SELECT...FROM (to select rows from a specific table)
WHERE (to filter rows - optional)
INSERT (to insert new rows)
UPDATE (to update existing rows)
DELETE (to remove rows)
GROUP BY (to group data into... well, groups)
CREATE TABLE (to create new tables)
ALTER TABLE (to make changes to tables like adding new columns)
JOIN (to join multiple tables together)

The JOIN keyword

I left the JOIN keyword until last in that list because it warrants its own section.

Merging multiple data sources is a staple data science operation, and that's no different when working with SQL. If you've used the merge function in pandas you'll have seen this already, but let's see how they compare.

Let's take the example of joining two data sources with pandas. One of them is a csv of people, with names, ages, heights and jobs. The other is a csv of phone numbers linked to people's names. The name column is common between both data sources.

import pandas as pd

df = pd.read_csv("people.csv")
phone_numbers = pd.read_csv("phone_numbers.csv")

merged = pd.merge(df, phone_numbers, on="name", how="inner")

That how keyword corresponds to the type of join in SQL. The same operation in SQL looks like this (assuming we have a people and phone_numbers table in a database, rather than csv files):

SELECT
  people.name, people.age, people.height, phone_numbers.phone_number
FROM
  people
  INNER JOIN phone_numbers ON people.name = phone_numbers.name

I've specifically stated in the SELECT clause where the columns come from, because both tables have a name column and SQL would have gotten confused otherwise.

The types of SQL join correspond to the valid values of the "how" keyword in the pandas merge function. They are:

Inner Join - only rows where both tables have a value are returned
Left Outer Join - only rows where the table on the left of the statement has a value are returned
Right Outer Join - only rows where the table on the right of the statement has a value are returned
Full Outer Join - all rows are returned from both tables

The outer joins let you keep rows from either table if there are no corresponding rows in the other table.

So a left join in the previous statements would have shown all people, regardless of whether they had a phone number in the second data source. The rows of people who don't have a phone number would have shown a NULL value for the phone number. Using an inner join wouldn't have returned them at all.

It can be helpful to see this visually, and the w3schools pages do that already, but here's another good example.

SQL Tools for Data Science

If you know the basic query syntax and the various join types, you're probably equipped enough to start pulling data out of any SQL database. Programmers working with SQL often use specific tools to access databases, such as Microsoft's SQL Server Management Studio.

However, as a data scientist you may want to do this straight from your code instead. You have a few options for this.

You can connect to sqlite
You can use SQL queries in pandas using pandasql
pandas also allows connecting to SQL sources

Also, if you're familiar with pandas but not with SQL, the pandas documentation has a section with pandas commands and the associated SQL queries.

Conclusion

I'd argue that's all you need to get up and running.

W3schools is a great interactive resource to test your sql queries, but there are plenty of other good learning resources like Codecademy.

There is more to know of course about relational databases. I haven't covered the concepts of primary keys, foreign keys, or indexes because these are more important for database design rather than data retrieval. Designing a relational database has its own set of skills and required knowledge, but if your only interaction is retrieving data, you shouldn't have to worry about it.

I may write a post about database design in the future, but I'd strongly argue that it's an optional skill for most data scientists.

More on K-means Clustering

2016-11-20T17:01:00+00:00

In Part 1 I described the k-means clustering algorithm and some of its uses along with a quick Python implementation. Going forward I recommend using the scikit-learn implementation.

Now let's see k-means in action!

Image Segmentation

One use of clustering is to segment images:

For example, in computer graphics, color quantization is the task of reducing the color palette of an image to a fixed number of colors k. From: k-means clustering (Wikipedia)

Given an image, we can use k-means clustering to find similar colours in the image, and re-draw it with fewer colours. This has uses in data compression for example.

This is the example I'll run through today.

We'll take this image of a puppy:

Puppy image from Greg on Flickr (https://flic.kr/p/oaj46)

and redraw it in much fewer colours using k-means clustering.

The Data

Let's start by defining our data. To represent this problem, we take each pixel in the image as a data point whose 3 features are the R, G and B values of the pixel.

For this particular image this gives us a dataset with 3 columns and 43,680 rows. Some of the pixels are the same colour, but we've still got over 10,000 unique colours in our image.

Running K-means

It is conceivable that we can group similar colours together and redraw the same image with fewer colours in a way that we can still tell what is in the image. This would reduce the amount of information needed to represent the image (and therefore the filesize) without visibly losing much detail.

Once we have our dataset (the details of extracting the pixel values will be in the accompanying Jupyter notebook) running the algorithm with scikit-learn is easy:

from sklearn.cluster import KMeans

kmeans = KMeans(n_clusters=k)
# our dataframe (df) is the image data
clusters = kmeans.fit_predict(df)
# "clusters" is a vector with a cluster assignment for each data point (pixel)
df['cluster'] = clusters
# use the K cluster centroids as new colours to represent the image
colours = [tuple([int(c) for c in x]) for x in kmeans.cluster_centers_]

In this instance the cluster centroids are points in 3D space, where the 3 dimensions are R, G and B, so the centroids can be thought of as colours themselves.

That means once we have the cluster assignment for each pixel, plus the centroid as the associated colour value, we can reconstruct our image pixel by pixel to get the \(k\)-colour representation.

The same image with only 3 colours looks like this:

The same puppy drawn with only 3 colours

As you can clearly see, we've reduced the number of colours required, and incidentally also reduced the filesize threefold, without losing too much information. The image is still clearly a puppy, despite the fact that we've only used 3 colours.

When we use 16 colours the image starts to resemble the original in much more detail:

The same puppy drawn with 16 colours

You can still see the background isn't smooth but we're getting close. In fact, we would get to an image that is indistinguishable from the original by using far less than the original 10,000 colours.

Here is the Jupyter notebook for drawing puppies.

I have a couple of points left to raise, namely some practical tips when using clustering.

Choosing K

How would we know which point to stop at? When is \(k\) at its optimal value?

As I mentioned in part 1, this is usually somewhat subjective, but there are some general heuristics.

In this case we could do it by visual inspection. That is, we could say \(k\) is high enough when we are no longer visually able to tell the difference between the original and the redrawn images.

Not all datasets will lend themselves to visual inspection like this though.

We can use what's called the elbow method to evaluate when to stop.

For each value of \(k\) you want to evaluate how much of the variance in your data is explained by the configurations of those \(k\) clusters. This value increases for each value of \(k\), but the idea is that we stop increasing \(k\) when increasing it gives us diminishing returns.

Let's think of the two extremes. When \(k\) = 1, it means every point will belong to the same cluster. This configuration explains 0% of the variance in your data, because it says all your data points are the same. Then \(k\) is equal to the number of data points you have, it means every point will belong to its own cluster. This configuration explains 100% of the variation in your data because it says each of your data points is different from every other one. A value in between will explain some % of the variation, because it will say some data points are equal to some other data points, and different from some others.

Here's a good explanation of the elbow method, although it uses the "error" for each \(k\) value, so the graph is upside down compared to the "variance explained" metric I discussed above.

Either way, there is usually an "elbow" where the increase/decrease is suddenly less sharp. That's usually a good point to stop and use that value for \(k\).

Normalisation

As I mentioned in the Self-Organising Maps tutorial, in practice you will want to normalise your data so all features are on the same scale. This is also true of k-means clustering. If all features are on the same scale, each feature will "contribute" to the algorithm equally, otherwise a feature with much larger values will dominate the others.

In the case of colours, the R, G, and B values are all on the same scale (0 to 255) so this is not necessary, but in real world examples your features will often be on different scales.

See more information on this in Sebastian Raschka's machine learning FAQ.

K-means and clustering in general have many more uses, and I hope these puppies have piqued your interest!

Introduction to K-means Clustering

2016-11-20T15:18:00+00:00

This is the first of a two-part post on K-means clustering.

Introduction

Unsupervised Learning

I've talked about unsupervised learning before when dealing with Self-Organising Maps, but just to recap. Unsupervised learning is when you have a dataset of features with no pre-defined outcomes. You give it to an algorithm to learn patterns in the data without knowing in advance what associations you want it to learn.

So you're not trying to teach it to tell the difference between images of cats and dogs; instead, you're trying to make it learn something about the structure of the images, so it can find similar images without explicitly knowing about cats and dogs.

K-means is a type of unsupervised learning method, specifically a type of clustering.

Clustering

Clustering deals with finding groups of similar data points.

There are two criteria that make a "good" set of clusters:

Intra-cluster similarity. That is, all the data points within a cluster should be similar to each other (we'll deal with what 'similar' means a bit later).
Inter-cluster dissimilarity. That is, data points in one cluster should be sufficiently different from data points in another cluster.

This is also what we're trying to achieve with k-means. It is only one of the many types of clustering algorithm, but I've chosen it as it's popular as well as being easy to understand and implement.

Uses of Clustering

What are some uses of clustering?

Finding similarities in your data that you couldn't do by inspection has a lot of uses. The classic example is "segmenting your customer base", that is identifying customers with similar buying behaviours for better targeted advertising. Another form of clustering, hierarchical clustering, is used in astronomy. You can even use it to find similar boroughs in London based on house-buying behaviour.

Potential Problems

Choosing K

Before you start training your data to learn the clusters, you need to choose a value for \(k\). That is, you have to decide beforehand how many groups there are going to be. This sounds like it defeats the purpose of being unsupervised, and it is indeed something that you have to set manually.

Due to the random nature of the initialisation of the algorithm, and the uncertainty in the correct value for \(k\), you might find re-running the algorithm multiple times gives you different results.

Subjective Evaluation

As with unsupervised algorithms in general, evaluating the outcome is partly subjective. There are objective measures with which we can compare different runs, but the final evaluation will be based on which one feels best.

Sometimes you can look at the characteristics of the clusters to see which one makes most sense. For example, if segmenting your customers into clusters results in two clusters that both contain mostly customers over 60, you might choose another run that better separates your customers based on age. Of course, the clusters will be created based on your data, so if there really are two distinct groups of over-60 customers then no amount of runs will change that!

The K-means Algorithm (with words)

Clusters have two properties: a centroid and a set of your data points that are assigned to the cluster.

The centroid is simply a point which is the mean of the data points that belong to it (hence, "k-means").

Mathematically, the centroids are a point in n-dimensional space, where n is the number of features your data has.

The basic idea behind the k-means clustering algorithm is simple:

Start with a chosen value of \(k\).
Choose \(k\) of your data points at random to be your starting centroids.
For each data point, assign it to a cluster based on which of the \(k\) centroids it is closest to. Closest can mean any distance measure. The Euclidean distance is often used.
Now you have \(k\) groups of data points assigned to a cluster. Re-calculate the position of each cluster centroid by taking the mean of the new points that are now associated with that cluster.
Repeat steps 3 and 4 until convergence. You are typically done when no points have changed clusters since the last iteration.

It's better to see this happen visually - here's a good interactive example.

The K-means Algorithm (with code)

Let's go through the steps again with code. Let's use the iris dataset.

Steps 1 & 2 - Initialisation

import pandas as pd
import numpy as np

df = pd.read_csv('iris.csv') # I'll supply this alongside the Jupyter notebook
# we don't need the target variable
df.drop('class', axis=1, inplace=True)


k = 3
centroids = []
clusters = []

# random initialisation of centroids = pick K data points at random as centroids
init_centroids = np.random.choice(range(len(df)), size=k, replace=False)

for i in init_centroids:
    # get the data point at index i
    pt = df.iloc[i,:]
    # append it to the centroids list
    centroids.append(pt.values)

Steps 3 & 4 - Learning

n_iterations = 10

def assign_to_cluster(idx):
    point = df.iloc[idx,0:4]
    # calculate distance (without sqrt) to each centroid
    distances = []
    for c in centroids:
        distances.append(((point - np.array(c)) ** 2).sum())
    # find index of closest cluster
    closest = np.argmin(distances)
    # assign point to that cluster
    clusters[closest].append(point.values)
    df.loc[idx,'cluster'] = closest

for _ in range(n_iterations):
    # first, reset the clusters
    clusters = []
    for i in range(k):
        clusters.append([])
    # assign each data point to nearest cluster
    for i in range(len(df)):
        assign_to_cluster(i)
    
    # now, recalculate the centroids
    for i in range(k):
        centroids[i] = np.array(clusters[i]).mean(axis=0)

First we've defined a function that calculates the distance between a data point and each of the cluster centroids. We can use the implementation trick where we don't need the actual Euclidean distance (to avoid the expensive square root operation). We then assign the data point to that cluster.

Then we iteratively assign points to the clusters and re-position the cluster centroids. With the new centroid positions, we assign points again, then re-calculate the centroids again and so on.

In this example I've forced it to run just 20 times, so there might still be room for improvement, but typically you'd run this until no points have changed cluster since the last iteration.

Once we've done that we can plot two of the data's dimensions and colour each point by its assigned cluster (and mark the cluster centroids).

We've gone from this plot of raw data:

Plot of raw data before clustering

To this plot where we've clustered our points into 3 groups:

3 clusters after just 20 iterations

There is still some overlap between the black and blue clusters, but just 20 iterations have quite effectively grouped our data into 3 clusters.

That's all there is to it!

Here is the associated Jupyter notebook.

In Part 2, I'll talk about another practical application of the k-means algorithm (using scikit-learn this time) as well as some implementation details such as how to pick the value of \(k\).

Turning Jupyter Notebooks into Reusable Scripts

2016-11-17T19:56:00+00:00

I read an article today called Data Scientists Need More Automation. No prizes for guessing what it was about.

A lot of the specifics were focused on sysadmin-type work like using SSH, but the main idea is one that applies to all data science tasks.

The thrust of the article was:

Someone please help data scientists be lazier, do less work, and reduce the mental overhead of dealing with computers!

From Data Scientists Need More Automation

As part of my commitment to occasionally talk about "programming for data scientists", I want to share ideas that will facilitate this to help data scientists focus on important stuff.

Laziness is a virtue when it comes to programming.

Always thinking "how can I do the same thing with less effort?" is a great way to be more productive and focus on the hard parts of data science.

For example, it's clear that you want to speed up and automate your data cleaning. That's not the important stuff you want to focus on. So in this post I want to share some thoughts on how to make your Jupyter notebooks easier to "productionise".

When you do data cleaning, notebooks are a great way to experiment with your code in an interactive way before you can create a script that runs on gigabytes of data. These thoughts are mostly concerned with how you take a notebook that can clean a specific file, and make it into a Python script you could run in the background to process many similar files automatically.

Start Small

If your datasets are big enough that processing them takes longer than a few seconds, you are going to lose a lot of time if don't you test your code on a smaller subset first.

If you are cleaning your data, you shouldn't be using your entire dataset until you can prove that your script will run on a smaller version of it. That might be as easy as just restricting your dataframe to its first N rows:

import pandas as pd

df = pd.read_csv("my_huge_file.csv")
df = df.head(1000) # delete this line later, but only when you're ready!

This might sound obvious but it's important to get into the habit of doing it.

Brief Digression: Subsets of Data for Machine Learning

For machine learning, if you're just testing your code to make sure it runs, you can do the same thing and take the first few hundred rows. Obviously if you're training predictive models you want to use your entire dataset.

However, if you want to just get a sense of which models are more accurate than others, in the case of classification problems you can use a stratified subset of your data. Instead of taking a random sample you can sample based on the frequencies of your classes, so that your smaller sample has the same class proportions. In scikit-learn you can use StratifiedShuffleSplit for example.

Back to Notebooks...

Once you've experimented enough with your code so that you know it works on your small subset, you'll want to ensure your code is general enough that it would run with any file you give it.

For example, if you have a dataset that covers one day's of data you might eventually want to let it loose and process months of data one day at a time.

The obvious way to do this is parametrisation.

Use Parameters

Stop hard-coding things.

Seriously, whenever you have a value that is likely to change when you run the script multiple times, make it a variable.

Turn this:

pd.read_csv("my_data_2016-01-01.csv")

Into this:

filepath = "my_data_2016-01-01.csv"
pd.read_csv(filepath)

It's an extra line but you're going to have to do it if you want to automate your script, so get in the habit of starting out like this.

Even better, create a separate cell at the top of your notebook just for parameters. That way you won't forget which things will need to change for each file.

Converting Parameters to Command Line Arguments

If you've created the right variables for automation, you can convert them to command line arguments.

This can be as simple as exporting your notebook to a Python file (File -> Download as -> Python) and replacing your parameters with command line arguments.

Assuming your notebook looks something like this:

date = "2016-01-01"
other_parameter = 16

# rest of your code here...
# ...
# ...

Export it to Python, then amend the script slightly:

import sys

if __name__ == "__main__":
    date = sys.argv[1]
    other_parameter = sys.argv[2]

    # the rest of your code from the notebook
    # can be pasted here UNEDITED

All we've done is replace the hard-coded parameter values with arguments from the command line (remember sys.argv[0] is the name of the script), so now you can do this on the command line:

python my_automated_script.py "2016-01-01" "15"

This is the "quick and dirty" way of doing it. For more robustness and better documentation of your arguments, use argparse or begins.

To make this workflow possible, you also need to make sure your notebook doesn't do too much.

Single-Purpose Notebooks

There might be a lot of code involved in cleaning your data. You might need to deal with things like missing values, but then perform transformations and computations.

Eventually your notebook might be hundreds of lines of code.

If you ever get to that point, break the notebook into multiple smaller ones. You could even make the first notebook output a semi-cleaned version of your data which your second notebook picks up.

You can then still combine your notebooks into one Python script by exporting them, and just removing the intermediate files that you were creating when experimenting.

Say your notebook workflow is like this:

Notebook 1 reads raw csv
Notebook 1 does some data cleaning
Notebook 1 exports semi-cleaned data (intermediate csv)
Notebook 2 reads in intermediate csv
Notebook 2 does data transformations
Notebook 2 exports final csv

You can see that if we export both notebooks, combine them into a single file and remove steps 3 and 4, we get our final automated script. In fact, if we encounter problems later on we can always go back and debug them using our notebooks, and re-export them to get an updated version of the script. As long as you make all your changes in the notebooks and not the final script, this will be a valuable approach.

Hopefully I've given you some ideas about how you can design your notebooks from the start with a view to future automation.

Now go forth and be lazy!

Duck Typing

2016-11-14T16:05:00+00:00

In an attempt to bridge the gap between the two disciplines of programming and data science I will occasionally talk about programming concepts useful for data scientists, and vice versa.

Today I want to discuss duck typing.

Duck typing is a concept that originated in the Python community. It is a way of checking an object's type not by testing its type directly, but testing its methods.

The idea is based on something called the duck test. You've probably heard it before:

"If it looks like a duck, swims like a duck, and quacks like a duck, then it probably is a duck."

How does this relate to programming?

Well, Python is a dynamically-typed language. That means that the types of objects (whether they're integers, strings etc.) is checked at runtime, not compile time.

A variable is allowed to have different types at different points of a program's execution. This is perfectly valid:

my_variable = 42
# do stuff with my_variable...
# and later...
my_variable = "now it's a string!"

You can't do this in a statically-typed language. Once you declare a variable as a certain type, it stays that way. Take this example in C#:

int my_variable = 42; // explicitly declare an integer
// do stuff
my_variable = "can I be a string?"; // this will produce a compiler error

Actually there are dynamic types in C# but we'll just gloss over that.

The point is you can be quite liberal with types in Python.

You can take this one step further with duck typing.

Duck Typing in Python

Say you have a function that makes a duck quack, like this:

def make_it_quack(something_duck_like):
    something_duck_like.quack()

We've taken an object in and called its quack method. We don't care what type of object this is, only that it is able to quack. So if we had a "real" duck and an impostor, they'd both work with this method:

class Duck(object):
    def quack(self):
        print("Quack quack")

class Ferret(object):
    # ferrets can't normally quack, but this one's cunning
    def quack(self):
        print("Quack quack")

donald = Duck()
fred = Ferret()

make_it_quack(donald)
make_it_quack(fred)

Both of these will produce the output "Quack quack" because all we did was make it quack. If it can do that, then as far as we're concerned it's a duck.

Duck Typing in Data Science

This is a concept that can be quite useful in data science.

For example, imagine that you have your own implementation of a machine learning algorithm but want to use a lot of the goodness built in to scikit-learn.

Well, you know how all scikit-learn implementations have a fit and predict function?

You can create your own object and make use of duck typing by "quacking like a scikit-learn duck".

First, create a class that has fit and predict methods:

class MyFakeClassifier():
    def fit(self, x, y):
        print("Working VERY HARD...")
    
    def predict(self, x):
        # predict 0 no matter what
        return [0 for item in range(len(x))]

Now you can fit and predict the same way as you could with, say, a Random Forest.

Imagine you already wrote a small function that takes in a machine learning classifier, does a train-test split and gets the accuracy and confusion matrix of the predictions.

from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score, confusion_matrix

# write a function to give us a train-test accuracy score
def get_accuracy(model, X, y):
    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, stratify=y)
    model.fit(X_train, y_train)
    y_pred = model.predict(X_test)
    print("Accuracy: {}\n{}".format(accuracy_score(y_test, y_pred),
                                    confusion_matrix(y_test, y_pred)))

We can use this for a built-in classifier, but also our new estimator:

from sklearn.datasets import load_iris
from sklearn.ensemble import RandomForestClassifier

iris = load_iris()
X = iris.data
y = iris.target

rf = RandomForestClassifier()
random_model = MyFakeClassifier()

# we can get the accuracy of our Random Forest
get_accuracy(rf, X, y)
# and our new model!
get_accuracy(random_model, X, y)

There you have it. All we had to do was create something that can "fit" and "predict" and Python doesn't need anything else for it to work.

Note: to use the full range of scikit-learn functions with your own estimator, you should do it properly, but the point is you can do a lot of it by duck typing.

Here's the associated notebook.

Happy quacking!

How to Connect to Google Sheets in Python

2016-11-13T13:54:00+00:00

In most data science and machine learning tutorials you typically encounter csv files. Either you connect to them locally, something like this:

import pandas as pd
df = pd.read_csv("my_local_data.csv")

Or you access them via a direct url like this:

import pandas as pd
df = pd.read_csv("http://www.lotsofdata.com/hosted_data.csv")

What I rarely see though is connecting to slightly more obscure data sources. You will probably end up doing this once you go out into the real world of data science.

One useful data source is Google Sheets. If you have a spreadsheet hosted on Google Drive, which is made available for public access, and want to access it, it's not immediately clear how to do that.

Let's go through an example of how to connect to one. I'll use a spreadsheet that has the Hacker News salary survey results from a couple of years ago.

You can't use the url directly, because the url isn't just pointing to the data, it's pointing to the entire Google Sheets interface.

Instead you need the sheet's export link.

To do this simply take the url until the /d/ part, and the unique ID that comes after, so this much:

https://docs.google.com/spreadsheets/d/17Mr201gfDoOTe5ONLS6LYJi1wQbtT26srXeSwUjMK0A

and add /export at the end with some parameters.

You can specify the sheet number (zero-indexed) using gid, and the format to be csv using format.

The full url then becomes:

https://docs.google.com/spreadsheets/d/17Mr201gfDoOTe5ONLS6LYJi1wQbtT26srXeSwUjMK0A/export?gid=0&format=csv

Try that in your browser and it will download the csv file directly.

You can then read it into pandas and it will be treated as a regular csv file.

Here is the associated Jupyter notebook to see it all in action.

Markov Chains for Text Generation

2016-11-12T22:09:00+00:00

Markov chains are a popular way to model sequential data. They form the basis of more complex ideas, such as Hidden Markov Models, which are used for speech recognition and have applications in bioinformatics.

Today I want to run through an implementation of Markov chains for generating text based on an existing corpus. First of course we need to understand what Markov chains are before we can implement one.

The Intuition

I've talked about intuition-first machine learning before, so I'll start with the intuition. The first thing we need to know about is what a Markov chain consists of, then we need to define the Markov assumption.

States and Transitions

In a Markov chain we are assuming our sequence is made up of discrete states. That is, every item in the sequence is one of a finite set of possible values. In text, the states could be every letter of the alphabet and our punctuation marks.

We assume that for every item in the sequence, there is a probability associated with what the next value will be or, more formally, what state we will transition to. This is called the transition probability.

This is different between pairs of states, so the probability of going from state A to state B might be different than going from B to A. There is also a probability of staying in the same state.

However, it would be too difficult to attempt to model the transition probabilities if we always used the entire sequence. This would be like trying to predict the last word in War and Peace and having to take into account every single word that came before it.

To make this easier, we make what's called the Markov assumption.

The Markov Assumption

The Markov assumption is when we assume the Markov property to be true.

The Markov property (of a sequence) is typically formulated like so:

The future is independent of the past, given the present.

That sounds a bit like an old proverb but it's a simple concept.

If a sequence has the Markov property it means that the value of the sequence at the next step depends only on the value at the past \(n\) steps. The value of \(n\) (how far back we assume we need to go) is called the order of a Markov chain.

So a second-order Markov chain is one where we use the current and the previous value to predict the next value. We are assuming that it does not matter what the rest of the text was before the previous word, we only need the current word and the one before it.

In effect, the Markov chain has no "memory" beyond the last \(n\) steps.

The Markov assumption simply states that we believe this Markov property to hold for a given sequence.

This is a simplifying assumption that means it is much easier to compute these Markov chains at the cost of some complexity and accuracy. However simple this assumption may feel, it performs remarkably well.

The Markov Assumption in Text

Let's look at an example. What does this Markov assumption look like for text? We'll work with second-order Markov chains, as defined above.

Let's imagine that our sequence is this string of animals:

cat cat cat dog cat cat cat fish cat dog fish cat dog dog cat

What we say if we assume the Markov property is that our prediction of the next word in the sequence depends only on the last two words: dog cat.

How do we use that for predicting?

We need to calculate the transition probabilities based on the text that we have, then assume that going forward those probabilities are what decide our next words.

Because we are using second-order Markov chains, we calculate the probabilities of each word following a two-word pair. To do this, we simply count what percentage of the time a word appeared after each possible pair.

If we take a single example, "cat cat", we'll see that this pair was followed by "cat" twice, "dog" once and "fish" also once. That means if we're in the state "cat cat" the next word will be "cat" with a 50% probability, or "dog" or "fish" with 25% probability each.

The Code

That's enough intuition, let's get into the code.

I've scraped the lyrics to all songs written by Muse and we'll use this as our corpus. We'll use it to generate some more text, specifically a new Muse song.

Preparation

The first step is to read the file in and clean it so we have a sequence of words and new lines. I won't go into the details, you can see how the cleaning is done in the accompanying Jupyter notebook. I also included the code I wrote to scrape the lyrics in the first place, but going through that is optional as I'll also include the final source file.

Training

When we do the "learning" for a Markov chain, we're just identifying all unique triplets of words, and creating our transition probabilities using the first two words as the current state and the third as the next state.

As an implementation detail, I've made the transition probabilities a Python dictionary where each key is a unique word pair, and the value is a list of all the words that have ever followed that pair. I haven't even explicitly calculated the probabilities, but instead included each word as many times as it appears.

Our "cat cat" example above would look like an entry in a Python dictionary like this:

{ "cat cat": [ "cat", "cat", "dog", "fish" ] }

To generate our transition probabilities we use a helper function:

def generate_triples(word_list):
    d = {}
    # loop through the text and generate triples
    for i in range(len(word_list)):
        if i == 0 or i == len(word_list) - 1:
            continue
        else:
            if (word_list[i-1], word_list[i]) in d:
                d[(word_list[i-1], word_list[i])].append(word_list[i+1])
            else:
                d[(word_list[i-1], word_list[i])] = []
                d[(word_list[i-1], word_list[i])].append(word_list[i+1])
    return d

So given a list of words (our entire text) we go through it word by word, creating dictionary entries for every pair we encounter, and adding the next word into the list associated with that pair.

To then simulate the probability of choosing a word given a pair of words, we randomly sample from the list associated with that pair.

This function gives us the next word each time. This is part of a Markov class, where self.words is our previously trained dictionary.

def get_random_word(self, phrase=None):
    if phrase:
        # find a phrase from the list of words associated with the last two words in the supplied phrase
        phrase_words = [x.lower() for x in phrase.split(' ')]
        if len(phrase_words) > 1:
            if (phrase_words[-2], phrase_words[-1]) in self.words:
                past = self.words[(phrase_words[-2], phrase_words[-1])]
            else:
                past = self.words[random.choice(list(self.words.keys()))]
        else:
            past = self.words[random.choice(list(self.words.keys()))]
    else:
        # no phrase supplied, return a word from our dict at random
        past = self.words[random.choice(list(self.words.keys()))]
    return random.choice(past)

The function will look at a phrase, check if it is at least two words long and for the last two words, it finds the appropriate dictionary entry and samples from the list associated with that pair.

So if we gave it "banana cat cat" it would look up "cat cat" in the dictionary and sample from the list.

Remember: the words in the list implicitly represent the transition probabilities. Because we have "cat" twice out of four words in our list we've defined the transition probability as 50%.

To generate the text we can then either give it some text to start it off, or let it start with a random pair.

I've added some structure so that the code creates a song title, a chorus and a couple of verses.

It came up with this:

hear me moan

Verse 1

you are just
too much attention
and its gonna be
show me mercy can someone rescue me
make me agitated

Chorus

escaped your world
no one is crying alone
i wont let them hurt
hurting you no

Verse 2

you know what youve done
bring me peace and wash away my dirt
spin me round and have me to

Chorus

escaped your world
no one is crying alone
i wont let them hurt
hurting you no

I dare say Matt Bellamy couldn't have written it better himself.

Here's the associated Jupyter notebook. I've also previously published the code as more of a plug and play library, which you can find here.

Next Steps

We could improve this code by doing any of the following:

Experiment with different orders for the Markov chain
Giving it more data is never a bad idea, you could mash Muse's lyrics up with another artist
You could try a character-level Markov chain to see if it would generate meaningful text that way, although for that you might want a Recurrent Neural Network
Adding support for punctuation

Why You Should Reinvent the Machine Learning Wheel

2016-11-11T17:17:00+00:00

When you're a data scientist, unless you're in a job that's very research-focused (and likely requires a PhD) you'll mostly be using machine learning algorithms invented anywhere between say 5 and 30 years ago. Similar to being a programmer, you'll be using many libraries made by other people based on other people's ideas.

There's nothing wrong with that.

In fact, it's usually more productive to use someone else's implementation of an idea. Why reinvent the wheel every time?

With machine learning, I want to make a case for why it's good to reinvent the wheel when you're still learning.

I want to stress that I still think it's a good idea to use scikit-learn 99% of the time. However, when you're learning about one of the most popular algorithms (I've linked to this KDNuggets article before because I think it's a good overview of the minimum you should know) I think it's worthwhile trying to implement them yourself.

So why go through the effort of implementing existing algorithms again?

Programming Practice

This is an obvious benefit of trying to implement anything: you get programming experience. Many data scientists don't come from a programming background, so a crucial part of the learning process is feeling at home when writing code. Coupling this with additional machine learning practice seems like a good way to do this.

Also, implementing machine learning algorithms is a harder coding challenge than the ones you'd face if you were doing an introductory programming course, so this is a good transition from FizzBuzz-type problems to more meaty challenges.

Deeper Understanding

I'd argue this is the most important benefit. It's one thing to conceptually understand an algorithm and it's another to understand it in enough depth to tell a computer how to do it. Computers don't deal with ambiguity so you need to understand every little implementation detail to get to the end. That can only be helpful in deepening your understanding. Once you move on and use in-built libraries you can also more easily debug any strange behaviour because you'll know more about what is happening under the hood.

Portfolio Pieces

Perhaps not a reason to do this outright, but a good byproduct of these coding exercises is getting little pieces to add to your GitHub profile. That's never a bad thing.

A few things to remember.

We are not trying to "beat" other implementations

If you're really into optimisation and want to spend time learning how to make your implementations faster, obviously I wouldn't advise against it. At the same time it might not be worthwhile spending too much time improving the performance/scalability of your code if your aim was to deepen your understanding of machine learning.

Don't overdo it and implement everything you read about

Again, I'm not suggesting you should actively not implement every algorithm you hear about, but some algorithms might be harder to implement than others. Start small to avoid getting frustrated by complex problems. For example, don't start by implementing a deep convolutional neural network.

Then which algorithms should I choose?

Here are the ones I've gone for so far, because I thought they were easy enough to implement but I wanted to dive in to the details.

Naive Bayes

This was one of the activities of the Becoming a Data Scientist Learning Club. Actually, the activity was to read about and use the algorithm, but I took this as an opportunity to go through and implement it from scratch.

The Naive Bayes classifier is conceptually quite straightforward and a good place to start.

K-Means Clustering

I will spend some time later this month diving into clustering, but for now it's enough to say that this is also a good choice to start with. Conceptually simple, but you need to understand the details to be able to code the whole thing.

This is another algorithm I implemented as part of the Learning Club.

Self-Organising Maps

I discussed this algorithm recently, and hopefully showed that two blog posts are enough to go through the details enough to actually write the code for it. This is perhaps a less mainstream choice but conceptually lends itself to a good coding exercise.

Markov Chains

A popular choice for modelling and predicting sequential data (text, audio, time series). It only requires simple probability theory and is another good choice to start with.

My implementation generates new text based on text you give it.

Other Choices

Some other algorithms I suggest might be reasonable choices:

Decision Trees (although you might end up needing the help of recursion)
K Nearest Neighbours

Or if you're feeling a bit more brave, try a Multilayer Perceptron. You'll need to understand and implement backpropagation, but it would be a good advanced programming challenge.

Hopefully I've convinced you that implementing machine algorithms from scratch is a worthwhile endeavour. I'm always interested in seeing other people's implementations so let me know if you've done any!

Realistic Machine Learning

2016-11-10T15:41:00+00:00

As most data scientists quickly realise, there's a difference between the kind of data science you do while learning about it, and the kind you do at a real job.

This is especially true of university degrees, and isn't unique to data science. However much you learn during your degree, reality will always be different in a way you weren't prepared for. I don't mean this in a scary way, just as a matter-of-fact appraisal of how things are.

This is equally true of data cleaning/wrangling and machine learning.

The "ML Hipster" summed this up very well.

… and that concludes Machine Learning 101. Now, go forth and apply what you've learned to real data! pic.twitter.com/D6wSKgdjeM
— ML Hipster (@ML_Hipster) August 19, 2015

There's also this quote I've seen (source unknown):

In data science, 80% of time is spent preparing data, and 20% of time is spent complaining about the need to prepare data

What are the main reasons for this disconnect?

Real Data is Messy

Yes some data science courses cover this and talk about what sorts of things you'd need to do to clean a dataset.

However, this usually amounts to converting strings to dates or dealing with missing values. These are important skills to have, but only make up 10% of the kind of data cleaning you might encounter.

Human-generated data is messy because people are not consistent, and often data entry systems allow multiple ways of entering data or simply free text.

Computer/machine-generated data is messy because things can go wrong, log files from systems can have weird values in them, etc.

Not only does this have an impact on your data cleaning efforts (by requiring 10x more work than anticipated) but also machine learning.

Scikit-learn random forests are not designed to deal with someone putting "next Tuesday" in a free-type date field.

The Question is Unclear

An underappreciated quality of a data scientist is the ability to frame the question.

Converting a business requirement, which is often nebulous and unquantifiable, into a machine learning problem is a difficult task. There's usually more than one way to approach it, but it boils down to making a human problem into a concrete, measurable task for an algorithm.

This means a shift from your studies where you're given the question. No longer are you identifying types of iris - you need to start with identifying what to identify!

A Difference of Purpose

This is the big one. The moment you're doing data science for a business, your aims are different.

Previously you were solving problems for the sake of it. Now, the focus is on adding value.

That means that at any point in a project the next step will depend on what makes sense for the business, not what is academically most interesting. The kind of work you do will by definition be different to what you're used to.

How can we better prepare?

The obvious way to smooth the transition between academia (or "learning" in general) and industry is to teach more realistic problems.

That's not to say you should start with solving a business problem, but you should certainly end on one.

A more rounded way of teaching data science would be to start with toy problems and gradually build up to more complex ones. Something like this scale of four difficulties:

Difficulty 1

You're given an "academic" dataset with perhaps some cleaning to do (date formats, missing values). There is an explicit target value so once the cleaning is done you know exactly what to do for machine learning (e.g. predict the type of iris).

Difficulty 2

You're given a "real" dataset obtained from some real system or somehow produced by real humans. There is not just some cleaning to do, but questions of interpretation, data transformations, maybe even some feature engineering. The machine learning task is still clear though.

Difficulty 3

A real dataset but with a less well-defined machine learning task. This could be an unsupervised problem, where part of the work is in interpreting the outcomes, or it could just be unclear whether you want a classification or a regression approach. For example, a dataset with sales figures where it's unclear if you should predict sales to the nearest unit or just which group the sales belong to (in terms of size).

Difficulty 4

The "real world problem". You're given a broad task to solve but no data, or a dataset that clearly needs enhancing somehow. This task would involve scraping or just trawling the net for open data, then defining your features and deciding on how best to frame it as a machine learning problem.

For the last type of task to work though, I'd argue the teaching has to be interactive. It's much easier to evaluate your approach by getting qualitative feedback, unlike the first 2-3 tasks where you can objectively measure your success.

This isn't always a practical way to teach, but it would go a long way in preparing people for the reality of data science in the wild.

"Intuition First" Machine Learning

2016-11-07T09:34:00+00:00

I read an article on Machine Learning Mastery recently about getting started with machine learning.

The reason it stuck in my mind was that it discussed a way of learning that I always felt was closer to me and a better way for me to learn, yet I hadn't encountered many resources that teach machine learning that way.

The article calls this approach "top down". I refer to the same approach as "intuition first".

The problem is that most books and online courses, while technically aimed at people with no background in machine learning, are usually too maths-heavy too soon. Andrew Ng's excellent Coursera machine learning course does a good job of introducing the intuition behind the algorithms without getting into too much detail about the underlying mathematics. However, especially in the chapter about support vector machines, I felt there was still a need to step away from the technical details a bit more.

There is a time and a place for deep dives into algorithms and maths, but it's not at the start of your learning.

I know from experience that if I want to learn a new concept, and the first thing I see is equations I'll be discouraged, because there's too much to understand too soon. By teaching the ideas in an abstract way first, you get a "feel" for the concept before you get into the nitty-gritty implementation details.

Here's the idea of top down machine learning, quoted straight from the article:

We can summarize this top-down approach as follows:

Learn the high-level process of applied machine learning.
Learn how to use a tool enough to be able to work through problems.
Practice on datasets, a lot.
Transition into the details and theory of machine learning algorithms.

I wholeheartedly agree with step 1 being the high-level process. You want to know exactly what you're trying to achieve before you need to formalise it mathematically.

The part I don't agree with is the order of steps 3 and 4. There is a slight danger in doing things that way around.

If you learn the intuition and start practising on datasets without knowing some more details you can get derailed, and it might be difficult to "debug" any problems you encounter. As with any new technique, superficial knowledge of it is fine at the start, but you need to understand it in more depth if you want to actually use it on a real problem.

This is what I propose instead, for an amended version of "top down" machine learning.

Learn the high-level process of applied machine learning.
Try some toy examples, get your hands dirty.
Transition into the details and theory of machine learning algorithms.
Practise on lots of datasets.

Not a big change, but the key difference is that you should understand the theory in more detail before you spend lots of time practising. Trying some toy problems before diving into the theory will help you put the theoretical concepts into context.

Getting exposed to more theory is mostly for practical reasons, so you understand things like:

Normalisation - does my data need to be normalised? If you understand the inner workings of a machine learning algorithm you'll intuitively know the answer rather than having to resort to trial and error.
Metrics - measuring success in machine learning is an important topic and you don't want to be using the wrong metrics to judge how well your algorithm is performing.
Cross-validation - this might be covered in the high-level process but the idea of training and test sets needs to be drilled home before you start getting too deep.

This might not work for everyone, the mathematically-minded could well prefer an approach where they're given the equations first, but I suspect they're the minority. If the imbalance is the way I perceive it, then there does seem to be a disconnect between the way people learn and the way machine learning is often taught.

Self-Organising Maps: In Depth

2016-11-06T19:44:00+00:00

In Part 1, I introduced the concept of Self-Organising Maps (SOMs). Now in Part 2 I want to step through the process of training and using a SOM - both the intuition and the Python code. At the end I'll also present a couple of real life use cases, not just the toy example we'll use for implementation.

The first thing we need is a problem to solve!

I'll use the colour map as the walkthrough example because it lends itself very nicely to visualisation.

Setup

Dataset

Our data will be a collection of random colours, so first we'll artificially create a dataset of 100. Each colour is a 3D vector representing R, G and B values:

import numpy as np

raw_data = np.random.randint(0, 255, (3, 100))

That's simply 100 rows of 3D vectors all between the values of 0 and 255.

Objective

Just to be clear, here's what we're trying to do. We want to take our 3D colour vectors and map them onto a 2D surface in such a way that similar colours will end up in the same area of the 2D surface.

SOM Parameters

Before training a SOM we need to decide on a few parameters.

SOM Size

First of all, its dimensionality. In theory, a SOM can be any number of dimensions, but for visualisation purposes it is typically 2D and that's what I'll be using too.

We also need to decide the number of neurons in the 2D grid. This is one of those decisions in machine learning that might as well be black magic, so we probably need to try a few sizes to get one that feels right.

Remember, this is unsupervised learning, meaning whatever answer the algorithm comes up with will have to be evaluated somewhat subjectively. It's typical in an unsupervised problem (e.g. k-means clustering) to do multiple runs and see what works.

I'll go with a 5 by 5 grid. I guess one rule of thumb should be to use fewer neurons than you have data points, otherwise they might not overlap. As we'll see we actually want them to overlap, because having multiple 3D vectors mapping to the same point in 2D is how we find similarities between our data points.

One important aspect of the SOM is that each of the 2D points on the grid actually represent a multi-dimensional weight vector. Each point on the SOM has a weight vector associated with it that is the same number of dimensions as our input data, in this case 3 to match the 3 dimensions of our colours. We'll see why this is important when we go through the implementation.

Learning Parameters

Training the SOM is an iterative process - it will get better at its task with every iteration, so we need a cutoff point. Our problem is quite small so 2,000 iterations should suffice but in bigger problems it's quite possible to need over 10,000.

We also need a learning rate. The learning rate decides by how much we apply changes to our SOM at each iteration.

If it's too high, we will keep making drastic changes to the SOM and might never settle on a solution.

If it's too low, we'll never get anything done as we will only make very small changes.

In practice it is best to start with a larger learning rate and reduce it slowly over time. This is so that the SOM can start by making big changes but then settle into a solution after a while.

Implementation

For the rest of this post I will use 3D to refer to the dimensionality of the input data (which in reality could be any number of dimensions) and 2D as the dimensionality of the SOM (which we decide and could also be any number).

Setup

To setup the SOM we need to start with the following:

Decide on and initialise the SOM parameters (as above)
Setup the grid by creating a 5x5 array of random 3D weight vectors

Here's the code:

network_dimensions = np.array([5, 5])
n_iterations = 2000
init_learning_rate = 0.01
# establish size variables based on data
m = raw_data.shape[0]
n = raw_data.shape[1]

# weight matrix (i.e. the SOM) needs to be one m-dimensional vector for each neuron in the SOM
net = np.random.random((network_dimensions[0], network_dimensions[1], m))

# initial neighbourhood radius
init_radius = max(network_dimensions[0], network_dimensions[1]) / 2
# radius decay parameter
time_constant = n_iterations / np.log(init_radius)

Those last two parameters relate to the 2D neighbourhood of each neuron in the SOM during training. We'll return to those in the learning phase. Like the learning rate, the initial 2D radius will encompass most of the SOM and will gradually decrease as the number of iterations increases.

Normalisation

Another detail to discuss at this point is whether or not we normalise our dataset.

First of all, SOMs train faster (and "better") if all our values are between 0 and 1. This is often true with machine learning problems, and it's to avoid one of our dimensions "dominating" the others in the learning process. For example, if one of our variable was salary (in the thousands) and another was height (in metres, so rarely over 2.0) then salary will get a higher importance simply because it has much higher values. Normalising to the unit interval will remove this effect.

In our case all 3 dimensions refer to a value between 0 and 255 so we can normalise the entire dataset at once. However, if our variables were on different scales we would have to do this column by column.

I don't want this code to be entirely tailored to the colour dataset so I'll leave the normalisation options tied to a few Booleans that are easy to change.

normalise_data = True

# if True, assume all data is on common scale
# if False, normalise to [0 1] range along each column
normalise_by_column = False

# we want to keep a copy of the raw data for later
data = raw_data

# check if data needs to be normalised
if normalise_data:
    if normalise_by_column:
        # normalise along each column
        col_maxes = raw_data.max(axis=0)
        data = raw_data / col_maxes[np.newaxis, :]
    else:
        # normalise entire dataset
        data = raw_data / data.max()

Now we're ready to start the learning process.

Learning

In broad terms the learning process will be as follows. We'll fill in the implementation details as we go along.

For a single iteration:

Select one of our 3D colour vectors at random from our dataset
Find the neuron in the SOM whose associated 3D vector is closest to our chosen 3D colour vector. At each step, this is called the Best Matching Unit (BMU)
Move the BMU's 3D weight vector closer to the input vector in 3D space
Identify the 2D neighbours of the BMU and also move their 3D weight vectors closer to the input vector, although by a smaller amount
Update the learning rate (reduce it at each iteration)

And that's it. By doing the penultimate step, moving the BMU's neighbours, we'll achieve the desired effect that colours that are close in 3D space will be mapped to similar areas in 2D space.

Let's step through this in more detail, with code.

1. Select a Random Input Vector

This is straightforward:

# select a training example at random
t = data[:, np.random.randint(0, n)].reshape(np.array([m, 1]))

2. Find the Best Matching Unit

# find its Best Matching Unit
bmu, bmu_idx = find_bmu(t, net, m)

For that to work we need a function to find the BMU. It need to iterate through each neuron in the SOM, measure its Euclidean distance to our input vector and return the one that's closest. Note the implementation trick of not actually measuring Euclidean distance, but the squared Euclidean distance, thereby avoiding an expensive square root computation.

def find_bmu(t, net, m):
    """
        Find the best matching unit for a given vector, t, in the SOM
        Returns: a (bmu, bmu_idx) tuple where bmu is the high-dimensional BMU
                 and bmu_idx is the index of this vector in the SOM
    """
    bmu_idx = np.array([0, 0])
    # set the initial minimum distance to a huge number
    min_dist = np.iinfo(np.int).max    
    # calculate the high-dimensional distance between each neuron and the input
    for x in range(net.shape[0]):
        for y in range(net.shape[1]):
            w = net[x, y, :].reshape(m, 1)
            # don't bother with actual Euclidean distance, to avoid expensive sqrt operation
            sq_dist = np.sum((w - t) ** 2)
            if sq_dist < min_dist:
                min_dist = sq_dist
                bmu_idx = np.array([x, y])
    # get vector corresponding to bmu_idx
    bmu = net[bmu_idx[0], bmu_idx[1], :].reshape(m, 1)
    # return the (bmu, bmu_idx) tuple
    return (bmu, bmu_idx)

3. Update the SOM Learning Parameters

As described above, we want to decay the learning rate over time to let the SOM "settle" on a solution.

What we also decay is the neighbourhood radius, which defines how far we search for 2D neighbours when updating vectors in the SOM. We want to gradually reduce this over time, like the learning rate. We'll see this in a bit more detail in step 4.

# decay the SOM parameters
r = decay_radius(init_radius, i, time_constant)
l = decay_learning_rate(init_learning_rate, i, n_iterations)

The functions to decay the radius and learning rate use exponential decay:

\(\sigma_{t} = \sigma_{0} \times \exp(\frac{-t}{\lambda})\)

Where \(\lambda\) is the time constant (which controls the decay) and \(\sigma\) is the value at various times \(t\).

def decay_radius(initial_radius, i, time_constant):
    return initial_radius * np.exp(-i / time_constant)

def decay_learning_rate(initial_learning_rate, i, n_iterations):
    return initial_learning_rate * np.exp(-i / n_iterations)

4. Move the BMU and its Neighbours in 3D Space

Now that we have the BMU and the correct learning parameters, we'll update the SOM so that this BMU is now closer in 3D space to the colour that mapped to it. We will also identify the neurons that are close to the BMU in 2D space and update their 3D vectors to move "inwards" towards the BMU.

The formula to update the BMU's 3D vector is:

\(w_{t+1} = w_{t} + L_{t}(V_{t} - w_{t})\)

That is to say, the new weight vector will be the current vector plus the difference between the input vector \(V\) and the weight vector, multiplied by a learning rate \(L\) at time \(t\).

We are literally just moving the weight vector closer to the input vector.

We also identify all the neurons in the SOM that are closer in 2D space than our current radius, and also move them closer to the input vector.

The difference is that the weight update will be proportional to their 2D distance from the BMU.

One last thing to note: this proportion of 2D distance isn't uniform, it's Gaussian. So imagine a bell shape centred around the BMU - that's how we decide how much to pull the neighbouring neurons in by.

Concretely, this is the equation we'll use to calculate the influence \(i\):

\(i_{t} = \exp(-\frac{d^{2}}{2\sigma_{t}^{2}})\)

where \(d\) is the 2D distance and \(\sigma\) is the current radius of our neighbourhood.

Putting that all together:

def calculate_influence(distance, radius):
    return np.exp(-distance / (2* (radius**2)))

# now we know the BMU, update its weight vector to move closer to input
# and move its neighbours in 2-D space closer
# by a factor proportional to their 2-D distance from the BMU
for x in range(net.shape[0]):
    for y in range(net.shape[1]):
        w = net[x, y, :].reshape(m, 1)
        # get the 2-D distance (again, not the actual Euclidean distance)
        w_dist = np.sum((np.array([x, y]) - bmu_idx) ** 2)
        # if the distance is within the current neighbourhood radius
        if w_dist <= r**2:
            # calculate the degree of influence (based on the 2-D distance)
            influence = calculate_influence(w_dist, r)
            # now update the neuron's weight using the formula:
            # new w = old w + (learning rate * influence * delta)
            # where delta = input vector (t) - old w
            new_w = w + (l * influence * (t - w))
            # commit the new weight
            net[x, y, :] = new_w.reshape(1, 3)

Visualisation

Repeating the learning steps 1-4 for 2,000 iterations should be enough. We can always run it for more iterations afterwards.

Handily, the 3D weight vectors in the SOM can also be interpreted as colours, since they are just 3D vectors just like the inputs.

To that end, we can visualise them and come up with our final colour map:

A self-organising colour map

None of those colours necessarily had to be in our dataset. By moving the 3D weight vectors to more closely match our input vectors, we've created a 2D colour space which clearly shows the relationship between colours. More blue colours will map to the left part of the SOM, whereas reddish colours will map to the bottom, and so on.

Other Examples

Finding a 2D colour space is a good visual way to get used to the idea of a SOM. However, there are obviously practical applications of this algorithm.

Iris Dataset

A dataset favoured by the machine learning community is Sir Ronald Fisher's dataset of measurements of irises. There are four input dimensions: petal width, petal length, sepal width and sepal length and we could use a SOM to find similar flowers.

Applying the iris data to a SOM and then retrospectively colouring each point with their true class (to see how good the SOM was at separating the irises into their distinct categories) we get something like this:

150 irises mapped onto a SOM, coloured by type

This is a 10 by 10 SOM and each of the small points is one of the irises from the dataset (with added jitter to see multiple points on a single SOM neuron). I added the colours after training, and you can quite clearly see the 3 distinct regions the SOM has divided itself into.

There are a few SOM neurons where both the green and the blue points get assigned to, and this represents the overlap between the versicolor and virginica types.

Handwritten Digits

Another application I touched on in Part 1 is trying to identify handwritten characters.

In this case, the inputs are high-dimensional - each input dimension represents the grayscale value of one pixel on a 28 by 28 image. That makes the inputs 784-dimensional (each dimension is a value between 0 and 255).

Mapping them to a 20 by 20 SOM, and again retrospectively colouring them based on their true class (a number from 0 to 9) yields this:

Various handwritten numbers mapped to a 2D SOM.

In this case the true classes are labelled according to the colours in the bottom left.

What you can see is that the SOM has successfully divided the 2D space into regions. Despite some overlap, in most cases similar digits get mapped to the same region.

For example, the yellow region is where the 6s were mapped, and there is little overlap with other categories. Whereas in the bottom left, where the green and brown points overlap, is where the SOM was "confused" between 4s and 9s. A visual inspection of some of these handwritten characters shows that indeed many of the 4s and 9s are easily confused.

Self-Organising Maps: An Introduction

2016-11-05T18:26:00+00:00

Introduction

When you learn about machine learning techniques, you usually get a selection of the usual suspects. Something like: Support Vector Machines, decision trees/random forests, and logistic regression for classification, linear regression for regression, k-means for clustering and perhaps PCA for dimensionality reduction.

In fact, KDNuggets has a good post about the 10 machine learning algorithms you should know.

If you want to learn about machine learning techniques, you should start there. The point is, on the subject of these algorithms the internet has you covered.

In this post I want to talk about a less prevalent algorithm, but one that I like and that can be useful for different purposes.

It's called a Self-Organising Map (SOM).

Brief History

SOMs are a type of artificial neural network. Some of the concepts date back further, but SOMs were proposed and became widespread in the 1980s, by a Finnish professor named Teuvo Kohonen. Unsurprisingly SOMs are also referred to as Kohonen maps.

Artificial Neural Networks

Artifical neural networks (ANNs) were designed initially to be a computational representation of what is believed to happen in the brain. The way signals are passed along an ANN is based on how signals pass between neurons in the brain.

ANNs are constructed as a series of layers of connected nodes. The first layer consists of your inputs, the last layer consists of your outputs, and there are any number of so-called hidden layers in between.

Neural network image from Wikipedia: https://commons.wikimedia.org/wiki/File:Colored_neural_network.svg

The broad idea of an ANN is that you give it a dataset and a set of desired outputs, and it learns to map the inputs to the outputs. A classic example is teaching an ANN to recognise handwritten characters by giving it pixel values as inputs and the correct digit (say a number from 0-9) as the output.

During the training phase it learns the associations between pixel values and the digits. Then, you can give it a new set of inputs, digits it hasn't seen before, and it will be able to recognise them.

Here is such a system recognising characters in real time. It was built by Yann LeCun in the 1990s.

The way most ANNs "learn" a particular problem is by error-correcting. That is, during the training phase they adapt and improve based on the errors they make, and incrementally get better at solving the problem.

This is a supervised machine learning problem because you are telling the algorithm the desired answer for each set of inputs it's trained on, so it knows if it makes errors.

The SOM as an ANN

There are three main ways in which a Self-Organising Map is different from a "standard" ANN:

A SOM is not a series of layers, but typically a 2D grid of neurons
They don't learn by error-correcting, they implement something called competitive learning
They deal with unsupervised machine learning problems

Competitive learning in the case of a SOM refers to the fact that when an input is "presented" to the network, only one of the neurons in the grid will be activated. In a way the neurons on the grid "compete" for each input.

The unsupervised aspect of a SOM refers to the idea that you present your inputs to it without associating them with an output. Instead, a SOM is used to find structure in your data.

What is a SOM used for?

This last point about unsupervised learning brings me to an important question, because abstract concepts like neural networks are great to talk about but I'm a practical kind of guy.

In that spirit then, what is a SOM used for?

Finding Structure

A classic example of what clustering algorithms are used for is finding similar customers in your customer base. SOMs can also do this. In fact, a SOM is meant to be a 2D representation of your multi-dimensional dataset. In this 2D representation, each of your original inputs, e.g. each of your customers, maps to one of the nodes on the 2D grid. Most importantly, similar (high-dimensional) inputs will map to the same 2D node, or at least the same region in 2D space. This is how the SOM finds and groups similar inputs together.

Dimensionality Reduction

Related to finding structure is the fact that by finding this structure a SOM finds a lower-dimensional representation of your dataset while preserving the similarity between your records.

That is, data points that are "nearby" in high-dimensional space will also be nearby in the SOM.

Visualisation

By creating a (typically) 2D representation of your dataset you can also more easily visualise it, which you can't do if your data has more than 3 dimensions.

Summary

To summarise, I'll quote an answer I gave on StackOverflow to a question about SOMs:

The idea behind a SOM is that you're mapping high-dimensional vectors onto a smaller dimensional (typically 2D) space. You can think of itas clustering, like in K-means, with the added difference that vectors that are close in the high-dimensional space also end up being mapped to nodes that are close in 2D space.

SOMs therefore are said to "preserve the topology" of the original data, because the distances in 2D space reflect those in the high-dimensional space. K-means also clusters similar data points together, but its final "representation" is hard to visualise because it's not in a convenient 2D format.

A typical example is with colours, where each of the data points are 3D vectors that represent R,G,B colours. When mapped to a 2D SOM you can see regions of similar colours begin to develop, which is the topology of the colour space.

Colours

I hope that sounds interesting, because in Part 2 of this post I'll discuss some concrete examples and walk through a Python implementation of Self-Organising Maps.

The example we'll be working with is using a 3D dataset of colours (where the 3 dimensions are R, G and B) and producing a 2D SOM where we visualise the "topology" of the 3D colour space.

Something like this:

A self-organising colour map

In Part 2, we'll look at an in-depth implementation of SOMs.

The Junk in Fallout 4 - a Web Scraping Tutorial

2016-11-03T22:36:00+00:00

This post originally appeared on my blog in 2016

Introduction

The Problem

A few months ago I was playing one of my favourite games of 2015 - Fallout 4.

If you haven't heard of it, it's an action-RPG set in a post-apocalyptic world. The story is engaging and you're sort of trying to rebuild civilisation after the world has been devastated by nuclear war. I mean "rebuild" quite literally - a lot of the game is spent scavenging for raw materials so you can build pretty shoddy housing for strangers. And that rebuilding requires raw materials, which is contained mostly in junk.

Lots and lots of junk.

You can end up spending a disproportionately long time looking for, buying and selling things like broken telephones, toy rocketships, teddy bears, and ashtrays all because they contain precious raw materials for your resettlement project, or for enhancing your guns and armour.

If you play it long enough eventually you start remembering which items contain which raw material (information which then replaces more important things in your brain...). However, not all items contain the same amount of raw material. For example aluminium, which is quite useful but relatively rare, is in oil cans (which contain 1, erm, "unit" of aluminium) but it's better to get it from surgical trays, which contain 3.

That means if you want to maximise your scavenging efforts you have to start remembering which item contains which material and how much it contains.

So I thought: is there a more efficient way of scavenging?

The answer is yes.

With data.

The solution

What I really wanted to do is look at each material, and figure out which items it made sense to find to maximise the amount of the material I get. The plan was:

Find a list of all the items in Fallout 4 and the materials they contain
Scrape and store the list
Analyse it and make some plots

The Tutorial

Prerequisites

First of all, we need some data and some Python libraries.

The data comes from a webpage of all Fallout 4 junk items, specifically the table marked "Other junk items".

The Python libraries we need are:

requests - for fetching the webpage content via HTTP
BeautifulSoup - for reading, parsing and extracting data from the returned HTML
pandas - the usual suspect for manipulating data
matplotlib - for plotting

Step 1 - Fetch the Data

Side note: This isn't a tutorial about HTML, CSS or Javascript, so I'll assume you know enough about them. However, if you want to learn/brush up, w3schools is a great resource.

We've identified our data source and it's already in a table format, which will make things easier. What we want to do is isolate the table in the HTML so we can extract only that table.

The easiest way to do this is directly on the website.

Right click and choose "Inspect Element".

I'm using Firefox but this should be applicable to any browser.

This will open the Inspector, or Chrome Dev Tools or the equivalent, where we can inspect the raw HTML. Then we want to look at the <table> element to see its ID.

Unlucky, no ID.

OK, well it doesn't have an ID, so we need to find a way to uniquely select that element. It has a class "va-table-center" - we can check if that's unique on the page. To do this, go into the Console tab of the Inspector (in the above screenshot it's the second tab, but your browser may differ). In the console you can type and evaluate arbitrary Javascript/jQuery expressions, so let's try selecting a table with the class va-table-center.

Apparently a script on the page finished 'shamefully'...

It looks like that class isn't unique, there are 5 tables that match it on the page. What we can do is then figure out which one of those 5 is the one we need. We can expand the output on the console to help us find it.

You can highlight each of the 5 items and the corresponding table will be highlighted on the page. Doing this reveals we're after the second of those 5 tables.

Note the Halloween-themed adverts about The Exorcist.

Great, now we've identified the HTML element we want to load in, and we can do the rest of the wrangling in Python.

Step 2 - Clean and Wrangle

Using requests and BeautifulSoup we read in the HTML and grab just the table we need.

import requests
from bs4 import BeautifulSoup

# load the entire page
req = requests.get('http://fallout.wikia.com/wiki/Fallout_4_junk_items')

# read the output as text
raw = req.text

# load it into BeautifulSoup for parsing
soup = BeautifulSoup(raw, "html.parser")

# select just the second table with the right class
junk_table = soup.select("table.va-table-center")[1]

Next step is to iterate over the table rows and extract what we need. For now, we'll focus on the item name and its components.

There are often multiple materials in a single item, so we need to extract them individually. What you'll notice in the table is that each new material starts with a capital letter, so we can split on that with some regex. Then we'll create a dictionary where the key is the item and the value is a list of its components.

junk_dict = {}

# start from the second table row (to skip the header)
for j in junk_table.findAll('tr')[1:]:
    # select all the table cells
    cells = j.select('td')
    # get item name and components cells
    item_name = cells[0].text.strip()
    components_cell = cells[5]
    # split components text at uppercase letters
    components = [x.strip() for x in re.findall('[A-Z][^A-Z]*', components_cell.text)]
    junk_dict[item_name] = components

Next we'll create a pandas DataFrame from the dictionary. The only quirk is some components have "x2" style additions if there are more than one units in the item, and the absence of this signifies 1 unit.

df = pd.DataFrame(columns = ('item', 'component', 'quantity'))
row_idx = 0

for k in junk_dict:
    for c in junk_dict[k]:
        quantity = 1 # default unless otherwise specified
        # extract the multiplier
        multiplier = re.search(' ?x ?([0-9]*)', c)
        component_name = c
        # if the multiplier is specified set the quantity to the right value
        if multiplier:
            if multiplier.group(1) != '':
                quantity = int(multiplier.group(1))
                component_name = c[:-3]
        # add as a row
        df.loc[row_idx] = [k, component_name.strip(), quantity]
        row_idx += 1

Step 3 - Plot

We now have our very own Fallout 4 junk dataset that we can analyse to our heart's content. For example we can plot the frequencies of each component to order them by rarity.

It's a strange world where concrete is so rare.

Conclusion

There you have it - data-driven video gaming. Here's the whole thing in Jupyter notebook form, and a direct link to the csv file if you want to analyse it yourself - if you do, let me know!

Post-script

Since writing this tutorial (in 2016) I discovered pandas has a read_html() method, which scans an entire HTML document and extracts all <table> elements as pandas DataFrames. You should definitely use that instead of looping over table rows and cells, but I'm keeping this tutorial as it was, since it's still instructive.

Analysing London House Prices

2016-10-23T16:33:00+01:00

This post originally appeared on my blog in 2016

Introduction

London is expensive. So much so that it's a trope now for those that live there. But what does the data show? Are things getting better or worse? How did the 2008 recession affect buying behaviour for example?

I wanted to find out. With data.

There are many directions you could go with a dataset of London house prices, so I focused on answering specific questions. I wanted to know:

Can the recession, centred around 2008, be seen in the house price data?
Which boroughs are on the rise economically, and which are exhibiting a downward turn?
How have housing prices changed over time overall?
Does the overall trend match the individual trend in each borough?
Which boroughs are similar in their economic trends?

The Data

The dataset was obtained from an online service called Enigma, which requires a free registration to access the data. It contained the address of each house and the postcode, but not latitude-longitude values. I could have geocoded each address individually to get accurate lat-long values, but life's too short and postcodes were granular enough. I added lat-long values by joining a geocoded postcode dataset.

The final housing price dataset consisted of 16 columns and around 1,500,000 rows. The main columns of interest were:

Date of Sale (to the nearest day)
Postcode (now with latitudes and longitudes)
Local Authority (i.e. borough)
Price (in GBP)
Property Type (Detached, Semi, Terraced, or Flat/Maisonette)
Build (Old or New)

The data covers the period between 1996 and 2014, so this post won't turn into a digression about Brexit, although that will be an interesting dataset in a few years.

There were a few boroughs with only a handful of rows in the dataset. One of them for example was a Welsh address, incorrectly labelled as 'London'. I removed 5 boroughs in total because they were either outside of London, or contained less than 20 examples.

The Tools

The data wrangling was done in Python and all the visualisations were created with Tableau.

The Analysis

Price Distribution

The very first question I wanted to answer is what the distribution of prices is like.

The median house price is £320,000 and the upper whisker of the box plot is £1.26million. After that there are over 39,000 outliers with the highest house price at over £50 million. This suggests there may be two tiers of house prices in London: one for the 'average' person and one for the more wealthy.

Heavy right tails make box plots look ridiculous

At this point my hunch was that the buying patterns of people who can afford to pay millions for a house will be different to people buying houses for around the median price. To this end I created an "outliers" category and analysed those nearly 40,000 rows separately.

Prices over Time

So how have prices changed over the years in the "normal" and "outlier" categories?

At this point it might be cheaper to invest in a time machine and go back to 1996 to buy a house

It turns out prices have steadily been climbing for both categories, but the number of sales dipped during the 2008 recession, but only for the non-outlier category. That makes sense - you'd expect the most wealthy to be unaffected by a recession.

What about each borough?

Does this overall trend look the same in each borough? Let's find out. For borough-level figures I looked at the non-outlier category to make the analysis more relatable.

Some boroughs have a small dip but even a recession couldn't make London more affordable

There is a more noticeable, but still very small, dip during the recession years (highlighted in dark red) but prices are climbing in all London boroughs. The dip in sales during the recession is also present in each borough, echoing the overall trend.

This graph would probably be a worrying EKG

Buying Behaviour

Is buying a house seasonal or do people buy with the same frequency throughout the year?

Apparently people rarely buy houses as Christmas presents.

The slight peak in March is interesting.

After I showed this to a few people, someone suggested it's because the tax year in the UK ends in April so people might end up buying with some urgency to keep the spending in the previous tax year. This general seasonal behaviour is also apparent in each borough.

Is it still a heatmap if it's blue?

It looks like some of the boroughs have a bigger peak in March than others. Can we quantify which boroughs have this behaviour and which don't?

Clustering

How do we find similar boroughs? We can make use of clustering.

First, we aggregate each borough into a 12-dimensional vector, where each item corresponds to the median percentage of total sales in that particular month. So in a slightly extreme example, a borough with half of all its sales in January and the other half in July would look like this:

[0.5, 0, 0, 0, 0, 0, 0.5, 0, 0, 0, 0, 0]

We need to use the percentage rather than actual values to ensure all boroughs are on a common scale. Otherwise, we would end up clustering boroughs together simply on sales volume rather than buying behaviour.

After trying multiple values of k in the k-means clustering algorithm, it looked like there were two distinct clusters with three outliers. Those outliers were boroughs that had much fewer rows in the dataset, so they're arguably not that representative.

It appears then that there are really just two types of behaviour - boroughs that have a prominent peak in sales in March and those that don't. But which boroughs are which? Let's colour a map of London by cluster:

Interestingly the different seasonal behaviour almost neatly splits London into East and West. The dataset unfortunately didn't have enough details to give any suggestions about whether this is a coincidence.

Conclusion

Unsurprisingly, the data confirms what we all complain about - buying in London is expensive and it is only getting worse.

The recession impacted the average person; the number of sales dipped dramatically between 2007 and 2009. However, the rich were unaffected.

Buying behaviour is seasonal peaking in July, with some boroughs displaying a small peak in March (possibly for tax purposes). These boroughs are typically in East London, which may or may not be a coincidence.

The next analysis on my list might be finding a new, more affordable city!