As SEOs, it’s our job to drive traffic online by developing strategies informed by data. But what happens when the data doesn’t tell the full story?
If you look at your Google Analytics data, over time, you will probably have noticed a steady increase in direct traffic to your site. Have you ever wondered why this is the case?
The answer: Dark traffic.
What exactly is dark traffic? How does dark traffic affect my site?
The technical explanation of the term is… any traffic that arrives at a site without a referral URL from where the source of the traffic can be traced. In laymen’s terms Google has no idea where the traffic has come from so attributes the sessions to direct traffic.
It is thus interpreted as ‘direct’ as it ostensibly looks like the user has typed in the URL directly, when it’s actually the result of organic search or a social share of some kind.
It comes in many forms, but it’s not as dastardly as its name suggests. In fact, it can be beneficial to both brands and marketers alike if utilised correctly.
Today on the blog, we’re going to tell you all about dark social & dark search.
What is Dark Social?
The term ‘dark social’ was introduced into the digital lexicon in 2012 by tech journalist Alexis C. Madrigal, with his seminal article Dark Social: We Have the Whole History of the Web Wrong. For those of us in the SEO field, it’s always been on the radar, but Madrigal gave it a name, and henceforth, it has been known as dark social.
Basically, it’s when traffic arrives at your site looking like it has just shown up directly but has rather arrived through a social share of some kind.
You might not know it but we’ve all engaged in the not-so-dark arts of dark social at some point. You could be doing it every day and not even realise. It happens when someone copy and pastes a link and sends it directly to someone else via any private communication tool. So, whether it’s by a colleague sending you a link over Slack, a friend sending you a link over WhatsApp, or your Dad sending you a link via text, you’ve more than likely dabbled in dark social.
Me Sending Whatsapp Message to Team Member:
Team Member Receiving Whatsapp Message:
Google Analytics Thought Process: No Referrer? Put in Direct!
Dark Social Definitions and Key Terms
So how does it all work in practice? First, let’s define a couple of key terms.
Direct traffic is when traffic arrives at a site without a referrer. For example, if I were to open up a new tab in my browser and type in www.boyddigital.co.uk, this would be correctly be identified to analytics tools as direct traffic, as there is no referral link.
Back when the Internet was a smaller, simpler place, you arrived at a link one of basically three ways; Search, link, or typing the URL into your browser directly. However, there are a multitude of other ways to arrive at a URL through and most ‘direct traffic’ isn’t direct at all, but is navigated through a combination of multiple platforms, devices, apps, and touch points.
A referral URL is the URL which was visited before arriving as the current address. For example, if I open up a new browser and do a Google search for Boyd Digital, then click on the search result for www.boyddigital.co.uk, the referral URL would be Google as I have used Google to reach our site.
A Practical Explanation of Dark Traffic and its Effects
Now, let’s give a practical explanation of how it can work.
Most analytics platforms break down traffic into categories which indicates where the traffic is coming from. These categories are:
- Organic
- Direct
- Referrals
- Social Media
- Paid
- Campaigns
For example, if someone clicks on the link to this blog post via Twitter, our analytics platform will be able to tell where that user came from because the referral URL will come with a message which essentially says ‘Hey, we came from Twitter,’ and this will be categorised under ‘social media’.
However, if someone arrived at this same blog post via an IM (instant messenger) from a friend, this will be identified as ‘direct traffic’ as there is no message from a referrer with information on where they came from, and the analytics platform won’t be able to tell the difference.
With short URLs, it’s pretty straight forward. If it looks like direct traffic, and it sounds like direct traffic; it’s direct traffic.
However, when direct traffic arrives at longer URLs, things get a little murkier.
Let’s say I check my analytics tool, and in the past hour, I have had 10 hits to the following URL as the result of direct traffic:
(https://www.boyddigital.co.uk/news/boyd-digital-lends-support-think-tanks-boost-start-competition/)
https://www.boyddigital.co.uk/news/boyd-digital-lends-support-think-tanks-boost-start-competition/
Given that this URL is 97 characters long, and no one’s memory is that good; it’s safe to assume that these hits arrived at this URL through a direct link via a social share of some kind.
Seems obvious to us, right? Well, an analytics platform can’t tell the difference.
Additionally, much of dark social shares happen on mobile devices, and mobile search is on the rise with Marshall Simmonds presenting data showing +115% growth in the 18 months between July 2014 and December 2015. This makes sense given that messaging apps are also on the rise, with data showing that over a third of all smartphones users using Facebook Messenger, WhatsApp or both.
It’s no surprise then, that dark social has become so important. Smartphone users send links back and forth all the time, but there is no way to track where they come from, which could have larger implications as you could be left with a large volume of unaccountable data.
What is Dark Search?
Similar to dark social, ‘dark search’ is another term which describes traffic categorised as direct traffic, but is rather the result of organic search by the user.
This could be the result of in-app searches, image searches, or secure searches as these types of searches don’t share any information on where they came from with analytics platforms. Some native search apps, such as the Android search app, report organic search traffic as direct, and as far as analytics is concerned, they’ve just showed up and there’s no good explanation why.
For example, if you use the search feature on your phone to search for ‘Boyd Digital’ you will be presented with a list of suggested websites as illustrated in example below.
If you click through to our site – or in infact, any of these links – your visit will be reported as ‘direct’ even though you have actually conducted a search to find this link.
The Dangers of Dark Search to SEO Reporting
As SEOs, we must be assiduous in our efforts to account for as much natural search traffic as possible. By doing so, we can determine if the strategies we implement are hits or misses. Dark search can be problematic because it can paint an inaccurate picture of the data and, if left unchecked, it can look like SEO is being unsuccessful when the opposite may be true.
This can be exacerbating SEOs, and when you throw dark social into the equation, it accounts for a lot of information which is at best inexplicable, and at worst misleading.
Why Dark Traffic Matters
In his article Revisiting Dark Social, Bob Cohn of the Atlantic offers a logical explanation to why understanding dark traffic is important, saying:
‘We need a clear understanding of where our audiences are coming from. There’s a false sense of security in believing a greater portion of our audience is coming in through typed/bookmarked than is really the case. Our audience, it turns out, may not be loyal repeat visitors after all.
‘On the other hand, if there are more visitors coming in through the social side door – the Facebook/Twitter axis plus everyday dark social – well, that tells us something about the content we are creating: It works! People are sharing it. People want others to see it. In the sharing economy, quality wins. And if that sharing economy is bigger than we realized, quality matters more than ever.’
As SEOs, we need to understand the big picture of ‘big data’ and understanding how to analyse dark traffic can help in determining which variables are driving traffic, and which are not.
For example, if your analytics tools are giving you inaccurate information on how much traffic is arriving at your site directly, you could be overvaluing the importance of direct traffic, and undervaluing other tools which are over performing.
Studies have underlined the importance of both dark social and dark search on current SEO practice, with research by RadiumOne showing that up to 70% of all social shares happen through dark social channels.
To give perspective, this is almost three times as much as social shares that happen through Facebook, which account for only 23% of the total amount of shares. This means that for every one share on Facebook, there are up to three other shares happening through dark social channels. Additionally, the same research shows that 32% of people who share content online do so only through dark social channels.
Additionally, a share over dark social is a direct share, which makes it a more meaningful share. Implicitly, this means that an individual has taken an interest in a topic and has engaged with it. They have made the calculation that someone else may also share an interest in it, and thus, have shared it with them which could lead to a higher conversion rate.
The takeaway from this research is that dark social is too big to ignore.
Groupon, the e-deals and coupons behemoth, conducted a study into dark search which endeavoured to find out what percentage of direct traffic was in fact direct and how big an impact dark search has.
The company deindexed themselves from Google completely for six hours, and analysed traffic to their longer URLs, which they defined as at least being in a subfolder so as to exclude the home page and URLs which consistently get high levels of direct traffic.
They found that during the six hours they ‘went dark’, organic search traffic fell to near zero, and direct traffic fell by 60%, indicating that dark search is responsible for a sizeable percentage of their direct traffic.
With this research in mind, bigger players are starting to recognise the importance of dark search and social.
Business Insider highlighted the increasing importance of messaging apps as content sharing devices in their 2016 Messaging App Report.
They note:
‘Users around the world are logging in to messaging apps to not only chat with friends but also to connect with brands, browse merchandise, and watch content. What were once simple services for exchanging messages, pictures, videos, and GIFs have evolved into expansive ecosystems with their own developers, apps, and APIs.’
Audible, the Amazon-owned Audiobook giant and your favourite podcasts’ sponsor, are also recognising the importance of dark traffic. The company’s Senior Director of Social Media & Digital Communications, Reid Armbruster, recently said: ‘We’re kidding ourselves if we think that most sharing happens out in the open, on social platforms.’
Giants like Adidas are also getting on board and are trying to leverage the power of dark social to their advantage. Senior director of global brand communications for Adidas Football, Florian Alt, told The Drum earlier this year:
‘WhatsApp was specifically chosen as our research shows that consumers already use the app to create their own micro-communities. Adidas wants to be the most personal brand, so we need to know and understand our consumer in order to have a meaningful relationship… There is huge potential in dark social.’
List of Dark Search and Social Sources
We’ve already covered the myriad of dark social and search channels which do not pass on referral data, and as the Groupon example showed us, some organic search traffic will be treated as direct traffic too. However, there are many other reasons in which browsers do not share referral information.
Bookmarks – If traffic arrives at your site through someone clicking on a bookmark, this will be treated as direct traffic.
Native search on mobile – Native search apps on mobile platforms do not come with a referrer, and will be attributed as direct traffic.
Apps – the vast majority of apps
Non-web documents – If you click on an untagged link on a non-web document (such as PDF files, documents, spreadsheets, presentations, etc) this will have no referral data and will be treated as direct traffic.
Browser privacy settings – Private browsing or browsing in ‘incognito’ mode will not pass a referral code.
IOS open in – On, i0S, if you select ‘open in Safari’ or ‘open in ‘Chrome’ then this won’t come with a referrer attached.
HTTPS to HTTP – When going from a secure HTTPS link to a non-secure HTTP link, no referral data is passed.
Desktop email clients – Clicking through an untagged link from a personal-email provider such as Gmail or Outlook will not pass a referral code.
Marketing campaigns with improper tags
IM – Traffic which comes from instant messaging on mobile devices, or desktop apps such as Skype of Google Hangouts, will be treated as direct traffic.
The Dark Elephant in the Room – What Can Be Done About Dark Traffic?
So what can be done about it?
The first thing to do is tag everything with UTM markups, which are simple tags attached to the end of the URL to help specify where they come from. Adding these tags to any links you post on Facebook, Twitter, or e-mails will help identify which campaign is driving the link, which will in turn allow you to determine the effectiveness of each.
Here’s a screenshot of a UTM code in action, showing the traffic which is the result of a click through on the Facebook link:
Additionally, you should consider using URL shorteners such as bit.ly when posting any links, as these will come with referral links.
Bit.ly is great at highlighting dark social traffic when you’re not sure where and how you’re getting traffic. We’ll frequently use it when we’re running any of our BDX events as we know many people will share links via email or direct message
In the case of this example below, just over half of the traffic that would’ve been unaccounted for we knew was dark traffic, which is sort of what we’d want for promoting an event when it relies a lot on word of mouth.
And don’t forget to check your direct traffic in Google Analytics too. Take your data and omit the homepage and any top level pages you can confidently say that a user may have typed in (or had autocomplete for them). Filtering these out should leave you with a number of pages with longer URLs that you would know a user didn’t manually type in word for word. Compare these pages against themselves over time to see if the level of direct traffic to them has increased. You can also check certain pages you’re unsure about against the browser accounting for a session. Ask yourself questions like:
– Are these direct sessions coming from iOS?
– Would someone really type the URL in full on their phone?
– Is my direct Android traffic on a continual rise?
For example, this is a list of pages on site that have been viewed on a mobile device today according to Google Analytics. Notice the URL we’ve highlighted in blue. This is a very long URL and one you wouldn’t ever directly type in word for word.
Now, let’s see if this page has any instances of showing up as direct traffic when we know that the page was accessed via mobile indirectly:
Yep. The session has shown up as direct traffic in Google Analytics even though we know it isn’t.
What Next?
With the rapid growth of private communications apps like Snapchat, WhatsApp, and Facebook Messenger showing no signs of slowing down, it’s unlikely that we will be able to penetrate beyond the tip of the iceberg, and find out where all our traffic is coming from. At least in the short term anyways.
However, as illustrated in this article, there are ways to make sense of the data and leverage the dark side to your advantage.
Here are a few key takeaways to keep in mind as you check your analytics platforms on a day-to-day basis:
– Search and social must work together to produce solid data.
– Google’s data can be opaque.
– As can social data.
– Filter and compare direct traffic to look