Skip Social Crawler Tracking

The "Skip social crawler tracking" feature allows you to control whether visits from social media crawlers are recorded in your analytics. When enabled, social media platforms can still access your links (unless blocked), but their visits won't appear in your click statistics.

This helps keep your analytics focused on real human visitors while still allowing social platforms to generate link previews and share cards.

Social media crawlers are automated bots used by platforms to:

• Generate link previews when you share a URL
• Extract Open Graph metadata and images
• Create rich social share cards
• Verify link safety and content

The following platforms are recognized as social media crawlers:

• Facebook (FacebookExternalHit, Facebot)
• Twitter (Twitterbot)
• LinkedIn (LinkedIn Corporation)
• Google (Googlebot, APIs-Google, AdsBot-Google)
• YouTube (Youtube, LLC)

Social media crawler visits link
    ↓
Link forwards to destination
    ↓
Click is recorded in database
    ↓
Appears in your analytics dashboard
    ↓
Counts toward total clicks
    ↓
Visible in "Robots" tab

Result:

• ✅ Social media crawlers can access the link
• ✅ Clicks are recorded in analytics
• ✅ You can see all social crawler activity
• ⚠️ Analytics include both human and bot traffic

Social media crawler visits link
    ↓
Link forwards to destination
    ↓
Click is NOT recorded in database
    ↓
Does not appear in analytics
    ↓
Does not count toward total clicks

Result:

• ✅ Social media crawlers can still access the link
• ✅ Link previews work on social platforms
• ✅ Analytics show only human visitors
• ✅ Reduced database usage
• ❌ No visibility into social crawler activity

• Skip social crawler tracking: ❌ Disabled

Use case: You want complete visibility into all traffic, including when social platforms fetch your links.

Benefits:

• See when Facebook/Twitter/LinkedIn fetch your links
• Verify that social previews are working
• Track all traffic sources including bots
• Useful for debugging social sharing issues

Drawbacks:

• Analytics include non-human traffic
• Total clicks include bot visits
• Larger database usage

• Skip social crawler tracking: ✅ Enabled

Use case: You want analytics focused only on real human visitors.

Benefits:

• Cleaner analytics showing only human clicks
• More accurate conversion metrics
• Reduced database storage
• Faster analytics queries
• No need to filter out bot traffic

Drawbacks:

• No visibility when social platforms fetch links
• Can't verify social preview generation
• Can't track which platforms are sharing your content

The "Skip social crawler tracking" feature works together with "Block bots & spiders":

• Block bots: ❌ Disabled
• Skip social crawler tracking: ❌ Disabled

Result:

• ✅ Social crawlers can access links
• ✅ Clicks are recorded in analytics

• Block bots: ❌ Disabled
• Skip social crawler tracking: ✅ Enabled

Result:

• ✅ Social crawlers can access links
• ❌ Clicks are NOT recorded in analytics

• Block bots: ✅ Enabled
• Skip social crawler tracking: ❌ Disabled

Result:

• ❌ Social crawlers are blocked (403 error)
• ❌ No clicks to record

• Block bots: ✅ Enabled
• Skip social crawler tracking: ✅ Enabled

Result:

• ✅ Social crawlers can access links (exception to bot blocking)
• ❌ Clicks are NOT recorded in analytics
• ❌ Other bots are still blocked

Use case: Block search engines and spam bots, allow social sharing with previews, but keep analytics clean.

Social media crawlers are identified by:

1. User Agent String: Matching against known crawler signatures

   • Example: facebookexternalhit/1.1
   • Example: Twitterbot/1.0

2. ISP Name: Matching against known platform ISPs (via GeoIP lookup)

   • Example: ISP = "Facebook"
   • Example: ISP = "LinkedIn Corporation"

If either the user agent OR the ISP matches a known social media platform, the visit is treated as a social crawler.

1. Request arrives at your short link
2. Link validation (spam check, expiry, etc.)
3. Bot blocking check (if enabled)
   ├─ Is it a social crawler?
   │  ├─ Yes + skip_social_crawler_tracking enabled → Allow through
   │  └─ Yes + skip_social_crawler_tracking disabled → Block (403)
   └─ Is it another bot? → Block (403)
4. Click recording decision
   ├─ Is skip_social_crawler_tracking enabled AND is social crawler?
   │  └─ Yes → Skip recording
   └─ Otherwise → Record click
5. Redirect to destination URL

With tracking disabled:

• No database INSERT operation for social crawler visits
• No BigQuery write for analytics
• Reduced server load on high-traffic links
• Faster response time (by ~10-20ms per social crawler visit)

Database savings example:

• Link shared on Facebook with 10,000 followers
• Facebook may fetch the link 5-10 times for preview generation
• Without skip tracking: 5-10 database records
• With skip tracking: 0 database records

• Disabled: Includes social crawler visits
• Enabled: Excludes social crawler visits

• Disabled: Shows spikes when social platforms fetch links
• Enabled: Shows only human visitor patterns

• Disabled: Lists all social crawler visits with platform names
• Enabled: Social crawlers don't appear (not recorded)

When "Skip social crawler tracking" is enabled, the "Filter Robots" button becomes less necessary since social crawler traffic is already excluded from your analytics.

Scenario: You're running a social media marketing campaign and want accurate ROI metrics.

Recommendation: Enable "Skip social crawler tracking"

Why:

• Focus analytics on actual customer clicks
• Accurate conversion tracking
• Clean funnel metrics
• Better understanding of campaign performance

Scenario: You regularly share links on Facebook, Twitter, and LinkedIn.

Recommendation: Enable "Skip social crawler tracking"

Why:

• Social platforms can still generate previews
• Your click counts reflect real people, not bots
• Easier to measure engagement
• No need to manually filter bot traffic

Scenario: You're troubleshooting why social previews aren't working correctly.

Recommendation: Disable "Skip social crawler tracking" temporarily

Why:

• See when Facebook/Twitter fetch your links
• Verify that crawlers can access your content
• Check timing and frequency of crawler visits
• Diagnose Open Graph metadata issues

Scenario: You have a viral link with millions of clicks.

Recommendation: Enable "Skip social crawler tracking"

Why:

• Reduce database load
• Lower BigQuery costs
• Faster analytics queries
• Focus on actual user engagement

Scenario: Sharing links within your organization or to a small private group.

Recommendation: Enable both "Block bots" and "Skip social crawler tracking"

Why:

• Block search engines from indexing
• Allow social previews for team members
• Keep analytics clean
• Maintain link privacy

1. Enable for Most Links: For typical use cases, enabling skip tracking provides cleaner analytics without downsides

2. Disable for Debugging: Only disable when actively troubleshooting social sharing issues

3. Consider Your Goals:

   • Want to track everything? → Disable skip tracking
   • Want clean analytics? → Enable skip tracking

4. Use with Bot Blocking: Combine both features for maximum control over bot access and analytics

5. Monitor Your Robots Tab: Periodically check what bots are visiting to understand your traffic

• Block Bots & Spiders - Prevent bots from accessing your links
• Blocking Social Media Crawlers - Detailed guide on bot blocking and social crawler exceptions
• Analytics Filtering - Filter bot traffic from analytics views

Need help configuring social crawler tracking?

• Visit our support documentation
• Contact support at [email protected]
• View the list of detected social media crawlers