Blocking Social Media Crawlers

When you enable the "Block bots & spiders" feature on your links, you can control whether social media crawlers are allowed to access your links. This guide explains how the feature works and how to configure it for your needs.

Social media crawlers are automated bots sent by platforms like Facebook, Twitter, LinkedIn, Google, and YouTube to:

• Generate link previews and thumbnails
• Extract Open Graph metadata
• Create social share cards
• Verify link content for safety

Common social media crawlers include:

• Facebook: FacebookExternalHit, Facebot
• Twitter: Twitterbot
• LinkedIn: LinkedIn Corporation
• Google: Googlebot, APIs-Google, AdsBot-Google
• YouTube: Youtube, LLC

When you enable "Block bots & spiders" on a link:

1. All known bots and spiders are blocked with a 403 Forbidden error
2. This includes search engines, social media crawlers, and other automated traffic
3. Only regular human users can access the link

By default, blocking bots will also block social media crawlers. However, you can allow social media crawlers while still blocking other bots by enabling the "Skip social crawler tracking" option.

• Block bots & spiders: ✅ Enabled
• Skip social crawler tracking: ❌ Disabled

Result:

• ❌ All bots are blocked (403 error)
• ❌ Social media crawlers are blocked
• ❌ Social platforms cannot generate link previews
• ✅ Maximum protection from automated traffic

Use case: Private links, internal tools, or links you don't want indexed or previewed anywhere.

• Block bots & spiders: ✅ Enabled
• Skip social crawler tracking: ✅ Enabled

Result:

• ❌ Search engines and other bots are blocked (403 error)
• ✅ Social media crawlers can access the link
• ✅ Social platforms can generate link previews
• ⚠️ Social crawler visits are NOT recorded in analytics

Use case: Links shared on social media that need previews, but you want to block search engines and other bots.

• Block bots & spiders: ❌ Disabled
• Skip social crawler tracking: (any setting)

Result:

• ✅ All traffic is allowed
• ✅ All bot visits are recorded in analytics (unless skip_social_crawler_tracking is enabled)

Use case: Public links, marketing campaigns, content meant to be widely accessible.

Social media crawlers are identified by two methods:

The bot's user agent string is checked against a list of known social media crawler patterns. For example:

• facebookexternalhit
• Twitterbot
• LinkedInBot

The request's ISP (Internet Service Provider) is checked using GeoIP lookup. For example:

• ISP: "Facebook"
• ISP: "LinkedIn Corporation"
• ISP: "Google"
• ISP: "Youtube, LLC"

If either the user agent OR the ISP matches a known social media platform, the request is treated as a social media crawler.

• Social media crawler visits are recorded in your analytics
• These visits appear in the "Robots" tab with their bot names
• They count toward your total clicks
• You can filter them out using the "Filter Robots" button

• Social media crawler visits are NOT recorded in your analytics
• No database record is created
• They don't count toward your total clicks
• This reduces database usage and keeps analytics focused on human visitors

Problem: You enable "Block bots & spiders" but your social media posts don't show previews.

Solution: Enable "Skip social crawler tracking" to allow social platforms to access your link and generate previews.

Problem: You want to prevent Google/Bing indexing but allow social sharing.

Solution: Enable both "Block bots & spiders" and "Skip social crawler tracking". Search engines will be blocked, but social media crawlers will be allowed.

Problem: You want complete analytics data including all bot traffic.

Solution: Disable "Block bots & spiders". All traffic will be allowed and recorded. Use the "Filter Robots" button in analytics to view human-only traffic when needed.

Problem: You need maximum security and don't want any automated access.

Solution: Enable "Block bots & spiders" and disable "Skip social crawler tracking". All bots including social media crawlers will be blocked.

When a bot is blocked, it receives:

• HTTP Status Code: 403 Forbidden
• Error Page: Custom bot_blocked.html template
• Information: The bot name that was detected

In addition to bot detection, the "Block bots & spiders" feature also blocks traffic from known cloud hosting providers (AWS, DigitalOcean, etc.) as these are often used for automated traffic or scraping.

• Bot detection is performed on every request when "Block bots & spiders" is enabled
• The check happens before analytics recording, minimizing database load
• GeoIP lookup is cached for performance

1. For Marketing Links: Disable bot blocking or enable skip_social_crawler_tracking to ensure social previews work
2. For Internal Tools: Enable bot blocking without social crawler exception for maximum security
3. For Public Content: Allow all bots to maximize discoverability and reach
4. For Analytics Accuracy: Enable skip_social_crawler_tracking to keep your metrics focused on human visitors

For additional help or questions about bot blocking:

• Visit our support documentation
• Contact support at [email protected]
• View the full list of detected bots