[FEATURE] RFC 9111 Compliant HTTP Response Caching

[dwisiswant0]

Summary

Introduces built-in HTTP response caching that follows the RFC 9111 spec. This feature reduces redundant network requests by caching responses based on standard HTTP caching headers (Cache-Control, Expires, ETag, Last-Modified), improving scan perf and reducing load on target servers.

Motivation

When scanning targets with multiple templates, nuclei often makes repeated requests to the same endpoints. Many responses (static assets, API responses, configuration files) are cacheable according to HTTP semantics. By respecting server-provided caching directives, nuclei can:

• Reduce scan time by avoiding redundant network round-trips.
• Lower bandwidth usage for both scanner and target.
• Decrease load on target servers — important for responsible scanning.
• Improve consistency when the same resource is referenced by multiple templates.

Implementation

Library

Uses github.com/sandrolain/httpcache pkg, a fully RFC 9111 compliant HTTP caching library with:

• Cache-Control directive parsing and enforcement.
• Conditional request support (If-None-Match, If-Modified-Since).
• Vary header separation for proper cache key generation.
• Stale-while-revalidate and stale-if-error support.
• Async revalidation for background cache refresh.

Cache Storage

• Primary: LevelDB-backed persistent cache at ~/.cache/nuclei/httpcache/
• Fallback: In-memory cache if LevelDB initialization fails

Caching Policy

Only 2XX responses are cached. This ensures:

• Error responses (4XX, 5XX) are never served from cache.
• Security-relevant responses (redirects, authentication challenges) bypass caching.
• Templates always see fresh error states.

Transport Configuration

transport := &httpcache.Transport{
    Transport:                 rt,
    Cache:                     c,
    MarkCachedResponses:       false,
    SkipServerErrorsFromCache: true,
    AsyncRevalidateTimeout:    10 * time.Second,
    IsPublicCache:             false,
    EnableVarySeparation:      true,
    ShouldCache:               shouldCache,
    CacheKeyHeaders: []string{
        "Authorization",
        "Cookie",
        "Accept-Encoding",
        "Accept-Language",
        "Accept",
        "Origin",
    },
    DisableWarningHeader: true,
}

Usage

HTTP caching is enabled by default. The cache persists across scans in ~/.cache/nuclei/httpcache/.

Cache Behavior Examples

Response Headers	Cached?	Duration
Cache-Control: max-age=3600	✅	1 hour
Cache-Control: no-store	❌	*
Cache-Control: private, max-age=300	✅	5 minutes
Expires: <future date>	✅	Until expiry
No caching headers	❌	*
Status 404	❌	*
Status 500	❌	*

Comparison with -project Flag

The -project flag provides a different caching mechanism that operates at the application layer.

How -project Works

1. Before HTTP request (request.go#L794-L801):

   if request.options.ProjectFile != nil {
       fromCache = true
       resp, err = request.options.ProjectFile.Get(dumpedRequest)
       if err != nil {
           fromCache = false
       }
   }

2. Cache key generation (project.go#L42-L47):

   • Takes raw dumped HTTP request bytes (method, path, headers, body)
   • Strips User-Agent header (regex: (?mi)\r\nUser-Agent: .+\r\n)
   • Strips interactsh markers (regex: (?mi)[a-zA-Z1-9%.]+interact.sh)
   • Computes SHA256 hash of sanitized request

3. After HTTP request (request.go#L946-L949):

   if request.options.ProjectFile != nil && !fromCache {
       request.options.ProjectFile.Set(dumpedRequest, resp, respChain.BodyBytes())
   }

4. Storage: Uses hybrid.HybridMap (disk-backed map) at path specified by -project-path

Key Differences

Aspect	HTTP Cache	Project Cache (-project)
Layer	Transport (http.RoundTripper)	Application (request.go, before/after HTTP call)
Cache Key	URL + Vary headers + CacheKeyHeaders (RFC 9111)	SHA256 of raw request bytes (sanitized)
Key Includes	URL, method, selected headers	Full request: method, path, ALL headers, body
Key Excludes	Body (for GET), non-Vary headers	User-Agent, interactsh markers only
Expiration	Respects Cache-Control, Expires, max-age	Never expires (persistent across runs)
Invalidation	Automatic via HTTP semantics	Manual only (delete project file)
Conditional Requests	Yes (If-None-Match, If-Modified-Since)	No (full response replay)
Storage	~/.cache/nuclei/httpcache/ (LevelDB)	-project-path (HybridMap)
Caches Errors	No (2XX only via ShouldCache)	Yes (all responses)
Cross-Template	Yes (same URL = same cache entry)	No (different template = different request bytes)

When to Use Each

HTTP Cache (default):

• Targets with proper HTTP caching headers
• Reducing load on well-behaved servers
• Cross-template resource sharing (same endpoint, different templates)
• Respecting server-defined cache semantics

Project Cache (-project):

• Long-running campaigns spanning multiple sessions
• Exact request deduplication (same template + same target = skip)
• Resumable scans after interruption
• Caching responses regardless of HTTP headers
• Avoiding re-testing identical requests across runs

Testing

Template:

# test-http-cache.yaml
id: test-http-cache

info:
  name: Test HTTP Cache
  author: dwisiswant0
  severity: info
  tags: test

http:
  - method: GET
    path:
      - "{{BaseURL}}/cache/300" # Cacheable for 300 seconds (Cache-Control: max-age=300)
      # - "{{BaseURL}}/cache/0" # not cacheable
      # - "{{BaseURL}}/get" # no cache headers (won't be cached)
      # - "{{BaseURL}}/response-headers?Cache-Control=max-age=60" # custom cache headers

    matchers:
      - type: status
        status:
          - 200

$ time ./bin/nuclei -duc -t test-http-cache.yaml -u https://httpbin.org -silent # cold, test with a cacheable endpoint
[test-http-cache] [http] [info] https://httpbin.org/cache/300

real	0m1.554s
user	0m0.341s
sys	0m0.121s
$ ls -la ~/.cache/nuclei/httpcache/
total 24
drwxr-xr-x 2 dw1 dw1 4096 Dec 20 22:05 .
drwxr-xr-x 3 dw1 dw1 4096 Dec 20 22:05 ..
-rw-r--r-- 1 dw1 dw1  871 Dec 20 22:05 000001.log
-rw-r--r-- 1 dw1 dw1   16 Dec 20 22:05 CURRENT
-rw-r--r-- 1 dw1 dw1    0 Dec 20 22:05 LOCK
-rw-r--r-- 1 dw1 dw1  359 Dec 20 22:05 LOG
-rw-r--r-- 1 dw1 dw1   54 Dec 20 22:05 MANIFEST-000000
$ time ./bin/nuclei -duc -t test-http-cache.yaml -u https://httpbin.org -silent # warmed, faster response (cached)
[test-http-cache] [http] [info] https://httpbin.org/cache/300

real	0m0.441s
user	0m0.425s
sys	0m0.120s

Future Considerations

• CLI flag to disable caching (-disable-http-cache or -no-http-cache).
• Cache statistics/metrics reporting.
• Cache size limits and eviction policies.
• Per-template cache control overrides.

References

Informative:

• RFC 9111 * HTTP Caching
• sandrolain/httpcache
• sandrolain/httpcache/leveldbcache

Normative:

• Add WrapTransport option for custom transport middleware retryablehttp-go#504
  • Add WrapTransport option for custom transport middleware retryablehttp-go#504 (comment)

[coderabbitai]

📝 CodeRabbit Plan Mode

Generate an implementation plan and prompts that you can use with your favorite coding agent.

• Create Plan

Examples

• Example 1
• Example 2

---

---

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord or schedule a call!