JavaScript Is a Recon Goldmine, Part 3: Automating the Pipeline

Part 1 and Part 2 covered the techniques. Run by hand, they take an afternoon per app. The value comes from running them every day and noticing what changed, which only happens if it’s automated.

This part puts the pieces together into a small pipeline you can run on a cheap server: collect, extract, remember, alert, and hand you a ranked list to test by hand. Every example uses example.com. Point it only at apps you are authorized to test.

What we’re building

1. 01Collectcollect.sh · crawl · archives · chunks
2. 02Extractextract-calls.mjs · method + path
3. 03Rememberjsr.py · SQLite per target
4. 04Alertnotify · only never-seen routes
5. 05Rankqueue · code-only, risky, writes first
6. 06Test by handyou decide what it means

Four small scripts and one SQLite file per target. The machine does the collecting, extracting and remembering every day; you only see the routes worth your time, ranked.

1. Project layout, with scope enforced in code

Keep everything for one program in one folder, and make scope a file the scripts read, not something you remember.

jsrecon/
├── scope.txt             one regex per line: URLs you may request
├── collect.sh            downloads every JS file for a target
├── extract-calls.mjs     the AST script from Part 2
├── jsr.py                the route database
├── run.sh                the daily job
└── targets/
    └── app.example.com/
        ├── js/           downloaded files
        ├── calls.tsv     today's extracted calls
        └── routes.db     everything seen so far

scope.txt lists what the program allows, as anchored regular expressions:

^https://app\.example\.com/
^https://static\.example\.com/

Every list of URLs passes through this filter before any tool requests it. Automation doesn’t read program policies, so the filter is what stops a crawler following a link to a third-party site or an out-of-scope subdomain.

in_scope() { grep -Ef "$JSR/scope.txt"; }

Set the rate limits for the program once, as variables at the top of run.sh, so every tool uses the same ones.

2. The collector

collect.sh combines Part 1’s collection step with Part 2’s chunk recovery. Every list passes through in_scope before any request, and every tool runs at the program’s rate limit.

#!/usr/bin/env bash
# collect.sh TARGET_URL OUTDIR   ->  OUTDIR/js/*.js
set -uo pipefail
URL="$1"; OUT="$2"
HOST=$(echo "$URL" | awk -F/ '{print $3}')
RATE="${RATE:-10}"
in_scope() { grep -Ef "$JSR/scope.txt" || true; }

mkdir -p "$OUT/js" && cd "$OUT" || exit 1

fetch_all() {   # download each URL, named by content hash
  while read -r u; do
    tmp=$(mktemp)
    curl -s --max-time 20 "$u" -o "$tmp" && mv "$tmp" "js/$(sha1sum "$tmp" | cut -c1-12).js"
    sleep 0.2
  done
}

# 1. JS from a crawl and from archives (Part 1, step 1)
{ katana -u "$URL" -jc -d 3 -rl "$RATE" -silent; gau "$HOST"; } \
  | grep -iE '\.js(\?|$)' | in_scope | sort -u \
  | httpx -silent -mc 200 -rl "$RATE" > js_urls.txt
fetch_all < js_urls.txt

# 2. Lazy chunks rebuilt from the webpack runtime (Part 2, section 2)
#    Adjust the URL pattern to match the app's real chunk names.
grep -ohE '\{([0-9]+:"[0-9a-f]{8,20}",?)+\}' js/*.js 2>/dev/null | head -1 \
  | grep -oE '[0-9]+:"[0-9a-f]+"' | tr -d '"' \
  | awk -F: -v base="$URL" '{print base "static/js/" $1 "." $2 ".chunk.js"}' \
  | in_scope | httpx -silent -mc 200 -rl "$RATE" > chunk_urls.txt
fetch_all < chunk_urls.txt

echo "$(ls js | wc -l) JS files in $OUT/js" >&2

Naming files by their content hash means a bundle that hasn’t changed is stored once, however many times you download it. The folder grows only when the app actually ships new code.

3. A route database that remembers

Text files and anew work for one target. Across many programs and months of runs you want to ask questions: what appeared this week, which routes the UI never calls, what you’ve already tested. A single SQLite file per target answers all of them, and Python’s standard library already includes SQLite.

#!/usr/bin/env python3
"""jsr.py - remember every API route seen in a target's JavaScript.

  jsr.py DB ingest-static calls.tsv        add routes found in the code
  jsr.py DB ingest-runtime role_calls.tsv  mark routes the UI really calls
  jsr.py DB new [HOURS]                    routes first seen in the last HOURS (default 24)
  jsr.py DB queue                          untriaged routes, highest score first
  jsr.py DB mark METHOD PATH STATUS        set status: new, testing, done, finding
"""
import re, sqlite3, sys
from datetime import datetime, timedelta, timezone

RISKY = re.compile(r"admin|internal|export|impersonat|role|permission|debug|delete|refund", re.I)

def norm(path):
    return re.sub(r"\{(var|id)\}", "{x}", path.strip())

def db(path):
    con = sqlite3.connect(path)
    con.execute("""CREATE TABLE IF NOT EXISTS routes(
        method TEXT, path TEXT, source TEXT,
        first_seen TEXT, last_seen TEXT,
        in_ui INTEGER DEFAULT 0, status TEXT DEFAULT 'new',
        PRIMARY KEY (method, path))""")
    return con

def now():
    return datetime.now(timezone.utc).isoformat(timespec="seconds")

def ingest(con, tsv, runtime):
    t, added = now(), 0
    for line in open(tsv, encoding="utf-8"):
        parts = line.rstrip("\n").split("\t")
        if len(parts) < 2:
            continue
        method, path = parts[0].upper(), norm(parts[1])
        source = parts[2] if len(parts) > 2 else ""
        cur = con.execute(
            "INSERT OR IGNORE INTO routes(method, path, source, first_seen, last_seen, in_ui) VALUES (?,?,?,?,?,?)",
            (method, path, source, t, t, int(runtime)))
        added += cur.rowcount
        con.execute("UPDATE routes SET last_seen=? WHERE method=? AND path=?", (t, method, path))
        if runtime:
            con.execute("UPDATE routes SET in_ui=1 WHERE method=? AND path=?", (method, path))
    con.commit()
    print(f"{added} new routes", file=sys.stderr)

def score(method, path, in_ui):
    s = 0 if in_ui else 3                                          # never called by the UI
    s += 2 if method in ("POST", "PUT", "PATCH", "DELETE") else 0  # changes data
    s += 2 if RISKY.search(path) else 0                            # sensitive-sounding name
    return s

def main():
    if len(sys.argv) < 3:
        sys.exit(__doc__)
    con, cmd, args = db(sys.argv[1]), sys.argv[2], sys.argv[3:]
    if cmd == "ingest-static":
        ingest(con, args[0], runtime=False)
    elif cmd == "ingest-runtime":
        ingest(con, args[0], runtime=True)
    elif cmd == "new":
        hours = float(args[0]) if args else 24
        since = (datetime.now(timezone.utc) - timedelta(hours=hours)).isoformat(timespec="seconds")
        for m, p in con.execute("SELECT method, path FROM routes WHERE first_seen >= ? ORDER BY path", (since,)):
            print(f"{m}\t{p}")
    elif cmd == "queue":
        rows = con.execute("SELECT method, path, in_ui FROM routes WHERE status='new'").fetchall()
        for m, p, ui in sorted(rows, key=lambda r: -score(*r)):
            print(f"{score(m, p, ui)}\t{m}\t{p}\t{'ui' if ui else 'code-only'}")
    elif cmd == "mark":
        con.execute("UPDATE routes SET status=? WHERE method=? AND path=?", (args[2], args[0].upper(), norm(args[1])))
        con.commit()
    else:
        sys.exit(__doc__)

if __name__ == "__main__":
    main()

The score is deliberately simple: three points if the UI never calls the route, two if it changes data, two if its name sounds sensitive. Tune the RISKY word list to each program’s vocabulary; a billing app’s sensitive words differ from a healthcare app’s.

4. The daily run

run.sh ties the pieces together for one target. The first run builds a baseline and stays quiet; after that, it alerts only on routes it has never seen.

#!/usr/bin/env bash
# run.sh TARGET_URL      e.g.  ./run.sh https://app.example.com/
set -uo pipefail
export JSR="$(cd "$(dirname "$0")" && pwd)"
export RATE=10                                   # requests per second, per the program's rules

URL="$1"; HOST=$(echo "$URL" | awk -F/ '{print $3}')
OUT="$JSR/targets/$HOST"; DB="$OUT/routes.db"
FIRST_RUN=0; [ -f "$DB" ] || FIRST_RUN=1

"$JSR/collect.sh" "$URL" "$OUT"
node "$JSR/extract-calls.mjs" "$OUT"/js/*.js > "$OUT/calls.tsv"
python3 "$JSR/jsr.py" "$DB" ingest-static "$OUT/calls.tsv"

# The first run is the baseline: everything is new, so don't alert on it
python3 "$JSR/jsr.py" "$DB" new 24 > "$OUT/new_routes.txt"
if [ "$FIRST_RUN" = 0 ] && [ -s "$OUT/new_routes.txt" ]; then
  { echo "New routes on $HOST:"; cat "$OUT/new_routes.txt"; } | notify -silent -bulk
fi

Schedule it once a day per target. Spread the start times so you never hit two programs at once:

# crontab -e
17 3 * * * /home/hunter/jsrecon/run.sh https://app.example.com/  >> /home/hunter/jsrecon/log.txt 2>&1
43 3 * * * /home/hunter/jsrecon/run.sh https://shop.example.org/ >> /home/hunter/jsrecon/log.txt 2>&1

When you record traffic for a role (Part 2, section 4), load it into the same database so the queue knows what the UI really calls:

python3 jsr.py targets/app.example.com/routes.db ingest-runtime user_calls.tsv

Re-record after big releases. A route the UI started calling yesterday shouldn’t stay ranked as “code-only”.

5. Triage: work the queue by hand

Automation stops here on purpose. The pipeline decides what to look at; you decide what it means.

$ python3 jsr.py targets/app.example.com/routes.db queue
7   PUT    /api/v2/admin/users/{x}/role   code-only
7   POST   /api/v1/export                 code-only
0   GET    /api/v2/orders/{x}             ui
0   GET    /api/me                        ui

For each route, top of the queue first:

1. Read the code that calls it. The source column names the file. Find what parameters it sends and what screen it belongs to.
2. Mark it as testing, so tomorrow’s queue doesn’t show it again: jsr.py DB mark PUT '/api/v2/admin/users/{x}/role' testing
3. Call it with the lowest-privilege account you have, using values that point at your own second test account, never at real users.
4. Compare the response with what that account should be allowed to do. A refusal (401 or 403) is the correct result. A success, or data that belongs to the other account, is a lead.
5. Mark the result: done if it behaved correctly, finding if it didn’t.

An hour a day on the top of the queue beats a weekend spent spraying requests at every route. The top of the queue stays short because routes the UI calls every day, which everyone has already tested, sink to the bottom.

6. From lead to report

A route that answered when it shouldn’t have is a lead, not yet a finding. Before you write anything, check it holds up:

• Reproduce it twice, from a fresh session each time, so it isn’t a caching or timing fluke.
• Prove impact with your own accounts only. Account A reading or changing account B’s data is a complete proof; real users’ data is never needed and never acceptable.
• Confirm the route is in scope against the program’s policy, not just your scope.txt.
• Check it isn’t intended. Some routes are public on purpose; read the program’s known-issues list and documentation.
• Stop at the proof. Don’t enumerate further records or chain into other systems unless the program explicitly asks for it.

Then write the report so a developer can fix it without asking you a single question:

The “how it was found” line is worth including. It tells the team their front-end code is advertising the route, which often leads them to find and fix similar routes themselves.

This is the last part of the JavaScript series. SecureDX runs this kind of continuous review for clients alongside manual pentests; get in touch at securedx.in/contact. Hunters who build pipelines like this can request an SDX Society invite at securedx.in/society.