browser4-web-miner

WebMiner — Convert Detail Web Pages into Interactive Views

Quick Start

browser4-cli webminer install            # one-time install (Java 17+ auto-detected)
browser4-cli webminer all <html-dir>     # full pipeline: encode → cluster → views

WebMiner groups similar web pages together and produces an interactive HTML report with clusters of related pages — plus Excel spreadsheets for further analysis. Give it a folder of downloaded HTML files, and it handles the rest. Everything runs locally; no data leaves your machine.

When to Use

Use WebMiner when you have a folder of downloaded HTML pages and want to cluster them into interactive views and Excel reports — fully local, no LLM tokens. It complements rather than replaces browser4-cli crawl/swarm (which acquire pages): WebMiner analyzes pages you already have. Not for single-page extraction — use htmlsnapshot for that.

How It Works

WebMiner runs a three-stage local pipeline: encode converts each HTML page into a 69-dimension feature vector, cluster groups similar pages with SMILE KMeans (k auto-detected), and views renders an interactive HTML report plus Excel spreadsheets. Everything runs locally on your machine — no data leaves it, and no LLM tokens are consumed.

Patterns

1. Full pipeline on a folder of pages

browser4-cli webminer all <html-dir>

2. Rebuild views from an existing run

browser4-cli webminer views <result-dir>

3. Try it on the sample dataset

browser4-cli webminer run-example

Flags

Flag	Applies to	Description
--max-files <n>	webminer all	Limit the number of HTML files processed (default 40)
--output <dir>	webminer all	Override the output directory
--resume [<project-id>]	webminer all	Resume a previous run

Errors & Recovery

Symptom	Cause	Fix
webminer install fails	No Java 17+ on PATH	Install JDK 17+ or point JAVA_HOME at it
webminer all finds no pages	Directory has no .html files	Check the input directory path and file extensions
Pipeline crashes on large corpora	Free tier limit (< 1,000 pages)	Reduce the corpus or use --max-files; see the commercial Spark tier for scale
Views land in an unexpected temp dir	The views stage uses the app task-output root	Use webminer views <result-dir> to rebuild beside the result dir

Using from the Browser4 CLI

WebMiner is a first-class Browser4 citizen: the browser4-cli webminer command installs, updates, and runs the tool natively (no PowerShell needed — the CLI locates a Java 17+ installation, preferring the JRE bundled with the Browser4 runtime, and launches scent-miner.jar directly). The JAR and its release metadata are installed to ~/.scent/webminer/.

browser4-cli webminer install            # Download and install the latest release
browser4-cli webminer update             # Check for and install the latest release
browser4-cli webminer version            # Show installed and latest available versions
browser4-cli webminer uninstall          # Remove the installed release
browser4-cli webminer run-example        # Sample dataset + full pipeline (needs 7-Zip)
browser4-cli webminer all <html-dir>     # Full pipeline (encode → cluster → views)
browser4-cli webminer views <result-dir> # Rebuild views from an existing run

• webminer all <dir> accepts the pipeline options directly (--max-files <n>, --output <dir>, --resume [<project-id>]).
• Any other command is forwarded verbatim to scent-miner.jar, e.g. browser4-cli webminer encode <dir>.
• Runs started through the CLI set -Dapp.name=webminer, so the views task-output root is %TEMP%\webminer-<user>\ml\tasks\... (<user> is the OS user name; see Output).
• The bare webminer panel and webminer version keep the update check quiet: the GitHub → OSS-mirror fallback notices (rate limit, HTTP status, unreachable) are suppressed, and the Published line is omitted entirely when the release carries no published_at. webminer install / update still report the fallback.

Installing WebMiner

browser4-cli webminer install downloads, verifies, and installs the latest release (GitHub Releases with an Aliyun OSS mirror fallback; works on Windows, Linux, and macOS — no PowerShell needed):

browser4-cli webminer install            # Download and install the latest release
browser4-cli webminer update             # Check for and install the latest release
browser4-cli webminer version            # Show installed and latest available versions
browser4-cli webminer uninstall          # Remove the installed release

Releases are installed to ~/.scent/webminer/ and checked against https://github.com/platonai/web-miner/releases. SHA-256 checksums are verified automatically on download.

You can also use the JAR directly if it's already available:

java -jar scent-miner.jar <command> <args>

Converting Pages to Views

Running the Example

The run-example command downloads a pre-uploaded test dataset of real web pages, extracts it, and runs the full pipeline — no manual setup required beyond Java 17 and 7-Zip:

browser4-cli webminer run-example

The dataset is cached at ~/.scent/test-data/amazon.com/ so subsequent runs skip the download.

Running on Your Own Pages

# Full pipeline (one-shot)
browser4-cli webminer all /path/to/html/files

# Or with the JAR directly
java -jar scent-miner.jar all /path/to/html/files

The cluster count is always auto-detected from the data — this produces better results than guessing a number.

Options

Flag	Default	Purpose
--max-files <n>	40	Maximum number of HTML files to process
--output <dir>	<html-dir>-ml-output	Where to write the clustered results (CSV + clustering info; the views stage uses the app temp root — see Output)
--resume [<project-id>]	—	Pick up where a previous run left off. If no project ID is given, the most recent project is used.

Building Views from an Existing Run

If clustering has already completed and you just need to (re)build the views:

java -jar scent-miner.jar views <html-dir>-ml-output/kmeans-result/p<timestamp>

Output

all produces two kinds of artifacts in two different places:

1. Clustered results — written to <html-dir>-ml-output/kmeans-result/p<timestamp>/ (or wherever --output points): one result.csv per feature view (predictionAnd{Final,Minimal,Original}Features/result.csv) plus clusteringInfo.txt.
2. Views (interactive HTML report + Excel + JSON) — the views stage of all writes them to the application's temp task-output root, NOT under <html-dir>-ml-output: %TEMP%\<app>-<user>\ml\tasks\unsupervised\result\p<timestamp>\predictionAndMinimalFeatures.views\ on Windows, and <java.io.tmpdir>/<app>-<user>/ml/tasks/unsupervised/result/p<timestamp>/predictionAndMinimalFeatures.views/ on Linux/macOS (/tmp/... on Linux, $TMPDIR on macOS) — the <app> prefix follows -Dapp.name (webminer when launched through browser4-cli webminer, pulsar for a direct java -jar run) and <user> is the OS user name. The end of the run prints the resolved absolute views path.

So after java -jar scent-miner.jar all ./html-pages/ the clustered results look like:

html-pages-ml-output/
  └── kmeans-result/
      └── p<timestamp>/
          ├── predictionAndFinalFeatures/result.csv
          ├── predictionAndMinimalFeatures/result.csv
          ├── predictionAndOriginalFeatures/result.csv
          └── clusteringInfo.txt

and the views (<project>.html, *.xlsx, *.json) live in the temp task-output directory printed by the run.

The real report is <project>.html (e.g. p<timestamp>.html), not index.html. The index.html inside the views directory is an auto-generated directory listing ("Index of predictionAndMinimalFeatures.views") — opening it shows a file list, not the interactive clustering report. Open <project>.html instead.

To place the views beside the clustered results (e.g. to archive them with the project), rebuild them from the result directory:

browser4-cli webminer views <html-dir>-ml-output/kmeans-result/p<timestamp>
# (equivalent to: java -jar scent-miner.jar views <html-dir>-ml-output/kmeans-result/p<timestamp>)

This writes predictionAndMinimalFeatures.views/ inside the given result directory — the recommended way to locate artifacts, since the output path is explicit instead of an opaque temp path. Open the generated <project>.html in a browser to explore the clustering results. The .xlsx files can be opened in Excel for sorting, filtering, or further analysis.

Tips

• Input files — only *.html and *.htm files are processed. Other files in the directory are ignored.
• Resume interrupted runs — if a pipeline stops partway through, use --resume to continue from the last completed stage instead of starting over.
• Offline only — WebMiner works with pre-downloaded HTML files. Use a browser, wget, or a crawler to fetch pages first.
• Java 17 is required. Make sure java is on your PATH.