Как через jsoup парсить javascript

jsoup: Java HTML Parser

jsoup is a Java library for working with real-world HTML. It provides a very convenient API for fetching URLs and extracting and manipulating data, using the best of HTML5 DOM methods and CSS selectors.

jsoup implements the WHATWG HTML5 specification, and parses HTML to the same DOM as modern browsers do.

• scrape and parse HTML from a URL, file, or string
• find and extract data, using DOM traversal or CSS selectors
• manipulate the HTML elements, attributes, and text
• clean user-submitted content against a safelist, to prevent XSS attacks
• output tidy HTML

jsoup is designed to deal with all varieties of HTML found in the wild; from pristine and validating, to invalid tag-soup; jsoup will create a sensible parse tree.

Example

Fetch the Wikipedia homepage, parse it to a DOM, and select the headlines from the In the news section into a list of Elements (online sample, full source):

Document doc = Jsoup.connect("https://en.wikipedia.org/").get(); log(doc.title()); Elements newsHeadlines = doc.select("#mp-itn b a"); for (Element headline : newsHeadlines)

Open source

jsoup is an open source project distributed under the liberal MIT license. The source code is available at GitHub.

Getting started

1. Download the jsoup jar (version 1.16.2)
2. Read the cookbook introduction
3. Enjoy!

Development and support

If you have any questions on how to use jsoup, or have ideas for future development, please get in touch via one of the discussion methods.

If you find any issues, please file a bug after checking for duplicates.

The colophon talks about the history of and tools used to build jsoup.

Development of jsoup happens on GitHub. There you can see the latest changes, and get the source to build an unreleased version.

Status

jsoup is in general release.

Parsing a body fragment

You have a fragment of body HTML (e.g. a div containing a couple of p tags; as opposed to a full HTML document) that you want to parse. Perhaps it was provided by a user submitting a comment, or editing the body of a page in a CMS.

Solution

String html = "
Lorem ipsum.
"; Document doc = Jsoup.parseBodyFragment(html); Element body = doc.body(); 

Description

The parseBodyFragment method creates an empty shell document, and inserts the parsed HTML into the body element. If you used the normal Jsoup.parse(String html) method, you would generally get the same result, but explicitly treating the input as a body fragment ensures that any bozo HTML provided by the user is parsed into the body element.

The Document.body() method retrieves the element children of the document’s body element; it is equivalent to doc.getElementsByTag(«body») .

Stay safe

If you are going to accept HTML input from a user, you need to be careful to avoid cross-site scripting attacks. See the documentation for the Safelist based cleaner, and clean the input with clean(String bodyHtml, Safelist safelist) .

Cookbook contents

Introduction

Input

1. Parse a document from a String
2. Parsing a body fragment
3. Load a Document from a URL
4. Load a Document from a File

Parse a document from a String

You have HTML in a Java String, and you want to parse that HTML to get at its contents, or to make sure it’s well formed, or to modify it. The String may have come from user input, a file, or from the web.

Solution

Use the static Jsoup.parse(String html) method, or Jsoup.parse(String html, String baseUri) if the page came from the web, and you want to get at absolute URLs (see Working with URLs).

String html = "" + "
Parsed HTML into a doc.
"; Document doc = Jsoup.parse(html); 

Description

The parse(String html, String baseUri) method parses the input HTML into a new Document . The base URI argument is used to resolve relative URLs into absolute URLs, and should be set to the URL where the document was fetched from. If that’s not applicable, or if you know the HTML has a base element, you can use the parse(String html) method.

As long as you pass in a non-null string, you’re guaranteed to have a successful, sensible parse, with a Document containing (at least) a head and a body element.

Once you have a Document, you can get at the data using the appropriate methods in Document and its supers Element and Node .

Cookbook contents

Introduction

Input

1. Parse a document from a String
2. Parsing a body fragment
3. Load a Document from a URL
4. Load a Document from a File

Parse JavaScript with jsoup

In an HTML page, I want to pick the value of a javascript variable.
Below is the snippet of HTML page:

My aim is to read the value of variable key from this page using jsoup .
Is it possible with jsoup ? If yes then how?

19.3k 13 13 gold badges 122 122 silver badges 143 143 bronze badges
asked Feb 15, 2013 at 22:58
6,170 18 18 gold badges 77 77 silver badges 156 156 bronze badges

You’d have to get the script content then either parse manually, or see if you could use Rhino to get context out of an executed JS fragment.

Feb 15, 2013 at 23:09

@Reimeus: no. Initialization can be done somewhere else here some value is being assigned to variable key .

Feb 15, 2013 at 23:19

Added kotlin tag because a similar Koltlin question is marked duplicate and is linked to this question.

Nov 14, 2021 at 15:24

2 Answers 2

Since jsoup isn’t a javascript library you have two ways to solve this:

A. Use a javascript library

B. Use Jsoup + manual parsing

• Pro:
  • No extra libraries required
  • Enough for simple tasks
  • Not as flexible as a javascript library

  Here’s an example how to get the key with jsoupand some «manual» code:

  Document doc = . Element script = doc.select("script").first(); // Get the script part Pattern p = Pattern.compile("(?is)key=\"(.+?)\""); // Regex for the value of the key Matcher m = p.matcher(script.html()); // you have to use html here and NOT text! Text will drop the 'key' part while( m.find() ) < System.out.println(m.group()); // the whole key ('key = value') System.out.println(m.group(1)); // value only >

  Output (using your html part):

  key="pqRjnA" pqRjnA