Using the Smart Browser DevTools IDE#

The Smart Browser DevTools IDE, available on the Enterprise plan, gives access to advanced features that can save you time when writing data extraction code.

For example, you can write, debug, and deploy browser scripts to use as browser actions in browser-based data extraction requests.

Browser scripts can be written from the Smart Browser DevTools IDE using our TypeScript scripting API.

Requirements#

To use the Smart Browser DevTools IDE you need a modern browser.

You also need to enable third-party cookies on the zyte.group domain. If your browser blocks them, you will see an error like the following:

Error loading webview: Error: Could not register service workers: NotSupportedError: Failed to register a ServiceWorker for scope …

Advantages of browser scripts#

Browser scripts are written using scripting API, which is basically a TypeScript API to expose the generic actions and special actions available to any Zyte API user.

The main advantage of browser scripts over using generic and special actions directly is that browser scripts support a non-linear flow thanks to the use of a programming language, TypeScript, which allows using conditional statements, loops, and so on.

For example, in a browser script you can check if an element is present in a webpage, and run different actions depending on that.

Other advantages over the HTTP API include:

Executing arbitrary JavaScript code with Page.evaluate().
Hiding content with Page.hide().
Accessing iframe elements through Page.getIframe().

`src/` and `dist/`#

On the Explorer view of the Smart Browser DevTools IDE you get 2 folders:

src/ is the development folder, where you develop your browser scripts.

Changes to this folder are saved on the Zyte API Smart Browser DevTools IDE cloud storage system.

Multiple developers can work on this folder at the same time, and see the work of one another in real time.
dist/ is the deployment folder.

It contains files built from the last version of src/ that has been deployed to Zyte API.

You should never write anything in the dist/ folder, all its contents are removed during deployment.

Creating a new script#

To create a new browser script:

Open the Smart Browser DevTools IDE.
Select (top-left corner) → File → New File….
On the Create New… dialog, select Smart Browser Interaction.
On the Use Case dialog, select one of the following:
- One of the special action interaction classes, to extend it in your new script.
- Others, to create a script from scratch.
- Utils, to create a file with utility code that you can reuse from browser scripts and other utility code files.
On the Domain dialog, enter the domain of the website for which you are writing the script (e.g. toscrape.com).

A TypeScript file is created in the src/ folder based on your input data, and will open on a tab of Smart Browser DevTools IDE.

For example, if you select the Others use case and the toscrape.com domain, a src/Others/toscrape.com.ts file is created with the following content:

import { BaseInteraction, Page } from "smartbrowser-core-interactions/index.ts";

interface Args {
    // add your arguments here
    // arg1: string;
}

export default class Interaction extends BaseInteraction {
    domains = ["example.com"];
    async do(page: Page, args: Args): Promise<void> {
        // implement your logic here
    }
}

You can now use the scripting API to implement your browser script, and then debug and, eventually, deploy that browser script.

Debugging a script#

After you create an interaction class, you can run your interaction class on a webpage from the Smart Browser DevTools IDE to see how it works:

Select (top-right corner).
On the URL dialog, enter a target URL (e.g. https://toscrape.com).
On the Interaction Parameters(in JSON) dialog, enter a JSON object of arguments for your interaction class. You can leave it empty to pass no parameters.

On pressing Enter, the Smart Browser DevTools IDE view splits vertically, and on the right-hand view a tab loads a tool, Smart Browser DevTools, which runs your interaction against the specified URL, showing you the result in real time, and offering you debugging tools and data.

When your interaction finishes, an “execution finished” message pops up on the bottom-right corner.

Once your interaction class is working as expected, you can deploy it and use it as a browser action in your data extraction requests.

Deploying your changes#

You can debug scripts from the Smart Browser DevTools IDE, but to use them as browser actions in your data extraction requests you must first deploy your changes to Zyte API.

To deploy all your changes to Zyte API:

Select on the Activity Bar (left-hand side).
On the Zyte Smart Browser DevTools side view that opens, select Deploy.

The deployment process empties the dist/ folder, builds files from the src/ folder into the dist/ folder, and deploys the files in the dist/ folder to Zyte API.

Using a script with Zyte API#

Once you have deployed a script, you can get the interaction ID as follows:

Select on the Activity Bar (left-hand side).
On the Zyte Smart Browser DevTools side view that opens, select Interactions, right-click on your interaction, and click Copy Interaction ID.

The interaction ID will be copied to your clipboard.

You can then invoke your script as a browser action from your data extraction requests:

Set the action field to "interaction".
Set the id field to the interaction ID from your clipboard.

For example, if your interaction ID is Others-toscrape.com, use:

{
    "action": "interaction",
    "id": "Others-toscrape.com"
}

To pass arguments to your script, set the args field to an object. That object is passed to your script as the args parameter of BaseInteraction.do(). For example:

{
    "action": "interaction",
    "id": "Others-toscrape.com",
    "args": {
        "foo": "bar"
    }
}