XPath Fundamentals

Last week we looked at a light introduction to Python. This week we are going to take a look at XPaths. We need to learn xpaths if we are to build a web crawler which can be used to scrape data from websites.

XPaths are fairly straightforward and easy to learn. Follow these steps:

1. Download the chrome browser and open it.
2. The default page should be Google so right-click on the "Google" icon
3. Select inspect. You should get the Developer Console to pop up as shown below:


4. Right-click on the highlighted element
5. Select Copy >> Copy XPath
6. Paste the path into any text editor, it should be this:
//*[@id="hplogo"]

There are two types of xpaths, absolute and relative. The xpath shown above is relative. The absolute xpath of the Google logo is:
/html[1]/body[1]/div[1]/div[8]/span[1]/center[1]/div[1]/img[1]

XPaths are subject to constant change so it is always good to use a relative xpath whenever possible. Breaking down the syntax for the relative XPath:

//* Select ALL elements in the DOM [@id="hplogo"] that have an attribute named id that is set to hplogo. When selecting elements in a DOM it is good practice to refer to their ids when possible. Ids are unique to elements and normally do not change.

More information on XPath syntax can be discovered here:
https://www.w3schools.com/xml/xpath_intro.asp

A good extension for creating, editing and testing XPaths is the ChroPath. You can learn all about it from this YouTube video:
https://www.youtube.com/watch?v=ikC4gLt0u8M

Next time, we will take what we've learned in last week's and this week's blog and combine it to build ourselves a python web scraper.

Comments

Post a Comment

Popular posts from this blog

Covering Your Tracks

The ability to remain stealth during a penetration exercise is what separates a sophisticated hacker from a script kitty. The phrase “The Quieter You Become, The More You Are Able to Hear” holds true, especially in the field of hacking. Being able to perform Ani-forensics and avoid detection is critical to the success of hacking and penetration testing and understanding these methods is beneficial, even for white hat hackers, as there will be times in a penetration where the system is already compromised. In such cases, understanding what the attacker is doing in order to cover their tracks can help determine the sophistication of the attack or bring prosecution to the attacker. All hacks begin with system enumeration and nmap is one of the best tools to use during this discovery phase. However, as I explained in Scanning With nmap blog, port scanning a bunch of IP Addresses or every port on a target system is noisy and can cause denial of service conditions (very bad if you are
Read more

Covering Your Tracks - Anti-forensics for the Cloud - Introduction

Image
Before delving into cloud forensics there are some concepts that need to be explained. The cloud consisted of two entities, provider (companies that host cloud technologies) and tenants (companies that use cloud technologies from the providers). The cloud generally consists of three service models:  Infrastructure-as-a-Service (IaaS) where the provider provides the hardware and network. Platform-as-a-Service (PaaS) where the provider provides all the components to a tenant's application. Software-as-a-Service (SaaS) where the provider supplies the software and the components to run it and the tenant uses and at most configures the application. Chris Brenton gives a great overview of the cloud and the challenges it poses in the link below: https://www.sans.org/blog/pen-testing-in-the-cloud/ Surprisingly there is not a lot of coverage out on the web for cloud forensics and this makes sense. Cloud forensics analysts, especially for tenants, may be somewhat limited on acc
Read more

Cross-Site Scripting (XSS) Introduction

Image
  Cross-Site Scripting (XSS) Introduction Note** The following examples utilize Web for Penetration Testers which I do not own any rights to. These examples provide solutions to the live image of Web for Pentester 1 which can be downloaded at https://pentesterlab.com/exercises/web_for_pentester/course . Please attempt these exercises on your own before reviewing my solutions to gain maximum benefit.   Introduction XSS is an often-overlooked vulnerability. The generic “Hello World” example of XSS (e.g. <script>alert(“XSS”);</script>) is an overly benign example. Once an XSS vulnerability is discovered it becomes possible for attackers to modify the webpage to re-direct users to malicious site, steal a user’s session or cookie, or bypass a log in. Because XSS is such a huge topic, a series will be developed with exploitations to follow. For this blog article, XSS detection will be focused on. Eventually this series will cover JSON, XML, XPath, JQuery, and other typ
Read more