XPath Fundamentals

Last week we looked at a light introduction to Python. This week we are going to take a look at XPaths. We need to learn xpaths if we are to build a web crawler which can be used to scrape data from websites.

XPaths are fairly straightforward and easy to learn. Follow these steps:

1. Download the chrome browser and open it.
2. The default page should be Google so right-click on the "Google" icon
3. Select inspect. You should get the Developer Console to pop up as shown below:


4. Right-click on the highlighted element
5. Select Copy >> Copy XPath
6. Paste the path into any text editor, it should be this:
//*[@id="hplogo"]

There are two types of xpaths, absolute and relative. The xpath shown above is relative. The absolute xpath of the Google logo is:
/html[1]/body[1]/div[1]/div[8]/span[1]/center[1]/div[1]/img[1]

XPaths are subject to constant change so it is always good to use a relative xpath whenever possible. Breaking down the syntax for the relative XPath:

//* Select ALL elements in the DOM [@id="hplogo"] that have an attribute named id that is set to hplogo. When selecting elements in a DOM it is good practice to refer to their ids when possible. Ids are unique to elements and normally do not change.

More information on XPath syntax can be discovered here:
https://www.w3schools.com/xml/xpath_intro.asp

A good extension for creating, editing and testing XPaths is the ChroPath. You can learn all about it from this YouTube video:
https://www.youtube.com/watch?v=ikC4gLt0u8M

Next time, we will take what we've learned in last week's and this week's blog and combine it to build ourselves a python web scraper.

Comments

Post a Comment

Popular posts from this blog

Covering Your Tracks

The ability to remain stealth during a penetration exercise is what separates a sophisticated hacker from a script kitty. The phrase “The Quieter You Become, The More You Are Able to Hear” holds true, especially in the field of hacking. Being able to perform Ani-forensics and avoid detection is critical to the success of hacking and penetration testing and understanding these methods is beneficial, even for white hat hackers, as there will be times in a penetration where the system is already compromised. In such cases, understanding what the attacker is doing in order to cover their tracks can help determine the sophistication of the attack or bring prosecution to the attacker. All hacks begin with system enumeration and nmap is one of the best tools to use during this discovery phase. However, as I explained in Scanning With nmap blog, port scanning a bunch of IP Addresses or every port on a target system is noisy and can cause denial of service conditions (very bad if you are ...
Read more

LDAP Vulnerabilities

Image
Note** The following examples utilize Web for Penetration Testers which I do not own any rights to. These examples provide solutions to the live image of Web for Pentester 1 which can be downloaded at https://pentesterlab.com/exercises/web_for_pentester/course . Please attempt these exercises on your own before reviewing my solutions to gain maximum benefit.   Web for Penetration Testers Example 1 Lightweight Directory Access Protocol (LDAP) can be thought of as a hierarchical directory structure. Using the live iso of Web for Pentester 1 lab we can find that Open LDAP service is running by running performing a nmap scan from our kali box (attacking machine): Figure 1 Using the ldapsearch command with the -b (base dn [top of hierarchy in this case]) -LLL (print responses in LDIF format without comments or version) and -x (simple authentication) flags (-h specifies the host) from our kali box (see Figure 2), we see that there are three common names (cn) entries: admin, ...
Read more

Setting Up a Proxy to Protect Your Public IP (An Introduction to Proxies)

Image
I recently purchased a static IP address. While this is great for setting up VPNs, websites, and other major benefits it bothered me that I normally visit CTFs and sites that contain malicious code to accomplish these CTF challenges. There had to be a way I could better protect my public IP from direct exposure in the event I did something stupid while surfing the web. This is where a proxies come into play. A proxy is exactly the same thing meant in the English language; think if computer A who set up the proxy wants to talk to computer C. An intermediary could be introduced (Proxy Server B) so that when computer C talks to computer A it has to go through Proxy Server B. Thus Computer C thinks its talking to Proxy Server B and only sees that IP address; when in reality all traffic is getting forwarded to computer A and vise versa. This creates a layer of abstraction. There are many types of proxies such as reverse proxies which is used mainly for public facing web sites. We are strict...
Read more