Magnet Weekly CTF Challenge Week 5 Writeup a.k.a. Fun with Hadoop

Another week (and month) has gone and the Magnet Weekly CTF Challenge has entered week 5. As announced by the CTF organisers, Linux images prepared by Prof. Ali Hadi would be used for November's CTF challenges and they could be downloaded from here.

The first November challenge question is not a long one, yet still an interesting one:


It is worth mentioning that the Linux images (3 in total, 1 HDFS primary and 2 HDFS secondary nodes) came from a "Hadoop Distributed File System" (HDFS) environment. Since the phrase "Had-A-Loop" rhymes with "Hadoop", it would be a no-brainer to look at the HDFS first. 

Diving into the HDFS Storage

The HDFS primary node image was examined first. The Hadoop was installed under "/usr/local/hadoop/" directory and the HDFS data storage location was designated by the directory value in "dfs.datanode.data.dir" property in a "hdfs-site.xml", which could be found in the "/usr/local/hadoop/etc/hadoop" directory:



From this point, it was clear that the HDFS data was stored under the "/usr/local/hadoop/hadoop2_data/hdfs/datanode" directory. However, there was no data under the HDFS data node directory on the HDFS primary node image. Therefore, the HDFS secondary mode 1 image was examined to look for the HDFS data nodes.

Spotting the HDFS Data Node Block File


A file called "blk_1073741825" could be found under one of the subdirectories under the datanode location (/usr/local/hadoop/hadoop2_data/hdfs/datanode/current/BP-1479872265-192.168.2.100-1510165901692/current/finalized/subdir0/subdir0/). Judging from the filename, "blk_1073741825" should be the file stored at block 1073741825 on the HDFS. The contents of the "blk_1073741825" looked like a typical Linux Apt SourceList file, which was usually found as the "/etc/apt/sources.list" file. 

At this moment, the answer of this challenge seemed to be "sources.list".

Much to my surprise, it was not the case. So there must be something different about this file. Was the challenge asking for the full filepath (i.e. "/etc/apt/sources.list")? Unsurprisingly, it was also wrong. 

Getting Back on the Right Track

I decided to take a look at the Hadoop logs folder at "/usr/local/hadoop/logs", so the Hadoop logs folders from all 3 hosts were exported and a bit of grep magic (grep -r 1073741825 *) was used. A file referred as "/text/AptSource._COPYING_" was copied from HDFS primary node to HDFS secondary node 1 and was allocated on block 1073741825.


With the support of log lines, I am very certain the answer should be "AptSource._COPYING_" and eventually became the first person to crack this week's challenge. To be honest, even though there was no bonus point awarded for being the 1st solver, it was still an enjoyable and pleasant moment for myself.

Tying Up All Loose Ends

I was still curious about the "._COPYING_" file extension as I have never seen this kind of file extension. Then this Hadoop Issues page explained that the  "._COPYING_" file extension would be created when a file was copied from HDFS or local to another file system implementation. Perhaps the perfect answer should be "AptSource", after all.


Lessons Learnt

Observe, don't just see. What you see may not be the whole truth.


Coincidence?

Evernote featured in week 4 had an elephant on its logo, whereas Hadoop featured in week 5 also had an elephant on its logo. Perhaps there would be also an elephant on week 6?😎











Comments

Post a Comment

Popular posts from this blog

Dumpster Diving in Google Photos Android App: "local_trash.db"

Image
  This article will explore the Trash within the Google Photos Android App. If you delete any media file from Google Photos on your mobile device, the deleted media file would then be relocated to the Trash within Google Photos. All items inside Trash will be permanently deleted after 60 days in Trash. In fact, the information of the Google Photos Trash items will be stored inside the " local_trash.db " under " /data/com.google.android.apps.photos/databases/ ": The " local " table of  " local_trash.db " contains most of the useful information, including " local_path " ( the original path of the deleted media file ), " trash_file_name " ( a UUID value that becomes the new name of the deleted media file ), " deleted_time " ( a Unix timestamp in milliseconds that recorded the delete time ), " is_video " ( where "0" = "no", "1" = "yes" ), etc. As soon as a media file was
Read more

Hexordia Weekly CTF Challenge 2024 - Week 1 Writeup

Image
 Recently, Hexordia has started a new weekly CTF challenge and I have signed up to join the challenge. The following is my writeup for the week 1 challenges: iOS - Crewmates are Sus (15 marks) What is Chad's user ID for the multiplayer social game? The terms "Crewmates" and "Sus" used in the question name suggests to the popular multiplayer social game "Among Us" From iLEAPP "Application State report" page, Among Us ( com.innersloth.amongus ) was installed and the sandbox path for Among Us can be found at  /private/var/mobile/Containers/Data/Application/AE23352D-C47B-43D9-87A7-6141653955A2 Using FTK Imager, the sandbox file path could be navigated to and I have decided to inspect iOS app Preferences related files first as we are looking for a gaming user ID: Using plistEditor Pro, the plist files could be opened and one of them ( com.innersloth.amongus.plist ) has a userid key with value 001381.5ced44f175f640fb9264ce19cc43683f.2043 which wa
Read more

An Android Casting (Device) Story: "cast.db"

Image
  This article will briefly document a SQLite database (" cast.db ") found from Android devices, which would store the information of detected casting devices on the Wi-Fi network. The said " cast.db " can be recovered at " /data/com.google.android.gms/databases/cast.db ".  Using Joshua Hickman's Android 10 image as an example, " DeviceInfo " table from  " cast.db "  would store the casting device ID (SSDP UDN), device friendly name, device model name, device IP address and port number, hotspot BSSID , etc. Apart from the basic information of the casting devices, some interesting Epoch timestamps in milliseconds like " last_published_timestamp_millis ", " last_discovered_timestamp_millis " and " last_discovered_by_ble_timestamp_millis " could be discovered as well.  There were some other intriguing tables such as " NetworkInfo ", which probably records the last connected time to a Wi-Fi networ
Read more